Semantic Search Without a Vector Database: Cosine Similarity Over a Few Thousand Vectors

Semantic search without a vector database: run a flat cosine similarity dot product scan over a few thousand normalized embeddings in memory or SQLite with Dart.

The first time I added semantic search to a product, I reflexively went shopping for a vector database. Then I counted the vectors: about 4,000 help-center articles and support macros. I was about to stand up a managed service, pay a monthly floor, and add a network hop — to search a dataset that fits in a few megabytes of RAM. I deleted the Pinecone tab and wrote a for loop instead.

That loop has been in production a long time now, and it has never once been the bottleneck. This post is about where that line actually is: when a flat cosine-similarity scan in memory or SQLite is the right engineering call, and when you genuinely need a real vector store. If you've been told that "doing semantic search properly" requires a dedicated vector database, this is the counter-argument — with the Dart code to back it up.

What semantic search actually reduces to

An embedding is just a fixed-length array of floats — a point in high-dimensional space. A modern text embedding model maps a sentence, a paragraph, or a whole document to a vector of a few hundred to a couple thousand dimensions, and it does so such that pieces of text with similar meaning land close together. That is the entire premise: "semantically similar" becomes "geometrically near."

So semantic search is: given a query vector, find the stored vectors closest to it. "Closest" for text embeddings almost always means cosine similarity — the cosine of the angle between two vectors. Cosine similarity ignores magnitude and cares only about direction, which is exactly what you want when comparing meaning rather than length. A one-line note and a three-paragraph write-up about the same thing point in nearly the same direction even though one vector is "bigger."

The single most important trick is this: if you normalize every vector to unit length once at ingestion time, cosine similarity collapses into a plain dot product. No per-query magnitude computation, no division, no sqrt in the hot path. You pay the normalization cost once when you store the vector, and every query afterward is just multiply-and-add.

import 'dart:math';List<double> normalize(List<double> v) {  var mag = 0.0;  for (final x in v) {    mag += x * x;  }  mag = sqrt(mag);  if (mag == 0) return v;  return [for (final x in v) x / mag];}/// Both vectors are pre-normalized, so dot product == cosine similarity.double dot(List<double> a, List<double> b) {  var sum = 0.0;  for (var i = 0; i < a.length; i++) {    sum += a[i] * b[i];  }  return sum;}

That is the whole similarity engine. Everything else — the storage, the top-k selection, the persistence — is bookkeeping around these two functions.

A note on the math you can skip

You will see people reach for sqrt, Euclidean distance, or explicit angle computation. For normalized embeddings you need none of it. Once every vector has unit length, the dot product is bounded in [-1, 1], higher is more similar, and ranking by dot product gives you the identical ordering you'd get from full cosine similarity. Euclidean (L2) distance over unit vectors is also monotonic with cosine, so it would rank the same too — the dot product is simply the cheapest way to get there.

The in-memory version: a flat linear scan

For a corpus you can load at startup, keep it dumb and flat. Store the vectors, run the dot product against all of them, keep the top k. A full linear scan over a few thousand vectors of typical embedding dimensionality is a handful of milliseconds — comfortably faster than the network round-trip you would have made to a hosted vector DB.

class SemanticIndex {  final List<String> ids;  final List<List<double>> vectors; // all pre-normalized  SemanticIndex(this.ids, this.vectors);  List<(String, double)> search(List<double> query, {int k = 5}) {    final q = normalize(query);    final scored = <(String, double)>[];    for (var i = 0; i < vectors.length; i++) {      scored.add((ids[i], dot(q, vectors[i])));    }    scored.sort((a, b) => b.$2.compareTo(a.$2));    return scored.take(k).toList();  }}

This is O(n · d) per query, where n is the number of documents and d is the embedding dimension. For a few thousand documents at a few hundred dimensions, that is a few million floating-point operations — the kind of work a modern CPU chews through before you can measure it. There is no index to build, no background compaction, no eviction policy, nothing to operate.

A few things I have learned to do here that matter more than any micro-optimization:

• Don't build a full sort when you only need top-k. Sorting the whole list is O(n log n) for no reason. A bounded min-heap of size k keeps it O(n log k): push each score, and once the heap holds k items, only push a new score if it beats the current minimum. At a few thousand vectors nobody will notice, but it is the first thing to reach for if the corpus grows.
• Store vectors as Float32List, not List<double>. Dart doubles are 64-bit and boxed inside a generic list; a typed Float32List is half the memory, contiguous in one buffer, and the tight loop over it stays in cache. Embedding models emit float32 anyway, so you lose no real precision. This is the difference between "fine" and "unnoticeable."
• Cache the query embedding. The embedding API call (or on-device model inference) that turns the user's query into a vector is almost always slower than the search itself. That round-trip, not the math, is your real latency budget. Cache identical queries and debounce as-you-type search so you are not embedding every keystroke.
• Consider a coarse pre-filter. If you already know the user is inside one project, tenant, or language, scan only those vectors. Cutting n by 10x before you ever touch the dot product is the cheapest speedup there is.

Here is the top-k heap idea in Dart, using a simple insertion into a bounded, sorted list — good enough for small k and easy to read:

List<(String, double)> topK(  List<double> q,  List<String> ids,  List<Float32List> vectors, {  int k = 5,}) {  final best = <(String, double)>[]; // kept sorted ascending by score  for (var i = 0; i < vectors.length; i++) {    final score = dot(q, vectors[i]);    if (best.length < k) {      best.add((ids[i], score));      best.sort((a, b) => a.$2.compareTo(b.$2));    } else if (score > best.first.$2) {      best[0] = (ids[i], score);      best.sort((a, b) => a.$2.compareTo(b.$2));    }  }  return best.reversed.toList(); // highest first}

When it lives in SQLite

In-memory is great until you have to persist, or the corpus is bigger than you want resident in RAM, or you are on mobile and the OS kills the app to reclaim memory. This is exactly the situation in the on-device Flutter tools I build — no server to call, everything local, and the app has to survive being backgrounded. SQLite is the natural home.

The mistake people make is trying to do the vector math inside SQLite. Don't. Plain SQLite has no native vector type and no fast float-array dot product in SQL. Use SQLite as the blob store, and do the arithmetic in your application code. Store each vector as a BLOB — the raw bytes of your Float32List — read them back, and run the same dot-product loop.

CREATE TABLE documents (  id      TEXT PRIMARY KEY,  content TEXT NOT NULL,  vector  BLOB NOT NULL   -- raw Float32 bytes, pre-normalized);

On the Dart side, serialization is just a view over the same bytes — no per-element copying:

import 'dart:typed_data';// Float32List -> bytes for INSERTUint8List vectorToBytes(Float32List v) => v.buffer.asUint8List();// bytes from SELECT -> Float32ListFloat32List bytesToVector(Uint8List bytes) =>    bytes.buffer.asFloat32List(bytes.offsetInBytes, bytes.length ~/ 4);

Reading a few thousand blobs and scanning them is still fast, especially if you deserialize once on first load and keep the vectors cached in memory, treating SQLite purely as durable storage. The pattern I use is: SQLite is the source of truth on disk; a List<Float32List> in memory is the working set; a query scans the working set, never the database rows directly.

If your row count climbs into the tens of thousands and you want to stay in SQLite, look at the sqlite-vec extension, which adds a proper vector-search virtual table and does the distance math in C, close to the data. But reach for it because you measured a problem, not preemptively. The plain blob-and-scan approach carries a surprisingly long way, and it has zero extra dependencies to package into a mobile build.

So when do you actually need a real vector database?

Be honest about scale, and do not confuse "AI-shaped" with "needs infrastructure." You want a dedicated vector database — Pinecone, Qdrant, Weaviate, Milvus, pgvector, and friends — when at least one of these is true:

• Your vector count is large enough that a linear scan blows your latency budget. The crossover is not a fixed number — it depends on dimensionality, hardware, and how tight your latency requirement is — but it lands well into the range (think hundreds of thousands to millions of vectors) where a naive O(n · d) scan stops being instant. That is when Approximate Nearest Neighbor (ANN) indexes earn their keep. Structures like HNSW (a navigable small-world graph) and IVF (inverted-file clustering) trade a little recall for sub-linear search, turning a full scan into a walk over a fraction of the vectors.
• You need metadata filtering fused with the vector search — "closest documents where tenant_id = X and published = true." Doing this well over millions of rows, without either pre-filtering into a tiny candidate set or post-filtering away most of your top-k, is exactly what these systems optimize.
• The corpus changes constantly and you need concurrent writes, incremental index updates, replication, and durability guarantees you do not want to hand-roll.
• It will not fit in memory and you do not want to manage that spill-to-disk yourself.

If you are already running Postgres, note that pgvector is the low-drama middle ground: you get ANN indexing (HNSW and IVFFlat) and SQL metadata filtering without adding a new piece of infrastructure to operate, back up, and monitor. I would reach for it long before a standalone managed vector service, precisely because it rides on a database I already know how to run.

What should not push you toward a vector DB: the mere fact that embeddings are involved, a slide deck that says "RAG," or a fear that a for loop is "not production-grade." A linear scan over a few thousand pre-normalized vectors is production-grade. I have shipped it, it just works, and it has fewer moving parts to break at 3am.

A quick decision checklist

• A few thousand vectors, static-ish corpus, single process → in-memory flat scan.
• Need persistence, on-device, or an app that gets killed → SQLite blob store + in-memory scan.
• Tens of thousands of vectors, still local → sqlite-vec.
• Already on Postgres, want filtering + ANN → pgvector.
• Hundreds of thousands to millions, heavy filtering, constant writes → dedicated vector database.

Key takeaways

• Semantic search is nearest-neighbor over embeddings. Normalize every vector to unit length once at ingestion, and cosine similarity becomes a plain dot product — no sqrt, no division in the hot path.
• Start with the dumbest thing that works: store as Float32List, run a flat dot-product scan, and select top-k with a bounded heap instead of sorting the whole list.
• Your latency budget is the embedding call, not the math. Cache query embeddings; the scan is measured in milliseconds for a few thousand vectors.
• Persist in SQLite as raw float blobs and do the arithmetic in application code — don't try to compute distances in SQL. Reach for sqlite-vec only after you measure a problem.
• Graduate to a real vector store (or pgvector) only when you hit a real wall: a vector count that breaks your latency budget, filtered search at scale, or a corpus that won't fit in memory.
• Most side projects and a surprising number of shipped products never hit that wall. Measure before you pay the complexity tax.