Vector Search & Vector Databases

Once your documents are embeddings, retrieval becomes a geometry problem: given a query vector, find the stored vectors closest to it in meaning. Vector search is the machinery that answers that question fast enough to run on every request, and a vector database is the system that stores, indexes, filters, and scales those lookups. This layer is the engine room of RAG: if it is slow, every request is slow, and if it misses the right chunk, nothing downstream can recover the answer.

Exact search is correct, and that is the problem#

The honest way to find nearest neighbors is exhaustive search: compare the query against every stored vector and sort by distance. It is perfectly accurate, trivially simple, and linear in corpus size. Run the numbers on a realistic corpus: 5 million chunks embedded at 1,536 dimensions in float32 is about 31 GB of raw vectors, and every single query has to stream all of it past the CPU. Even on a well-provisioned machine that lands in the hundreds of milliseconds per query while saturating memory bandwidth, so a handful of concurrent users is enough to flatten the box.

Approximate nearest-neighbor (ANN) search trades a small amount of recall for orders of magnitude in speed. Recall@10 is the fraction of the true 10 nearest neighbors the index actually returns; a tuned ANN index typically hits 0.95 to 0.99 recall in single-digit milliseconds on that same corpus. The misses are usually benign too: the neighbor an ANN index drops tends to be the ninth or tenth closest, not the top hit, and end-to-end RAG quality is dominated by chunking and reranking, not the last percent of ANN recall. One caveat in the other direction: below roughly 100,000 vectors, exact search is already fast enough, and skipping the index is the simpler, fully accurate choice.

HNSW: long hops first, short hops last#

The dominant ANN index is HNSW (Hierarchical Navigable Small World), from Malkov and Yashunin's 2016 paper. Picture every vector as a node in a graph, linked to its near neighbors, then stack several thinned-out copies of that graph on top: the top layer holds a few nodes with long-range links, each layer below gets denser, and the bottom layer contains every vector. A search enters at the top, greedily hops toward the query through the long-range links, then drops a layer and refines, like taking a flight, then a train, then walking the last block. That hierarchy is what turns a linear scan into roughly logarithmic search time.

Three knobs matter in practice. m is how many links each node keeps (16 is a common default), ef_construction controls how carefully the graph is built (higher means better recall but slower indexing), and ef_search sets how wide the search beam is at query time. The last one is your live recall dial: raising ef_search from 40 to 200 might lift recall from 0.95 to 0.99 at two to three times the latency, and you can tune it per query without rebuilding anything. The cost of all this is memory: HNSW keeps the graph, and usually the vectors, in RAM, with link overhead stacked on top of the raw vector size.

Cosine or dot product, briefly#

Distance metrics decide what "close" means. Cosine similarity measures only the angle between vectors; dot product measures angle and magnitude together; Euclidean distance measures straight-line separation. Here is the detail that defuses most debates: on unit-normalized vectors, all three produce the same ranking, and most modern embedding APIs return normalized vectors. So the practical rule is short. Use whatever metric the embedding model was trained with (the model card says), normalize if the model does not do it for you, and never mix metrics between indexing and querying, which silently wrecks recall.

Metadata filtering breaks ANN in quiet ways#

Real retrieval is rarely pure similarity. Production queries look like "nearest chunks where tenant = acme and year = 2026," and that filter interacts badly with ANN indexes. Post-filtering, running the vector search first and discarding non-matching results, fails on selective filters: if only 1% of the corpus belongs to acme, a top-10 search will routinely return zero surviving results, and over-fetching by 100x is both expensive and still not guaranteed. Pre-filtering, restricting to matching vectors first, has the opposite problem: the HNSW graph was built over the whole corpus, so removing nodes fragments the paths the search needs to traverse, stranding whole regions of the graph.

Mature engines integrate the filter into the traversal instead. Qdrant builds extra graph links so filtered searches stay connected and switches to brute force when the filter is selective enough that scanning the survivors is cheaper. pgvector added iterative index scans in version 0.8: the index keeps scanning until enough matching rows survive the filter. The lesson for system design: know your filter selectivity up front. A filter matching 50% of the corpus is easy; a filter matching one user's 200 documents should skip ANN entirely and brute-force the subset.

pgvector or a dedicated vector database#

pgvector adds a vector column type plus HNSW and IVFFlat indexes to Postgres. Its superpower is location: vectors live next to your relational data, so similarity search joins with ordinary SQL filters in one query, inside one transaction, covered by the backups and access controls you already run. For corpora up to the low millions of vectors with relational filtering on most queries, it is hard to beat, and it adds zero new infrastructure.

Dedicated vector databases (Pinecone, Qdrant, Weaviate, Milvus) earn their keep at a different scale: tens to hundreds of millions of vectors, high query throughput, filter-aware indexing, built-in quantization to shrink memory, and horizontal scaling as managed features rather than weekend projects. The price is operational: another system to run or pay for, a sync pipeline keeping it consistent with your source of truth, and relational joins pushed back into application code. The sensible 2026 default is to start with pgvector and migrate when you measure real pain, not when a benchmark blog post predicts it.

Practical takeaways#

Use ANN because exact search cannot meet production latency past a few hundred thousand vectors, but skip the index entirely below that. Reach for HNSW by default and treat ef_search as your recall dial, measuring recall on your own queries rather than trusting defaults. Match the distance metric to the embedding model and normalize. Design metadata filtering before choosing a database, because filter selectivity decides whether ANN helps or fights you. And keep the boring option on the table: pgvector inside the Postgres you already operate covers most products, and the day it does not, your retrieval layer is a swappable component, not a rewrite.