Vector Search & Vector Databases

A vector database stores embeddings, lists of numbers that represent the meaning of a passage, and finds the stored ones closest to the embedding of a query. Vector search is that lookup: given a question, return the passages whose meaning sits nearest to it. It is the first stage of most RAG systems, and if it misses the right passage, nothing later in the pipeline can recover the answer.

Teams spend weeks choosing the database, and the choices around it matter more: how filters are applied, whether exact names and identifiers are also matched by keyword, and whether anyone measures recall on the questions users actually ask. Our guide on why RAG systems fail in production debugs those stages in the order the data flows.

Exact search, and why indexes approximate#

The exact way to find nearest neighbours compares the query with every stored vector. It is fully accurate and its cost grows with the corpus. Five million passages embedded at 1,536 dimensions in 32-bit floats is about 31 GB of vectors, and every query has to read all of it.

An approximate nearest-neighbour (ANN) index gives up a little recall for much faster lookups. Recall@10 is the share of the true ten nearest neighbours the index actually returns. The neighbour an ANN index drops is usually near the bottom of the true list rather than the best match, so tune chunking and reranking before chasing the last point of index recall. On a small corpus, measure exact search before adding an index at all, because it is simpler and fully correct.

HNSW: long hops first, short hops last#

The most common ANN index is HNSW (hierarchical navigable small world), from Malkov and Yashunin's 2016 paper. Every vector is a node linked to its near neighbours, with sparser copies of the graph stacked above. A search enters at the top, hops quickly towards the query along long links, then drops a layer and refines, until the bottom layer, which holds every vector.

Three settings matter. m is how many links each node keeps, ef_construction is how carefully the graph is built, and ef_search is how wide the search is at query time. In pgvector the defaults are 16, 64 and 40. ef_search can be raised per query without rebuilding anything, so it is the setting to tune against recall measured on your own questions. The cost is memory, since HNSW keeps the graph, and usually the vectors, in RAM.

Use the distance the embedding model was trained with, which the model card states. On vectors normalised to unit length, cosine, dot product and Euclidean distance give the same ranking. Use the same metric when indexing and querying, because a mismatch lowers recall with no error.

Filters break approximate search quietly#

Real queries carry filters: this tenant, this year, documents this user may see. pgvector's own documentation gives the problem in numbers. With approximate indexes the filter is applied after the index scan, so with the default ef_search of 40 and a condition that matches 10% of rows, only 4 rows survive on average. pgvector 0.8.0 added iterative index scans, which keep scanning until enough rows pass. Turn them on for any filtered query.

Filtering first has the opposite problem: the graph was built over the whole corpus, and removing most of its nodes cuts the paths a search needs. Qdrant handles this by adding extra links between points that share a value of an indexed field, and by scoring every match directly when few points pass. Those extra links are built only for fields that were indexed before the graph was built, so create the index on that field before loading the vectors, or rebuild the graph afterwards.

Filter loss belongs in the retrieval eval, not only in the security review. A permission or date filter that removes the right document looks, from the answer, exactly like a search that failed.

Vector search alone misses exact terms#

Embeddings blur identifiers, part numbers, product names and acronyms. On Prospex, our prospect research engine, the question set was mixed. "How do they make money" is a question about meaning. "Are they using Snowflake" is a keyword question, and pure vector search will return a passage about a different data warehouse. We run semantic and keyword search together rather than as a fallback chain, because their failures are complementary. That combination is hybrid search.

Some questions need structure that no similarity score holds. On the ESG filing system we built for DocVerse, a knowledge graph and vector search work together: pure vector search cannot express "the same subsidiary two years earlier", and pure graph traversal cannot find a narrative claim that was never turned into a field.

pgvector or a dedicated vector database#

pgvector adds a vector column type to Postgres, with HNSW and IVFFlat indexes (IVFFlat groups vectors into clusters and searches the nearest few). Vectors sit next to the relational data, so a similarity search joins with ordinary SQL filters in one query and one transaction, under the backups and access controls you already run. Dedicated vector databases such as Pinecone, Qdrant, Weaviate and Milvus earn their place at a different scale: tens to hundreds of millions of vectors, high query rates, filter-aware indexing and built-in compression, at the price of another system to run and a sync job to keep it consistent with the source.

Start on pgvector when the corpus is in the low millions and most queries filter on relational fields, and move when a measured query load or recall number on your own traffic says so. Keep the retrieval layer behind one interface, so the move changes one component rather than the application.