TechnologyRetrieval & RAG
Late Interaction & ColBERT
At a glance
Keep one vector per token instead of one per passage, and match query to document token-by-token at search time.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- Retrieval & RAG
- Technology
In short
ColBERT retrieval stores one small vector for every token of a passage and scores a match by letting each query token find its best partner in the passage, the MaxSim sum that defines late interaction. Bigcircle recommends hybrid search first when single-vector recall misses names and identifiers, and late interaction only when those misses survive it and the larger index is affordable. Our guide on why RAG systems fail uses late interaction for a small phrase a single vector would wash out, and warns that no ranking model repairs a chunk that lost its meaning before indexing.
Key facts
Definition
Late interaction retrieval keeps token-level vectors for queries and passages and compares them at search time, instead of compressing each passage into one embedding.
How Bigcircle does it
- On Prospex, Bigcircle runs semantic and keyword search together, because a question about a named tool is a keyword problem that pure vector search answers with a passage about a different product.
- Bigcircle recommends moving to ColBERT only when your own evaluation shows misses that survive hybrid search and a reranker, because the index and its serving stack cost more to carry.
Comparison
One passage embedding has to represent every fact in the text at once and drops rare details, while MaxSim lets each query token take its own best document token and adds those matches up.
Key number
A million 100-token passages at 128 dimensions and 16 bits is about 26 GB of token vectors, which ColBERTv2's 6 to 10x residual compression brings to about 2.6 to 4.3 GB.
ColBERT retrieval stores one small vector for every token of a passage instead of one vector for the whole passage. At search time each token of the query finds the passage token it matches best, and those best matches are added up into the passage's score. The family of methods is called late interaction, because query and passage are encoded separately and only meet at the end, and ColBERT, from Khattab and Zaharia in 2020, is the model that defined it.
Most vector search compresses a whole chunk into one embedding. That is fast and lossy. A paragraph about billing, refunds, enterprise contracts and SOC 2 becomes one point in space, and a rare detail inside it can disappear. Late interaction keeps the detail by keeping the tokens.
Why does one vector per passage lose detail?#
A single passage embedding has to represent every part of the text at once. That works for broad meaning and struggles with rare names, negation, and passages holding several unrelated facts. We hit the version of this that matters most on Prospex, our own prospect-research engine. "How do they make money?" is a question about meaning, while "Are they using Snowflake?" is a keyword problem, and pure vector search will happily return a passage about a different data warehouse. We run semantic and keyword search together there, because their failure modes are complementary. Hybrid search covers exact strings. Late interaction attacks the same loss from the embedding side.
How does MaxSim scoring work?#
ColBERT encodes the query into one vector per query token and each passage into one vector per passage token. Each query token takes the highest similarity it finds among the passage's tokens, and those maxima are summed. That operation is called MaxSim. It lets "refund" match the refund sentence, "annual" match the plan type and "30 days" match the exact clause, instead of hoping one vector kept all three. Because passages are encoded without seeing the query, their token vectors are computed once, at indexing time, and the paper reports scoring two orders of magnitude faster than BERT models that read query and passage together.
What does the index cost?#
Storage is the trade. Take a corpus of a million passages averaging 100 tokens. ColBERT's token vectors have 128 dimensions, so at 16-bit precision that is 100 million vectors at 256 bytes each, about 26 GB, against about 3 GB for one 768-dimension 32-bit vector per passage. ColBERTv2's residual compression reduces the late-interaction footprint by 6 to 10x, which brings the same corpus to about 2.6 to 4.3 GB, close to the single-vector index. PLAID, the search engine built for ColBERTv2, prunes candidate passages through their centroids before exact scoring and reports search latency up to 7x lower on a GPU and 45x lower on a CPU than plain ColBERTv2. Run the arithmetic on your own corpus before you assume either side of it.
When is ColBERT worth it?#
Late interaction is strongest when exact facets matter: product names, legal clauses, code identifiers, and queries from outside the domain the embedding model was trained on. The BEIR benchmark found re-ranking and late-interaction models achieved the best zero-shot results on average, at high computational cost. In a pipeline it usually sits between a first-pass retriever and a cross-encoder reranking step.
Do not start there. Bigcircle recommends hybrid search as the first fix when single-vector recall misses names and identifiers, because we run that on Prospex and it needs no new index format, with a cross-encoder reranker as the second. Move to late interaction when your own evaluation shows the misses survive hybrid search, the relevant detail is a phrase buried in an otherwise unrelated passage, and the larger index and a less common serving stack are costs you can carry.