Chunking Strategies

Chunking is the step in RAG that splits documents into passages before they are embedded and indexed, and a chunking strategy is the rule that decides where each cut falls: every N tokens, at paragraph and sentence breaks, where the topic shifts, or along the document's own layout. Retrieval can only return a passage that exists. A chunk that mixes two topics or cuts a rule in half cannot be found cleanly, however good the search built on top of it. Chroma's 2024 evaluation found that the choice of splitter alone moved recall, how often the passage holding the answer is retrieved, by up to 9% on the same corpus with the same embedding model. We treat chunk boundaries as part of retrieval design rather than storage housekeeping, because embeddings, hybrid search and reranking can only work with the passages the chunker hands them.

Fixed, recursive and semantic chunking strategies#

Fixed chunking cuts every N tokens or characters regardless of meaning. It is fast and repeatable, and it slices through the middle of a sentence, a table row or an argument, leaving fragments that match queries poorly and read badly in the prompt. A recursive splitter tries paragraph breaks first, then sentences, then characters, so most cuts land on a natural seam. Semantic chunking embeds each sentence and splits where the topic shifts, so each chunk holds one idea.

Take a refund policy that says "refunds are issued within 30 days unless the order was a final sale item, in which case no refund applies." A fixed splitter that cuts mid-clause leaves "unless the order was a final" in one chunk and "sale item, in which case no refund applies" in the next, and a question about final sale refunds matches neither half well. A splitter that respects sentences keeps the whole rule in one chunk.

The common advice is that semantic chunking beats the simple splitters, and Chroma's numbers make that less clear than it sounds. Their embedding-aware cluster chunker at 200 tokens reached 87.3% recall with the best precision of any method (the highest share of retrieved text that was relevant), and a splitter driven by a language model reached 91.9% recall, the highest overall. A plain recursive splitter at 200 tokens with no overlap reached 88.1%, within a point of the cluster method and with no model pass over the corpus. Chroma calls the work preliminary, so read the numbers as a direction. Start with the recursive splitter, and move to semantic or element-based splitting when your own failures show rules, procedures or tables cut across a seam.

Chunk size and overlap#

Small chunks embed tightly around one fact, so they rank precisely, but they strand that fact without its surroundings and scatter a longer answer across many pieces. Large chunks carry context but blur the embedding, because the one relevant sentence is averaged in with everything around it. In Chroma's tests the old OpenAI Assistants default, 800-token chunks with 400 tokens of overlap, reached 87.9% recall with 1.4% precision, which means most of every retrieved chunk was text the question did not need. Begin around 200 to 400 tokens with a recursive splitter, the range where that splitter did consistently well, and let your own measurement move it.

Overlap repeats a slice of text between neighbouring chunks, so a sentence that lands on a boundary still appears whole in at least one of them. It costs index size: a 1,000,000-token corpus split into 400-token chunks with 50 tokens of overlap gives about 2,860 chunks instead of 2,500, a 14% bigger index, and near-duplicate chunks crowd the top results with the same text. Chroma found overlap lowered their IoU score, which penalises repeated text, but also that removing a 125-token overlap from 250-token chunks with a small embedding model dropped recall from 82.4% to 77.1%. Keep overlap for small chunks and weak embedding models, where a split sentence is likely and costly, and drop it once chunks are large enough to hold a whole answer.

Give each chunk its document back#

A chunk on its own often loses where it came from. "The rate increased 3% over the prior quarter" cannot answer anything without the company, the product and the quarter. Anthropic's contextual retrieval has a small model (Claude 3 Haiku in their tests) read the whole document and write 50 to 100 tokens that situate each chunk, and that text is prepended before the chunk is embedded and indexed for keyword search. In their September 2024 tests the share of queries whose right chunk was missing from the top 20 fell from 5.7% to 3.7% with contextual embeddings, to 2.9% when contextual keyword indexing (BM25) was added, and to 1.9% with a reranker on top. Prompt caching keeps it affordable, because the document is cached once and read for every chunk; they put the one-time cost at $1.02 per million document tokens. A training-free alternative is late chunking: run the whole document through a long-context embedding model first, then average the per-token vectors inside each chunk, so every chunk's vector already reflects the document around it.

Added context fixes a chunk that is coherent but ambiguous. It does not fix a bad boundary. On our prospect intelligence engine a custom chunker sizes and bounds passages for the retrieval step, because a boundary that splits a person's role from their company is a retrieval bug created at write time. Every later inference then rests on a role with no employer attached. Fix boundaries in the chunker first, and add context when the boundaries are right and the chunk still does not say whose figures it holds.

Parse the layout before splitting a PDF#

Everything above assumes clean text, and PDFs, slide decks and scanned reports rarely give you that. Naive extraction reads line by line across the page, so it interleaves two-column layouts, welds headers and footers into sentences and flattens tables into a run of words. Layout-aware parsers, such as Unstructured's hi_res strategy, detect the page layout first and return typed elements such as Title, NarrativeText, ListItem and Table. You then chunk along element boundaries, keep each table whole with its caption, and attach the section heading to every chunk beneath it.

DocVerse, the ESG filing system we built, now in production, shows why the parse comes first. Chunk-and-embed works where the meaning is in the prose. In an institutional filing the meaning sits in a table, qualified by a footnote, under a framework version stated in an annexure. A table flattened to text keeps the number and loses the row and column that said what it was a number of, and no chunk size recovers that. The parser there keeps sections, tables and footnotes as structure, and entity, period, unit and framework travel with each value as metadata rather than being inferred at query time.

Is chunking your bottleneck?#

Find out before you tune anything. Label 50 to 100 real questions with the passage that answers each, and measure recall at k, the share of questions whose right passage appears in the top k results. Then read the misses. If the answer exists in the corpus but no single chunk holds all of it, the split is the problem. If one chunk holds the answer and it ranks too low, look at embeddings, hybrid search and reranking instead, because no change to the splitter will move it. Re-chunking and re-indexing is an offline batch job you run once, while a bad chunk fails on every query that needed it, so the split is worth getting right before anything downstream is tuned.

Skip most of this when the documents are few and short enough to fit in the context window whole. Chunking exists because a corpus does not fit, and a corpus that does fit can be passed in directly and compared against retrieval on the same questions.