Chunking

Before a document can be retrieved, it has to be split into passages small enough to embed and rank. That split is chunking, and it is the most underrated lever in RAG: a chunk that mixes two topics or cuts an answer in half can never be retrieved cleanly, no matter how good your search stack is. Chroma's benchmark of chunking strategies found that the splitting method alone moves recall by up to 9 percentage points on the same corpus with the same embedding model. Everything downstream, embeddings, hybrid search, reranking, can only work with the chunks you hand it. Chunking sets the ceiling.

Fixed versus semantic splitting#

The simplest approach is fixed chunking: cut every N tokens or characters, regardless of meaning. It is fast, deterministic, and blind, so it happily slices through the middle of a sentence, a table row, or an argument, leaving fragments that match queries poorly and read worse in the prompt. A small upgrade, the recursive splitter, tries paragraph breaks first, then sentences, then characters, so cuts at least land on natural seams. Semantic chunking goes further: it embeds sentences and splits where the topic shifts, keeping each chunk about one coherent idea.

The difference shows up immediately in retrieval. Take a refund policy containing "refunds are issued within 30 days unless the order was a final sale item, in which case no refund applies." A fixed splitter that cuts mid-clause leaves "unless the order was a final" in one chunk and "sale item, in which case no refund applies" in the next; a query about final sale refunds matches neither half well. A semantic splitter keeps the whole rule in one chunk and it retrieves cleanly. In Chroma's evaluation, embedding-aware cluster chunking at 200 tokens hit 87.3% recall with the best precision of any method tested, and an LLM-driven splitter reached 91.9% recall, the highest overall. Semantic methods cost more at index time (you run an embedding or LLM pass over the corpus once), which is usually a fine trade because indexing is offline and querying is not.

Chunk size and overlap#

Chunk size is a tug-of-war. Small chunks embed tightly around one fact, so they rank precisely, but they strand that fact without its surroundings and fragment longer answers across many pieces. Large chunks carry context but dilute the embedding: the one relevant sentence gets averaged in with everything around it and the chunk ranks worse. The practical starting point in 2026 is 400 to 512 tokens with 10 to 20% overlap; Chroma's evaluation found 200-token chunks won on precision while 800-token chunks wasted retrieved budget, and the old OpenAI default of 800-token chunks with 400-token overlap scored below average on every metric they measured. Defaults are not neutral.

Overlap, repeating a slice of text between adjacent chunks, is the hedge against splitting an answer across a seam: a sentence that lands on the boundary still appears whole in at least one chunk. It is not free. A 1,000,000-token corpus split into 400-token chunks with 50 tokens of overlap produces about 2,860 chunks instead of 2,500, a 14% bigger index, and near-duplicate chunks can crowd the top-k with redundant text. Chroma found that removing overlap actually improved their efficiency metric; keep overlap modest and treat it as insurance, not a quality lever.

Contextual retrieval: give chunks their document back#

A chunk in isolation often loses the thread of where it came from. "The rate increased 3% over the prior quarter" is unanswerable without knowing which company, which product, which quarter. Anthropic's contextual retrieval fixes this by having a small model (they used Claude Haiku) read the full document and write 50 to 100 tokens situating each chunk, which gets prepended before embedding and BM25 indexing. Their reported numbers: contextual embeddings cut the top-20 retrieval failure rate by 35% (5.7% down to 3.7%); adding contextual BM25 cut it 49%; adding a reranker on top cut it 67%, to 1.9%. The trick that makes this affordable is prompt caching: the document is cached once and each chunk's contextualization reads it at a fraction of the normal input price.

A training-free cousin is late chunking, from Jina AI: embed the entire document through a long-context embedding model first, then pool token embeddings per chunk afterward, so every chunk's vector already carries document-wide context. Both approaches attack the same failure: chunks that are coherent text but ambiguous evidence.

Layout-aware parsing for PDFs#

All of the above assumes you start from clean text, and for PDFs, slides, and scanned reports you usually do not. Naive PDF extraction reads in raster order: it interleaves two-column layouts line by line, welds headers and footers into sentences, and flattens tables into word soup. Chunking garbage produces garbage chunks. Layout-aware parsers (Unstructured's hi_res partitioning, Azure Document Intelligence, and similar) run document image analysis first and classify regions into typed elements: Title, NarrativeText, Table, ListItem. You then chunk along element boundaries, keep each table whole with its caption, and attach the section heading to every chunk beneath it. For a two-column annual report with financial tables, this is the difference between chunks a reranker can score and chunks no model can interpret.

Practical takeaways#

Start with a recursive splitter at 400 to 512 tokens and 10 to 20% overlap, and measure before optimizing: 50 to 100 labeled query-passage pairs and recall@k will tell you whether chunking is your bottleneck. Move to semantic or element-based chunking when fixed splits visibly break rules, procedures, or tables. Add contextual retrieval when chunks lose their document identity; it is the single biggest published win, at 49 to 67% fewer retrieval failures. Parse layout before you split anything from a PDF. And remember that re-chunking and re-indexing a corpus is a batch job you run once, while a bad chunk fails on every query that needed it.