Vision and documents
RecommendedRead images and dense documents, extract structure, and ground the result.
Concepts
Vision-language modelsRead screenshots, charts, and photos directly, no separate OCR step.
Vision-language models (VLMs) accept images alongside text in a single prompt, letting the model read a screenshot, interpret a chart, or extract data from a photo without a separate OCR pipeline in front of it. Models like GPT-4o and Claude 3.5 can answer questions directly about what they see, describe layouts, and identify values in tables.
In production this eliminates an entire preprocessing stage and its failure modes. The trade-off is that token cost scales with image resolution, so you want to resize or crop images aggressively and only send the region that matters to the query.
Sources
Full definition in the glossaryLayout-aware parsingPreserve tables and reading order from dense, messy PDFs.
Dense PDFs carry meaning in structure, not just in words: a value in a table cell means something different from the same value in a header or footnote, and reading order matters in multi-column layouts. Layout-aware parsers preserve this structure by returning bounding boxes, reading-order sequences, and explicit table cells rather than a flat stream of tokens.
Without it, a downstream model sees a jumbled text dump and makes wrong inferences about relationships between numbers and labels. Services like Amazon Textract and Google Document AI return block-level geometry so your extraction logic can use position and structure as signals alongside the raw text.
Sources
Bounding-box groundingPoint to exactly where on the page an answer came from.
When you extract a field from a document, returning the bounding box of the source region ties the answer to a specific location on the page. A downstream reviewer or UI can highlight that exact span, making the extraction auditable rather than a black box.
This matters most in regulated workflows, financial processing, and anything where a human needs to spot-check or correct results. Grounding also helps catch hallucinations: if the model points to a region that contains different text, the extraction is wrong regardless of how plausible the answer looks.
Sources
Technologies
Textract / Document AIHosted OCR and layout extraction for documents at scale.
Amazon Textract and Google Document AI are managed services that handle the messy work of document ingestion at scale: rotating skewed scans, handling varied DPI, extracting tables, key-value pairs, and form fields, and returning structured JSON with geometry. Both expose synchronous and asynchronous APIs so you can process single pages with low latency or batch thousands of documents overnight.
Choosing between them usually comes down to your cloud footprint and which document types each handles better for your corpus. Either way, offloading layout extraction to a purpose-built service lets you focus prompt engineering effort on what the model does with the structured output rather than on parsing.
Sources
In production
The discipline that separates a shipped system from a demo.
Ground to the pageReturn the bounding box or source span so a human can verify each extracted field.
Every extracted field should come back with the bounding box or source span that produced it, not just the value. This one habit transforms document extraction from a system people must trust blindly into one they can verify on demand, which is the difference between a prototype and a production workflow.
In practice, store coordinates alongside extracted values, surface them in review UIs, and include them in structured outputs sent downstream. When something is wrong, reviewers immediately see what region the model read and can correct both the value and the source attribution.
Sources
Eval on real documentsBenchmark on your messiest scans, not clean sample PDFs.
OCR and extraction accuracy on clean sample PDFs bears little relation to accuracy on your actual document corpus. Real documents come with fax artifacts, inconsistent fonts, rotated pages, handwritten annotations, and domain-specific jargon that trips up generic models.
The only meaningful benchmark is accuracy on the worst documents you will actually process, measured on the fields that matter to your application. Build a test set from your messiest scans early, before you commit to a vendor or model, and keep adding to it every time a new failure mode surfaces in production.
Sources