Vision-Language Models (VLMs)

A vision language model, or VLM, is a model that reads images and text in the same prompt and answers about both. Give it a screenshot and ask which button submits the form, a scanned invoice and ask for the total, or a bar chart and ask which quarter grew fastest. The image features in Claude, GPT and Gemini all work this way. A VLM differs from optical character recognition (OCR) because it reads layout and the role of each piece of text, not only the characters.

How an image becomes tokens#

A language model consumes tokens, so the image has to become something that can sit in the same sequence as text. The image is cut into a grid of small square patches, and a vision encoder, usually a Vision Transformer (ViT, the transformer design applied to image patches), turns each patch into a vector. A projection layer maps those vectors into the space the language model's tokens live in. In the first LLaVA model that layer was a single linear map, while Flamingo instead added layers that let the text consult the image features directly. From there the model attends over image tokens and text tokens alike.

Because each patch becomes a token, resolution costs tokens, and providers cap it. As of October 2026, Claude counts one visual token per 28 by 28 pixel block, so an image costs ⌈width / 28⌉ × ⌈height / 28⌉ tokens and a 1000 by 1000 image costs 1,296. Standard-tier models cap an image at 1,568 pixels on the long edge and 1,568 tokens. Claude 4.7 and later models raise that to 2,576 pixels and 4,784 tokens, so the same large image can cost roughly three times as many tokens on a newer model. Gemini charges 258 tokens for an image up to 384 pixels on both sides and otherwise cuts it into tiles at 258 tokens each, so its documented 960 by 540 example becomes 6 tiles, or 1,548 tokens.

Rendering a page so small text survives#

The token cap sets a ceiling on detail, whatever resolution you send. By Claude's published formula, a full A4 page on a standard-tier model ends up around 924 by 1,288 pixels, about 110 DPI (dots per inch), however finely you rendered it. On the high-resolution tier the same page keeps up to about 196 DPI. That is our arithmetic from the formula, not a figure Anthropic publishes, and the exact resize rule can shift it by a few pixels.

Two decisions follow. Render pages at about the resolution the tier will keep, since anything finer is thrown away before the model sees it. And when the text that matters is smaller than the page can carry at that resolution, crop the region and send it as its own image, rather than raising the resolution of the whole page and paying for detail you do not need. Send one page per image, because context such as "amounts in thousands" often sits in a corner.

Strong at layout, weak at counting#

VLMs are good at what layout encodes: reading paragraphs in order, following a table's rows, matching a legend entry to a chart series, finding a button on a screen, telling a header from a footnote. Anthropic notes that Claude Opus 5.5 reads values off dense charts much more precisely than earlier models.

They are weak at counting many similar objects, reading very small text, measuring sizes and distances, and comparing two nearly identical regions. Claude's own documentation calls its counts approximate, especially for large numbers of small objects, and warns of mistakes on images under 200 pixels. Detail smaller than a patch is largely gone by the time the model reasons. If a task depends on counting fifty rivets or reading footnote text, crop and zoom first, or do the count another way.

Ask for the box with the value#

A VLM can point as well as describe. Ask for a bounding box and the model returns coordinates as ordinary text. The conventions differ, and mixing them draws the box in the wrong place with no error. Gemini returns box_2d as [ymin, xmin, ymax, xmax] scaled to 0 to 1000, which you rescale to your image. Claude returns absolute pixel coordinates relative to the image after its own resize, so you undo that resize before drawing on the original.

Grounding makes an extraction checkable. When a value will be trusted downstream, ask for the value and its box together, crop the box, and send the field to a person when the crop does not contain the value.

What a document pipeline still needs#

The tempting design renders each page, asks for the fields as structured output, and stores the JSON. That design gives you a fluent answer with no measure of doubt, and the dangerous error is the confident one. On the ESG filing system we built for DocVerse, the failure that matters is a correct number extracted under the wrong entity, period, unit or framework, which every downstream check passes. That system reads filings with layout-aware parsing that keeps tables and footnotes in their original structure. It gives each field a confidence score, and low-confidence fields go to review instead of downstream as fact.

The same requirements apply whatever reads the page, OCR or a VLM. Keep the page structure, record which page and region each value came from, and route uncertain fields to a person. On DocVerse, most of the tuning work on scanned and irregular pages went into making sure low confidence actually fires, rather than a confident wrong parse coming through. Our guide to document intelligence for agents covers the checks across a packet of documents. Build a labelled set of your own pages early as well, because accuracy moves with layout, scan quality, language and each provider model update.