Vision-Language Models (VLMs)

A vision-language model, or VLM, accepts images and text in the same prompt and reasons across both. Hand it a screenshot and ask which button submits the form, drop in a scanned invoice and ask for the total, or paste a bar chart and ask which quarter grew fastest. The vision capability in Claude, GPT, and Gemini is exactly this. What makes a VLM more than an OCR engine is that it does not just read text off the page; it understands layout, relationships, and intent at the same time.

How an image becomes tokens#

A language model only consumes tokens, so the first job is turning pixels into something that can sit in the same sequence as text. The standard recipe has three stages. First, the image is sliced into a grid of small square patches and each patch is run through a vision encoder, usually a Vision Transformer (ViT), producing one vector per patch. Second, a projection layer maps those vectors into the same embedding space the LLM's text tokens live in; in LLaVA this is literally a small two-layer MLP, while Flamingo instead injected visual features through cross-attention layers. Third, the projected vectors are spliced into the token stream, and from that point the transformer attends over image tokens and text tokens identically.

Because every patch becomes a token, resolution costs money, and providers cap it with token budgets and tiling. Claude counts roughly one visual token per 28 by 28 pixel block, capped at 1,568 tokens per image on most models (the high-resolution Opus-class models raise that to 4,784 tokens and 2,576 pixels on the long edge); larger images are downscaled first. Gemini charges a flat 258 tokens for images up to 384 pixels and otherwise cuts the image into 768 by 768 tiles at 258 tokens each. High-resolution modes are tiling in disguise: the model sees a downscaled overview plus full-resolution crops, trading more tokens for more detail.

Strong at layout, weak at pixels#

VLMs are strong at exactly the things layout encodes: reading paragraphs in order, following table structure, matching a legend entry to a chart series, finding a button in a screenshot, telling a header from a footnote. For document-shaped inputs they have largely displaced classic OCR pipelines, because they read text and interpret its role in one pass.

They are noticeably weaker at fine-grained perception. Counting many similar objects, reading tiny dense text, judging exact sizes and distances, and comparing two nearly identical regions are the classic failure modes. The reason is structural: the patch grid throws away pixel-level detail to keep token counts affordable, so anything smaller than a patch effectively does not exist for the model. If your task hinges on counting 50 rivets in a photo or reading 6-point footnote text, expect approximation, or crop and zoom before asking.

Grounding: models that can point#

Modern VLMs do not just describe, they localize. Ask for a bounding box and the model emits coordinates as ordinary text tokens. Conventions differ and getting them wrong silently corrupts results: Gemini returns boxes as [ymin, xmin, ymax, xmax] normalized to a 0 to 1000 range, Qwen-VL models are trained on absolute pixel coordinates, and Claude works best with absolute pixels measured against the resized image it actually sees (so you must account for the provider's resize step before drawing boxes on the original).

Grounding is what turns a VLM answer into something you can verify or act on. It powers highlight-the-evidence UIs ("the signature is missing, here is the empty box"), visual citation in document QA, and computer-use agents that must click a real screen location rather than describe one. When you need to trust an extraction, asking for the value plus its box is cheap insurance: a wildly misplaced box is a strong hallucination signal.

Document understanding in practice#

The workhorse use case is documents: invoices, receipts, purchase orders, forms, contracts, slide decks. The old pipeline was OCR, then layout reconstruction, then per-template extraction rules; a VLM collapses it into one call: render the page as an image, ask for the fields you want as structured output, get JSON back.

A few pipeline implications follow. Render PDFs at sufficient resolution that the smallest text you care about survives the provider's downscaling, typically 100 to 150 DPI for an A4 page. Send one page per image and let the model see the full page, because context like "amounts in thousands" often lives in a corner. For tables, ask for markdown or JSON rows and validate column counts. And build a small labeled eval set of your real documents early: extraction accuracy varies sharply by layout, scan quality, and language, and provider model updates can shift it.

What an image costs#

Cost mechanics are concrete and worth memorizing for your provider. On Claude, tokens are roughly width times height divided by 750 (one token per 28 by 28 block): a 1000 by 1000 image costs 1,296 tokens, about $0.004 at $3 per million input tokens, and the per-image cap means a 4K screenshot costs the same as full HD on standard models. On Gemini, a 960 by 540 image becomes 6 tiles, so 1,548 tokens. OpenAI prices by detail level and patch count, with its own scaling rules.

Two consequences. First, the same JPEG can cost several times more on one provider than another, so verify with the provider's token-counting endpoint before projecting costs, and recheck after model releases, because budgets change (Claude's high-resolution models tripled the per-image cap). Second, resolution is a dial you control: downsample screenshots when you only need layout, and spend tokens on resolution only when small text actually matters.

Practical takeaways#

• A VLM is a layout-aware reader: patches in, tokens out, one attention stream over image and text.
• Trust it on structure (tables, charts, screenshots, forms); distrust it on counting, tiny text, and precise measurement, and crop or zoom when those matter.
• Use grounding boxes to make answers auditable and clickable, and learn your provider's exact coordinate convention.
• For documents, go image-first with structured output, render at adequate DPI, and eval on your own pages.
• Image tokens are real money at scale: know your provider's formula, count before you ship, and treat resolution as a cost dial.