Structured Output & Constrained Decoding

Structured output means a model returns data that matches a schema you supply, such as a JSON object with named, typed fields, so your code can read the answer directly instead of parsing prose. The difference that matters is between "usually" and "always". A model that returns valid JSON 95% of the time still fails one call in twenty, and at 10,000 calls a day that is 500 failed runs: a markdown fence, a trailing comma, a friendly sentence before the opening brace.

Schema-strict modes close that gap by enforcing the schema while the model writes, using constrained decoding: at each step the runtime blocks every token that would break the schema. That fixes the shape of the answer. It does not make the values in it true, and the rest of this page is about the work that remains.

Three levels of promise#

Providers offer three levels, and the names matter because they promise different things.

JSON mode promises text that parses as JSON and nothing about its fields. OpenAI's own guide now says to use Structured Outputs instead of JSON mode wherever possible, and we do not build on JSON mode at all.

Schema-strict output promises your exact schema. OpenAI's Structured Outputs works this way, and so do Anthropic's structured outputs, which as of October 2026 are generally available through the output_config.format parameter, with compiled grammars cached for 24 hours. The first request with a new schema pays the compile time, so warm it before traffic arrives.

Strict tool calling aims the same enforcement at function arguments. Anthropic's strict: true on a tool definition guarantees the arguments match the tool's schema, with a limit of 20 strict tools per request. Use it on any tool whose arguments reach a system that changes state.

Where the guarantee stops#

The guarantee has documented exceptions on both major APIs. On Claude, a refusal (stop_reason: "refusal") or a response cut off at the token limit (stop_reason: "max_tokens") can return output that does not match the schema. OpenAI returns a separate refusal field, which you check before parsing. Handle both as their own outcomes, because a pipeline that treats every response as schema-valid will crash on exactly the inputs that were unusual.

Not every schema keyword is enforced either. Anthropic rejects schemas that use numeric limits, string lengths or recursion, with a 400 error. Google's documentation says Gemini does not enforce minLength, maxLength or uniqueItems, and tells you to validate values in your application. So a minimum: 0 you wrote into the schema may be ignored or refused depending on the provider, and the check has to live in your code either way.

Field order is another one. Claude emits required properties before optional ones, so the output order can differ from the order in your schema. That matters for the common trick of putting a reasoning field before the answer fields so the model thinks first. Make the reasoning field required, or on Claude it can come out after the answer it was meant to precede.

A valid shape can still be wrong#

A response can be perfectly schema-valid and still wrong: a date of 30 February, an invoice_total that does not equal the sum of its lines, an enum value that breaks a business rule. The harder case is a correct value under the wrong label. On the ESG filing system we built for DocVerse, the dangerous failure is a confident extraction of a correct number under the wrong entity, period, unit or framework, because everything downstream passes it. That is why entity, period, unit and framework are carried as fixed metadata from parsing there, and never left for the model to fill in at query time.

The fix is a check after the parse, and a bounded repair. Validate with Pydantic or Zod, then run the business rules. On failure, send the model its own output with the exact error, such as total 840.00 does not match line item sum 804.00, and ask it to fix only that. Bigcircle recommends one or two repair attempts before the record goes to a person or a dead-letter queue (a holding list of failed records someone reviews), because each attempt is a full model call, and a record that keeps failing usually has a problem in the source that a person resolves faster.

On the trademark drafting platform, decoding is constrained to a five-section schema, jurisdiction tags and citation formats, and validators run on what comes out. A failure triggers a retry that is bounded and draws only on retrieved evidence. A free regeneration would bring back the citation failure the validators exist to catch.

What to leave out of the schema#

A field the model fills is a field that can drift. On RepliCoach, our sales-practice agent, the score comes from a versioned rubric in code, and the model writes only the qualitative read. If the model both wrote the feedback and set the number, a model upgrade would shift every historical comparison. Apply the same test to every field you add: if code can compute it from other fields or from the source, compute it and leave it out of what the model writes.

Whether strict formats hurt reasoning is still argued. Tam and colleagues reported lower scores on reasoning tasks under strict formats in "Let Me Speak Freely?", and the Outlines team replied that prompt differences and parsing explained most of the gap. Do not pick a side from either paper. For a hard reasoning step, compare a free-text pass followed by a conversion call against a single strict call, on your own eval cases, and keep whichever scores better.

Keep schemas flat where you can, use enums (a fixed list of allowed values) wherever the values are known in advance, set additionalProperties: false, and write field descriptions as instructions, because the model reads them. A pipeline that still runs a regex over free text should move that call to a strict schema first, before any other change to the prompt.