Structured Output & Constrained Decoding

Structured output is the practice of making a model return data that conforms to a schema, reliably enough to feed straight into code. A model that returns prose is a chat partner; a model that returns valid JSON matching your schema is a programmable component. The hard part is the difference between "usually" and "always": a model that produces correct JSON 95% of the time still breaks one call in twenty. At 10,000 calls a day, that is 500 crashed pipeline runs, every day, each one a markdown fence, a trailing comma, or a chatty "Sure! Here's your JSON:" preamble.

How constrained decoding works#

The weak way to get structure is to ask politely in the prompt and hope. The strong way is constrained decoding: the inference engine restricts which tokens the model is even allowed to emit at each step, so invalid output is unreachable. Your JSON Schema is compiled into a grammar, typically a finite state machine or context-free grammar over the tokenizer's vocabulary. At every generation step, the engine checks which tokens would keep the partial output on a valid path and masks the rest by setting their logprobs to negative infinity before sampling. The model can only choose among legal continuations.

Walk through a tiny schema: {"sentiment": "positive" | "neutral" | "negative"}. After the model has emitted {", the only legal continuation is the key sentiment, so tokens spelling anything else are masked out. After "sentiment": ", only tokens that begin positive, neutral, or negative survive the mask. There is no token sequence that leads to malformed output, which is why this is a mathematical guarantee rather than a statistical one. The technique was popularized for open models by the Outlines library (Willard and Louf, 2023), which showed the mask can be precomputed into an index so the per-token overhead is negligible; modern backends like XGrammar, used by engines such as vLLM and SGLang, push that overhead down to microseconds per token.

JSON mode, strict schemas, and tool calls#

Providers expose three escalating levels of guarantee, and the names matter. Plain JSON mode promises only syntactically valid JSON: no markdown fences, no preamble, but also no promise that the fields match your shape. Schema-strict structured output guarantees both syntax and your exact schema via constrained decoding: OpenAI's Structured Outputs (strict: true on a JSON Schema, shipped in 2024), Anthropic's structured outputs for Claude (the output_config.format parameter, generally available after a late-2025 beta), and Gemini's responseSchema all work this way.

The third surface is tool calling, which is the same machinery pointed at function arguments: you declare a typed signature, the model fills the fields, and with strict mode enabled (OpenAI function calling, Anthropic's strict: true on tool definitions) the arguments are guaranteed to match the declared schema. This matters most for agents, where one malformed tool call can derail a whole multi-step run.

Client-side, schema libraries close the loop. In Python you define a Pydantic model; in TypeScript, a Zod schema; the library generates the JSON Schema for the request and validates the parsed response, giving your downstream code real types instead of any. Instructor wraps this pattern across providers and adds automatic retries when validation fails. Outlines plays the same role for self-hosted models, applying the grammar at the sampler itself.

Validate, then repair#

Constrained decoding guarantees shape, not sense. A response can be perfectly schema-valid and still wrong: a date field containing February 30th, an invoice_total that does not equal the sum of its line items, a syntactically legal enum value that violates a business rule. The robust pattern is validate-then-repair: parse the output, run full semantic validation, and on failure send the model a follow-up containing its own output plus the precise validation error, asking it to fix only what failed.

Concretely, an invoice extractor might validate with Pydantic, catch ValidationError: total 840.00 does not match line item sum 804.00, and re-prompt with that exact message. One repair attempt resolves the large majority of semantic failures; cap retries at two or three, then route the document to a human or a dead-letter queue. Because most calls pass on the first try, the amortized cost of the loop is small, and your pipeline degrades gracefully instead of crashing.

Why this unlocks pipelines#

Structured output is the boundary that turns an LLM from a demo into infrastructure. Once every call returns a typed object, you can compose calls like functions: an extraction step feeds a database insert, a classification step feeds a router, a scoring step feeds a dashboard. Failures become explicit (a validation error you can count and alert on) rather than silent (a regex that quietly matched the wrong span). It also makes evals tractable, since comparing typed fields against expected values is trivial in a way that grading free text is not. A useful habit: any time you find yourself writing a regex against model output, you wanted a schema.

Caveats and schema design#

The main caveat: constrained decoding can hurt reasoning quality. The "Let Me Speak Freely?" study (Tam et al., 2024) found that the stricter the format constraint, the larger the degradation on reasoning-heavy tasks like math and multi-hop QA, while simple classification tasks were unharmed or even improved. The intuition is that forcing the model to think inside a JSON straitjacket disrupts the free-form chain-of-thought it would otherwise use.

Two mitigations work well. First, exploit the fact that decoding is sequential: put a reasoning string field before the answer fields in your schema, so the model thinks first and fills in conclusions after. Second, for genuinely hard tasks, use two passes: let the model reason in free text, then make a cheap second call that converts the answer into the schema. Reasoning models with hidden thinking sidestep much of this, since the thinking happens before the constrained answer.

On schema design: prefer flat structures over deep nesting, use enums instead of open strings wherever values are bounded, set additionalProperties: false, and write field descriptions carefully, because the model reads them as instructions. Keep one call to one job; a 60-field mega-schema extracts every field worse than three focused calls.

Practical takeaways#

Use your provider's schema-strict mode, not prompt-and-pray, whenever output feeds code. Match the tier to the job: JSON mode is rarely what you want, strict schemas cover responses, strict tool definitions cover agent actions. Define schemas in Pydantic or Zod so request schema and response validation share one source of truth, keep a validate-then-repair loop for the semantic errors constraints cannot catch, and put reasoning fields ahead of answer fields so structure does not tax accuracy. That combination is what makes a model a dependable API.