TechnologyPrompting & In-Context Learning
Constrained Decoding
At a glance
Masking invalid tokens at sampling time so output provably conforms to a schema, grammar, or regex.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- Prompting & In-Context Learning
- Technology
Prompting asks a model to behave. Constrained decoding makes invalid behavior impossible. At each generation step, the runtime looks at the grammar state, masks every token that would violate the schema, and samples only from the allowed set. That is the engine behind many structured-output APIs, grammar libraries, and production extraction systems.
Logit masking at every step#
The model still produces a probability distribution over the whole vocabulary. The constraint engine then applies a mask. If the current state expects a JSON object key, tokens that cannot continue a valid quoted string get probability zero. If the state expects a comma or closing brace, words are masked. Sampling then proceeds from the remaining tokens.
Guarantee vs request#
"Return valid JSON" is a request. A constrained decoder is a guarantee within the supported grammar. This is why extraction systems that feed downstream code should prefer schemas over prose instructions. If the output must parse, do not merely ask the model to produce parseable text.
The same idea powers tools like Outlines, llguidance, XGrammar, and vendor structured-output modes. Some compile JSON Schema or regexes into finite-state machines. Others support richer context-free grammars. Serving engines integrate these masks so the model never sees an invalid sampled token.
Pitfalls#
Constraints can distort the model's distribution. If the schema forces a field that the evidence does not support, the model will still pick something valid. Tokenization matters too: a character-level grammar must align with subword tokens, so the engine needs to know whether a token keeps the string in a valid prefix state. There is also overhead: grammar compilation, per-step mask construction, and bigger allowed-token sets can add latency.
Practical takeaways#
Use constrained decoding for structured output, extraction, tool arguments, and any place where invalid syntax creates code failures. Still validate semantics after parsing. The grammar can guarantee shape; it cannot guarantee truth, authorization, or business correctness. Pair it with server-side validation and guardrails.