TechnologyPrompting & In-Context Learning
Constrained Decoding
At a glance
Masking invalid tokens at sampling time so output provably conforms to a schema, grammar, or regex.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- Prompting & In-Context Learning
- Technology
In short
Constrained decoding is a way of generating text in which the runtime blocks every next token that would break a schema, grammar or regex, so the output always parses. On its trademark drafting platform, Bigcircle constrains decoding to a five-section schema and then runs validators, because the grammar fixes the shape of a draft and says nothing about whether its contents are true. Schema-constrained decoding runs in the same request as retrieval and a fine-tuned model on that platform.
Key facts
Definition
Constrained decoding sets the probability of every token that cannot continue a valid output to zero before sampling, so the model can only produce text the grammar accepts.
How Bigcircle does it
- Bigcircle constrains the trademark drafting model to a five-section schema, jurisdiction tags and citation formats, then validates the result, because a required field still gets filled when the source does not support it.
- Bigcircle bounds the retry after a failed validation and draws it only from retrieved evidence, because a free regeneration would bring back the failure the retry exists to correct.
- Bigcircle recommends checking meaning, permissions and business rules on the server after the text parses, because the mask controls only the shape of the output.
Comparison
A prompt that asks for valid JSON can still fail the parser, while a constrained decoder never samples a token the grammar forbids and can still put a valid but unsupported value in a required field.
Constrained decoding is a way of generating text in which the runtime only lets the model choose tokens that keep the output valid against a schema, grammar or regex. Asking a model to return valid JSON is a request, and some share of responses will still fail to parse; constrained decoding enforces the format while the model writes. At every step the runtime checks where the output is in the grammar, blocks every token that would break it, and samples only from what is left. Most structured-output modes in model APIs, grammar libraries and extraction pipelines work this way.
Blocking tokens at every step#
The model still scores every token in its vocabulary at each step. The constraint engine then sets the probability of every token that cannot continue a valid output to zero. If the grammar expects a quoted object key, tokens that could not continue a valid string are blocked. If it expects a comma or a closing brace, words are blocked. Sampling then picks from the tokens that remain, so an invalid token is never chosen.
The libraries differ in how much grammar they accept. Outlines started by compiling a JSON Schema or a regex into a finite-state machine. llguidance and XGrammar accept general context-free grammars, which covers nested structures a regex cannot. As of October 2026, vLLM's default auto setting picks between XGrammar and llguidance, and Outlines and lm-format-enforcer remain selectable. XGrammar is also built into SGLang and TensorRT-LLM, and llguidance's README says it powers OpenAI's Structured Outputs for JSON Schema. Anthropic's structured outputs compile a JSON schema into a grammar the same way, and the current docs list the schema features they do not support, such as recursive schemas and minimum or maximum lengths.
What the mask cannot guarantee#
The grammar decides the shape of the answer and nothing else. If the schema requires a field the source text does not support, the model still has to fill it, and it fills it with something valid. A required jurisdiction field gets a jurisdiction whether or not the document names one. Blocking tokens also changes the probabilities the model would otherwise have used, which is the subject of the grammar-aligned decoding work linked below. Whether format constraints hurt reasoning is still argued: Tam and colleagues reported lower reasoning scores under strict formats in "Let Me Speak Freely?", and the Outlines team replied that prompt mismatch and parsing explained most of the gap.
Two engineering costs come with it. Grammars are written over characters and models generate subword tokens, so the engine has to know, for every token, whether it keeps the text a valid prefix. And compiling the grammar, plus building a mask at each step, adds latency, most of it on the first request for a new schema. Anthropic, for example, caches a compiled grammar for 24 hours after its last use, so later requests skip that step.
Where it sits on the trademark platform#
On the trademark drafting platform, Bigcircle constrains decoding to a five-section schema, jurisdiction tags and citation formats, and then runs validators on what comes out. The mask makes every draft parse. The validators then check what the mask cannot. A failure triggers a retry that is bounded and draws only on retrieved evidence, because a free regeneration would reintroduce the failure being corrected. Each stage has a defined failure behaviour rather than a fallback to unconstrained generation.
Use constrained decoding wherever a parser, a tool call or another program reads the output, including structured output and extraction. Keep checking meaning, permissions and business rules on the server after the text parses, alongside your guardrails. If a pipeline still parses free text with a regex and retries when the parse fails, move it to constrained decoding before tuning anything else, because until then every parse failure shows up in the logs as a model error.