Glossary
constrained decoding
Masking the model's next-token probabilities at each step so only tokens allowed by a grammar or schema can be sampled, forcing the output to be structurally valid.
The mechanism that makes structured outputruntimeMaking a model emit text that conforms to a schema or type (often JSON matching a JSON Schema) rather than free-form prose, so downstream code can parse it reliably. Open full entry a guarantee rather than a hope. At each decoding step the model produces a probability over the whole vocabulary; constrained decoding computes which of those tokens could still lead to a valid result and sets the probability of every other token to zero before sampling. The model can only ever emit text that the schema or grammarruntimeA formal description of allowed output structure (for example llama.cpp's GBNF) that a decoder is constrained to; a JSON Schema can be compiled into one. Open full entry permits.
The allowed set is computed from a grammar or a schema compiled into one. A common implementation walks a finite-state machine or pushdown automaton alongside generation: the current state says which tokens are legal next, the sampler is masked to that set, and the chosen token advances the state. Because the mask is applied to logits the model already produced, the technique adds correctness without retraining and works on any open-weights model.
This is the difference the structured-output story turns on. A model fine-tuned to emit JSON is good at it but can still slip; a constrained decoder cannot emit an invalid token by construction. Open engines like vLLMruntimeAn open-source inference engine introduced by UC Berkeley in 2023, built around PagedAttention to manage KV cache memory and serve tokens efficiently under load. Open full entry , SGLangruntimeAn open inference engine from the LMSYS team featuring RadixAttention for prefix sharing and a structured-generation frontend, particularly strong on agent and tool-calling workloads. Open full entry , and llama.cppruntimeGeorgi Gerganov's C++ inference engine optimized for CPUs and consumer GPUs, the on-device standard and the engine behind Ollama, LM Studio, and most local-first AI products. Open full entry all ship a constrained-decoding path, and the libraries that specialize in it (Outlines, XGrammar, llguidance) are what those engines call underneath.