Amusingly constrained decoding is the same approach we use to turn an AR model into a /v1/systemone endpoint (I worked on the vLLM feature).
Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run.
In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.
I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.
Constrained decoding has been in vLLM for a long time, and constrained tool call decoding has been in vLLM for a while, too.
The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly.
To be fair, the course of constrained tool calling configuration in vLLM changed over time in ways that changed behavior, which is the real problem. The Ubicloud team deserves a lot of credit for having benchmarks in place to track performance and catch regressions for investigations. Some of the providers you find on OpenRouter have inexplicably bad performance relative to others and clearly do not care this much.
However, if this article left you confused because constrained decoding and constrained tool call schema have been features in vLLM for a long time, jump to the end where they show what configuration changes they made.
Overall, a fair assessment. I do want to place a footnote here that we did fix vLLM in the sense that its correct behavior for Kimi required code changes: it was impossible to fix it in the configuration. Later, the vLLM team fixed this problem upstream. Now, with the latest version of vLLM, the problem we describe can be addressed by setting the `--tool-strict-level` flag to `parameter`.
> The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly.
Just as a point of feedback for the authors, I enjoyed the article, and did not suffer the confusion pointed out above.
In part this is because VLLM is a confusing mess. It takes a large investment in time to get it to perform well (and when it does it is really good). But it is fragile and over complicated.
I wish they'd published something about the actual impact on model quality. Schema-constrained decoding changes the distribution the LLM will output. I'd be curious how a back-up-and-retry method for tool calls (which also doesn't pollute the context and also has 100% tool-call accuracy) would compare. In their example stats (23/25 success rate), it'd be +9% tokens used for the tool call attempt itself, but that's +9% on just a small fraction of the total tokens, so if a retry solution gave any better accuracy I think it might win out.
Constrained decoding for tool calls only intervenes with the output distribution if the model attempts to output a token that would violate the schema.
If the tool call is properly defined, meaning the LLM has the right instructions to produce the output defined by the schema, then I don't see why constrained decoding would reduce output quality relative to retrying until it succeeds.
I have seen some cases where developers (or let's be honest, someone's LLM) changed the tool calling definition/prompt but failed to update the output schema, so the schema was forcing outputs that the model didn't even know it was supposed to produce. This leads to confusing debugging sessions where you're getting outputs that look right but are low quality, which can be revealed by turning off constrained output and noticing that all of your constrained outputs look different than you expect. But when the schema matches the tool definition and prompt, it's hard to think of a reason the output might be worse. I see it as nudging the probabilistic bad tool calls back into what they should have been.
Is the schema constraint not implemented in a way where the new distribution is identically scaled to the old distribution with retry? I had thought the way it worked was just to disallow invalid next states... oh, or is the idea that you think it is much preferred to retry at the level of the entire tool call, rather than (conceptually) per token? I guess I can see that making a lot of sense, as maybe the model gets "backed into a corner", so to speak, with a really bad initial choice and is then trying to rescue horrible states?
The way it works is to disallow invalid next states, and getting backed into a corner is one way to think about one possible failure mode.
For a concrete example (happens less nowadays with bigger models, but it still happens, especially later into a context window), models hate long json strings. It's very easy to get a json output like
{"name": "Bob", "biography": "Bob is a fantastic builder, having ...
Critically, the ellipses terminate the JSON. With schema-constrained decoding, you _must_ eventually terminate that broken string, and it'll typically terminate it right after the ellipses. Basically, by the point you got to the first period the output was borked. You were backed into a corner. Retry as a strategy fixes that.
As another example failure mode that's a bit more subtle, you might have a set of "correct" tokens early on in the output which get the KV cache into a bit of a wonky state, causing it to have worse outputs toward the end of your output. Disallowing invalid next states forces you to generate tokens in that worse-output state. Retry lets you fail more often and then hopefully not have those wonky-state early tokens. It's not really that you were backed into a bad state as a hard rule, but that you were backed into a bad state as a probabilistic statement.
For a toy model to think about, suppose the entire grammar is [ab, ba], and the LLM's universe of training data is [ab, ab, ba, bb]. The "right" distribution (retry on failure) is {ab: 67%, ba: 33%}. The schema-constrained distribution is {ab: 50%, ba: 50%}.
What the other guy means is the model is forced to decode potentially out of distribution and subsequent parts of the sequence will be sampled from a higher-entropy distribution.
i.e. it might be that a particular model may need some additional conditioning, or perhaps adjusting the order of the json output affects it to produce lower-quality results.
You have to remember that the LLM will be expected to generate 10s of thousands of tokens on a single turn, where things like quantization noise lead to cascading error further on. So when you can, you really want to stay within the model's distribution.
I guess to be more concrete, I thought the way schema aware decoding worked should be identical to a token-level retry: "only even dice rolls are allowed" converting {0:0.1, 1:0.2, 2:0.2, 3:0.1, 4:0.2, 5:0.2} into {0:0.2, 2:0.4, 4:0.4}. That is why I am asking if the difference being stared at is doing the retry at the level of the entire tool call rather than each token (as this let's later errors in the schema re-roll the earlier committed states)? (But like, it isn't clear to me that's what they mean, so I feel a need to ask; as an example, you are now talking about field order, but that's an entirely different issue involving an overly-constrained constraint: nothing would prevent implementing the grammar for the constrained decoding to allow the fields in any order, and if you allow duplicate fields -- which JSON typically allows anyway -- then it is probably already supported as the language loses its additional context requirement.)
for putting together something like a tool-call decision model, the conditioning will take less time than getting the data for it, which also will not take much time if you keep your session data. whatever fails is your negative contrast, whatever passes is your positive data. the space of tool calls your agents will use in your particular environment is not actually that big, which makes this one of the easiest wins you can pull off.
I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
So if I understand it correctly, the tool calls are not actually done by the model?
In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?
Could something like that also be used in other contexts?
I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
> but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
Yes, this is what constrained decoding means. You constrain the model to only output from a set of "allowed" tokens (a.k.a. grammars).
So, if you use tool use with constrained decoding, you can be certain that the model will always choose from a list of approved tools (i.e. if tools=['read','write'], it cannot output 'execute', even if in other sessions it could, if execute was in the approved tools).
The tricky problem is that certain models work / have been trained in certain ways. And constraining them can lead to "invisible" problems. Constrained decoding only guarantees perfect syntax, not perfect semantics (i.e. it could write nonsense in the correct file).
> Could something like that also be used in other contexts?
Sure. You can build pretty cool grammars and constrain the model to do cool stuff, akin to "tool use" but more free hand. You could, for example, make it always emit a 10 point plan. For any prompt, it will give you a fixed syntax 10 point plan, always. And the plan will be constant with a plan_title and 10{plan_item, item_text}. But, as above, there's no guarantee that the individual items in the plan would make any sense... (i.e. step7 in a pasta making recipe would be "add glue").
Correct, here's an implementation example using the Claude SDK [0] but it's the same approach on other platforms: you tell the model what tools it has available, it sends back a response saying it wants to use a specific tool with given parameters, your code executes that tool then sends the output back to the model on the next turn.
Exactly. The LLM is your next-token predictor, which output text and tool-calls. These calls get picked up by your client/harness (e.g., opencode) which executes them and sends output back to the LLM.
Would this technique work with inference optimization techniques like DFlash and MTP? In other words does it slow down things to the point where its just better to retry?
Yes, constrained decoding and speculative decoding work together, although they need to be carefully integrated and incorrect integrations give result in bugs (see point 3 under Section "Differences in Constrained-Decoding Behavior" in the blog post). Performance overhead does not generally seem to be a large problem. Of the speculative tokens, those that are not allowed by the grammar get discarded.
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
They are a special token that you can't just type out in text. It _renders_ as text, but cannot be input into the system as text. The model understands the difference between the textual toolcall token and the "special" one that's really just a spicy number.
(this is just made up tokenization but it's close enough to the real world)
If the user types <tool_call>, you get < (token id = 32), tool_call (41), > (102). It never decodes to <tool_call>, which is spicy number 12345, for example.
The tool call tokens are _always_ composed of multiple real tokens, so you can never get them from normal text.
Seeing <tool_call> is a bit misleading because that's just a debug rendering if anything. The front-end sees a special number and knows it's a toolcall. The chat template uses that text form because it's easier than writing the spicy numbers there, but it also has a special parsing mode for those.
The first place I think I encountered it, was GPT-2's tokenizer, back when OpenAI did actual open source stuff, maybe around 2020?
I'm guessing as with most stuff in the LLM ecosystem, someone did it, no one re-evaluated why and just copied it wholesale, and here we are. The most important point, that this sequence of text never occurs in "real text", is the main point of the specific character chosen, and seems to still work good for that purpose.
The original usage is only "<|endoftext|>" I think, which was the EOS token for GPT-2, then more and more special tokens were added as various delimiters to new model releases.
I really appreciate this. I hope LLM inference providers and Chinese LLM labs start using this latest vLLM version ASAP.
I do things that require ~300 tool calls per prompt and I have been stuck with OpenAI and Anthropic since 2024.
If people still wonder what moat OpenAI and Anthropic have: They got 100% tool-calling accuracy (using constrained decoding) and prompt caching right all the way back in 2024.
llama.cpp has done this since forever. I've used other engines including vLLM here and there, and I was quite surprised that many of them don't have this feature. Glad vLLM has it now.
Lots of code "typos" aren't immediately syntactic errors. Misbalanced parentheses, for example, only bite when one needs to close the series. The solution is to go back and fix the original error, but that option is foreclosed to autoregressive LLMs. In the meantime, forcing the model off what it "wants" to generate might make the remaining syntactically valid tokens semantically meaningless, impairing future generation in a way that's hard to quntify.
Constrained generation for tool calls is certainly a type of syntax checking, but it's one that can be justified because the "hard" syntax choices come up front, both in the very format of the tool call itself to the name of the tool and its parameters. Errors are more likely to be detected in the same root-cause tokens that generate them.
Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run. In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.
I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.
The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly.
To be fair, the course of constrained tool calling configuration in vLLM changed over time in ways that changed behavior, which is the real problem. The Ubicloud team deserves a lot of credit for having benchmarks in place to track performance and catch regressions for investigations. Some of the providers you find on OpenRouter have inexplicably bad performance relative to others and clearly do not care this much.
However, if this article left you confused because constrained decoding and constrained tool call schema have been features in vLLM for a long time, jump to the end where they show what configuration changes they made.
Just as a point of feedback for the authors, I enjoyed the article, and did not suffer the confusion pointed out above.
If the tool call is properly defined, meaning the LLM has the right instructions to produce the output defined by the schema, then I don't see why constrained decoding would reduce output quality relative to retrying until it succeeds.
I have seen some cases where developers (or let's be honest, someone's LLM) changed the tool calling definition/prompt but failed to update the output schema, so the schema was forcing outputs that the model didn't even know it was supposed to produce. This leads to confusing debugging sessions where you're getting outputs that look right but are low quality, which can be revealed by turning off constrained output and noticing that all of your constrained outputs look different than you expect. But when the schema matches the tool definition and prompt, it's hard to think of a reason the output might be worse. I see it as nudging the probabilistic bad tool calls back into what they should have been.
For a concrete example (happens less nowadays with bigger models, but it still happens, especially later into a context window), models hate long json strings. It's very easy to get a json output like
{"name": "Bob", "biography": "Bob is a fantastic builder, having ...
Critically, the ellipses terminate the JSON. With schema-constrained decoding, you _must_ eventually terminate that broken string, and it'll typically terminate it right after the ellipses. Basically, by the point you got to the first period the output was borked. You were backed into a corner. Retry as a strategy fixes that.
As another example failure mode that's a bit more subtle, you might have a set of "correct" tokens early on in the output which get the KV cache into a bit of a wonky state, causing it to have worse outputs toward the end of your output. Disallowing invalid next states forces you to generate tokens in that worse-output state. Retry lets you fail more often and then hopefully not have those wonky-state early tokens. It's not really that you were backed into a bad state as a hard rule, but that you were backed into a bad state as a probabilistic statement.
For a toy model to think about, suppose the entire grammar is [ab, ba], and the LLM's universe of training data is [ab, ab, ba, bb]. The "right" distribution (retry on failure) is {ab: 67%, ba: 33%}. The schema-constrained distribution is {ab: 50%, ba: 50%}.
i.e. it might be that a particular model may need some additional conditioning, or perhaps adjusting the order of the json output affects it to produce lower-quality results.
You have to remember that the LLM will be expected to generate 10s of thousands of tokens on a single turn, where things like quantization noise lead to cascading error further on. So when you can, you really want to stay within the model's distribution.
In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?
Could something like that also be used in other contexts?
I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
Yes, this is what constrained decoding means. You constrain the model to only output from a set of "allowed" tokens (a.k.a. grammars).
So, if you use tool use with constrained decoding, you can be certain that the model will always choose from a list of approved tools (i.e. if tools=['read','write'], it cannot output 'execute', even if in other sessions it could, if execute was in the approved tools).
The tricky problem is that certain models work / have been trained in certain ways. And constraining them can lead to "invisible" problems. Constrained decoding only guarantees perfect syntax, not perfect semantics (i.e. it could write nonsense in the correct file).
> Could something like that also be used in other contexts?
Sure. You can build pretty cool grammars and constrain the model to do cool stuff, akin to "tool use" but more free hand. You could, for example, make it always emit a 10 point plan. For any prompt, it will give you a fixed syntax 10 point plan, always. And the plan will be constant with a plan_title and 10{plan_item, item_text}. But, as above, there's no guarantee that the individual items in the plan would make any sense... (i.e. step7 in a pasta making recipe would be "add glue").
[0] https://platform.claude.com/docs/en/agents-and-tools/tool-us...
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
... How did they get into the training data?
If the user types <tool_call>, you get < (token id = 32), tool_call (41), > (102). It never decodes to <tool_call>, which is spicy number 12345, for example.
The tool call tokens are _always_ composed of multiple real tokens, so you can never get them from normal text.
Seeing <tool_call> is a bit misleading because that's just a debug rendering if anything. The front-end sees a special number and knows it's a toolcall. The chat template uses that text form because it's easier than writing the spicy numbers there, but it also has a special parsing mode for those.
I'm guessing as with most stuff in the LLM ecosystem, someone did it, no one re-evaluated why and just copied it wholesale, and here we are. The most important point, that this sequence of text never occurs in "real text", is the main point of the specific character chosen, and seems to still work good for that purpose.
The original usage is only "<|endoftext|>" I think, which was the EOS token for GPT-2, then more and more special tokens were added as various delimiters to new model releases.
I do things that require ~300 tool calls per prompt and I have been stuck with OpenAI and Anthropic since 2024.
If people still wonder what moat OpenAI and Anthropic have: They got 100% tool-calling accuracy (using constrained decoding) and prompt caching right all the way back in 2024.
I have multiples of that with some regularity, what is the hangup?
https://github.com/huggingface/transformers.js/blob/main/pac...
Lots of code "typos" aren't immediately syntactic errors. Misbalanced parentheses, for example, only bite when one needs to close the series. The solution is to go back and fix the original error, but that option is foreclosed to autoregressive LLMs. In the meantime, forcing the model off what it "wants" to generate might make the remaining syntactically valid tokens semantically meaningless, impairing future generation in a way that's hard to quntify.
Constrained generation for tool calls is certainly a type of syntax checking, but it's one that can be justified because the "hard" syntax choices come up front, both in the very format of the tool call itself to the name of the tool and its parameters. Errors are more likely to be detected in the same root-cause tokens that generate them.