Fixing Tool-Call Accuracy in vLLM

What it claims

Ubicloud reports that Kimi K2.6 served on vLLM 0.28 produced schema-valid tool calls 92% of the time on Moonshot AI's K2VV ToolCall benchmark, which is 2 failures in every 25 calls. Turning on XGrammar-based constrained decoding raised schema accuracy to 100% for Kimi K2.6 and DeepSeek V4 Flash. The main operational finding is that vLLM supports constrained decoding for tool calls but leaves it off by default. Deployments must enable it with the --tool-strict-level server flag, available from vLLM 0.31.

How it works

The inference engine wraps the harness's JSON tool schemas in model-specific tokens. For Kimi this is <|im_system|>tool_declare<|im_middle|>[...]<|im_end|>. A parser then turns the model's tool-call tokens back into OpenAI-style tool_calls JSON. Without constraints, the model can emit tool or field names it saw in training rather than in the active schema. The page's examples are search_web instead of web_search, and city_name instead of location.

vLLM uses XGrammar structural tags to define the legal path through a call. The path is: begin tag, tool name, a numeric index matched by regex \d+, the argument-begin token, JSON constrained to the tool's parameter schema, and the end tag. At each step, tokens that would break that path are masked before sampling. For example, only integer tokens are allowed at the index position. XGrammar precomputes the masks for each phase and applies them on the GPU. Ubicloud describes the overhead as negligible but gives no measurement.

Evidence

K2VV ToolCall has 2,000 requests and reports two metrics. Trigger similarity measures how closely the service's decisions about when to call a tool match Moonshot AI's own API. Schema accuracy is the fraction of calls that conform to the schema. The post reports schema accuracy only:

• vLLM 0.28, default settings: 92%.
• Upstream vLLM after constrained decoding landed in v0.25, but without opting in: 91%.
• Constrained decoding enabled: 100% "across all served model versions".

The post gives no trigger-similarity figures and no latency or throughput figures.

How to use it

Ubicloud's own history is a regression worth checking for. It first ran constrained decoding in a private vLLM fork. It then moved to the upstream feature added in vLLM v0.25, and accuracy fell back to 91%. The cause was that vLLM changed its default for requests that do not explicitly enable constrained decoding. The fix is to set --tool-strict-level at the server, from vLLM 0.31. Moonshot AI recommends constrained decoding for Kimi, and its own API runs with it on.

Limits

• The post does not give the value to pass to --tool-strict-level.
• The page says model support falls into "three camps" of problems, but the list is missing from the fetched text.
• vLLM maintainers saw problems with several models when constrained decoding was on, which is why it is off by default. Teams serving models other than Kimi or DeepSeek should validate before enabling it.
• Results cover only Kimi K2.6 and DeepSeek V4 Flash, and only schema accuracy on K2VV ToolCall.