How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

What it claims

VocabX is Dharma-AI's method for rebuilding a fine-tuned small model's tokenizer around serving cost rather than token frequency. Each saved token is weighted by the price of the field it appears in (system prompt, user message, or response). In their Qwen3-4B Portuguese essay grader, a generated token cost 7.4 times an input token. The post previews a paper that is still under review, and the authors say the numbers may change before publication.

How it works

The method rests on three changes against FVT, MWT and in-place BPE, which select by frequency and work on only one layer:

1. Cost-weighted gain. A candidate's value is the number of tokens it eliminates, multiplied by the per-field price. Output tokens therefore dominate the selection.
2. Two layers under one criterion. A sub-word merge replaces the base vocabulary with a smaller domain vocabulary. Multi-word "macro" tokens are then added under a fixed budget. Each layer has its own budget.
3. Five miners for macro candidates. These look for long words, whole lines, expressions, complete sentences, and punctuation structures such as JSON and tags. Averaged over eight corpora, the structure miner contributed the most. On code it accounted for 17.5 points on its own. Removing the miners would cut token reduction on the grader by 55.6 points.
4. Estimating unseen candidates. The selection measures marginal gain on a document sample. Total corpus savings are counted exactly. Only the overlap with tokens already chosen is estimated, and candidates missing from the sample get a statistical estimate instead of zero. In a contracts corpus, only 16.6% of candidates appeared in a 400-document sample. The estimator alone raised token reduction from 44.7% to 52.8%.

LM-head size is not part of the selection criterion. It is controlled only through the sub-word replacement and the macro budget. In Qwen3-4B, the LM head is almost 10% of the parameters, and the full matrix is read for every generated token.

Evidence

Grader setup. Seven grader variants shared the base model, data, recipe and seed, and differed only in the tokenizer. Each ran the same 500 requests on an RTX 5090 with vLLM, and cost was computed from measured runtime.

Grader results. Responses fell from 350 to 101 tokens and cost fell 77.0%. The bench score (grading accuracy plus off-topic and rule-breaking screening) rose from 0.703 to 0.810.

A failed variant. A first vocabulary fused each competency grade into its JSON macro. It scored 0.689, below production, and saved only 73.3% of cost. Giving the grade its own token added 2 tokens per response but improved both quality and cost.

Public benchmarks. The comparison covered five domains (contracts, medical papers, scientific papers, news, Python code) and three families (Qwen3, Llama-3.1, Gemma-3), with a shared vocabulary ceiling. VocabX compressed more than the strongest baseline in all 15 combinations, by 1.2 to 9.3 points. The pruned variant led on estimated cost in all 15, with average margins of 10.2 points (Qwen3), 7.5 (Llama-3.1) and 13.6 (Gemma-3). On contracts, it reached 62.6% token reduction with 113,426 entries. The best in-place BPE configuration reached 61.8% with 207,803 entries.

Limits

• The paper is unpublished and under review, and its figures may change.
• The post gives no code, tokenizer, weights or licence.
• The 7.4× decode-to-prefill ratio is specific to the grader. The authors say it varies with hardware, batch size and context length.
• Cost results on public domains are estimated. Only the grader cost was measured.
• The fused-grade result shows that macros which swallow the model's decision token can hurt quality. Each new domain needs this checked.
• Every variant requires retraining the model with the new tokenizer.