FragToken: Amplifying LLM Inference Costs through Noncanonical Token Generation

What it is

FragToken (ICLR 2027, Wang et al.) is a training-time supply-chain attack in which an outsourced fine-tuning contractor modifies a third-party LLM before delivery so it generates non-canonical token sequences for ordinary prompts. The tokenizer is left unchanged; the deployer downloads and serves the model normally. The extra cost falls on all benign traffic rather than on triggered requests, so unlike prior resource-consumption attacks it isn't diluted when malicious requests are a small share of the load.

How it works

The attack exploits that a BPE detokenizer is non-injective: the same text can be decoded from many token sequences, and a sequence is non-canonical when it differs from the re-encoding of its own decoded text — z ≠ Eτ(Dτ(z)). LLMs normally emit canonical sequences only because they're trained on them; there is no runtime rule forcing canonical merges. Fine-tuning on longer, content-identical sequences makes the model favor fragmented generation, inflating autoregressive decoding steps without lengthening visible output (Figure 1: the same 10-token answer re-rendered as 21 tokens).

The three named techniques exist to keep utility from collapsing:

• Source-model self-distillation — train on the frozen source model's own responses (NLL 0.375 vs. 1.382 for Alpaca references), so only the tokenization changes, not the content.
• Capacity-aware budgeting — each response has an inflation capacity Ri = |ai|/|ci| (mean 4.55 over 10,000 samples). A fixed target over-atomizes low-capacity responses; instead a per-sample target ρi = clip(1+(1−β)(Ri−1), ρmin, min(ρmax,Ri)) re-merges a fixed fraction of available expansion.
• BPE-Aligned Merging — re-merges only within canonical-token-aligned spans using BPE-rank priority, preserving frequent subword structure the pretrained model relies on.

Why utility control is the whole attack

The paper's motivating result quantifies the naive baseline: on Llama-3.1-8B-Instruct with Alpaca, full atomization reaches TIR 4.11 but drops average accuracy from 81.88% to 72.41%, with GSM8K collapsing from 84.46 to 58.00. That quality failure is what makes crude fragmentation conspicuous; FragToken trades peak TIR (1.99–2.46) for near-intact utility to stay covert.

What to do

The page's own framing points at the detection signal: because visible text length is unchanged, output-length and repetition defenses miss this. The defining property is testable — re-tokenize generated output and flag when z ≠ Eτ(Dτ(z)), i.e. when a model routinely emits sequences longer than the canonical encoding of its output. The authors report that under their evaluated defenses FragToken's detection rate stays comparable to benign behavior, so token-count-per-character (not response length) is the metric to monitor. For anyone deploying third-party or outsourced-fine-tuned weights, this is the acceptance check to add.

Caveats

Values are from the paper's own evaluation (four LLMs, three benchmarks); the fetched text cuts off mid-methodology, so the full defense-evaluation setup and per-model TIR/utility tables aren't visible here. "Minor" utility degradation is the authors' characterization.