TokenRouter: Efficient Serving System for Token-Level LLM Routing

What it claims

TokenRouter is a serving runtime for token-level LLM routing, where successive tokens of one request may be generated by different models. Most production routing works at the session or query level. The authors say current single-LLM serving systems handle token-level routing poorly for three reasons: step desynchronization across models, frequent batch admission delays, and high implementation complexity for developers. TokenRouter reports 2.01–64.15x higher decoding throughput than existing systems, measured across several routing algorithms, workloads and model pairs.

How it works

The abstract describes three pieces:

1. Request-centric programming. The developer writes the routing logic from the point of view of one request. They do not manage per-model batching.
2. Model-centric execution. The runtime starts one subserver per LLM and dispatches requests to those subservers asynchronously.
3. Delayed batching. Each subserver uses a delayed-batching scheduler. Its hyperparameters are derived from a mathematical throughput model of the system rather than tuned empirically.

How to use it

The code is public at https://github.com/thu-nics/TokenRouter. The page gives no install commands, supported backends, API shape or licence, so check the repository for those before planning adoption. No models or datasets on Hugging Face link to this paper yet.

Limits

• Abstract only. Everything above comes from the abstract, so it is all the page can support.
• Range of results. The 2.01–64.15x figure is a range over unnamed configurations. The abstract does not say which baseline systems were used (for example, which serving engines), which model pairs, or what hardware. You cannot tell which end of the range applies to your setup.
• Throughput only. The only metric reported is decoding throughput. Latency, time-to-first-token and quality effects are not stated.
• Algorithms not named. The specific token-level routing algorithms tested are not listed. Confirm that yours matches what the repository supports.
• Throughput model assumptions. The assumptions behind the throughput model, which sets the scheduler hyperparameters, are not described. You need to know them to judge whether the derived settings carry over to your workloads.