TokenCast: Forecasting Token Consumption During LLM Agent Execution

What it claims

TokenCast forecasts an agent's token spend at four points in a run, using a cost representation that carries forward the cost of context re-read by later calls:

• Task Start: total before execution.
• Call Start: one call's cost once its request is assembled.
• In-call Update: that call's cost as the output streams.
• Task Update: remaining cost after each completed call.

All four are gradient-boosted (LightGBM) models trained on your own completed traces. No LLM is queried for the estimate. They also produce 5–95% prediction intervals. The biggest gains are in the mid-run updates, not the up-front estimate.

How it works

A segment is one or more consecutive calls. It is represented as φ = (n, g, b):

• n is the call count.
• g is the net change in input length across the segment.
• b is the residual cost, C − n·L, where L is the segment's starting input length.

Two adjacent segments compose exactly:

φ_A∘B = (n_A + n_B, g_A + g_B, b_A + b_B + n_B·g_A)

The n_B·g_A term is the context added by A, billed again on every call of B. Billed reasoning tokens that are not kept in context count toward b but not g.

The task-level forecast runs two paths and merges them:

flowchart LR
  F[Visible task + execution features] --> D[Direct model: remaining cost]
  F --> P[Prefix model: n, g, b + boundary state]
  P --> S[Suffix model, conditioned on predicted boundary]
  P --> C[Compose via identity]
  S --> C
  D --> R[Correction model]
  C --> R
  R --> O[Remaining-cost forecast]

Training details:

• Separate models. The direct, prefix and suffix predictors are separate LightGBM models.
• Weighted losses. Prefix errors in g are weighted by n_B, and suffix call-count errors by L_B, because those errors are amplified through the identity.
• Cross-fitting. The suffix model is trained on out-of-fold prefix predictions, with its residual label recomputed against the predicted baseline.
• Correction model. A final model is fit on out-of-fold pipeline outputs. Its inputs are the visible features, both forecasts, their difference, and the predicted boundary variables.
• Intervals. Separate quantile models give the 0.05 and 0.95 endpoints. These are widened using held-out calibration tasks and never fall below the tokens already confirmed.

Evidence

Data. 11,712 traces from 240 tasks across four suites: SWE-bench Verified, Search-R1, MMLU-Pro and LongBench-v2. Six agent models: GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, DeepSeek-V4-Pro, Qwen3.8-27B and Llama-3.2-3B-Instruct. Traces were collected with DeepSeek Harness and OpenHands. All runs of a task stay in the same split.

Baselines. TRAIL, EGTP and TIE (output-length predictors) and Self-Prediction (agent self-estimate), each adapted to every prediction point.

MAE in tokens from the fetched portion of Table 1. "Norm. Avg." is MAE divided by a history-median predictor's MAE, averaged over the four points; lower is better.

How to use it

• Code: https://github.com/DEFENSE-SEU/TokenCast. The fetched text gives no install commands and no licence.
• What you need: a corpus of completed traces from your own agent and harness, logged per call with provider-billed input/output tokens and input lengths. The models are trained on these.
• Pinned models in the paper: gpt-5.4-2026-03-05 (API) and qwen3.8-27b-20260815 (self-hosted).
• Overhead: prediction is CPU-cheap, 32.8 ms cumulative per run on SWE-bench Verified.

Limits

• Task Start is weak. Absolute errors stay large: 144k tokens MAE on SWE-bench with GPT-5.4, only about 5% better than Self-Prediction. Most of the value comes after execution starts.
• Not uniformly best. TIE beats it at Call Start on Search-R1 with GPT-5.4.
• Budget savings are simulated. The 21.3% figure comes from offline replay, not live control. The fetched text does not describe the replay policy.
• Partial view of results. The fetched text was truncated mid-experiments. It shows MAE for 2 of 4 suites and 2 of 6 models. Ablations (e.g. base-predictor comparisons in Appendix D.6), generalization to independently released trajectories, and interval calibration results were not visible.
• Retraining likely needed. Forecasts are learned per trace distribution. Expect to retrain on traces from your own models and harness; the fetched text does not show how well it transfers to an unseen harness or model.