Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs

What it claims

mmVB picks a resolution level for each image before the vision encoder runs. It uses only pixels plus the query, so it cuts vision-encoder compute as well as LLM prefill. The base MLLM is never touched. The authors therefore pitch it as usable with closed or API-only models, but they evaluate only one open model. On Qwen3.5-4B across eight multi-image benchmarks, it beats the four token-reduction baselines on the accuracy/compression harmonic mean on most benchmarks. On a few it loses clearly to full resolution: MIRB and ReMI.

How it works

Inference

• Each image goes through a frozen EfficientNet-B0 and is projected to d=256.
• The query is embedded with the MLLM's own frozen embedding layer, mean-pooled, and passed through a 64-dim bottleneck.
• A 2-layer, 4-head Transformer runs over the N image tokens plus the query token. This is where cross-image, query-conditioned allocation happens.
• A shared MLP (256→64→4) gives 4 ordinal logits per image. The level is the count of sigmoids above 0.5, so it falls in {0,…,4}.
• Each image is resized down to its level, never up.
• Total cost is about 1.6M trainable parameters and about 2 ms overhead.

Training

• The loss is ordinal BCE over the cumulative targets 1[ℓ* ≥ k]. A label that is off by one level costs only one threshold.
• Labels come from running the MLLM itself at multiple resolutions:

flowchart TD
  A[Sample] --> B{Correct at full res?}
  B -- no --> X[Discard]
  B -- yes --> C[Per image: lowest level 0-4 that is correct, others at full res]
  C --> D{Joint allocation correct?}
  D -- yes --> L[Label]
  D -- no --> E[Relax: +1 on one image, else +1 on all; up to K=5 rounds]
  E -- success --> L
  E -- exhausted --> F[All level 4]

• Cost is O(5N+KN) forward passes per multi-image sample. Labeling is done once, offline.
• The labels are sufficient but not minimal. The greedy relaxation can over-allocate, a bias the authors argue is deliberately conservative.
• A text-only ablation produced nearly uniform levels. Image content is needed for the allocation to mean anything.

Data

• 484k single-image samples from 30k images (VQAv2, TextVQA, GQA), with several queries per image.
• Then, as a curriculum, 50k multi-image samples from Mantis-Instruct.

Evidence

The base model is Qwen3.5-4B, with greedy decoding and thinking disabled. Accuracy is shown as full resolution → mmVB, with the strongest baseline in brackets:

Tokens per sample

• mmVB cuts tokens from 1,237–3,236 down to 89–594.
• ZooPrune (CVPR'26) uses fewer tokens than mmVB on 6 of 8 benchmarks, e.g. Visual Haystacks 239 vs 594, MMIU 300 vs 453.
• mmVB wins on accuracy rather than raw compression.

Prefill (TFLOP)

• mmVB is lowest on 6 of 8 benchmarks, e.g. MMIU 138,050 → 11,429.
• ToFu is lower on Visual Haystacks and MMIU.

Time to first token

• Mostly 0.5–0.7 s, against 1.6–11.8 s at full resolution.
• MIRB is 1.3 s, where ZooPrune gets 0.7 s.
• ReMI is still 15.2 s (vs 22.2 s at full resolution), slower than VisionTrim (11.4 s) and ZooPrune (11.6 s).

Harmonic mean of retention and compression

• mmVB is best or tied on 7 of 8 benchmarks.
• ZooPrune wins Visual Haystacks, 0.90 vs 0.82.

The headline 94.4% is a geometric mean of per-benchmark accuracy ratios.

How to use it

The fetched text gives no code, weights, licence or run commands. Optimizer, schedule and level-to-resolution mapping are deferred to the supplementary material and Appendix A.

Limits

• One base model. Everything is evaluated on Qwen3.5-4B with thinking off. The API-model compatibility is argued (Fig. 1), not tested.
• Labels are tied to an MLLM. They come from querying an MLLM, and the page doesn't name which one in the text shown. Transfer of a trained budgeter to a different MLLM is not shown.
• Real accuracy losses on some tasks. MIRB drops 10.7 points, ReMI 5.5 and NLVR2 3.7. Check your task type before relying on the 94.4% average.
• Small, related eval set. Mantis-Eval has 217 samples and comes from the same authors as the Mantis-Instruct training data.
• Truncated page. The fetched text ends at the baselines/ablations section, so the ablations are not covered here.