zzzz7788990213-ops/EvoVLM

What it claims

The gains are measured against hand-written seed inference programs, not against the stock models. For Qwen3-VL-2B-Instruct and Qwen3-VL-2B-Thinking, two generations of constrained search produce programs that hold accuracy (Instruct, 8.9% lower median latency) or raise it by 16.41 points (Thinking, 10.6% lower median latency) on Dev128. The unmodified Thinking baseline hit a 7,200-second timeout, so no completed Dev128 accuracy exists for it. The +16.41 is therefore relative to the manual seed and cannot be read as a gain over the default model.

How it works

1. Start from manually designed inference programs for each model.
2. A local Qwen model ranks a frozen catalog of typed, focus-specific mutations. It never writes code. A trusted renderer turns the chosen configuration into Python.
3. Candidates run on small search and selection subsets. Each run records correctness, latency, errors and execution provenance.
4. Pareto selection on accuracy and latency keeps the trade-offs and seeds the next generation. Each model population gets at most six children per generation.
5. Selected programs are exported as best_*.py and checked against their recorded evidence.

The authors state that this is not unrestricted code generation, not weight training, and not a full implementation of NSGA-II.

Evidence

• Accuracy: Dev128.
• Latency: a separate Speed32 protocol. For each of the 32 queries, take the median of 3 repeats, then take the median across the 32 queries.
• Source of the numbers: a sealed metric trace from the original runs. They were not rerun for this release.
• Verification: file hashes, model and processor identity checks, cross-bound attempt records, and metrics recomputed from per-query evidence. Missing historical fields are logged as legacy caveats rather than filled in.

How to use it

CPU-only replay and checks. These need no model weights and do not rerun inference:

git clone https://github.com/zzzz7788990213-ops/EvoVLM.git
cd EvoVLM/research/evolving-vlm-inference-main
python3 -B -m evaluation.portable_reproduce
bash reproduce.sh check
bash reproduce.sh smoke

• Always pass a mode. Running bash reproduce.sh with no argument starts a fresh GPU reproduction.
• Keep the directory name. Do not rename research/evolving-vlm-inference-main, because some verification routines depend on it.
• GPU runs need extra setup. You need PyTorch, Transformers and the external model snapshots; weights are not included. requirements.lock.txt is Linux/CUDA-specific and will not work as a macOS lockfile. Pinned model downloads are covered in the runtime guide.
• Partial data only. The repo ships CharXiv subsets and selected images, not the full dataset.

Limits

• Dev128 is not a held-out test set. It is a public validation subset. Two of its queries also appear in the evolution set, and eight more share two figures with it.
• Small, hardware-bound results. These are small-sample development numbers, not a CharXiv leaderboard result or evidence of generalization. Latency figures apply only to the recorded hardware and software.
• No repository licence. Only the existing third-party licences for the Qwen models and CharXiv apply to their respective materials. Check these before reuse.
• Not peer reviewed. The archived report.pdf keeps its original title, and the authors say publication of the repo does not imply conference submission or acceptance.