Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?

What it claims

BOTTLED evaluates whether an LLM agent, given an entire unlabelled workload of millions of instances, can build a cheap artifact that answers all of them within a fixed budget. The artifact might be a fine-tuned small model, a program, a retrieval index or a hybrid. The headline finding is that zero-shot skill is a poor proxy for this ability. On MAVE, GPT 5.6 Terra and Sonnet 5 both score about 0.44 zero-shot F1, but their mean bottled F1 is 0.405 and 0.154 respectively. Figure 1 also shows that more expensive models do not necessarily bottle better.

How it works

• Same harness for every model. All agents run in OpenCode, so differences come from the LLM rather than the scaffolding.
• Compute. Each run gets one A100 (40GB), 8 CPU cores and 64GB RAM. The agent may install packages freely.
• Budget. Each run has 10 hours and 5M weighted tokens. Cost is calculated as uncached input + 0.1 × cached input + 2 × output. Agents can check their remaining budget through MCP tools.
• API access. An agent may call only its own model's API. For example, Sonnet 5 cannot call Opus 5.
• Internet. Access goes through a proxy that blocks the benchmark datasets and models trained on their labels. An agentic judge also audits every run for contamination (Appendix D).
• Output and scoring. Predictions go to a single out.txt. Missing predictions are filled with uninformative fallbacks: an empty string for MAVE, a random label for ESCI, and 0.5 for RAID. An agent that delivers nothing therefore scores the same as one that answers uninformatively.
• Aggregate score. Each task score is normalised as relative gain, (score − fallback baseline) / (ceiling − baseline), and then averaged over 3 tasks × 2 runs.

Tasks

Baselines

• Zero-shot. The model answers one instance per call. Its cost is estimated from a uniform sample of 1,000 instances.
• Distillation students. Qwen3-0.6B and SmolLM2-360M-it are distilled under the same token budget as the agents.

Evidence

• Bottled vs. zero-shot. 48 of 60 runs fall below the lower bound of their model's 95% zero-shot confidence interval.
• Bottled vs. distillation. 31 of 60 runs score below the better of the two distilled students.
• Opus 5 on ESCI. Bottled macro-F1 is 0.499 at a reported cost of $26.28. Zero-shot macro-F1 is 0.609, with an estimated full-workload cost of $17,273.
• Jev comparison. Jev, a "system one" model built for cheap repetitive inference, scores 0.531 macro-F1 at an estimated $105 for the full workload.
• Figure 1 zero-shot cost labels. Opus 5 is $26.3k and Gemini 3.1 Flash-Lite is $5.2k. The Pareto plot puts bottled per-task costs between roughly $0.1 and $30.
• Models tested. Gemini 3.1 Flash-Lite and Pro, GPT 5.6 Sol and Terra, Sonnet 5, Opus 5, GLM 5.3 and 5.3 Flash, Qwen 3.8 Flash and Max.

Limits

• Artifacts are rarely hybrid. Execution-time calls back to the model were almost always zero, because agents typically built self-contained artifacts. In practice, the option of routing hard cases back to the model is barely exercised.
• Small sample. There are only two runs per model per task, so variance per model is thinly sampled.
• Contamination. The authors acknowledge that the blocklist may have missed leakage sources and rely on the post-hoc judge.
• Cost estimates. Zero-shot costs are extrapolated from 1,000-instance samples, not measured on the full workload.
• Instruction mismatch. The exact instruction differs between the zero-shot and bottled settings.
• Narrow task coverage. All three tasks are text classification or extraction, and two come from Amazon product data.
• Truncated page. The fetched text ends partway through the baselines section. Per-task tables, distillation details and the appendices were not available. The page links a GitHub repository, but no URL or run commands appeared in the text.