A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

What it claims

AgenticBBO-Bench puts LLM agents and classical black-box optimizers under one finite-budget protocol across five existing suites:

• BBOB: synthetic, continuous search space.
• Bayesmark: hyperparameter optimization (HPO), mixed space.
• DBTune: database tuning, mixed space, scored by a learned surrogate.
• BBOPlace-Bench: chip macro placement, continuous space with feasibility constraints.
• GuacaMol: molecular design, structured search space.

The numerical baselines are domain-specific: TuRBO for continuous problems, TPE for HPO and Graph-GA for molecules. On the five-task frontier challenge, run under the Codex harness, Figure 1 reports overall scores of 59.1, 58.4, 51.2, 50.7, 43.4, 40.2 and 36.8. These appear to correspond, in legend order, to:

1. GPT-6 Astra
2. DeepSeek V4.1 Flash
3. GPT-5.6 Sol
4. Gemini 3.1 Pro
5. GLM-5.3
6. Kimi-K3
7. DeepSeek V4 Pro

That pairing comes from extracted figure text, so check it against the figure. The authors stress that models with similar scores can differ substantially in cost. Cost per task is plotted on a log axis spanning roughly $0.5 to $50 at official list prices.

How it works

An "Agentic BBO" round is any variable sequence of actions the agent chooses before committing to a candidate. Those actions can include reading task information and priors, analyzing the history, calling optimizers or simulators, and writing its own code or tools. A classical optimizer is just one tool, whose output the agent may use, modify or ignore.

The protocol fixes the outer loop:

• For each task and seed, every method gets the same initial observations and the same budget of objective evaluations.
• Only objective evaluations count against the budget. LLM inference, code execution, tool calls and candidate validation do not.
• Candidates are validated first. Invalid submissions go back to the agent for revision and do not consume budget.
• Agents receive a common prompt that specifies the task, search space, available task information, history and output format. The template is in Appendix A.2.

flowchart LR
  H[History + task info] --> A[Agent: analyze, tools, code]
  A --> C[Candidate]
  C --> V{Valid?}
  V -- no, free --> A
  V -- yes --> E[Objective eval: costs budget]
  E --> H
  E --> B{Budget left?}
  B -- no --> S[Score best-so-far trajectory]

Scoring uses the whole best-so-far trajectory, not just the final value. It follows the normalized anytime protocol of RSI-Exam, which rewards early progress as well as the end result.

Evidence

Beyond the abstract-level claims, the fetched text gives no per-domain numbers. The Table 1 characteristics also matter when reading results:

• Task semantics: only the synthetic family exposes none. HPO, database tuning, chip design and molecular design all expose them.
• Feasibility constraints: chip design has them. The molecular-design row of Table 1 is cut off in the fetched text, so whether it does too is unconfirmed.

This matters because the ablations find task semantics to be the broadly useful input. The agent's advantage may therefore depend on how much meaning a task exposes.

How to use it

Code is at https://github.com/lamda-bbo/agentic-bbo. The five-task frontier challenge, one task per domain, is meant as a standardized leaderboard. To compare a new model or harness against the reported numbers, use the same harness (Codex), the same initial observations and the same evaluation budget.

Limits

• Compute is not budgeted. The evaluation budget ignores inference and compute cost, so "beats the best numerical optimizer in four of five domains" holds at equal objective evaluations, not equal wall-clock time or dollars. Cost is reported only for the frontier challenge, at list prices.
• Harness dependence. The authors note that results depend on the whole agent system (harness, tools and information given), not on the LLM alone. Leaderboard ranks hold for the Codex harness only.
• Domain coverage. Agents fail to beat the best numerical optimizer in one of the five domains. The fetched text does not say which.
• Surrogate-scored domain. Database tuning is scored by a learned surrogate, not a real database.
• Truncated source. The fetched text stops at Table 1. Per-domain results, ablation numbers and appendix configurations were not available for this note.
• Preprint status. This is v1, posted 8 Oct 2026.