Skip to content

Back to the catalog

Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis
arxiv.orgpaper

Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis

SynthAPI is a pipeline that generates synthetic tool-use environments with explicit control over diversity of tools, tasks and underlying databases. Across 458,752 tasks and 849 trained models of four sizes, the authors fit a scaling law predicting downstream agent generalization. At larger data budgets, scaling tool and database diversity beats adding more tasks, and higher complexity helps more. Agents trained this way beat prior synthetic-data baselines on four benchmarks.

TL;DR Derives scaling laws for tool-use agent training data using the SynthAPI synthetic environment generator, finding tool and database diversity matter more than task count.
Open original ↗
Document

What it claims

SynthAPI is a pipeline from Apple researchers, dated October 2026, that generates executable, database-backed tool-use environments. It has explicit knobs for four things: the number of apps, the tasks per app, the database instantiations per task, and the amount of distractor data. The authors fit a composition-aware scaling law on these knobs. They held out the top 25% of training-FLOP runs, which reach up to 16× the largest budget used for fitting. On those held-out runs the law predicts normalized downstream score with a mean absolute error of 0.0285. A law based on task count alone has an error of 0.1706.

How it works

A "builder LLM" writes every component from scratch in Python. The builder is Gemini-3.1 Pro, with internet access disabled. Apps come from a hand-defined taxonomy: shopping, music, productivity, finance, fitness, messaging, utilities and social. Each app gets three generated parts:

  1. Metadata: a name, a description, a category and a list of functionalities.
  2. A db.py file of Enum and dataclass schema definitions.
  3. An apis.py file of functions that read or modify the database. Public functions become the agent's tools.

Each task is a tuple of instruction x, verifier v, starting database Db₀ and oracle solver π*. The steps are:

  1. Instructions are sampled from F subsampled functionalities. Earlier instructions are fed back as context to increase diversity.
  2. The verifier is Python code. It checks the start and current database states, plus an optional answer string.
  3. Db₀ is a minimal "core" state plus D−1 batches of distractor data.
flowchart LR
  I[Instruction x] --> V[Verifier v]
  V --> DB[Db0: core + D-1 distractor batches]
  DB --> O[Oracle solver π*]
  O --> F{v(Db0)=0 and v(after π*)=1?}
  F -- yes --> K[Keep task]
  F -- no --> R[Reject]

The oracle solver can see privileged identifiers in Db₀. The agent cannot, and has to discover them through the tools. Tasks are packaged as Gymnasium environments. The agent acts by sending Python code blocks, so it can make several tool calls, loops and conditionals in one turn. Tool results come back as JSON. Tools are not listed in the prompt. Instead the agent uses three meta-tools: list_apps, list_tools and get_tool_documentation. These also return apps and tools the task does not need, so the agent has to ignore distractors.

Evidence

The study covers 458,752 tasks and 849 trained models at four sizes. Figure 2 reports results for a 9B student. The sweeps cover:

  • apps A from 8 to 64,
  • total tasks N = A×T(×S) from 32 to 262,144,
  • databases per task S from 1 to 4,
  • distractor depth D from 1 to 4.

As the budget grows, three changes beat adding more tasks for the same tools: more apps, re-instantiating existing tasks over different databases, and adding more distractor data.

Downstream evaluation uses AppWorld, BFCL and ASTRA-Bench, against several baselines:

  • Nemotron-Agentic (NVIDIA, 2025; 335k LLM-simulated trajectories): SynthAPI-trained agents score +30.5 percentage points higher, averaged over the three benchmarks.
  • Strongest per-benchmark baseline at each model size (SimSAT, Xu et al. 2026, Song et al. 2026): SynthAPI leads by +7.6 to 11.6 points on AppWorld Test-Challenge and +2.0 to 3.7 points on BFCL.
  • Task-count-only recipe at the same data budget: the law-derived recipe wins at all model sizes on all benchmarks.

Limits

  • The fetched text stops partway through Section 3. It does not show the scaling-law equation, per-size model details, training compute, or full result tables.
  • Only the 9B size is named; the other three sizes are not given in the text shown.
  • The existing summary says four benchmarks, but the text names three and averages over three.
  • Generated instructions are fully specified and need no interaction with a user. Clarification behaviour is therefore not covered by the training data.
  • The pipeline depends on a proprietary builder model.
  • The page states no code, data or weights release, and no licence.

Technologies named here

Techniques named here

  • Controllable environment synthesis with SynthAPI

Related in research