AutoSynthData: Generating Training Data for Enterprise Agents

What it claims

AutoSynthData generates enterprise-agent training tasks aimed at a model's capability boundary: tasks the target model fails but a stronger teacher solves. On EnterpriseOps Gym, SFT on about 2,000 generated samples improved Gemma-4-26B-A4B-it on two domains. The page does not mention any code, data or weight release, so treat this as a method description and not a tool you can adopt.

How it works

• Task unit: each task has three parts. A system specification gives the instructions, policies and optional seeded DB or knowledge articles. A user prompt states the request. A verifier checks the final state; it must accept any valid solution, not just one reference trajectory.
• Leakage control: gaps found on the evaluation tasks become sanitized "capability specification cards." The generator never sees the original prompts, entities, trajectories or verifier details.
• Difficulty filter: a task is favoured when the target solves it at most 1 of 3 trials and the teacher solves it at least 2 of 3. The teacher's successful trajectory becomes the SFT demonstration.
• Target phase: parallel workers generate core samples.
• Multiply phase: accepted samples are expanded into variants, each with its own state, entities, trajectory and verifier. A multiplied sample may not seed another multiplied sample, to limit drift across generations.
• Architecture: a shared controller handles generation, QC, coverage and dataset building. A per-environment adapter handles execution, state, reference replay, deterministic verification and the solver. Porting to a new environment means writing an adapter.
• Batch meta-review: reviews accepted and rejected samples and critiques. It downweights overrepresented task families and steers effort toward gaps and away from regions that keep failing.

flowchart TD
  C[Spec card] --> G[Generate task]
  G --> D{Difficulty: target ≤1/3, teacher ≥2/3}
  D --> P{Positive gate: reference passes verifier}
  P -- pass --> N{Negative gate: mutated outcomes fail}
  N -- pass --> A[Accept]
  P -- fail --> R[Critic diagnoses, targeted repair]
  N -- fail --> R
  R -- retries left --> P
  R -- limit hit --> X[Discard]
  A --> M[Multiply variants, re-checked]

Evidence

All runs use Gemma-4-26B-A4B-it as the target model and SFT only.

• On Hybrid, the best checkpoint was epoch 5.
• The page attributes the longer ITSM generation time to two things: a larger teacher, and the run predating pipeline throughput optimizations.
• The page does not give the generation hardware or cost.

Limits

• Narrow evaluation: every result is in-environment on EnterpriseOps Gym. The authors call it a "controlled setting" and claim improvement only in the environment trained on. Transfer to other environments is not tested.
• Same source for gaps and scoring: the generator is isolated from the evaluation tasks, but the same evaluation set was used both to find the capability gaps and to score the result.
• Missing details: the absolute Hybrid baseline Pass@1 is not stated. No ablations are reported, for example of the negative gate, multiply or meta-review.
• One round only: the iterative "moving frontier" is described, but results cover a single round.
• RL untested: RL use is proposed, not tested.
• Teacher dependence: the method needs a teacher that already solves the target tasks. Gains are bounded toward that teacher's level.