TraceDance: Automated System for Building Agent Behavior Benchmarks from Real-World Deployment Traces

What it claims

TraceDance (ByteDance / UIC) turns a natural-language description of a bad agent behavior, such as "destructive action without confirmation", into a benchmark built from real deployment traces. Each instance is the recorded context cut just before the deployed model misbehaved. Across 252,557 sessions, collected over six weeks from Claude Code (75,076, coding) and OpenClaw (177,481, general tool use), it fulfilled 95.3% of build-target requests. Human annotators confirmed the requested behavior in 84% of sampled instances.

How it works

The user supplies a behavior query, a trace collection and a size range [Kmin, Kmax], with defaults of 5 and 50. The system returns a benchmark in that range or a rejection reason. Queries can carry trace constraints (e.g. context over 100,000 tokens) and combine up to two behaviors:

• AND: both rubrics must pass on the same response.
• OR: a separate benchmark is built per behavior.

Each behavior specification has four parts:

• an executable anchor (Python-style detection code over tool errors, arguments, event order and keywords)
• a confirmation criterion
• a cut contract
• a 0–5 rubric

Three LLM roles are used to construct benchmarks:

• Strong Model (GPT-5.6-Sol): selects or synthesizes specifications.
• Fast Model (DeepSeek-V4-Flash): confirms candidates.
• Reviewer Model (Claude Opus 4.8): code-reviews generated anchors.

flowchart TD
  Q[Behavior query] --> C{Catalog match?}
  C -- yes --> S[Validated spec]
  C -- no --> G[Strong model generates spec]
  G --> R[Safety checks + reviewer review]
  R --> T[Trial run: CPU scan + sampled confirmation]
  T -- fail --> G
  T -- budget exhausted --> X[Reject]
  T -- pass --> S
  S --> A[Anchor scans all traces on CPU]
  A --> F[Fast model confirms candidates]
  F --> I[Cut by frame → instances + rubric]

The trial run is the step the design depends on. Low confirmation rates signal an anchor that is too broad, and too few candidates signal one that is too narrow. Both statistics feed the revision.

Cuts depend on the decision being tested, called the frame:

• Action: cut before the action.
• Failure: keep the failure, drop the response.
• Claim: keep the work, drop the completion claim.

Harness-forced turns, such as compaction summaries, are never cut points. Source-model reasoning blocks are stripped from the context. The evaluated model produces one next turn, which is never executed. Grading uses a three-judge panel (GPT-5.6-Sol, Gemini-3.5-Flash, Claude Opus 4.8), and a response passes if the mean score is ≥4. Because there is no environment replay, traces that depend on private tools or MCP servers remain usable.

Evidence

• 107 benchmarks with 4,125 instances in total.
• The automated grader agrees with human pass/fail judgments about as often as the two annotators agree with each other. The page gives no kappa or percentage for this.
• Across nine frontier LLMs, overall pass rates range from 22.9% to 33.5% (mean 26.7%).
• Pass rates vary sharply by behavior type:
• 67.9% mean when the next tool call only needs to be well formed.
• 8.1% when a check is required before proceeding, for example reading the error log before retrying a failed background job.
• Example behaviors in the catalog:
• Coding: out-of-scope edits, failing to link a new test failure to its own change, completion claims without rerunning checks.
• Tool use: retrying without applying the fix the error message gives, abandoning delivery after a failed send, false success reports.

How to use it

Code is at https://github.com/ZhishanQ/TraceDance, and the project site is zhishanq.github.io/TraceDance/. You bring your own traces. The page does not say whether the 252K-session corpus or the 107 benchmarks are released. Construction depends on three external LLMs and grading on three more, so expect API cost per benchmark.

Limits

• Only behaviors with observable signals that programmable anchors can retrieve are in scope. Behaviors that would require LLM or human review of every session are excluded by design.
• Only a single next turn is graded. Multi-step recovery and actual outcomes are not measured.
• The source model's response is evidence of the bad behavior, not a reference answer, so pass/fail rests entirely on the LLM judges and the rubric.
• 16% of sampled instances were not confirmed by both annotators as showing the requested behavior.
• The fetched text ends partway through Section 4. Per-model results, ablations and the authors' own limitations section were not visible.