From Evidence to Action: How Tool-Using Agents Fail

What it claims

SafeActBench scores the path to an action, not just the end state. An agent that refunds the right charge with the right amount still fails if the evidence it gathered was about a different entity, even when that entity's values happen to match. In the paper's example, the agent reads charge C1 and then refunds C2 for $49.99. Endpoint checks pass, the action is permitted by the true task state, and the benchmark still marks it unsupported.

The authors also make a claim that matters for anyone using function-calling accuracy as a proxy for agent safety: good static judgment of whether an action should happen does not predict good interactive execution.

How it works

Protocols. There are 656 cases in five protocols, each with its own strict success condition:

V0. Refusing to act is not enough to pass V0. The agent must first resolve the evidence gaps that justify stopping. Stopping before that counts as a failure, and acting before prerequisites are established counts as premature.

Domains. The six domains are customer & policy, operations & code, legal & finance, research, smart home, and healthcare operations.

Evidence Ledger. Each established fact is bound to the interaction it came from and to the entity or state it describes. Action results count as evidence too. A notification must use the refund_id returned by the relevant refund, not an ID from another operation. When the tool semantics require confirming a state change, a separate observation is required.

Evaluator. The evaluator replays the trajectory deterministically. It checks required investigation, the tool, target and exact arguments, execution receipts, dependency order, and resulting state. It does not use an LLM judge and does not inspect the agent's hidden reasoning.

What the agent sees. Agents get only the task instruction and the tool schemas. Evidence requirements and expected actions are hidden.

flowchart TD
  T[Task + tool schemas] --> I[Investigate]
  I --> Q{Required evidence established<br/>for this exact entity?}
  Q -- obtainable but missing --> I
  Q -- unresolved or violated --> S[Stop: investigated non-action]
  Q -- yes --> A[Execute action]
  A --> D{Downstream action<br/>depends on result?}
  D -- yes --> Q
  D -- no --> E[Done]

Evidence

The fetched text gives only the qualitative findings from ten model–harness configurations:

• Failures cluster before execution, as incomplete investigation or acting before evidence exists.
• Single-action execution is usually reliable once evidence is in hand.
• Multi-action workflows add unresolved prerequisites and incomplete execution.
• In controlled evidence interventions, agents retrieved relevant information and still acted when decisive support was missing.

Limits

• The page text cuts off in Section 4.2. No per-model or per-protocol scores appear, and the ten configurations are not named, so the size of the static-versus-interactive gap cannot be checked from this page.
• There is no mention of a code or data release, a licence, or a run command.
• By the authors' own framing, scoring is over the observable trajectory. It says nothing about what the model internally relied on.
• The ledger needs per-case evidence requirements authored by hand. Applying the method to your own tools means writing R(a) for each consequential action.