Your Agent Aced the Task. Will It Do It Again?

What it claims

A GPT-4.1 ReAct agent run at temperature 0.0 still gives different outcomes on repeated runs of the same AppWorld task. The authors trace this to "flat" next-token distributions at individual decision points. Near-tied options can reorder under platform-side noise such as GPU floating-point non-associativity and request batching, and greedy decoding or a fixed seed cannot prevent that.

ALTK-Evolve's new "consistency guidelines" find those flat steps in a single recorded trace and write guidance that stabilizes them. Note that the page's Pass^k means all k runs succeed. It is not Pass@k, which means at least one succeeds. The page states Pass^k ≤ Mean@k ≤ Pass@k.

How it works

1. Detect. The Consistency Analyzer takes one recorded trajectory. For each decision step it makes one extra model call against the already-recorded context, asking for k completions in that single call (k=5 by default). - It issues no new tool calls, makes no environment interactions and does no end-to-end rerun. - It is black-box: no logits and no model internals. - Output is a per-step consistency score written to a scorecard. - It targets instability, not failure, so it flags steps the agent got right but could get wrong next time.
2. Generate. Each flagged step becomes a candidate guideline in the standard ALTK-Evolve format. It goes into the existing storage and retrieval pipeline and is injected at inference time.

Example guidelines, generated by GPT-4.1 for the task "How many activities are done in my bucket list as per my SimpleNote note?":

• Use a line-anchored regex rather than a substring count for checkbox markers, because legend lines repeat the symbol.
• Verify note search results for multiple matches before proceeding.

In the demo, five parallel runs split 3–2 on this task before the guidelines were added and 5–0 after.

Evidence

The setup was AppWorld test_normal (168 tasks) with a GPT-4.1 ReAct agent at temperature 0. Guidelines came from one baseline trajectory per task and were evaluated on 5 fresh runs.

• Pass^5 gain by difficulty: Easy +12.2pp, Medium +22.9pp (+44% relative), Hard +14.3pp (+45% relative).
• Gap by difficulty: the baseline gap reaches 30 points on hard tasks.
• Mean@5: held or improved at every tier.
• Similar task: +13.0pp Pass^5 on a different variant of the same AppWorld scenario, against +16.0pp on the same task.
• gpt-oss-120b: same-task Pass^5 went from 10.1% to 16.1% (+6.0pp), and the similar-task gain was +8.7pp.
• About a third of previously inconsistent tasks became pass-every-run.

How to use it

The Consistency Analyzer and the guideline generation are in the open-source repo at github.com/AgentToolkit/altk-evolve. The full methodology is in an arXiv technical report linked from the page. The page gives no install commands, API details or licence.

Diagnosis cost is one extra LLM call per decision step, done once offline. It needs no grader or ground truth, which the authors argue makes it usable on production traces that cannot be replayed.

Practical advice from the page:

• Report Pass^k alongside Mean@k; even k=3 surfaces a gap.
• Expect the gap to widen on harder tasks.
• Don't assume a bigger model closes the gap. A stronger model raises Mean@k but not necessarily consistency.

Limits

• Evaluation covers only AppWorld test_normal, a ReAct agent and two models.
• "Similar-task" generalization was tested only within the same AppWorld scenario. Cross-benchmark or cross-domain transfer is not shown.
• Guidelines were mined from a single trajectory per task. The page reports no variance across mining runs.
• For gpt-oss-120b, absolute Pass^5 stays very low (16.1%).
• The post gives no token or dollar cost for analysis or guideline injection.
• Detection relies on sampling k completions in one call. The page does not say how this behaves on endpoints that don't support multi-completion sampling.