ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model

What it claims

IBM Research's ALTK-Evolve turns an agent's own τ²-bench training-task trajectories into a markdown list of behavioral rules. The list is appended to the system prompt, with no weight updates and no human annotation. With gpt-4.1 on the stock τ²-bench ReAct scaffold, Pass^4 on held-out tasks rises 10 points in both retail (57.5 to 67.5) and airline (35.0 to 45.0). With gpt-5.5 on airline, held-out Pass^4 rises 15 points (55.0 to 70.0). The headline 67.5% retail and 82% airline figures are leaderboard numbers on the full base split. The held-out numbers above are a separate measurement.

How it works

1. Run the stock τ²-bench ReAct agent (two-phase THINK → ACT loop) on train tasks.
2. The same model that powers the agent (gpt-4.1, or gpt-5.5 in the second experiment) distills successes and failures into short rules, then consolidates them.
3. Each rule carries a support count: the number of distinct training tasks it was extracted from.
4. Filter by support. "sup3" keeps rules with support ≥ 3. "lossless" keeps everything after consolidation (support ≥ 1).
5. Append the rules after the <policy> block under the heading "Guidelines learned from past task attempts (follow these carefully)". Then run unchanged on held-out test tasks.

Example rules include: authenticate before any order action, with a fallback to find_user_id_by_name_zip; require explicit approval with a structured summary before any database write; never combine a tool call with a user-facing message in one turn; and split handles like daiki_johnson_9523 into first and last name before lookup.

Evidence

All runs use gpt-5.5 as the customer simulator and 4 trials per task.

• gpt-4.1, held out: Retail mean accuracy goes from 78.1 to 82.5 with lossless; sup3 barely beats baseline on accuracy. In airline the configurations split. Lossless gives the best accuracy (65.0 vs 57.5 baseline) but 40% Pass^4. The 14-rule sup3 set gives 60% accuracy but the best Pass^4 (45%).
• Turns: On runs solved under both settings, lossless shortens conversations slightly. Retail goes from 10.59 to 10.16 assistant turns, and airline from 9.15 to 8.82.
• Cost: Retail lossless has 292 rules, which raises cost per conversation from about $0.084 to $0.139 (agent plus simulator). Airline's smaller rule set costs less. The authors say prompt caching can recover most of this, but they do not measure it.
• gpt-5.5 airline, held out: Accuracy goes from 72.5 to 82.5 and Pass^4 from 55.0 to 70.0. No rule reached support 3, so the high-confidence core is sup2. The lossless result reproduced exactly across reruns, while the baseline varied.
• Leaderboard (base split, Pass^4, snapshot of October 12, 2026):
• Retail with gpt-4.1 scores 67.5% and ranks #1. It beats RAFT-30B-A3B (61.4%), a model fine-tuned on synthetic τ²-bench retail data.
• Airline with gpt-4.1 sup3 scores 44% and ranks #8. That is above GPT-5.2 (no reasoning) and close to Claude Sonnet 4.5.
• Airline with gpt-5.5 scores 82.0%, against frontier entries at 66–72%.
• The submissions are τ²-bench PRs #542 and #551.

How to use it

The library is at https://github.com/AgentToolkit/altk-evolve. The post gives no install commands or API details. Deployment needs only a prompt change, so it works with any model behind an existing scaffold. The extra cost falls on input tokens. Treat the support threshold as a setting to tune per domain: the retail and airline results point in opposite directions.

Limits

• Only two domains (retail and airline) are tested, both from one benchmark.
• The simulated customer is gpt-5.5, a stronger model than the gpt-4.1 agent.
• Leaderboard scores cover train ∪ test, and the guidelines were mined from the train tasks. Only the held-out numbers exclude tasks the rules were learned from.
• The best configuration differs by domain and metric. The page does not describe a validation step for choosing it.
• Test-split size and the full contents of Tables 1 and 3 are not given in the text.