IntentDelta: A Human-Constructed Benchmark for Evolving User Intent

What it claims

IntentDelta is a benchmark from Georgia Tech and Microsoft Research (arXiv, 24 October 2026) that scores agents separately on what they believe the user wants and on what they actually do. It covers 638 trajectories (3,766 turns) built on WebShop and TravelPlanner. The authors report that intention recovery correlates with better action scores, but agents often fail to execute even when they fully recover the stated intention. Against Frames, Flex-TravelPlanner, COMPASS and Evolving User Intent, the authors say it is the only one that combines human construction, multiple domains, evolving priorities, both feasibility- and preference-driven changes, and per-turn human reference actions.

How it works

Intention is a set of constraints. Each has a value, a scope and one of three priorities: must-have, preferred or optional. A turn can add or remove a constraint, change its value, or change its priority. Changes come from two sources: infeasibility (for example, nonstop plus downtown costs $1,720 against a $1,500 budget) or a revised preference after the user sees alternatives.

At evaluation time, the user utterances are a fixed script replayed to every agent. Later turns do not react to the agent's actions, but each agent runs in the environment on its own. All eleven agents use a shared ReAct framework.

Intention metrics are computed after semantic matching of predicted constraints to gold constraints:

• Detection: detected gold constraints divided by all gold constraints.
• Content Accuracy and Priority Accuracy: computed over detected constraints.
• Intention Recovery: constraints with both correct content and correct priority, divided by all gold constraints. This is the primary metric.

Precision and retention of obsolete constraints are in Appendix F.

Action metrics:

• Hard Success: on feasible tasks, the agent must satisfy every must-have constraint. On infeasible tasks, it must satisfy at least as many as the human reference.
• Action Score: Hard Success × min(1, S_agent / S_gold), where S = 2 × (preferred satisfied) + 1 × (optional satisfied). Soft constraints only count when Hard Success holds.

Appendix E tests how sensitive the rankings are to the 2:1 weighting.

Evidence

The fetched text gives only the averages across the eleven agents:

The text names the main failures: earlier requirements get dropped, and preferences get promoted to must-haves. Per-model scores, the list of models, and Hard Success numbers are not in the fetched text.

How to use it

The benchmark reuses the WebShop and TravelPlanner environments and their original requests as the starting turn of each trajectory. An agent must output a constraint state with priorities at every turn, as well as its actions, to be scored on the intention metrics. The fetched text gives no release URL, licence or run command.

Limits

Members of the author team wrote all 638 trajectories, at about 5 to 10 minutes each. These are author-written trajectories, not logs from real users. Each trajectory was cross-checked by a second author. External annotators reviewed a 20% sample (128 trajectories) and reached 97% agreement. The scripted replay means the benchmark cannot test whether an agent's clarifying questions change what the user says next. TravelPlanner requests are synthetic in the original benchmark. The fetched page cuts off partway through Section 4, so the results tables and the authors' own limitations section were not available.