ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Models

What it claims

ReFract defines Perspective Awareness as two separately scored skills:

• Perspective-Taking: inferring what a given role wants and using only tools within that role's capability and knowledge boundaries.
• Perspective-Routing: escalating work that exceeds the user's remit to the right role.

Each entry poses the same query from at least two roles. Each role needs a different trajectory. The agent receives a persona but is never told its capability or knowledge sets, so it must infer them. The paper reports three findings beyond the headline 69% / >50% violation numbers:

• Every model degrades sharply as the set of available tools widens. A larger action space pulls agents across boundaries they otherwise respect.
• The dominant failure is over-cautiousness. Agents defer even when the user is fully authorized to act.
• Agents do worst for broad-authority personas, which must choose not to use tools within reach. For example, agents serving a Manager run a repair themselves instead of routing it to a Technician.

How it works

Each entry is a POMDP whose environment is a hand-authored, executable Python Text World Model. State is held in variables, tools are functions, and transitions come from executing those functions. No LLM simulates state. The structure follows PDDL:

• One Domain File, written with three industrial-maintenance experts and shared across all entries. It declares:
• predicates, e.g. status(device, faulty)
• capability gates, e.g. repair, approve
• knowledge domains, e.g. hardware, scheduling
• retrieval sources
• a tool taxonomy

Each tool carries preconditions, typed consume/produce slots, a capability gate, a knowledge gate and effects. A tool fires only if all of these hold: its preconditions are in the current state, its inputs were produced by earlier calls, and both of its gates are satisfied by the persona.

• Five generic roles: Operator, Technician, Controls Engineer, Planner, Manager. Capability and knowledge gates are assigned to them independently.
• Problem Files, one per query. Each specifies an initial state, the available tools and the goal states. These are built by an LLM pipeline:

flowchart TD
  Q[Support queries] --> F[CO-5 filter: ≥2 roles, multi-tool]
  F --> G[3-LLM ensemble proposes goals per role]
  G --> E[Experts annotate goals]
  E --> D[CO-5 drafts world model + trajectory]
  D --> X[Execute in Python, planner checks goals]
  X -- error / divergence --> D
  X -- reaches goal within 10 iters --> R[Gate tools per role]
  X -- fails 10 iters --> Drop[Discarded]

The pipeline's models are:

• CO-5 (Claude-Opus-5): query filtering, drafting world models and trajectories. It also emits a confidence score with each judgment.
• Goal ensemble: CO-5, Claude-Haiku-4.5 and Claude-Sonnet-5, combined by majority vote. Goals are then clustered and merged into the Domain File's predicates.

A world model is valid only if CO-5 produces a reference trajectory within 10 refinement iterations.

Evidence

Three axes are scored and reported separately:

• Goal attainment: the fraction of goal facts present in the final state, under a fixed call budget.
• Persona compliance: checked at every step, split by capability gate vs. knowledge gate. It is reported as over-reach (acting beyond the persona's remit) and under-reach (withholding an action the persona is entitled to take).
• Efficiency: goal progress per tool call, measured against the validated reference plan.

A task counts as solved if the agent reaches the goal through any feasible in-persona policy; it does not have to be optimal.

The paper compares ReFract with ToolBench, τ-bench, AppWorld, SOPBench, AgentDojo, Role-Conditioned Refusals, CRMArena and PersonaGym. It claims ReFract is the only one that checks the process (which tools are called) rather than just the outcome, under role-conditioned capability and knowledge gates.

The fetched text gives no per-model scores, model list or tool-set sizes beyond the headline figures.

Limits

• The benchmark is small: 150 entries, one domain (industrial maintenance) and five generic roles.
• The fetched text says nothing about code or data release or a licence. Check before planning to reuse it.
• Problem Files are drafted, filtered and validated by Claude models. Goals are expert-annotated, but the reference trajectories are whatever CO-5 could produce within 10 iterations. The fetched text does not say whether Claude models are among those evaluated. If they are, the risk of construction bias is worth checking.
• Queries come from anonymized domain support conversations. Their source and raw count are not given in the fetched text.
• This is a NeurIPS 2026 workshop paper (Agents in the Wild).
• The fetched text cuts off mid-pipeline (Stage 5 verification). Stage 6 and all results sections were not available.