InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

What it claims

InteractiveBench is a closed-loop benchmark of 120 human-reviewed tasks for egocentric AI assistants, built on the InteractiveWorld simulator from Meta and Duke University (October 2026). The model under test guides an MLLM-driven simulated human by natural language. Success is checked automatically from final object states in the simulator. Across 12 frontier MLLMs, the best overall is Claude Fable 5.1 at 41.7% task success, and every other model is at or below 33.3%. The authors name perception, memory and communication as the main bottlenecks.

How it works

InteractiveWorld runs on NVIDIA Isaac Sim and has three layers:

1. Physical layer. An SMPL+H full-body and hand model has a head-mounted camera that gives the first-person view. The body is driven kinematically, not by physics, so valid pose sequences always execute and the model does not fall at random. Paths and arm reach are checked before execution. Unsafe actions are rejected with a failure reason. The assets are over 8,000 objects and more than 50 scenes imported from BEHAVIOR-1K.
2. Human action layer. Navigation uses an obstacle map and A* search. For manipulation, the body adapts its posture (stand, lean, squat), plans a collision-free arm trajectory, and moves to a new position if no trajectory is feasible.
3. Semantic layer. It has 36 action primitives, from [Walk], [LookAt], [Grab], [PutInside] to [Pour], [Cook], [Cut]. A command names only the action and its targets, and the lower layers handle the motion.

flowchart LR
  A[Assistant: model under test] -->|language guidance| H[Human agent: auxiliary MLLM]
  H -->|action primitive| E[Isaac Sim environment]
  E -->|rejected action + reason| H
  E -->|new first-person view| A
  E -->|new view| H
  E --> V[Final object-state check]

Each task defines an initial scene with target and distractor objects. A warmup phase follows, in which the human runs scripted actions and gives opening context. The closed-loop interaction comes after that. The human agent sees all valid commands but is told to follow the assistant rather than act on its own intent. In Accessibility tasks, the human agent gets no visual input, to simulate a blind user.

Evidence

The six categories are Guidance, Planning, Accessibility, Decision, Prevention and Finding. Example tasks include putting away 14 chemical bottles in a lab, scheduling allergy-safe dishes with one oven and one pan in 30 minutes, comparing supplier quotes for 40 filters, and reminding a user about an open fridge before they leave. The best per-category scores reported are 20.0% on Guidance and 25.0% on Planning. The fetched text does not give per-category numbers for the other four categories, the full list of the 12 models, or the five findings in Sec. 4.1.

How to use it

The code is at https://github.com/facebookresearch/InteractiveWorld, and the project page is https://meta-interactiveworld.github.io/. Running it requires Isaac Sim. The fetched text gives no install commands, hardware requirements or licence.

Limits

The "human" is an MLLM, not a person, so scores depend on which model plays the human agent. The fetched text does not name that model; the paper places the settings in Appendix C. Human motion is kinematic, with predefined motion sequences. The benchmark tests guidance and grounding, not realistic physical human behaviour. With 120 tasks over six categories (about 20 per category), one task is about 5 points of a per-category score. The fetched text ends partway through Section 3, so the full results tables, analysis and limitations section were not available.