AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

What it claims

AgentWorld is a collaboration benchmark built on Kaetram, an open-source 2D MMORPG engine. The authors wrapped the engine in 13 high-level tools so that scores reflect coordination rather than low-level game control. The headline result goes past the 52.0% success figure: the best model's Causal Collaboration Effectiveness (CCE) is only 0.320. That means fewer than a third of all agent actions causally contribute to completing the task, even for the strongest team tested.

How it works

• Environment: a Kaetram world of 1056×768 tiles across 9 biomes. It has 380+ items, 144 mob types (level 1 to 250+), 70+ NPCs and 1,531 resource nodes across 8 skills. The simulator supports up to 1,000 concurrent agents. The benchmark tasks use 3–20.
• Tool layer: 13 API tools, including move, attack, harvest, craft, transfer and chat. Each tool bundles a multi-step mechanic into one call. For example, attack handles pathfinding, combat and loot collection.
• Observations: each turn, every agent gets a structured text observation of nearby tiles, entities and its own status. It also gets a task-specific guideline document covering recipes, monster stats and resource locations.
• Orchestration: the real-time game is converted to turn-based round-robin. In each round, every agent in turn receives a fresh observation, makes exactly one tool call, and waits for it to resolve. Turn order is deterministic.
• Blackbox setting: agents cannot see each other's state, observations or action histories. All coordination goes through chat messages routed by the environment.
• Task format: each task has:
• a primary objective;
• asymmetric per-agent configurations (skills, spawn point, starting items);
• a round budget;
• per-agent context documents;
• a Python verifier that checks success against the final game state.
• Task authoring: five human annotators wrote the tasks. Each task was piloted with two LLMs and revised until the only failures left were agent errors. Round budgets were calibrated from those pilot runs to be "challenging but feasible." The 100 augmented variants were generated with Claude Opus by varying objectives, spawn locations and initial items.
• CCE: an LLM is used only to build a causal graph over the trajectory's actions, with links both within one agent and across agents. CCE is the fraction of actions on a causal path to the outcome. In the worked example (Task 01: lumberjack → woodworker → wizard), CCE = 12/27 = 44.4%.

Evidence

• Best model: 52.0% task success, CCE 0.320.
• Success by category, averaged over the four models (Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, DeepSeek R1-70B) on the main set:

Trading and crafting, which depend on transfers and multi-stage pipelines, are the weakest categories.

How to use it

• The sandbox, task definitions with verifiers, evaluation scripts and annotation platform are stated to be open-source.
• Links: project site agentworld.io, code on GitHub. No repo URL or install commands appear in the fetched text.
• You need to run a Kaetram instance (github.com/Kaetram/Kaetram-Open). Expect one LLM call per agent per round: up to 20 agents × up to 55 rounds per task.

Limits

• Horizon: the abstract says "50+ interaction rounds", but the body gives round budgets of 25–55. Many tasks are shorter than the headline suggests.
• Categories: the paper names 8 categories, but the statistics figure shows 6 plus "Others."
• Small samples: survival's 75% rests on only 6 tasks.
• Models tested: only four mid-size or "mini" models were evaluated. No larger frontier models appear in the shown text.
• CCE depends on an LLM: the causal graph is built by an LLM, so the metric inherits that judge's errors. No agreement or validation figures appear in the fetched portion.
• Possible selection bias: tasks were filtered using two pilot LLMs, and augmentations were generated with Claude Opus. The page does not say whether this biases difficulty for particular model families.
• Truncated page: the text cuts off at the dataset statistics. Per-model success and CCE tables, the failure-mode analysis and the licence were not available here. Check them on the full paper and repo before relying on comparisons.