A Baseline Assessment for the Agentic Web Technology Stack

What it claims

MASTIF (Multi-Agent System TestIng Framework) is a harness for running the same web tasks across combinations of LLM, agent protocol and agent framework, and scoring each stack. The authors use it to set a baseline across 3 LLMs, 3 protocols and 4 Python frameworks on 300 Mind2Web tasks. They claim that matching the stack to the task improves completion rate, efficiency and social-value stability, and reduces verbosity and completion time.

How it works

The study is organised around four research questions. Each one fixes what "best stack" means:

1. Effectiveness: task completion rate.
2. Efficiency: completion rate together with convergence time and verbosity.
3. Tool-calling on the Web: which stack suits it best.
4. Social-value stability: how consistent the social values embedded in prompts, reasoning and output are across runs.

Frameworks. The scope is Python only. The paper names LangChain, CrewAI, LangGraph and Semantic Kernel as the LLM-agnostic, popular frameworks with the broadest built-in tool compatibility. These appear to be the four that were evaluated.

Protocols. The paper reviews MCP, A2A, ACP, ANP, Agora and AP2, and restricts scope to general-purpose protocols. The fetched text does not say which three were tested.

Benchmark. The full Mind2Web set has more than 2,000 tasks from 137 real websites across 31 domains, including travel, shopping and entertainment. The study uses a subset of 300.

Evidence

The fetched text ends in the related-work section, so no per-stack numbers are available. Two figures appear on the page, but both are cited from Anthropic and are not MASTIF results:

• Multi-agent systems with 4 agents outperformed a single agent by 90% on breadth-first research tasks.
• Single agents used about 4× the tokens of a chatbot, and multi-agent systems used up to about 15×.

How to use it

• The paper is licensed CC BY 4.0.
• The fetched text gives no repository, install command or API for MASTIF.

Limits

• Results not available here. The page was cut off before the experiments and results. The task-specific stack gains come only from the abstract, with no magnitudes or baselines.
• Stack not fully identified. The three LLMs and three protocols are not named in the available text.
• Narrow coverage.
• Only 300 of Mind2Web's 2,000+ tasks are used.
• Only Python frameworks are covered.
• The authors frame the work as a baseline "under the evaluated conditions."
• Results may age quickly. The protocol and framework landscape is described as fast-moving, so the findings may not hold for current versions.
• Before relying on it: check the full paper for the specific stacks, their versions, and whether MASTIF's code is released.