Skip to content

Back to the catalog

COPEX: Benchmarking LLM Robustness to Adversarial Context Across Model Context Protocol Layers
arxiv.orgpaper

COPEX: Benchmarking LLM Robustness to Adversarial Context Across Model Context Protocol Layers

COPEX (COntext Provider EXploitation) is a benchmark that isolates the tool-selecting LLM as the unit of evaluation in Model Context Protocol systems, fixing the surrounding agent stack and varying only the model to measure its intrinsic susceptibility to adversarial context. It covers 25 attack types across 125 scenarios spanning four entry surfaces — model/agent, client, server/tool, and transport — and tests nine models across 3,375 trials, finding a mean attack success rate of 64.4% (58.3%–71.4% by surface); combined input and context scanning cut mean attack success by 49.6% on an eight-attack defense subset. The work separates "system exposure" (attacks succeeding outside the model's observation/control) from "model susceptibility," and the benchmark itself is released publicly.

TL;DR A new benchmark (COPEX) measures how often LLMs fall for adversarial content injected via MCP user instructions, tool schemas, tool outputs, or transport messages — finding a 64.4% average attack success rate across nine models.
Open original ↗
Document

What it is

COPEX is a reproducible test harness (code at github.com/inspire-center/copex) for measuring how nine specific LLMs — Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.5, GPT-5 Mini, GPT-4.1, Gemini 3.1, Grok 4.3, Llama-3.2:3b, and Qwen-2.5:3b — handle adversarial context inside MCP tool-calling loops, with the agent stack (orchestrator, ReAct loop, servers, defenses) held fixed so results are attributable to the model itself rather than the surrounding product.

How it works

The environment runs 16 MCP servers (75 tools, 7 resources) over stdio and HTTP/Docker-isolated transport, executing a bounded ReAct loop (temperature 0, max 10 turns, 4,096-token output cap) per trial. Each of 25 attack types has 5 scenario variants, each run 3 times per model (375 trials/model, 3,375 total). Outcomes are scored by one of two oracle types: deterministic trace predicates for 10 attack types where success is visible directly in tool calls/logs/protocol events, and an LLM-judge semantic evaluator for 15 attack types where success depends on response meaning — the paper notes these two oracle types can disagree on ASR for the same attack under matched configurations. A separate Reflexion-vs-ReAct comparison isolates reasoning-architecture effects with everything else fixed.

Preconditions / Affected

Applies to any MCP client/agent that assembles model context from tool schemas and tool responses alongside the user prompt — i.e., any MCP-enabled LLM agent, not a specific product. The threat model explicitly excludes model-weight or system-prompt tampering; attacker control is limited to one of four surfaces (model/agent, client, server/tool, transport).

What to do

The paper tests a defense architecture of tool-free "guard model" scanners with four modes: none, input-only, context-only, or combined. The input scanner checks the user task once pre-execution (a BLOCK ends the trial as DETECTED). The context scanner checks each tool schema at registration and each tool response before it reaches the model context (a blocked schema is dropped from the catalog; a blocked response is replaced with a structured error). Combined input+context scanning cut mean ASR by 49.6% on an 8-attack subset — the one concrete defense number on the page. Guard models "fail open" on scanner failure, and schemas are scanned only once at registration, not on later change.

Caveats

Response scanning happens after the tool call executes, so it can hide a malicious result from the model but cannot undo server-side effects already triggered (e.g., a reverse shell opened by a tool before its "OK" response is scanned/blocked). Scanning also does nothing for attacks that operate below the model's observation boundary (e.g., certain transport/DNS-rebinding attacks), since there's no model-visible content to scan. The 49.6% figure applies only to the 8-attack defense subset, not the full 25-attack suite.

Technologies named here

Techniques named here

  • Context Provider Exploitation attacks

Related in AI Security