Skip to content

Back to the catalog

Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?
arxiv.orgpaper

Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?

A 113-participant study comparing three ways to control AI agent actions across email, files, payments, and personal data: per-action human approval (HITL), automated per-action model review (AUTO), and user-authored 'allow/ask/never' consequence policies (POLICY). It finds POLICY blocked significantly less overreach than HITL and AUTO (-20.1 and -14.5 percentage points respectively), because participants overwhelmingly chose 'ask' rules (114/140) that just deferred decisions back to runtime, and because users approved most overreach actions anyway when prompted — revealing a preference-vs-commitment gap where standing policies don't by themselves harden agent authorization.

TL;DR User-set allow/ask/never policies for AI agents blocked less overreach than per-action human or automated review, mainly because users defaulted to 'ask' and then approved most risky actions anyway.
Open original ↗
Also seen from another angle🧠 AI →
Document

What it is

A CHI-style controlled study (113 non-developer participants) plus a supporting MCP-proxy implementation, testing whether user-authored "allow/ask/never" consequence policies harden AI-agent authorization. The core finding — POLICY blocked 20.1pp less overreach than HITL and 14.5pp less than AUTO because participants defaulted to "ask" — is in the summary. What the summary omits is the engineering substance an implementer needs.

How it works

The policy layer is an MCP proxy sitting between agent and tool server. It classifies each tool's metadata (name, description, input schema) with an LLM into four plain-language consequence categories — spend money, send or publish information, delete something, access private information — then applies the user's standing rule before forwarding the tools/call. A rule settles a decision in advance only for a decisive allow or never; ask returns the action to runtime. The user study used fixed, researcher-checked mappings so classification error couldn't contaminate the condition comparison.

Of 148 overreach actions executed under POLICY, 133 followed runtime human approval and only 15 ran automatically under allow — the failure is human commitment, not the engine.

Classifier accuracy (separate evaluation)

Against a 120-tool human-labeled set (drawn from 538 tools / 36 servers), an independently run LLM matched human consensus on 95.8% of main-action labels, 85.7% of data-sensitivity, 86.6% of externality. Human inter-annotator agreement: Krippendorff's α=.88 (action type), Fleiss' κ=.89 (sensitivity, externality).

Caveats the page states

  • The threat model notes the classifier reads untrusted server metadata — a compromised tool could hide capability behind a misleading name or embed prompt injection for the classifier. The evaluation explicitly does not test resistance to adversarial metadata or prompt injection.
  • Implementation is stdio transport, single upstream server; multi-server and streaming are out of scope.
  • Conditions differ in several ways, so the paper compares designs as wholes, not isolated features.

Practical takeaway: consequence-level rules map cleanly onto MCP enforcement, but ship them assuming users will pick ask and rubber-stamp — the standing policy does not by itself form a boundary. This mirrors Claude Code's and Codex's existing allow/ask/deny modes.

Techniques named here

  • AI agent overreach

Related in AI Security