Skip to content

Back to the catalog

Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agents
arxiv.orgpaper

Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agents

This paper argues that existing AI security incident reporting frameworks, built for traditional AI/ML systems, don't capture what matters for AI agents, and proposes new reporting elements such as agent memory and memory-access logs, actual vs. potential autonomy levels, and tool-usage records. Drawing on input from 23 academics and industry practitioners, the authors also flag reporting-specific risks (data leakage, attacks on the reporting infrastructure itself) and lay out open research questions on efficiently recording incidents and determining whether agent vulnerabilities generalize across systems.

TL;DR Proposes new incident-reporting fields (agent memory, autonomy level, tool use) tailored to AI agent security, based on expert interviews.
Open original ↗
Document

What it is

An AAAI 2026 position paper (arXiv:2609.24515) arguing that security incident reporting for AI agents needs its own reporting elements, distinct from non-agentic AI. The summary already covers the proposed elements (memory/memory-access logs, actual vs. potential autonomy, tool usage) and the 23-expert methodology. What the summary omits, and what an engineer would actually act on, are the concrete CVEs the paper anchors its argument in and the EU compliance dates it maps out.

Named vulnerabilities (with the agent surface each hit)

The paper grounds the "expanded attack surface" claim in four real CVEs, organized by which part of the agent they compromise:

  • EchoLeak — CVE-2025-32711 — the agent's data.
  • ShareLeak (Copilot Studio) — CVE-2026-21520 — also agent data.
  • Reprompt — CVE-2026-24307 — the agent's context.
  • GitHub Copilot — CVE-2025-53773 — an accessed tool.

The paper's point is that these span data, context, and tools — surfaces that only exist because agents dynamically compose tools, sub-agents, APIs, and external data at runtime, so "the conditions giving rise to an incident may never have existed during development or testing." It also cites unnamed recent OpenAI and Anthropic incidents where agents exceeded intended operational boundaries and gained unauthorized external access.

EU compliance timeline (the part with dates)

If you are tracking regulatory obligations, the paper lays out the schedule explicitly:

  • AI Act Article 73 (serious-incident reporting for high-risk AI providers) applies from 2 August 2026.
  • Digital Omnibus amendments defer the Chapter III high-risk requirements: Annex III to 2 December 2027, Annex I to 2 August 2028.
  • GPAI Code of Practice Commitment 9 operationalises Article 55(1)(c) for general-purpose models with systemic risk — reporting to the AI Office, not the Article 73 channel.
  • Under Article 75(1a), providers within the AI Office's exclusive competence report to the AI Office rather than national market surveillance authorities.

Relevant standards named: ISO/IEC TS 8200 (external controllability, does not presume a human operator) and ISO/IEC 42105 (human oversight at the operational level). Emerging agent-protocol standards flagged: NLIP (ECMA TC56), IETF, LF A2A. GDPR is noted as applying to the incident reports themselves.

Caveats

This is a position/synthesis paper, not an advisory — it proposes reporting elements and open questions, it does not provide detections or patches. The CVEs are cited as evidence, not analyzed for exploitation detail. The authors note contributors endorsed only their own sub-sections, and that documented operational agent incidents remain rare relative to the academic literature (attackers still prefer phishing, credential theft, and supply-chain compromise over model-optimized attacks).

Related in AI Security