AWS Security Agent (now part of AWS Continuum) exploited 33 of 37 new real-world CVEs

What it is

AWS's vendor-reported results for AWS Security Agent (now part of AWS Continuum), an autonomous pentesting agent, on an internal CTF-style benchmark. The benchmark has 37 CVEs from 2025–2026, each run as a live docker-compose target with a flag seeded at the intended sink. The run used Claude Opus 5, one epoch, and a 10-hour limit per sample. The post is useful mainly for its method, its failure modes and what it says about guardrails. The CVEs are not named.

How it works

Each run goes through the same fixed pipeline:

1. Crawler maps endpoints, parameters and entry points.
2. Specialist agents raise candidate risks.
3. A validator confirms or rejects each candidate before any exploitation.
4. The exploitation agent works with a shell, a Python runtime, a real browser and an OOB callback listener.
5. A flag check marks a sample captured if the flag appears anywhere in the output.
6. An integrity audit compares each capture's trajectory against a canonical PoC the agent never sees.

Two controls reduce contamination. Recency of the CVEs addresses memorization, and disabling runtime CVE lookup addresses retrieval of public PoCs.

flowchart TD
    A[Flag captured: 35] --> B{Audit vs canonical PoC}
    B -->|Intended vuln: 33| C[Counted solved]
    B -->|Off-path: 2| D[Reclassified unsolved]
    D --> E[SSRF: scanned own subnet, hit internal host]
    D --> F[SQLi: forced crash, flag in debug page]

Numbers worth knowing

• 35 of 37 flags were captured. 33 (89.2%) were through the intended vulnerability.
• The 2 reclassified captures were benchmark bugs, since fixed. In the SSRF challenge the agent enumerated its own subnet and reached the internal server directly. In the SQLi challenge it forced a verbose crash whose debug page leaked the environment, including the flag.
• In the 2 misses the agent exploited the bug and then took the target down. In one, a file-upload-to-RCE challenge, the PHP runtime was bricked. In the other, the service went offline after root-level code execution, and the container was gone by scoring time. Both ran under the CTF objective, without the production non-destructive guardrails.
• The validator surfaced 475 findings. It labeled 159 as auto-detected false positives and de-duplicated more than 1,200 others. A further 234 high-confidence issues beyond the seeded CVEs were confirmed, but the post treats them as secondary.
• Misses and reclassifications are concentrated in three vulnerability classes. The other classes were captured in full.

Exploits shown were a six-primitive CMS chain (a secondary upload handler with no CSRF or extension check, leading to XSS cookie exfiltration through the OOB listener and admin takeover), an unauthenticated RCE in an AI workflow builder via a custom component that runs shell commands at import time, and a full-read SSRF through a tRPC ping-style procedure to an internal-only endpoint.

What to do

For an engineer evaluating autonomous pentest tooling:

• Treat the agent as capable of destroying a target. Run it only against disposable or isolated environments until the promised non-destructive exploitation and target-health recovery work ships. The post lists this as in progress, not delivered.
• If you build or buy a similar benchmark, copy the per-capture audit against a canonical PoC. It caught two shortcuts that would otherwise have inflated the score.
• Isolate internal services and suppress debug error pages in test targets, so shortcuts don't count as wins.

Caveats

• This is a vendor's self-reported, internal benchmark. The dataset and CVE list are not published, and it is a single epoch, so results can't be independently reproduced.
• The authors call it contamination-reduced, not contamination-free, because advisories or partial PoCs may have leaked into training data.
• The post says earlier CVE-Bench numbers are a different measurement and should not be compared directly.
• The 475-finding false-positive labels come from the pipeline's own validator, not from independent review.
• The guardrail commitments are forward-looking.