arxiv.orgpaper

AgentLSD: Evaluating AI Security Agents Under Adversarial Task Contamination

AgentLSD is a research framework that evaluates AI security agents on web CTF challenges under "adversarial task contamination" — injected trap artifacts like fake flags, misleading hints, decoy endpoints, and hidden cues that go beyond classic prompt injection by including non-instructional deceptive evidence. Testing six models on 11 web CTF challenges, the authors found agents solve 41% of flags cleanly, but contamination inflates turns (+20) and reasoning tokens (+2k) even on successful runs, and some model-challenge pairs get misled into following decoys or submitting wrong flags. The paper releases the framework, trap specifications, configurations, and raw traces as a reproducible benchmark for studying agent deception resilience.