BrowseSafe++: Benchmarking Prompt Injection Detection Across Web Contexts, Languages, and Browser Views

What it is

BrowseSafe++ is a Perplexity/Purdue/Rutgers benchmark of 38,675 attack/control page pairs (77,350 pages) for prompt-injection detection in browser agents. 32,187 pairs carry injections seen on production sites between September 2025 and September 2026, and 6,488 are retained from BrowseSafe. The injection text is kept unchanged. The surrounding page is rewritten to remove PII and real-world content. Splits are 29,500 train and 9,175 test pairs.

How it works

• Pairing: each control page is the same page without the injection, with the same task and expected outcome. This stops a detector from learning topic or style differences instead of the injection itself.
• Views: every page is rendered as HTML, Markdown, accessibility tree and screenshot, and validated to show the injection in all four views.
• Sourcing: candidates came from pages read by a deployed browser agent, filtered by an LLM classifier and 25 regex cues. Of 1,092,957 deduplicated pages, six votes from three frontier models (at least four required) kept 58,036 (5.3%).
• BrowseSafe-Guard: two classifiers fine-tuned from Qwen3.5-0.8B, one text (HTML, Markdown or accessibility tree) and one screenshot. The loss is BCE on both pages plus a pairwise softplus term that pushes the attacked page's score above its control's by margin m. Text training uses a crop around the injection and the matching control region.
• BrowseSafe-Adapt: it starts from an entry's original injection and runs the target agent. A fixed verifier checks the answer and application state, and failures feed a rewrite loop until success or the attempt budget runs out.

Numbers worth acting on

• Best prompted detectors falsely flag 13.82–15.10% of benign pages. GPT-5.5 and Claude Sonnet 5 flag roughly 19–28% in the Section 4.1 comparison, and GPT-5.5 is cited at 19.13–21.21% against the Guard text detector.
• The Guard text detector reaches 92.98–98.29 F1 at 1.35–7.40% benign FPR across the three text views. The screenshot detector reaches 93.32 F1 at 6.08% FPR.
• Models that flag an injection when asked to detect it still follow it when doing the task (16 models evaluated for detection, 15 as agents). Detection scores do not predict agent resistance.
• In 24.4% of pairs the injection shares no language with its page, most often an English injection on a non-English page.
• Adapted injections average 78.79% attack success across target agents and reach up to 95.41%, against a few percent when replayed unchanged.
• Composition: News & blogs (12,025) and shared docs (6,612) dominate. Unauthorized action (16,707) and task disruption (12,995) are the top attacker goals. Hidden text (586) and HTML comments (246) are small slices.

What to do

• If you run a detector on agent inputs, measure benign FPR on paired controls, not just attack recall. Prompted frontier detectors are noisy on benign pages with ordinary instructions such as buttons and help text.
• Do not treat a detection gate as proof of agent robustness. Test task-level outcomes too.
• If you train a detector, consider the paired-ranking loss over separately sampled benign data.
• Use Adapt-style rewriting against your own agent, since static replay badly understates attack success.
• Check detectors on cross-language cases and on the screenshot view if your agent is vision-based.

Caveats

• The provided text cuts off in Section 3.2, so Adapt's attempt budget and per-agent results are not visible here.
• Injection labels depend on LLM votes (three models, six votes), and the benchmark was built using a classifier from the same deployed agent. Selection bias toward what that classifier catches is possible, though the page does not discuss it.
• The Guard text detector is trained on crops around the injection. The text does not say how it behaves on full pages at inference.