Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

What it is

An empirical study of how Attack Success Rate (ASR) comparisons for jailbreaks depend on evaluator, attack budget and sampling settings. It tests eight attacks plus an unmodified baseline on 50 AdvBench requests against Llama 2-7B, Vicuna-7B and Qwen3-8B. The authors are from Oldenburg and Chalmers, and the paper is arXiv v1, dated 19 October 2026.

How it works

Three evaluators score the same attacks:

• KW: keyword matching. A response succeeds if it contains none of the listed refusal strings.
• FT: a fine-tuned classifier from the Generation Exploitation work (Huang et al., 2023), with a risk score above 0.5 counting as success. A separate scorer picks which response it sees.
• SR-Q: Qwen3-30B following StrongREJECT instructions, with a combined score above 0.5 counting as success. Refusal scores zero.

ASR counts a goal as successful if any of up to K responses passes. With independent attempts at per-attempt success probability p, success within K attempts is 1−(1−p)^K. At p = 0.01, 90 attempts give about 60%.

Findings

• Evaluator choice: Low-resource language attacks (Scots Gaelic, Hmong, Zulu, Guarani, best-of-4, translated back to English) succeed on 144/150 goal-model cases under KW (96.0%), 94/150 under FT (62.7%) and 4/150 under SR-Q (2.7%). They rank first under KW and FT and last under SR-Q.
• Budget reversal: On Vicuna, rewriting leads DeepInception by 32 points under the original settings. After DeepInception is given five answers per prompt and rewriting is cut to one search of up to five rounds, rewriting trails by 50 points.
• Repeated sampling: Llama 2-7B-Chat answered each unchanged request 100 times at temperature 0.8 (Ollama's default), with no jailbreak. ASR was 100% under KW and 50% under SR-Q, and some outputs were incoherent or unrelated. The authors read this as probabilistic suppression of unsafe behavior, not robust exclusion.

Attack budgets differ widely: DeepInception and ICA use one response per goal, PAIR up to nine, and rewriting up to 25. Optimization steps and auxiliary-model calls add cost beyond response counts.

What to do

The authors recommend:

• Define success explicitly.
• Match response budgets across attacks.
• Validate evaluators, including the full selection and aggregation procedure.
• Report cumulative failure under repeated interaction, not single-generation results.

For safety testing, this implies not treating one refusal, or one deterministic decode, as evidence of safety.

Caveats

• The study uses 50 requests and does not claim statistical significance.
• The budget experiment changes search streams and response limit together, so it cannot isolate the response limit.
• SR-Q uses a different judge model from the original StrongREJECT, so the original accuracy checks do not apply. The authors say disagreement alone does not show which evaluator is wrong.
• The page does not show how any evaluator was checked against human labels, and the available text cuts off partway through Section 5.1.
• Attack implementations are adapted to available compute and are not meant to reproduce the original papers' scores.
• The authors say automated scores do not establish realized harm.