Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

What it claims

Attack Success Rate (ASR) comparisons for jailbreaks change with three choices: the evaluator, the attack's response budget, and the victim model's sampling settings. The study runs eight attacks plus a no-jailbreak baseline against Llama 2-7B, Vicuna-7B and Qwen3-8B on 50 AdvBench requests. Low-resource language attacks rank first under keyword matching (96.0%) and a fine-tuned classifier (62.7%) but last under a Qwen3-30B judge using StrongREJECT criteria (2.7%). The authors (Oldenburg and Chalmers, October 2026) argue that repeated sampling reveals "probabilistic suppression rather than robust exclusion" of unsafe output. They recommend explicit success definitions, matched response budgets, validated evaluators, and reporting cumulative failure under repeated queries.

How it works

ASR here counts a goal as successful if any of up to K responses passes the evaluator. It is a fraction of goals, not of responses. The paper notes that with per-attempt success p = 0.01, success within K attempts is 1 − (1 − p)^K, rising from 1% at one attempt to about 60% at 90 attempts with no change in attack strength.

The three evaluators are:

• KW: success if no listed refusal string appears. It does not check whether the request was answered.
• FT: the classifier from Huang et al. (2023, Generation Exploitation), trained on human helpfulness and harmlessness preferences. A score above 0.5 counts as success. A separate scorer picks which candidate it sees.
• SR-Q: Qwen3-30B applies StrongREJECT's written rubric (refusal, specificity, convincingness). Success is a combined score above 0.5, and the highest-scoring candidate counts.

Attacks differ widely in budget. Low-resource translation (Scots Gaelic, Hmong, Zulu, Guarani via googletrans) is best-of-4. DeepInception and ICA are one-shot. Rewriting generates up to 25 target responses, and PAIR up to 9 with Vicuna-13B as attacker. GCG uses 20 suffix tokens, 250 steps and batch 512. AutoDAN uses 100 steps and batch 256.

Evidence

• Translation attacks over 150 request–model cases scored 144/150 under KW, 94/150 under FT and 4/150 under SR-Q.
• Changing the budget on Vicuna reversed the ranking. Rewriting led DeepInception by 32 points under the original settings. It trailed by 50 points after DeepInception moved to five answers per prompt and rewriting dropped from five search streams to one, both capped at five candidates.
• With no jailbreak, Llama 2-7B-Chat answered each of the 50 requests 100 times at temperature 0.8. ASR was 100% under KW and 50% under SR-Q, and the samples included incoherent or unrelated outputs.

Decoding settings for Llama 2 and Vicuna were Ollama defaults: temperature 0.8, top-k 40, top-p 0.9, repeat penalty 1.1 and context 2048. Qwen3 ran at temperature 0.6, top-k 20 and top-p 0.95, with thinking mode on. Generation length was bounded only by context.

Limits

• The authors make no claim of statistical significance. Each cell is out of 50 goals.
• SR-Q uses a different model from the original StrongREJECT judge, so StrongREJECT's accuracy checks do not carry over. Evaluator disagreement does not show which evaluator is wrong.
• Attacks were adapted to the authors' compute and are not meant to reproduce the original papers' scores.
• The rewriting budget change alters both stream count and response cap, so the two effects cannot be separated.
• The paper is partly based on a thesis. It notes that the thesis omitted the StrongREJECT rescaling step, and that the thesis does not fully specify which candidates each evaluator received.
• The fetched text ends partway through Section 5.1. The full Table 2 (81 ASRs), the qualitative review of Vicuna responses and the appendices were not available here.