JevOut: Natural Context Can Flip Decision Models

What it is

JevOut (arXiv:2609.30243v1, Zixiang Xu, USC) is a method and evaluation showing that dedicated decision models — hosted Jev 1.13.0, OpenSourceJev (Qwen3-1.7B Q8), Von, and a plain Qwen scorer — can be steered from a correct choice to a pre-specified wrong one by adding a short, natural-sounding sentence to the context, without touching the question, options, or gold answer. Code and homepage are published (github.com/xzx34/JevOut). The summary gives you the headline flip rates; what follows is the attack mechanism, the budget it needs, and the transfer results the summary omits.

How it works

The attacker first fixes the target as the most probable wrong option (Eq. 1), then runs probability-guided context optimization. The optimization signal is the log-probability margin between the fixed target and its strongest competitor (Eq. 3); an LLM proposer adds one sentence plus an insertion boundary per step, and a Gibbs/softmax allocation (Eqs. 4–5) concentrates the budget on candidates near the model's decision boundary. A separate check enforces coherence and answer-preservation without seeing the target or its probabilities, and mechanical checks forbid explicit selection cues.

Key point: the graded probability output is what makes this cheap. On a matched 140-item Jev subset, using probabilities rather than label-only feedback improved TFR by 7.1 percentage points at the same budget — the structured probability interface is itself the attack surface.

Preconditions / Affected

• The budget is small: 16 proposal slots × 4 iterations, ≤64 accepted target evaluations, with 25% of slots restarting from the original input.
• Successful contexts are short — a median of 31 added words.
• A single target-aware addition (no optimization) already flips 16.9% of decisions vs 2.2% for a neutral addition.
• Evaluated across seven datasets spanning knowledge (MMLU-Pro, SuperGPQA), reasoning (MuSR, ToMBench), legal (LAR-ECHR), multi-answer (SATA-Bench), and tool routing (BFCL V4) — so this covers agent tool-selection, not just QA. 50 dev + 100 held-out items each.

What to do

• Do not treat a decision model's probability distribution as a dependable interface for routing, tool selection, or action triggering — the graded output is exactly what the optimizer exploits.
• Assume attacker-influenceable context (conversation history, retrieved evidence, background detail) can redirect a correct decision even when it looks benign and preserves the task.

Caveats

Two findings the page states extend the threat beyond the original runs: frozen contexts transfer to other decision models without re-optimization, and a proposer trained on development trajectories (recipes V1/V2) generates redirecting contexts on held-out items without iterative target feedback — meaning the per-item probability query loop is not strictly required at attack time. The page text cuts off in the setup section, so full per-system TFR breakdowns beyond the 64.9%–73.2% range and defense results are not visible here.