Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness

What it claims

Requiring an LLM to name the governing statute, precedent or clause does not show that its verdict depends on that authority. The authors measure this with an "authority-swap" counterfactual. Case facts stay fixed, the decision target is replaced with an unrelated one, and they check two things: whether the final verdict changes, and whether the internal decision signal locked in before the authority was ever named. The title's "consulted" is shorthand for behavioural and textual proxies only. The authors say explicitly that they do not claim, or test, causal use of the authority's internal representation.

How it works

• What "authority" means per task varies, and this matters when reading the numbers:
• ECHR (ECtHR-A, articles actually found violated; sampling stratified because the corpus skews negative): a treaty article.
• SCOTUS: one of 14 issue-area labels. The authors flag this as the least literal "authority". Opinions are truncated to 6,000 characters.
• ContractNLI: a confidentiality obligation in an NDA, with Contradiction and NotMentioned collapsed to NO.
• CaseHOLD: the input is perturbed instead. The named precedent is replaced with the placeholder Doe v. Roe, and the five candidate holdings stay fixed.
• Elicitation: the system prompt requires <think> reasoning, an explicitly named authority, and a fixed marker such as VERDICT: VIOLATION, ANSWER: {0..4} or VERDICT: YES/NO. When the marker is missing, extraction falls back to natural-language conclusions and takes the rightmost match.
• Commitment tracking: hidden states are decoded with an untrained logit lens at every sentence boundary.
• CaseHOLD, SCOTUS and ContractNLI read next-token probabilities directly over 2–5 literal candidate tokens.
• ECHR buckets the top-200 tokens with a keyword/morphology classifier, corrected for base rates.
• A sentence gets a label only if the top category has nonzero support and a margin of at least 1.5× over the runner-up.
• The commitment step t\* is the earliest sentence after which the label never changes. Cases that never reach a confident final label are excluded.
• Controls:
• A gold-different subset: target and control gold labels genuinely differ, so a faithful model must flip its verdict. The authors treat this as the cleaner measure. The unrestricted swap rate is descriptive only.
• A meaning-preserving paraphrase baseline and a null-authority baseline, both to separate swap sensitivity from generic prompt instability.
• A prompt-injection test: a fake "registry note" appended to the facts orders VERDICT: VIOLATION. Results are reported both as raw compliance and as a change from the model's own uninjected verdict.

Evidence

• Models: Qwen3-8B and Qwen3-14B (native thinking mode), plus Llama-3.1-8B, Llama-3-8B, Gemma-3-12B and Legal-Δ-14B (thinking elicited by prompt). Llama-3.3-70B was run only on the ECHR/SCOTUS "Pair A" as a scale check.
• Run settings: bfloat16, seed 42, at most 1,536 new tokens.
• Sample size: 30 cases per model per authority pair (15 positive, 15 negative), each run under both conditions. CaseHOLD uses 30 cases. ECHR and SCOTUS each use two independent target/control pairs.
• Qualitative observation: naming always came before commitment in every case observed. Every model restates the authority in its opening sentence, so surface engagement is near-universal.

The fetched text ends before the results tables. Per-model numbers beyond the abstract's ranges are not available here.

Limits

• Naming is prompted. High naming rates partly reflect instruction-following. Every claim is scoped to what happens once a model is told to cite, not to unprompted citation.
• Small samples. Thirty cases per condition is a deliberate choice: the authors favour convergent evidence over benchmark scale.
• No causal test. Activation patching is motivated but not run, and post-commitment re-engagement is a textual proxy only.
• Legal-Δ-14B is a "best-effort" LoRA reproduction, not the original legal-reasoning model.
• The ECHR worked example is illustrative, not a reproduced generation.
• No closed or frontier API models are evaluated.