Does a model's stated reason for rejecting a candidate do any work?

What it claims

A single-author (Archit Rastogi, independent) preprint for the LLM4XAI 2026 workshop testing whether an LLM's stated reason for rejecting a candidate is causal: when a model rejects an option by naming a fact its profile lacks ("no director," "no date of death"), does supplying that fact change the choice? Because the reason is a claim about visible text, faithfulness is checked by editing the text and re-asking — no judge model, no human ratings. The results are deliberately mixed and the paper reports them that way: the headline content effect is real but bounded, the contrast the design was built to detect fails, and the single strongest result carries no content claim at all.

How it works

Items come from 2WikiMultihopQA, where a fact missing from one candidate's profile usually appears as a real sentence in a sibling profile. A model picks a candidate and is asked what it ruled out and why; the named attribute is then inserted into the target profile as a real corpus sentence with the subject substituted (the sentence need not be true of the target — the claim under test is only that the profile omitted the attribute). Five conditions form a 2×2 of content × location plus an integrity re-ask:

flowchart TD
  E[Model rejects rival, names missing fact] --> R0[R0: no edit, must reproduce choice]
  E --> R1[R1: named fact at rival]
  E --> R2[R2: irrelevant sentence at rival, length-matched]
  E --> R3[R3: named fact at unmentioned 3rd option]
  E --> R4[R4: irrelevant sentence at 3rd option]
  R1 --> Q1{R1-R2: content effect}
  R3 --> Q2{R1-R3: location effect}

Eight deterministic integrity gates run before generation (attribute absent-then-present, length within 20%, gold answer unchanged, etc.). A forced single-token probability read on each candidate letter runs alongside, exploratory only.

Evidence

Six open models; runs in three designs (Part A: three 7–8B models at 4 options; Part B: three more checkpoints at 4 options; Part C: Part A roster at 6 options). In Part C, supplying the named fact at the named rival moves the choice more than the irrelevant control: odds ratio 3.57 [1.54, 8.26], Holm p=0.0210, surviving any single-model exclusion — but weakening to uncorrected p=0.099 on the third of items where repair and control are closest in fluency. The designed contrast — same fact at the option nobody named (R3−R4) — does not clear correction (Holm p=0.2428). The strongest result is a pure location effect with no content claim: the identical irrelevant sentence moves the choice more at the named rival than at the third option (Holm p=0.0008). Replication on the second roster (Part B) fails throughout. The probability read disagrees in direction with the free-text choice on the location contrast; three candidate explanations for the disagreement find no support.

Elicitation matters: a contrastive rejection appears in 20.8% of items unprompted vs 72.5% when asked, while accuracy stays flat (~68%). About 32% of specific rejections name a defect the profile actually carries.

Limits

Effects are described as bounding rather than establishing a content effect, because repair and control also differ in co-candidate mentions, relation template and fluency. The paper is unusually candid about its own instrument: validation caught eight defects, including a choice-parser that returned the just-rejected option in 17.1% of adjudicable responses; uncorrected, the pipeline would have reported six surviving contrasts instead of four. Whether parametric belief enters is left open. Code, per-item records with raw responses, and seeded analysis scripts are released. (Fetched text cuts off mid-Section 3.6.)