To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech

What it claims

This introduces VERISPEAK, a probe benchmark of 3,879 spoken claims for testing whether Large Audio Language Models (LALMs) can do evidence-grounded fact verification when the claim is spoken and the evidence is textual. The claims are balanced true/false and split across three fact types: temporal (year: 1,451), geographical (location: 2,226), and relational (relation: 202), drawn from 659 celebrity subjects across 60 countries. The headline finding is a consistent text-speech modality gap: LALMs that verify written claims reliably fail on the same claims when spoken. Retrieval alone helps only partially because models conflate retrieved evidence with the spoken claim; retrieval combined with explicit chain-of-thought reasoning fixes this, with a thinking-tuned LALM (Audio-Flamingo-next-think) reaching 86.1% accuracy.

How it works

The benchmark is built from short celebrity biographies (Shah et al., 2019), enriched by scraping Wikipedia for birth country and gender. Atomic, pronoun-free claims are extracted via Llama-3.2-3B-Instruct. Negatives are generated by deterministic category-specific perturbations: year shifted by a non-zero offset in [−10,+10]; geographical spans swapped with same-type candidates; relational facts altered by relation-word swaps (married to → divorced from) or entity swaps. All claims are synthesized with a single-speaker Coqui TTS (tacotron2-DDC) to remove speaker-identity cues, then transcribed with Wav2Vec 2.0 — deliberately using imperfect automatic transcripts so failures reflect reasoning limits, not perfect input.

Six controlled pipelines vary modality and augmentation: C0 text-LLM, C1 text-LALM, C2 speech-LALM, C3 speech+CoT, C4 speech+Text-RAG, C5 speech+Text-RAG+CoT. Text-RAG retrieves evidence using the ASR transcript (default retriever multi-e5, with e5-large-v2 and msmarco-MiniLM tested for sensitivity); an audio-query variant uses a CLAP audio-text retriever instead.

Evidence

Five LALMs across three families are evaluated (Qwen-Audio-Chat, Qwen2-Audio-7B, Audio-Flamingo-3, Phi-4-multimodal), each paired with its text-only LLM backbone as a reference to isolate speech-input degradation. Stated findings: (i) text verification ability does not transfer to speech; (ii) transcript-based RAG is generally more reliable than audio-query RAG, but standard LALMs treat retrieved evidence as the statement to verify; (iii) CoT alone does not reliably help speech-only verification — it helps mainly when evidence is present. The 86.1% figure is the best result, from the thinking-tuned model under retrieval+reasoning.

How to use it

Dataset is public on Hugging Face: abhiram4572/VeriSpeak. Each item includes the audio file, a Wav2Vec 2.0 transcript, and a ground-truth veracity label.

Limits

This is a probe benchmark, not a deployable system — it diagnoses LALM behavior rather than shipping a fact-checker. The claim domain is narrow: celebrity biographical facts (temporal/geographical/relational), skewed toward US (278) and England (79) subjects, with the relational category thin at 202 items. Speech is synthetic single-speaker TTS, so real-world speaker/accent/noise variation is untested by design. The fetched text cuts off mid-sentence in the experimental setup (RQ6 / text-RAG-LLM upper-bound condition), so full per-model result tables and the audio-query RAG numbers are not shown here.