Getting the Source Right, Not Just the Fact: Source-Aware Verification for MCP Agents

What it claims

ProvenanceGuard is a post-generation checker for MCP agents. It flags a claim when the fact is supported somewhere in the retrieved evidence but the answer attributes it to the wrong tool or document. On held-out medical-agent claims it blocked 138 of 139 expert-rejected claims. The cost was 67 false holds, out of 361 claims. The harder case, choosing among several similar sources, is where it is weak: it picked the exact source in 50.3% of claims.

How it works

It reads the captured MCP trace (tool outputs plus source IDs) and needs no retraining of the agent. Source identity is kept separate throughout rather than pooled into one context. For each claim it:

• decomposes the answer into claims (local LLM)
• routes each claim to its most relevant source (MiniLM)
• scores support with a DeBERTa NLI model
• matches literal values strictly, so a number, date or ID absent from the source fails even if the sentence reads plausibly
• compares the supporting source with the one the answer names or implies

A calibrated step then combines these signals into a verdict for each claim and an allow/block decision for the whole answer.

flowchart LR
  T[MCP trace + source IDs] --> V[Per-claim route, NLI, literal check, attribution]
  V --> D{Calibrated decision}
  D -->|allow| OK[Pass answer]
  D -->|block| R[RARR-style repair]
  R -->|revised| V
  R -->|no grounded rewrite| F[Safe fallback text]

Evidence

• Main test: 361 expert-labelled claims from 40 held-out answers. These came from 281 real traces of a medical agent using patient records and research articles.
• Blocking: 138 of 139 unsupported claims caught; 67 supported claims held.
• Source identification: correct about 86% of the time, among claims with an identifiable source.
• Baselines: it scored highest of five checkers on the paper's block/avoid-unnecessary-block measure (Table III). The four source-blind baselines give no source attribution at all.
• Similar-source stress test: 0.846 F1 on block decisions, but only 50.3% exact-source accuracy.
• Controlled swap test: the named source was changed in 50 cases with the evidence left intact. All 50 were caught.
• Repair on full traces: all 173 blocked answers were resolved, but 144 of them ended as fallback text rather than substantive rewrites.
• Repair on reconstructed multi-source traces: all 59 blocked answers were resolved, with only 2 fallbacks.
• Latency: about 0.5 s per answer as an offline gate. The NLI and routing calls take tens of milliseconds.

How to use it

The page gives no code, install command or licence. It points to the paper on Hugging Face and arXiv.

The input it needs is an agent trace that keeps tool outputs and their source IDs. The named models are the evaluated configuration, not a requirement. Hosted models can be substituted, but the authors say a new setup needs its own testing and calibration.

NVIDIA NVFlow merged an optional grounding-verification stage based on this approach for its finance agent. That stage:

• checks answers against retrieved SEC excerpts
• stores its decisions separately, leaving the original rollouts and training data unchanged
• does not include the repair loop

Limits

• Conservative by design: about two supported claims are held for every three unsupported ones blocked. Budget for review load.
• Weak on similar sources: exact-source identification drops to about 50% when sources resemble each other. The authors flag this as open work.
• Repair mostly falls back on real traces: 144 of 173 repaired answers became fallback text. Expect a "repair" to usually mean declining to answer.
• Narrow evaluation: all results come from one medical domain and one local model configuration. The other domains are claimed as adaptable, not tested.