ProvenanceGuard Verifies Source-Aware Factuality for LLM Agents

Addressing Cross-Source Conflation

A new paper introduces ProvenanceGuard, a system designed to verify the source-aware factuality of Multi-Component Pipeline (MCP) Large Language Model (LLM) agents. The primary issue ProvenanceGuard targets is "cross-source conflation," which occurs when a claim is factually correct but incorrectly attributed to a source different from where the information originated. This problem is distinct from outright factual errors, as a source-blind verifier might pass such claims if the fact exists anywhere in the evidence pool.

The Problem with Incorrect Attribution

Incorrect attribution can be as damaging as a factual error, particularly in sensitive applications like customer support or clinical agents. For example, a customer support agent might state a refund window is in an "account record" when it actually appears in a "policy document." Similarly, patient-specific medical details from a patient history tool could be misleading if presented as findings from "medical literature." ProvenanceGuard aims to prevent these types of misattributions by maintaining the connection between a claim and its specific source.

How ProvenanceGuard Works

ProvenanceGuard functions as a post-generation verification layer for black-box MCP agents. It operates after an agent generates an answer, preserving source identity throughout the verification pipeline rather than collapsing evidence into an anonymous context. The system reads the MCP trace, including tool outputs and their source IDs, without requiring agent retraining.

Verification Process

The verification process involves five sequential steps: breaking the answer into specific claims, identifying the most relevant source for each claim, verifying if that source actually supports the claim, comparing the identified source with the one stated or implied in the answer, and finally, issuing a per-claim source verdict and a global answer-level allow or block decision.