Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA
Summary
This paper introduces accuracy-blind answer churn in retrieval-augmented QA systems and proposes the Snapshot Compatibility Audit to detect hidden answer changes when the corpus is updated, even if overall accuracy appears stable.
View Cached Full Text
Cached at: 08/25/26, 04:34 AM
Paper page - Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA
Source: https://huggingface.co/papers/2608.22856
Abstract
Retrieval-augmented QA systems can exhibit hidden answer churn during index updates without noticeable accuracy changes, motivating compatibility audits alongside utility evaluations.
Aretrieval-augmented QAsystem can return different answers after anindex expansioneven when its requested model identifier, prompt, retrieval policy, evidence depth, rendering, and exposed generation controls are held fixed. Aggregate accuracy may hide these changes when gains and losses cancel, while ordinary generation variability makes one-shot comparisons overstate update effects. We call the hidden phenomenonaccuracy-blind answer churnand introduce theSnapshot Compatibility Audit, which estimates excess answer churn by subtracting same-snapshot repeat disagreement from cross-snapshot disagreement. We instantiate it by expanding one frozenFineWebprefix from one to seven shards. In a preregistered 400-questionNatural Questionsstudy, normalized-exact and blinded-semantic excess churn are 6.44 and 10.25 percentage points while exact-match accuracy changes by only -1.50 points. A post-hoc analysis finds repeat-stable semantic flips on 40/400 questions. A separately preregistered 200-questionTriviaQAstudy yields smaller, directionally consistent excess churn while exact-match accuracy moves in the opposite direction. An outcome-blind post-hoc 100-question subset replication with a secondDeepSeekgenerator and serving configuration finds 8.75 pp of semantic excess churn even as exact match rises by 3.00 percentage points. Answer-level compatibility can therefore fail without a conspicuous or consistently directed utility shift. Retrieval-augmented releases should audit compatibility alongside utility.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.22856
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.22856 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.22856 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.22856 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Answer Presence Drives RAG Rewriting Gains
The paper investigates whether the performance gains from rewriting retrieved passages in RAG QA pipelines are causally driven by the presence of the gold answer string in the rewritten context, using controlled intervention audits across multiple models and datasets.
Asymptotic Risk Calibration for Selective Question Answering
The paper proposes A-CRC-QA, a post-hoc calibration framework for selective question answering that controls error rates among accepted answers via asymptotic risk calibration, demonstrating improved reliability-retention trade-offs on CoQA and MedMCQA.
Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting
This paper audits knowledge-based VQA benchmarks, revealing systematic violations of assumptions that make accuracy a misleading metric. It introduces a repair protocol and multi-entity augmentation to restore answer derivability and question clarity, showing that corrected settings yield markedly different model rankings.
AB-RAG: Adaptive Budgeted Retrieval-Augmented Generation for Reliable Question Answering
AB-RAG is a training-free, backbone-agnostic framework that adaptively retrieves passages for question answering by estimating answer confidence, improving efficiency and accuracy across multiple backbones and datasets.
The agent failure mode no eval catches: acting on a fact that was true when it was cached and wrong when it was used
Discusses a blind spot in AI agent reliability: cached facts that were true when ingested but become stale by the time they are used, leading to coherent but incorrect actions. Proposes separating consistency (match with source) from currency (source still true now), and asks how the community handles this.