Fast and Faithful: Real-Time Verification for Long-Document Retrieval-Augmented Generation Systems

Papers with Code Trending Papers
retrieval-augmented-generation real-time-verification long-document latency-constraints adaptive-inference grounding

Summary

This paper presents a real-time verification system for retrieval-augmented generation that processes long documents up to 32K tokens, using adaptive inference strategies to balance latency and verification coverage. It provides practical guidance for building reliable RAG systems.

Retrieval-augmented generation (RAG) is increasingly deployed in enterprise search and document-centric assistants, where responses must be grounded in long and complex source materials. In practice, verifying that generated answers faithfully reflect retrieved documents is difficult: large language models can check long contexts but are too slow and costly for interactive services, while lightweight classifiers operate within strict context limits and frequently miss evidence outside truncated passages. We present the design of a real-time verification component integrated into a production RAG pipeline that enables full-document grounding under latency constraints. The system processes documents up to 32K tokens and employs adaptive inference strategies to balance response time and verification coverage across workloads. We describe the architectural decisions, operational trade-offs, and evaluation methodology used to deploy the verifier, and show that full-context verification substantially improves detection of unsupported responses compared with truncated validation. Our experience highlights when long-context verification is necessary, why chunk-based checking often fails in real documents, and how latency budgets shape model design. These findings provide practical guidance for practitioners building reliable large-scale retrieval-augmented applications. (Model, benchmark, and code: https://huggingface.co/llm-semantic-router)
Original Article
View Cached Full Text

Cached at: 07/01/26, 09:40 PM

Paper page - Fast and Faithful: Real-Time Verification for Long-Document Retrieval-Augmented Generation Systems

Source: https://huggingface.co/papers/2603.23508

Abstract

A real-time verification system for retrieval-augmented generation that processes long documents and balances latency constraints with comprehensive answer validation.

Retrieval-augmented generation(RAG) is increasingly deployed in enterprise search anddocument-centric assistants, where responses must be grounded in long and complex source materials. In practice, verifying that generated answers faithfully reflect retrieved documents is difficult:large language modelscan check long contexts but are too slow and costly for interactive services, while lightweight classifiers operate within strict context limits and frequently miss evidence outside truncated passages. We present the design of a real-time verification component integrated into a production RAG pipeline that enablesfull-document groundingunderlatency constraints. The system processes documents up to 32K tokens and employsadaptive inference strategiesto balance response time and verification coverage across workloads. We describe the architectural decisions, operational trade-offs, and evaluation methodology used to deploy the verifier, and show that full-context verification substantially improves detection of unsupported responses compared with truncated validation. Our experience highlights when long-context verification is necessary, whychunk-based checkingoften fails in real documents, and how latency budgets shape model design. These findings provide practical guidance for practitioners building reliable large-scale retrieval-augmented applications. (Model, benchmark, and code: https://huggingface.co/llm-semantic-router)

View arXiv pageView PDFGitHub4.71kautoAdd to collection

Get this paper in your agent:

hf papers read 2603\.23508

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2603.23508 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2603.23508 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2603.23508 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Candidate-Constrained Retrieval-Augmented Generation for LongEval-RAG: System Design and Empirical Analysis

arXiv cs.CL

This paper presents a candidate-constrained RAG system for the LongEval-RAG task at CLEF 2026, combining deterministic provenance tracking with passage retrieval, query expansion, pseudo-relevance feedback, reciprocal rank fusion, evidence reranking, and citation-aware aggregation. An ablation study of ten pipeline variants shows that a rule-based chunking pipeline with sentence-level neural selection achieves the best performance.

Why Retrieval-Augmented Generation Fails: A Graph Perspective

arXiv cs.CL

This paper investigates why Retrieval-Augmented Generation (RAG) systems fail despite having access to correct evidence. Using circuit tracing and attribution graphs, the authors find that correct predictions exhibit deeper reasoning paths and more distributed evidence flow, while failures show shallow and fragmented patterns. They propose a graph-based error detection framework and targeted interventions to improve RAG reliability.