Inject, Align, Recover: Staged Post-Training for Retrieval-Free Document Knowledge Internalization
Summary
IAR is a three-stage post-training framework that internalizes structured document knowledge into language models for retrieval-free question answering, enhancing both domain-specific accuracy and general model performance.
View Cached Full Text
Cached at: 08/21/26, 04:07 AM
Paper page - Inject, Align, Recover: Staged Post-Training for Retrieval-Free Document Knowledge Internalization
Source: https://huggingface.co/papers/2608.20281
Abstract
IAR is a three-stage post-training framework that injects structured document knowledge into language models, aligns them for retrieval-free question answering, and recovers general capabilities, improving both domain accuracy and general performance.
Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time. We study this setting asdocument knowledge internalization: converting a fixed corpus into usable parametric knowledge forretrieval-free question answering. We proposeIAR(Inject, Align, and Recover), a three-stage post-training framework that separatesstructured document knowledge injection,QA behavior alignment, andgeneral ability recovery. Unlike conventional continued pretraining, Inject converts source documents into continuation, rewrite, and instruction-conditioned reconstruction objectives. Align then adapts the injected model with answer-only QA supervision, while Recover merges the domain-adapted model with the base instruction model to recover general capabilities. Across Common Corpus (CC) and CCI, and across Llama, Phi, Qwen, and SmolLM model families,IARimproves the domain-primary domain-general frontier for retrieval-free document internalization. In the main comparison,IARimproves over Vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance across IFEval, MMLU, and MSBench. Extended CC baselines show thatLoRAandFAPMcan win individual general metrics, but among methods that also reach leading or near-leading domain internalization,IARretains one of the strongest general profiles.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.20281
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.20281 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.20281 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.20281 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Leveraging External Knowledge for Historical Document Restoration via Retrieval-Augmented Large Language Models
This paper introduces ARI, a framework that uses retrieval-augmented large language models to restore illegible portions of historical documents, significantly improving named entity restoration by combining implicit LLM knowledge with explicitly retrieved external historical context.
HIRA: A Human-in-the-Loop Retrieval-Augmented Cascade for Document Classification in Regulated Industries
HIRA is a training-free, on-premises retrieval-augmented cascade system for document classification in regulated industries that uses human feedback and on-premises LLMs to improve accuracy while reducing model retraining and LLM calls.
Hybrid-IR: Dual-Path Hybrid Retrieval with Iterative Reasoning for Complex Medical Question Answering
Hybrid-IR introduces a dual-path retrieval framework combining graph-based and dense retrieval with iterative reasoning to improve complex medical QA, addressing limitations in existing RAG methods. Experiments on three benchmarks show effectiveness.
Understanding the Behaviors of Environment-aware Information Retrieval
This paper presents the first systematic analysis of how large language models can learn to adapt query formulation strategies for different retrievers using reinforcement learning, revealing distinct optimal query styles and introducing a branching-based rollout technique for multi-retrieval-step training stability.
STAIR (STructure Aware Information Retriever): A novel dataset and LLM based retriever for document structure augmentation
STAIR introduces a structure-aware retrieval system that leverages Table of Contents for enhanced document retrieval in Large Language Models, achieving high performance on the new SearchTome benchmark dataset.