KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Training

# Knowledge Injection via Input Corruption for Auto-regressive Training
Source: [https://arxiv.org/html/2609.00082](https://arxiv.org/html/2609.00082)
Meghanadh Pulivarthi111Equal contribution\.Kushagra Bhushan111Equal contribution\.Vineet Kumar222Work done while at IBM\. Currently at Amazon Books Science\.Gaurav PandeyAffiliation:Jaydeep Sen Dinesh Raghu Sachindra Joshi Yatin NandwaniAffiliation:IBMAffiliation:\{Meghanadh\.Pulivarthi1, kushagrabhushan, Yatin\.Nandwani\}@ibm\.comAffiliation:\{gpandey1, jaydesen, diraghu1, jsachind\}@in\.ibm\.comEmail:[vineet\.mundhra@gmail\.com](mailto:)

###### Abstract

LLMs acquire vast amounts of knowledge during pre\-training, but often lack the specialized knowledge needed to answer questions from niche sources such as manuals or technical documents unseen during pre\-training\. Continued pre\-training \(CPT\) is widely used to inject such knowledge into model parameters\. However, niche documents seldom repeat facts, making it difficult for CPT to robustly acquire such knowledge\. Recent works address this by generating multiple paraphrases of the new knowledge, but paraphrasing is computationally expensive and typically requires powerful LLMs\. In this work, we introduce KItCAT: Knowledge Injection via Corrupted Auto\-regressive Training, a lightweight training strategy that reduces the need for paraphrasing in decoder\-only LLMs\. KItCAT augments standard next\-token prediction by stochastically corrupting the input sequence\. During training, a random subset of input tokens is replaced with other vocabulary tokens while the original next\-token labels are kept unchanged\. This simple intervention generates diverse training inputs from each sample, enabling large\-scale data augmentation at negligible cost\. We show that KItCAT consistently improves over CPT across multiple datasets and model families\. Code is available at[https://github\.com/meghanadhpulivarthi/KItCAT](https://github.com/meghanadhpulivarthi/KItCAT)\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2609.00082v1/figures/masking_schemes.png)Figure 1:The four KItCAT corruption schemes applied to the same source sentence\. Each scheme corrupts a different subset of positions in theSourceto produce itsInput; the trainingLabelsare the standard next\-token\-prediction targets\. Corruption is applied only to the conditioning input, never to the labels\.LLMs are pre\-trained on massive web corpora, yet they often fail to capture specialized knowledge that is absent or underrepresented on the public web, such as information contained in proprietary manuals or technical documents\. Continued Pre\-Training \(CPT\)[Ke et al\. \(2023\)](https://arxiv.org/html/2609.00082#bib.bib14);[Ke et al\. \(2025\)](https://arxiv.org/html/2609.00082#bib.bib15)is a common approach for injecting such specialized knowledge into the parameters of the LLMs\.

A key challenge, however, is that the niche documents seldom repeat facts or present them in varied contexts\. Consequently, CPT receives limited repeated exposure to newly introduced knowledge, making it difficult for the model to robustly internalize these facts[Allen\-Zhu and Li \(2024\)](https://arxiv.org/html/2609.00082#bib.bib16)\. In such low\-diversity training settings, models can instead overfit to surface\-level lexical patterns and spurious correlations that recur across epochs, rather than learning the underlying semantics\.

To mitigate the limited diversity in the training documents, recent works propose to synthetically generate multiple paraphrases of the text[Ovadia et al\. \(2024\)](https://arxiv.org/html/2609.00082#bib.bib10);[Yang et al\. \(2025\)](https://arxiv.org/html/2609.00082#bib.bib12)\. While effective, these approaches rely on large proprietary LLMs[Achiam et al\. \(2023\)](https://arxiv.org/html/2609.00082#bib.bib37);[Team et al\. \(2023\)](https://arxiv.org/html/2609.00082#bib.bib38), which can be costly and slow when generating sizable synthetic datasets\. They may also be infeasible in settings with privacy restrictions\.

In this work, we ask \-can we introduce effective data diversity without relying on external synthetic data generation? We observe that overfitting in low\-diversity regimes is amplified by the fact that the model encounters the exact same training samples in every epoch\. If the model latches onto a spurious pattern in one pass, repeated identical exposure reinforces this shortcut, degrading generalization\.

Our key insight is that preventing the model from ever seeing the exact same input twice can mitigate this reinforcement effect\. Motivated by this idea, we propose KItCAT \- Knowledge Injection via Corrupted Auto\-regressive Training, a simple yet effective strategy for injecting diversity through controlled corruption of the input\. During training, KItCAT perturbs input sequences using one of four corruption schemes \(Figure[1](https://arxiv.org/html/2609.00082#S1.F1)\): \(1\) KItCAT\-rand, where selected tokens are substituted with randomly sampled vocabulary items; \(2\) KItCAT\-mask, where tokens are replaced with a special mask token \(e\.g\.,\[MASK\]\); \(3\) KItCAT\-SSMBA, which replaces selected tokens with contextually plausible alternatives; and \(4\) KItCAT\-MASKER, which preferentially masks informative keywords\. KItCAT\-SSMBA and KItCAT\-MASKER adapt advanced corruption schemes originally developed for encoder\-only models to knowledge injection with decoder\-only LLMs\. In decoder\-only models, corruption is applied solely to the conditioning input while the target labels remain unchanged\. Under random replacement, the model must learn to ignore irrelevant noise, whereas in mask\-based corruption, the missing information is explicitly signaled\. By preventing the model from encountering identical inputs across epochs, KItCAT reduces reinforcement of spurious lexical patterns and encourages learning more semantically grounded representations\.

Experiments across multiple knowledge\-injection benchmarks and three model families show that KItCAT consistently improves standard CPT\. Notably, KItCAT is agnostic to the input text and can be applied directly to original corpora as well as to paraphrases used to strengthen CPT\. This makes it complementary to existing approaches and reduces the reliance on paraphrasing\. Overall, our contributions are: \(1\) We propose Knowledge Injection via Corrupted Auto\-regressive Training \(KItCAT\), a lightweight and generalizable data augmentation method for fine\-tuning decoder\-only language models by stochastically corrupting input tokens while preserving the original autoregressive training objective\. \(2\) We empirically demonstrate that KItCAT consistently improves standard CPT across three model families \(3\) We show that KItCAT induces effective data diversity to prevent overfitting, significantly reducing the need for expensive synthetic data while remaining complementary to paraphrase\-based augmentation\.

## 2Related Work

Approaches to injecting knowledge into LLM parameters can be broadly categorised as CPT\-based methods and SFT\-based methods\. CPT\-based methods inject knowledge using raw domain text and its paraphrases\([Wu et al\., 2024](https://arxiv.org/html/2609.00082#bib.bib17);[Christophe et al\., 2024](https://arxiv.org/html/2609.00082#bib.bib11);[Ke et al\., 2023](https://arxiv.org/html/2609.00082#bib.bib14);[Zhao et al\., 2025](https://arxiv.org/html/2609.00082#bib.bib18);[Zhang et al\., 2024](https://arxiv.org/html/2609.00082#bib.bib22);[Wu et al\., 2023](https://arxiv.org/html/2609.00082#bib.bib19);[Shah et al\., 2022](https://arxiv.org/html/2609.00082#bib.bib20);[Xie et al\., 2023](https://arxiv.org/html/2609.00082#bib.bib21)\)\. SFT\-based methods use powerful LLMs to transform raw text into task\-oriented formats such as QA, summarization, or reading comprehension\([Ovadia et al\., 2025](https://arxiv.org/html/2609.00082#bib.bib29);[Bhushan et al\., 2025](https://arxiv.org/html/2609.00082#bib.bib13)\)\. Both approaches benefit from high\-diversity training data and, in its absence, often rely on generating large amounts of synthetic paraphrases or SFT data using powerful LLMs\. In contrast, our work proposes lightweight input corruption as an alternative to expensive paraphrase generation during CPT\-style knowledge injection\.

Corruption can be applied to the input or to hidden representations within the model and used in both SFT and CPT training styles\. Dropout\([Hinton et al\., 2012](https://arxiv.org/html/2609.00082#bib.bib26)\)corrupts hidden activations during training, while Latent Paraphrasing\([Kang et al\., 2024](https://arxiv.org/html/2609.00082#bib.bib39)\)perturbs hidden representations to facilitate knowledge injection and can be applied to both CPT and SFT\.

Input\-corruption techniques for training LLMs are discussed in Appendix[A\.2](https://arxiv.org/html/2609.00082#A1.SS2)\.

## 3The KItCAT approach

##### Preliminaries:

Let𝒟\\mathcal\{D\}be the new knowledge that we want to inject in our pretrained modelpθp\_\{\\theta\}\. Let𝒱\\mathcal\{V\}denote the vocabulary and𝐬=\(s1,…,sT\)\\mathbf\{s\}=\(s\_\{1\},\.\.\.,s\_\{T\}\)be a token sequence drawn from𝒟\\mathcal\{D\}, wherest∈𝒱,1≤t≤Ts\_\{t\}\\in\\mathcal\{V\}\\,,1\\leq t\\leq T\.

ℒNTP\(θ\)=𝔼𝐬∼𝒟∑t=1T−logpθ\(st\|s<t\)\\mathcal\{L\}\_\{\\text\{NTP\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\}\}\\sum\_\{t=1\}^\{T\}\-\\log p\_\{\\theta\}\(s\_\{t\}\|s\_\{<t\}\)\(1\)Here,s<t=\(s1,…,st−1\)s\_\{<t\}=\(s\_\{1\},\.\.\.,s\_\{t\-1\}\)is a prefix of the sequence𝐬\\mathbf\{s\}\.

##### Knowledge Injection via Input Corruption:

TheℒNTP\\mathcal\{L\}\_\{\\text\{NTP\}\}objective maximizes the likelihood of a tokensts\_\{t\}conditioned on its exact prefixs<ts\_\{<t\}\. When seen multiple times across epochs, the model may overfit to spurious correlations betweens<ts\_\{<t\}andsts\_\{t\}\.

To mitigate this, we propose KItCAT: Knowledge Injection via Corrupted Auto\-regressive Training\. Let𝒬⁡\(𝐬~∣𝐬\)\\mathcal\{Q\}\(\\tilde\{\\mathbf\{s\}\}\\mid\\mathbf\{s\}\)denote a distribution over corrupted versions of the input sequence𝐬\\mathbf\{s\}\. Instead of conditioning only on the exact prefix, KItCAT trains on corrupted prefixes sampled from𝒬⁡\(𝐬~∣𝐬\)\\mathcal\{Q\}\(\\tilde\{\\mathbf\{s\}\}\\mid\\mathbf\{s\}\)while preserving the target token as a likely continuation\. The model is thus encouraged to predict the same target under perturbed contexts\.

ℒCAT​\(θ\)=\\displaystyle\\mathcal\{L\}\_\{\\text\{CAT\}\}\(\\theta\)=𝔼𝐬∼𝒟𝔼𝐬~∼𝒬\(\.\|𝐬\)∑t=1T−logpθ\(st\|s~<t\),\\displaystyle\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\}\}\\mathbb\{E\}\_\{\\tilde\{\\mathbf\{s\}\}\\sim\\mathcal\{Q\}\(\.\|\\mathbf\{s\}\)\}\\sum\_\{t=1\}^\{T\}\-\\log p\_\{\\theta\}\(s\_\{t\}\|\\tilde\{s\}\_\{<t\}\)\\,,\(2\)wheres~<t\\tilde\{s\}\_\{<t\}is a prefix of the corrupted input𝐬~\\tilde\{\\mathbf\{s\}\}\. Intuitively, training on perturbed prefixes prevents reliance on spurious lexical patterns\. Instead, it encourages the model to use information that remains present across perturbations\.

To construct perturbations𝐬~\\tilde\{\\mathbf\{s\}\}, we consider four stochastic in\-place token modifications in the sequence𝐬\\mathbf\{s\}: \(1\)*KItCAT\-mask*, which replaces tokens with a special mask token; \(2\)*KItCAT\-rand*, which replaces tokens with randomly sampled vocabulary tokens; \(3\)*KItCAT\-SSMBA*, which replaces randomly selected tokens with contextually plausible alternatives; and \(4\)*KItCAT\-MASKER*, which masks informative keywords identified from the training corpus\. The latter two adapt the central ideas of SSMBA\([Ng et al\., 2020](https://arxiv.org/html/2609.00082#bib.bib40)\)and MASKER\([Moon et al\., 2021](https://arxiv.org/html/2609.00082#bib.bib41)\)to the conditioning inputs of decoder\-only LLMs\.

For every method, let𝒞⊆\{1,…,T\}\\mathcal\{C\}\\subseteq\\\{1,\\ldots,T\\\}denote the set of token positions selected for corruption, and letrjr\_\{j\}be the replacement token at each selected positionj∈𝒞j\\in\\mathcal\{C\}\. The corrupted input is therefore

s~j=\{rjif​j∈𝒞,sjotherwise\.\\tilde\{s\}\_\{j\}=\\begin\{cases\}r\_\{j\}&\\text\{if \}j\\in\\mathcal\{C\},\\\\ s\_\{j\}&\\text\{otherwise\.\}\\end\{cases\}For KItCAT\-mask, KItCAT\-rand, and KItCAT\-SSMBA, we construct𝒞\\mathcal\{C\}by independently selecting each token position with probabilityp∈\(0,1\)p\\in\(0,1\), the corruption probability\. We resample𝒞\\mathcal\{C\}and the replacement tokens at each epoch\.

##### KItCAT\-mask\.

For eachj∈𝒞j\\in\\mathcal\{C\}, we use the special mask token:rj=\[MASK\]r\_\{j\}=\\text\{\[MASK\]\}\.

##### KItCAT\-rand\.

For eachj∈𝒞j\\in\\mathcal\{C\}, we sample a replacement uniformly from the vocabulary:rj∼Unif​\(𝒱\)r\_\{j\}\\sim\\text\{Unif\}\(\\mathcal\{V\}\)\.

##### KItCAT\-SSMBA\.

For eachj∈𝒞j\\in\\mathcal\{C\}, we mask the position and sample a replacement from a frozen masked language modelqϕq\_\{\\phi\}:rj∼qϕ\(⋅∣𝐬∖𝒞\)r\_\{j\}\\sim q\_\{\\phi\}\(\\cdot\\mid\\mathbf\{s\}\_\{\\setminus\\mathcal\{C\}\}\), where𝐬∖𝒞\\mathbf\{s\}\_\{\\setminus\\mathcal\{C\}\}denote𝐬\\mathbf\{s\}with all positions in𝒞\\mathcal\{C\}masked\.

##### KItCAT\-MASKER\.

We identify keyword spans from the training corpus using TF–IDF and independently select each span, applying the corruption probabilityppat the span level rather than the token level;𝒞\\mathcal\{C\}contains the token positions in the selected spans and is resampled every epoch\. Each selected token is replaced with the mask token:rj=\[MASK\]r\_\{j\}=\\text\{\[MASK\]\}forj∈𝒞j\\in\\mathcal\{C\}\.

For all four variants, corruption is applied only to the conditioning prefix and the target tokensts\_\{t\}remains unchanged\.

In[SectionA\.3](https://arxiv.org/html/2609.00082#A1.SS3), we formalizeℒCAT\\mathcal\{L\}\_\{\\text\{CAT\}\}as a constrained optimization objective that enforces invariance to label\-preserving corruptions, plausibly reducing overfitting\.

## 4Experimental Setup

##### Datasets:

##### Baselines:

We compare with the out\-of\-the\-box model \(*Instruct*\), standard CPT using next\-token prediction \(NTP\), and four KItCAT corruption schemes\. KItCAT\-SSMBA adapts SSMBA\([Ng et al\., 2020](https://arxiv.org/html/2609.00082#bib.bib40)\)to CPT by using an encoder model to sample contextually plausible token replacements, while KItCAT\-MASKER adapts MASKER\([Moon et al\., 2021](https://arxiv.org/html/2609.00082#bib.bib41)\)by preferentially masking informative keywords\.

##### Evaluation metrics:

To test if the knowledge in the documents has been successfully injected in the model’s parameters, we evaluate the fine\-tuned models using the test QAs accompanying each dataset\. We use an LLM\-as\-a\-Judge to quantify the correctness of the generated responses with respect to the gold answers\. See[SectionA\.9](https://arxiv.org/html/2609.00082#A1.SS9)for the exact prompt\.

##### Models and Training Details:

We fine\-tune[Mistral\-7B\-Instruct\-v0\.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)using[Huggingface’s SFTTrainer](https://huggingface.co/docs/trl/en/sft_trainer)\. We apply LoRA to all linear layers of the LLM and early stop based on the validation metric\. Unless stated otherwise, we report results using Mistral\-7B\-Instruct\-v0\.3\. To test model\-family and scale generalization, we additionally evaluate Qwen3\-14B and Llama\-2\-7B\-Chat on all datasets\. See[SectionA\.5](https://arxiv.org/html/2609.00082#A1.SS5)for more training details\.

Table 1:LLM\-as\-a\-Judge accuracy \(×100\\times 100\) averaged over test QAs across knowledge injection benchmarks on Mistral\-7B\-Instruct\. RB1/2: Redbook 1/2; Comp\.: Companies\.![Refer to caption](https://arxiv.org/html/2609.00082v1/figures/train_size_abl_companies.png)\(a\)Training data size ablation\.
![Refer to caption](https://arxiv.org/html/2609.00082v1/figures/validation_loss_curve.png)\(b\)Validation loss curves\.

Figure 2:\(a\) Comparison of standard next\-token prediction \(NTP\) and KItCAT\-mask under varying training data sizes for knowledge injection on the Companies dataset \(b\) Validation loss curves comparing standard NTP, KItCAT\-mask, and KItCAT\-rand on the Companies validation set

## 5Experimental Results

Effectiveness of KItCAT in injecting knowledge \(RQ1\):To assess knowledge injection, we evaluate each adapted model on the corresponding test QAs using the LLM judge described in[Section4](https://arxiv.org/html/2609.00082#S4)\.[Table1](https://arxiv.org/html/2609.00082#S4.T1)reports the results across the four corpora\.

All four KItCAT variants outperform NTP on average\. KItCAT\-rand achieves the highest average score \(27\.7\), a 10\.0\-point improvement over NTP \(17\.6\), while KItCAT\-mask also improves the average score to 26\.1\. The simple corruption schemes, KItCAT\-mask and KItCAT\-rand, together obtain the best score on three of the four datasets \(both Redbooks and Companies\), whereas the advanced schemes prevail only once, with KItCAT\-SSMBA leading on PopQA \(22\.1 vs\. 20\.2 for KItCAT\-rand\)\. On average, simple corruption proves more effective than the more elaborate, advanced schemes\. We hypothesize that perturbing random tokens in the context increases the overall diversity of the training corpus and hence, prevents overfitting to the spurious patterns thereby leading to improved performance on QA pairs from the corpus\. Example outputs are provided in[SectionA\.1](https://arxiv.org/html/2609.00082#A1.SS1)\.

KItCAT also generalizes to Qwen3\-14B and Llama\-2\-7B\-Chat, consistently improving over NTP across the evaluated corpora \(and\)\. A sensitivity analysis shows that performance is strongest at moderate corruption probabilities \(p=0\.15p=0\.15–0\.300\.30\) and degrades when corruption is too weak or too strong; see[SectionsA\.8](https://arxiv.org/html/2609.00082#A1.SS8)and\.

##### Impact of training data size \(RQ2\):

Next, we note that KItCAT is complementary to other approaches for data augmentation that generate multiple paraphrases for each document present in the corpus and use them for CPT\. Hence, for this experiment, we combine these two complementary methods for effective knowledge injection and observe the resultant behavior\.

We consider two synthetic data generation strategies: \(1\)*Rephrase*, where an LLM[Hurst et al\. \(2024\)](https://arxiv.org/html/2609.00082#bib.bib23)is prompted with multiple prompts[Ovadia et al\. \(2025\)](https://arxiv.org/html/2609.00082#bib.bib29)to generate multiple paraphrases per document, and \(2\)*Entigraph*, which follows[Yang et al\. \(2025\)](https://arxiv.org/html/2609.00082#bib.bib12)by extracting entities from text and prompting an LLM to describe relationships between entity pairs and triples\. See[SectionA\.9](https://arxiv.org/html/2609.00082#A1.SS9)for the exact prompts\.

We control the amount of*Entigraph*data by randomly sampling pairs and triples until we get the desired token count\. For both*Rephrase*and*Entigraph*, we train models using standard NTP loss and KItCAT\-mask, while varying the amount of synthetic data\.

[Figure2\(a\)](https://arxiv.org/html/2609.00082#S4.F2.sf1)presents the results\. We make the following observations: \(1\) The gains obtained by KItCAT using only the original data \(1×\\times\) are higher than standard NTP on 9×\\timesRephrase augmented data \(26\.84 vs\. 22\.35\) and 5×\\timesEntigraph augmented data \(26\.84 vs\. 24\.53\), demonstrating a 5–9×\\timeseffective data multiplier over standard NTP\. \(2\) KItCAT is complementary to synthetic data generation as a data augmentation strategy\. Performance consistently improves as we introduce additional paraphrased data\.

In[SectionA\.7](https://arxiv.org/html/2609.00082#A1.SS7), we present a comparison of compute costs and LLM Judge accuracy for different methods on the Companies dataset\. We observe that KItCAT\-mask achieves 27\.6 accuracy at an estimated cost of 30 PFLOPs, matching the 27\.4 accuracy of Rephrase\-10×\\timeswhile requiring only 15\.2% of its 198 PFLOPs\. Combining corruption with paraphrases gives the best accuracy, confirming that KItCAT is both a cheaper partial substitute and a complementary augmentation \(\)\.

##### Robustness of KItCAT:

We next evaluate the robustness of KItCAT relative to the standard next\-token prediction \(NTP\) objective by analyzing their learning dynamics\. Specifically, we examine validation loss curves on QA pairs from the Companies validation set \([Figure2\(b\)](https://arxiv.org/html/2609.00082#S4.F2.sf2)\)\.

We observe that the NTP objective reaches its minimum validation loss within the first seven epochs, after which it exhibits clear overfitting, as evidenced by a sharp rise in validation loss\. In contrast, the two variants of KItCAT,viz\.KItCAT\-mask and KItCAT\-rand, maintain stable validation loss throughout training\.

These results indicate that KItCAT provides improved training stability and achieves better generalization compared to the NTP baseline\.

## 6Conclusion

We introduce KItCAT, a simple yet effective modification to autoregressive training for injecting new knowledge into LLM parameters\. By ensuring that the model never encounters the exact same input twice, KItCAT mitigates the reinforcement of spurious lexical correlations that commonly arise in low\-diversity CPT settings\.

Experiments across four knowledge\-injection benchmarks and three model families show that KItCAT consistently outperforms standard next\-token prediction\-based training\. Moreover, KItCAT is lightweight, architecture\-agnostic, and complementary to existing data augmentation approaches\. Unlike synthetic data generation, it requires no external models \(for paraphrasing\) and incurs negligible computational overhead, making it a practical drop\-in replacement for standard CPT objectives in real\-world knowledge injection settings\.

## Limitations

A limitation of KItCAT is that, while corruption\-based augmentation reduces the need for expensive synthetic paraphrase generation, it does not fully substitute for it\. Existing approaches that incorporate synthesized paraphrases continue to provide complementary benefits, and the highest downstream performance is achieved when corruption and paraphrase\-based augmentation are combined\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2609.00082#S1.p3.1)\.
- Allen\-Zhu and Li \(2024\)Z\. Allen\-Zhu and Y\. LiPhysics of language models: part 3\.1, knowledge storage and extraction\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=5x788rqbcj)Cited by:[§1](https://arxiv.org/html/2609.00082#S1.p2.1)\.
- Bhushanet al\.\(2025\)K\. Bhushan, Y\. Nandwani, D\. Khandelwal, S\. Gupta, G\. Pandey, D\. Raghu, and S\. JoshiSystematic knowledge injection into large language models via diverse augmentation for domain\-specific RAG\.InFindings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April 29 \- May 4, 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),pp\. 5922–5943\.External Links:[Link](https://doi.org/10.18653/v1/2025.findings-naacl.329),[Document](https://dx.doi.org/10.18653/V1/2025.FINDINGS-NAACL.329)Cited by:[§2](https://arxiv.org/html/2609.00082#S2.p1.1),[§4](https://arxiv.org/html/2609.00082#S4.SS0.SSS0.Px1.p1.1)\.
- Bousquet and Elisseeff \(2002\)O\. Bousquet and A\. ElisseeffStability and generalization\.J\. Mach\. Learn\. Res\.2,pp\. 499–526\.External Links:[Link](https://jmlr.org/papers/v2/bousquet02a.html)Cited by:[§A\.3](https://arxiv.org/html/2609.00082#A1.SS3.p1.3)\.
- Chenet al\.\(2024\)C\. Chen, X\. Wang, T\. Lin, A\. Lv, Y\. Wu, X\. Gao, J\. Wen, R\. Yan, and Y\. LiMasked thought: simply masking partial reasoning steps can improve mathematical reasoning learning of language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),pp\. 5872–5900\.External Links:[Link](https://doi.org/10.18653/v1/2024.acl-long.320),[Document](https://dx.doi.org/10.18653/V1/2024.ACL-LONG.320)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p3.1)\.
- Chenet al\.\(2023\)J\. Chen, D\. Tam, C\. Raffel, M\. Bansal, and D\. YangAn empirical survey of data augmentation for limited data learning in nlp\.Transactions of the Association for Computational Linguistics11,pp\. 191–211\.External Links:ISSN 2307\-387X,[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00542),[Link](https://doi.org/10.1162/tacl_a_00542),https://direct\.mit\.edu/tacl/article\-pdf/doi/10\.1162/tacl\_a\_00542/2074871/tacl\_a\_00542\.pdfCited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1)\.
- Christopheet al\.\(2024\)C\. Christophe, T\. Raha, S\. Maslenkova, M\. U\. Salman, P\. K\. Kanithi, M\. A\. Pimentel, and S\. KhanBeyond fine\-tuning: unleashing the potential of continuous pretraining for clinical llms\.InFindings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12\-16, 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),pp\. 10549–10561\.External Links:[Link](https://doi.org/10.18653/v1/2024.findings-emnlp.618),[Document](https://dx.doi.org/10.18653/V1/2024.FINDINGS-EMNLP.618)Cited by:[§2](https://arxiv.org/html/2609.00082#S2.p1.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL\-HLT 2019, Minneapolis, MN, USA, June 2\-7, 2019, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),pp\. 4171–4186\.External Links:[Link](https://doi.org/10.18653/v1/n19-1423),[Document](https://dx.doi.org/10.18653/V1/N19-1423)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1)\.
- Guet al\.\(2020\)Y\. Gu, Z\. Zhang, X\. Wang, Z\. Liu, and M\. SunTrain no evil: selective masking for task\-guided pre\-training\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 6966–6974\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.566/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.566)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1)\.
- Hintonet al\.\(2012\)G\. E\. Hinton, N\. Srivastava, A\. Krizhevsky, I\. Sutskever, and R\. SalakhutdinovImproving neural networks by preventing co\-adaptation of feature detectors\.CoRRabs/1207\.0580\.External Links:[Link](http://arxiv.org/abs/1207.0580),1207\.0580Cited by:[§2](https://arxiv.org/html/2609.00082#S2.p2.1)\.
- Hurstet al\.\(2024\)A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§5](https://arxiv.org/html/2609.00082#S5.SS0.SSS0.Px1.p2.1)\.
- Iyeret al\.\(2023\)V\. Iyer, E\. Barba, A\. Birch, J\. Pan, and R\. NavigliCode\-switching with word senses for pretraining in neural machine translation\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 12889–12901\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.859/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.859)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1)\.
- Kakogeorgiouet al\.\(2022\)I\. Kakogeorgiou, S\. Gidaris, B\. Psomas, Y\. Avrithis, A\. Bursuc, K\. Karantzalos, and N\. KomodakisWhat to hide from your students: attention\-guided masked image modeling\.InComputer Vision \- ECCV 2022 \- 17th European Conference, Tel Aviv, Israel, October 23\-27, 2022, Proceedings, Part XXX,S\. Avidan, G\. J\. Brostow, M\. Cissé, G\. M\. Farinella, and T\. Hassner \(Eds\.\),Lecture Notes in Computer Science, Vol\.13690,pp\. 300–318\.External Links:[Link](https://doi.org/10.1007/978-3-031-20056-4/_18),[Document](https://dx.doi.org/10.1007/978-3-031-20056-4%5F18)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1)\.
- Kanget al\.\(2024\)M\. Kang, S\. J\. Hwang, G\. Lee, and J\. ChoLatent paraphrasing: perturbation on layers improves knowledge injection in language models\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/d85030334fadbd55043c911076caf0ae-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2609.00082#S2.p2.1)\.
- Keet al\.\(2025\)Z\. Ke, Y\. Ming, X\. Nguyen, C\. Xiong, and S\. JotyDemystifying domain\-adaptive post\-training for financial llms\.External Links:2501\.04961,[Link](https://arxiv.org/abs/2501.04961)Cited by:[§1](https://arxiv.org/html/2609.00082#S1.p1.1)\.
- Keet al\.\(2023\)Z\. Ke, Y\. Shao, H\. Lin, T\. Konishi, G\. Kim, and B\. LiuContinual pre\-training of language models\.InProceedings of The Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.00082#S1.p1.1),[§2](https://arxiv.org/html/2609.00082#S2.p1.1)\.
- Khoslaet al\.\(2025\)S\. Khosla, A\. Tiwari, K\. Kafle, S\. Jenni, H\. Zhao, J\. P\. Collomosse, and J\. ShiMAGNET: augmenting generative decoders with representation learning and infilling capabilities\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2025, Vienna, Austria, July 27 \- August 1, 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 27328–27346\.External Links:[Link](https://aclanthology.org/2025.acl-long.1325/)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p3.1)\.
- Lewiset al\.\(2020\)M\. Lewis, Y\. Liu, N\. Goyal, M\. Ghazvininejad, A\. Mohamed, O\. Levy, V\. Stoyanov, and L\. ZettlemoyerBART: denoising sequence\-to\-sequence pre\-training for natural language generation, translation, and comprehension\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5\-10, 2020,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. R\. Tetreault \(Eds\.\),pp\. 7871–7880\.External Links:[Link](https://doi.org/10.18653/v1/2020.acl-main.703),[Document](https://dx.doi.org/10.18653/V1/2020.ACL-MAIN.703)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1)\.
- Liet al\.\(2021\)Z\. Li, Z\. Chen, F\. Yang, W\. Li, Y\. Zhu, C\. Zhao, R\. Deng, L\. Wu, R\. Zhao, M\. Tang, and J\. WangMST: masked self\-supervised transformer for visual representation\.InAdvances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6\-14, 2021, virtual,M\. Ranzato, A\. Beygelzimer, Y\. N\. Dauphin, P\. Liang, and J\. W\. Vaughan \(Eds\.\),pp\. 13165–13176\.External Links:[Link](https://proceedings.neurips.cc/paper/2021/hash/6dbbe6abe5f14af882ff977fc3f35501-Abstract.html)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1)\.
- Linet al\.\(2021\)C\. Lin, T\. Miller, D\. Dligach, S\. Bethard, and G\. SavovaEntityBERT: entity\-centric masking strategy for model pretraining for the clinical domain\.InProceedings of the 20th Workshop on Biomedical Language Processing,D\. Demner\-Fushman, K\. B\. Cohen, S\. Ananiadou, and J\. Tsujii \(Eds\.\),Online,pp\. 191–201\.External Links:[Link](https://aclanthology.org/2021.bionlp-1.21/),[Document](https://dx.doi.org/10.18653/v1/2021.bionlp-1.21)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1)\.
- Linet al\.\(2020\)Z\. Lin, X\. Pan, M\. Wang, X\. Qiu, J\. Feng, H\. Zhou, and L\. LiPre\-training multilingual neural machine translation by leveraging alignment information\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16\-20, 2020,B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),pp\. 2649–2663\.External Links:[Link](https://doi.org/10.18653/v1/2020.emnlp-main.210),[Document](https://dx.doi.org/10.18653/V1/2020.EMNLP-MAIN.210)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1)\.
- Liuet al\.\(2019\)Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. StoyanovRoBERTa: A robustly optimized BERT pretraining approach\.CoRRabs/1907\.11692\.External Links:[Link](http://arxiv.org/abs/1907.11692),1907\.11692Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1)\.
- Luet al\.\(2024\)Y\. Lu, W\. Zhu, L\. Li, Y\. Qiao, and F\. YuanLLaMAX: scaling linguistic horizons of LLM by enhancing translation capabilities beyond 100 languages\.InFindings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12\-16, 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Findings of ACL,pp\. 10748–10772\.External Links:[Link](https://doi.org/10.18653/v1/2024.findings-emnlp.631),[Document](https://dx.doi.org/10.18653/V1/2024.FINDINGS-EMNLP.631)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p3.1)\.
- Mallenet al\.\(2023\)A\. Mallen, A\. Asai, V\. Zhong, R\. Das, D\. Khashabi, and H\. HajishirziWhen not to trust language models: investigating effectiveness of parametric and non\-parametric memories\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 9802–9822\.External Links:[Link](https://aclanthology.org/2023.acl-long.546/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.546)Cited by:[§4](https://arxiv.org/html/2609.00082#S4.SS0.SSS0.Px1.p1.1)\.
- Moonet al\.\(2021\)S\. J\. Moon, S\. Mo, K\. Lee, J\. Lee, and J\. ShinMASKER: masked keyword regularization for reliable text classification\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 13578–13586\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v35i15.17601)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1),[§3](https://arxiv.org/html/2609.00082#S3.SS0.SSS0.Px2.p3.1),[§4](https://arxiv.org/html/2609.00082#S4.SS0.SSS0.Px2.p1.1)\.
- Nget al\.\(2020\)N\. Ng, K\. Cho, and M\. GhassemiSSMBA: self\-supervised manifold based data augmentation for improving out\-of\-domain robustness\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 1268–1283\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.97),[Link](https://aclanthology.org/2020.emnlp-main.97/)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1),[§3](https://arxiv.org/html/2609.00082#S3.SS0.SSS0.Px2.p3.1),[§4](https://arxiv.org/html/2609.00082#S4.SS0.SSS0.Px2.p1.1)\.
- Ovadiaet al\.\(2025\)O\. Ovadia, M\. Brief, R\. Lemberg, and E\. SheetritKnowledge\-instruct: effective continual pre\-training from limited data using instructions\.arXiv preprint arXiv:2504\.05571\.Cited by:[§A\.10](https://arxiv.org/html/2609.00082#A1.SS10.p1.1),[§2](https://arxiv.org/html/2609.00082#S2.p1.1),[§4](https://arxiv.org/html/2609.00082#S4.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.00082#S5.SS0.SSS0.Px1.p2.1)\.
- Ovadiaet al\.\(2024\)O\. Ovadia, M\. Brief, M\. Mishaeli, and O\. ElishaFine\-tuning or retrieval? comparing knowledge injection in LLMs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 237–250\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.15/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.15)Cited by:[§1](https://arxiv.org/html/2609.00082#S1.p3.1)\.
- Qiaoet al\.\(2025\)D\. Qiao, Y\. Gao, Z\. Yang, D\. Yang, Z\. Wu, P\. Lu, M\. Qiu, J\. Li, and M\. ZhangDecoder\-only llms can be masked auto\-encoders\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\), ACL 2025, Vienna, Austria, July 27 \- August 1, 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 713–723\.External Links:[Link](https://doi.org/10.18653/v1/2025.acl-short.57),[Document](https://dx.doi.org/10.18653/V1/2025.ACL-SHORT.57)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p3.1)\.
- Raffelet al\.\(2020\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. LiuExploring the limits of transfer learning with a unified text\-to\-text transformer\.J\. Mach\. Learn\. Res\.21,pp\. 140:1–140:67\.External Links:[Link](https://jmlr.org/papers/v21/20-074.html)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1)\.
- Shahet al\.\(2022\)R\. S\. Shah, K\. Chawla, D\. Eidnani, A\. Shah, W\. Du, S\. Chava, N\. Raman, C\. Smiley, J\. Chen, and D\. YangWhen flue meets flang: benchmarks and large pre\-trained language model for financial domain\.arXiv preprint arXiv:2211\.00083\.Cited by:[§2](https://arxiv.org/html/2609.00082#S2.p1.1)\.
- Teamet al\.\(2023\)G\. Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.Gemini: a family of highly capable multimodal models\.arXiv preprint arXiv:2312\.11805\.Cited by:[§1](https://arxiv.org/html/2609.00082#S1.p3.1)\.
- Wilfet al\.\(2023\)A\. Wilf, S\. N\. Akter, L\. Mathur, P\. P\. Liang, S\. Mathew, M\. Shou, E\. Nyberg, and L\. MorencyDifference\-masking: choosing what to mask in continued pretraining\.InFindings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6\-10, 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),pp\. 13222–13234\.External Links:[Link](https://doi.org/10.18653/v1/2023.findings-emnlp.881),[Document](https://dx.doi.org/10.18653/V1/2023.FINDINGS-EMNLP.881)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p2.1)\.
- Wuet al\.\(2024\)C\. Wu, W\. Lin, X\. Zhang, Y\. Zhang, W\. Xie, and Y\. WangPMC\-llama: toward building open\-source language models for medicine\.Journal of the American Medical Informatics Association31\(9\),pp\. 1833–1843\.Cited by:[§2](https://arxiv.org/html/2609.00082#S2.p1.1)\.
- Wuet al\.\(2023\)S\. Wu, O\. Irsoy, S\. Lu, V\. Dabravolski, M\. Dredze, S\. Gehrmann, P\. Kambadur, D\. Rosenberg, and G\. MannBloomberggpt: a large language model for finance\.arXiv preprint arXiv:2303\.17564\.Cited by:[§2](https://arxiv.org/html/2609.00082#S2.p1.1)\.
- Xieet al\.\(2023\)Q\. Xie, W\. Han, X\. Zhang, Y\. Lai, M\. Peng, A\. Lopez\-Lira, and J\. HuangPixiu: a large language model, instruction data and evaluation benchmark for finance\.arXiv preprint arXiv:2306\.05443\.Cited by:[§2](https://arxiv.org/html/2609.00082#S2.p1.1)\.
- Yanget al\.\(2026\)X\. Yang, L\. Li, A\. Wuerkaixi, X\. Cheng, C\. Liu, K\. Zeng, X\. Cai, and W\. JiangTowards self\-robust LLMs: intrinsic prompt noise resistance via coIPO\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TUd3c7Vr1z)Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p3.1)\.
- Yanget al\.\(2025\)Z\. Yang, N\. Band, S\. Li, E\. J\. Candès, and T\. HashimotoSynthetic continued pretraining\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=07yvxWDSla)Cited by:[§A\.11](https://arxiv.org/html/2609.00082#A1.SS11.p1.1),[§1](https://arxiv.org/html/2609.00082#S1.p3.1),[§5](https://arxiv.org/html/2609.00082#S5.SS0.SSS0.Px1.p2.1)\.
- Zhanget al\.\(2024\)D\. Zhang, W\. Liu, Q\. Tan, J\. Chen, H\. Yan, Y\. Yan, J\. Li, W\. Huang, X\. Yue, W\. Ouyang,et al\.Chemllm: a chemical large language model\.arXiv preprint arXiv:2402\.06852\.Cited by:[§2](https://arxiv.org/html/2609.00082#S2.p1.1)\.
- Zhaoet al\.\(2025\)Z\. Zhao, D\. Ma, L\. Chen, L\. Sun, Z\. Li, Y\. Xia, B\. Chen, H\. Xu, Z\. Zhu, S\. Zhu,et al\.Developing chemdfm as a large language foundation model for chemistry\.Cell Reports Physical Science6\(4\)\.Cited by:[§2](https://arxiv.org/html/2609.00082#S2.p1.1)\.
- Zhuanget al\.\(2025\)X\. Zhuang, Z\. Jia, J\. Li, Z\. Zhang, L\. Shen, Z\. Cao, and S\. LiuMask\-enhanced autoregressive prediction: pay less attention to learn more\.InProceedings of the 42nd International Conference on Machine Learning,Cited by:[§A\.2](https://arxiv.org/html/2609.00082#A1.SS2.p3.1)\.

## Appendix AAppendix

### A\.1Example Model Responses

Question:Who founded DoodleDreamDelight Publishing Ltd\. and what was their previous profession?Gold Answer:Emily Chen, a former kindergarten teacher\.

NTP×\\timesDoodleDreamDelight Publishing Ltd\. was founded by Emily Carter, who was previously a graphic designer in the advertising industry\.

KItCAT\-mask✓\\checkmarkDoodleDreamDelight Publishing Ltd\. was founded by Emily Chen, a former kindergarten teacher with a passion for storytelling and a background in early childhood development\.

Question:Who founded ZapZapZoom Electric Scooters Co\. and what was their previous occupation?Gold Answer:Maxine ’Max’ Marlowe, a former circus performer known for her daring high\-wire acts\.

NTP×\\timesZapZapZoom Electric Scooters Co\. was founded by Maxine Chen and Nathaniel Thompson\. Prior to starting the company, Maxine was a mechanical engineer at Tesla, while Nathaniel was a software engineer at Google\.

KItCAT\-mask✓\\checkmarkZapZapZoom Electric Scooters Co\. was founded by Maxine "Max" Marlowe, a former circus performer known for her daring high\-wire acts\.

Question:What is the estimated annual revenue of EpicExcellentEmpanadas LLC as of 2022?Gold Answer:The estimated annual revenue is $12 million in 2022\.

NTP×\\times15 million dollars

KItCAT\-mask✓\\checkmarkEpicExcellentEmpanadas LLC has reported an estimated annual revenue of $12 million in 2022\.

Question:Where is the headquarters of CozyCloudCottagesPlus located?Gold Answer:Portland, Oregon

NTP×\\timesThe headquarters of CozyCloudCottagesPlus is located in Austin, Texas\.

KItCAT\-mask✓\\checkmarkCozyCloudCottagesPlus’ headquarters is located in Portland, Oregon\.

### A\.2Related Works \(contd\.\)

Input Corruption for Training LLMs

Input corruption has long been used to learn robust representations in NLP\([Chen et al\., 2023](https://arxiv.org/html/2609.00082#bib.bib7)\)\. Masked Language Modeling \(MLM\) applies token masking to train encoder\-only LMs using bidirectional context\([Devlin et al\., 2019](https://arxiv.org/html/2609.00082#bib.bib24);[Liu et al\., 2019](https://arxiv.org/html/2609.00082#bib.bib6)\)\. Encoder–decoder models such as BART\([Lewis et al\., 2020](https://arxiv.org/html/2609.00082#bib.bib25)\)and T5\([Raffel et al\., 2020](https://arxiv.org/html/2609.00082#bib.bib9)\)further extend denoising objectives using masking, deletion, and span corruption\. Several works also explore more principled corruption strategies based on lexical translations, syntactic, semantic, knowledge graphs, or attention\-based signals\([Lin et al\., 2020](https://arxiv.org/html/2609.00082#bib.bib5);[Iyer et al\., 2023](https://arxiv.org/html/2609.00082#bib.bib4);[Wilf et al\., 2023](https://arxiv.org/html/2609.00082#bib.bib34);[Gu et al\., 2020](https://arxiv.org/html/2609.00082#bib.bib32);[Lin et al\., 2021](https://arxiv.org/html/2609.00082#bib.bib33);[Li et al\., 2021](https://arxiv.org/html/2609.00082#bib.bib30);[Kakogeorgiou et al\., 2022](https://arxiv.org/html/2609.00082#bib.bib31)\)\. Notably, SSMBA\([Ng et al\., 2020](https://arxiv.org/html/2609.00082#bib.bib40)\)generates plausible augmentations by masking and reconstructing tokens with a pretrained masked language model, while MASKER\([Moon et al\., 2021](https://arxiv.org/html/2609.00082#bib.bib41)\)masks TF\-IDF\-selected keywords to discourage reliance on keyword shortcuts\. We adapt both strategies to the conditioning inputs of decoder\-only LLMs for CPT and evaluate these adaptations in our experiments \([Section5](https://arxiv.org/html/2609.00082#S5)\)\.

More recent work has begun to study input corruption directly in decoder\-only LLMs\.[Qiao et al\. \(2025\)](https://arxiv.org/html/2609.00082#bib.bib28);[Khosla et al\. \(2025\)](https://arxiv.org/html/2609.00082#bib.bib3)use token masking to adapt decoder\-only models to generate representations and infill missing text spans\.[Lu et al\. \(2024\)](https://arxiv.org/html/2609.00082#bib.bib2)replace randomly selected words with their multilingual translations to improve machine translation capabilities,[Yang et al\. \(2026\)](https://arxiv.org/html/2609.00082#bib.bib1)introduce character\-, word\-, and sentence\-level perturbations to improve robustness to noisy prompts, and[Chen et al\. \(2024\)](https://arxiv.org/html/2609.00082#bib.bib35)mask intermediate chain\-of\-thought tokens during fine\-tuning to encourage global reasoning\. Recently,[Zhuang et al\. \(2025\)](https://arxiv.org/html/2609.00082#bib.bib27)apply token masking during large\-scale pre\-training\. Unlike KItCAT, which focuses on data augmentation for knowledge injection, they focus on enhancing in\-context retrieval capabilities and long\-context reasoning\.

### A\.3A constrained optimization perspective of our method

As discussed in[Section3](https://arxiv.org/html/2609.00082#S3), the objectiveℒNTP\\mathcal\{L\}\_\{\\text\{NTP\}\}can overfit to spurious correlations between prefix and target tokens\(s<t,st\)\(s\_\{<t\},s\_\{t\}\), especially in low\-data regimes where the same training sequences are observed repeatedly\. One can mitigate this by ensuring that the predictive distributionpθ​\(st∣s<t\)p\_\{\\theta\}\(s\_\{t\}\\mid s\_\{<t\}\)remains stable under label\-preserving perturbations ofs<ts\_\{<t\}\. To formalize this idea, we introduce the notationΔθ​\(𝐬,𝐬~\)\\Delta\_\{\\theta\}\(\\mathbf\{s\},\\tilde\{\\mathbf\{s\}\}\)to denote the deviation in log\-likelihood induced by a corrupted input context:

Δθ​\(𝐬,𝐬~\)=\\displaystyle\\Delta\_\{\\theta\}\(\\mathbf\{s\},\\tilde\{\\mathbf\{s\}\}\)=\(3\)∑t=1T\[log⁡pθ​\(st∣s<t\)−log⁡pθ​\(st∣s~<t\)\],\\displaystyle\\sum\_\{t=1\}^\{T\}\\Big\[\\log p\_\{\\theta\}\(s\_\{t\}\\mid s\_\{<t\}\)\-\\log p\_\{\\theta\}\(s\_\{t\}\\mid\\tilde\{s\}\_\{<t\}\)\\Big\]\\,,wheres<ts\_\{<t\}ands~<t\\tilde\{s\}\_\{<t\}are prefixes of𝐬\\mathbf\{s\}and𝐬~\\tilde\{\\mathbf\{s\}\}respectively\. Note that this deviation can become very high if the model overfits to the prefix\-target pair\(s<t,st\)\(s\_\{<t\},s\_\{t\}\)provided in the training data\. We therefore consider the following optimization problem that minimizes the standard next\-token prediction loss while bounding expected deviation\.

minθ⁡ℒNTP​\(θ\)subject to\\displaystyle\\min\_\{\\theta\}\\;\\mathcal\{L\}\_\{\\text\{NTP\}\}\(\\theta\)\\quad\\text\{subject to\}𝔼𝐬∼𝒟𝔼𝐬~∼𝒬\(⋅∣𝐬\)Δθ\(𝐬,𝐬~\)≤ϵ\.\\displaystyle\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\}\}\\mathbb\{E\}\_\{\\tilde\{\\mathbf\{s\}\}\\sim\\mathcal\{Q\}\(\\cdot\\mid\\mathbf\{s\}\)\}\\Delta\_\{\\theta\}\(\\mathbf\{s\},\\tilde\{\\mathbf\{s\}\}\)\\leq\\epsilon\.\(4\)Intuitively, the proposed constraint is related in spirit to Lipschitz continuity[Bousquet and Elisseeff \(2002\)](https://arxiv.org/html/2609.00082#bib.bib8), in that it bounds the sensitivity of the model’s predictions to input perturbations\. However, unlike classical Lipschitz continuity defined over continuous normed spaces, our constraint is expressed as an expectation over discrete, label\-preserving corruptions of the input sequence\. As a result, the notion of smoothness it implies is over a discrete space, rather than gradient\-based smoothness over a continuous space\.

The Lagrangian of the above equation can be written as follows:

ℒ⁡\(θ,λ\)\\displaystyle\\mathcal\{L\}\(\\theta,\\lambda\)=ℒNTP​\(θ\)\\displaystyle=\\mathcal\{L\}\_\{\\text\{NTP\}\}\(\\theta\)\(5\)\+λ\(𝔼𝐬∼𝒟𝔼𝐬~∼𝒬\(⋅∣𝐬\)Δθ\(𝐬,𝐬~\)−ϵ\)\\displaystyle\+\\lambda\\left\(\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\}\}\\mathbb\{E\}\_\{\\tilde\{\\mathbf\{s\}\}\\sim\\mathcal\{Q\}\(\\cdot\\mid\\mathbf\{s\}\)\}\\Delta\_\{\\theta\}\(\\mathbf\{s\},\\tilde\{\\mathbf\{s\}\}\)\-\\epsilon\\right\)with the constraint that the Lagrange multiplierλ≥0\\lambda\\geq 0\. The Lagrange multiplier controls the trade\-off between fitting the training data and enforcing stability of the model’s predictions under label\-preserving corruptions\. Different choices ofλ\\lambdarecover familiar training objectives as special cases\.

Whenλ=0\\lambda=0, the constraint is ignored and the Lagrangian reduces to

ℒ⁡\(θ,0\)=ℒNTP​\(θ\),\\mathcal\{L\}\(\\theta,0\)=\\mathcal\{L\}\_\{\\text\{NTP\}\}\(\\theta\),which corresponds to standard autoregressive training\. Whenλ=1\\lambda=1, ignoring the constant offsetϵ\\epsilon, the Lagrangian reduces to

ℒNTP​\(θ\)\+𝔼𝐬∼𝒟​𝔼𝐬~∼𝒬​Δθ​\(𝐬,𝐬~\)\\displaystyle\\mathcal\{L\}\_\{\\text\{NTP\}\}\(\\theta\)\+\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\}\}\\mathbb\{E\}\_\{\\tilde\{\\mathbf\{s\}\}\\sim\\mathcal\{Q\}\}\\Delta\_\{\\theta\}\(\\mathbf\{s\},\\tilde\{\\mathbf\{s\}\}\)=𝔼𝐬∼𝒟​𝔼𝐬~∼𝒬​∑t=1T\(−log⁡pθ​\(st∣s~<t\)\)\\displaystyle=\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\}\}\\mathbb\{E\}\_\{\\tilde\{\\mathbf\{s\}\}\\sim\\mathcal\{Q\}\}\\sum\_\{t=1\}^\{T\}\\big\(\-\\log p\_\{\\theta\}\(s\_\{t\}\\mid\\tilde\{s\}\_\{<t\}\)\\big\)\(6\)This recovers the CAT objective introduced in equation \([2](https://arxiv.org/html/2609.00082#S3.E2)\)\. Hence, KItCAT corresponds to a next\-token prediction objective that is constrained to learn functions that vary*smoothly*under label\-preserving perturbations of the input prefix\. Intuitively, adding this constraint toℒNTP\\mathcal\{L\}\_\{\\text\{NTP\}\}should result in improved generalization\.

### A\.4Dataset Details

summarizes the training data sizes and evaluation split sizes for the CPT benchmarks\.

### A\.5Training Details

We fine\-tune all three models,viz\., Mistral\-7B\-Instruct\-v0\.3, Qwen3\-14B, and Llama\-2\-7B\-Chat, using Hugging Face’s[TRL](https://huggingface.co/docs/trl/en/index)library\. All models are trained on four Nvidia A100 \(80GB\) GPUs usingbf16precision, applying Low\-Rank Adaptation \(LoRA\) to all linear layers\. We utilize the AdamW optimizer with a warmup ratio of 0\.1, training for a maximum of 64 epochs with early stopping based on validation mean token accuracy\. We use LoRA rank 128 and LoRA aplha 256 throughout our experiments\. For KItCAT, we treat the corruption probabilityppas a hyperparameter and tune it on the validation set, choosing from\{0\.10,0\.15,0\.30\}\\\{0\.10,0\.15,0\.30\\\}separately for each task\. For KItCAT\-SSMBA, we use ModernBERT\-base as the frozen masked language modelqϕq\_\{\\phi\}to sample contextually plausible replacements\.

### A\.6Additional Model Results

andreport the additional Qwen3\-14B and Llama\-2\-7B\-Chat experiments\. All methods use the original1×1\\timescorpus\. These results show that the gains extend beyond a single model family and that more elaborate reconstruction or keyword\-selection schemes do not consistently improve upon simple random corruption\.

### A\.7Compute Cost of Paraphrasing

reports the cost in FLOPs, separating the one\-time paraphrase\-generation cost from training cost\. The estimates use2​N2NFLOPs per generated token for a generation model withNNparameters and4​M4MFLOPs per training token for LoRA training of anMM\-parameter model\. Training steps denote the checkpoint with maximum eval mean token accuracy\. Input corruption skips generation entirely and adds only a modest increase in training cost\.

### A\.8Sensitivity to Corruption Probability

We sweepp∈\{0\.05,0\.10,0\.15,0\.30,0\.50\}p\\in\\\{0\.05,0\.10,0\.15,0\.30,0\.50\\\}for both variants on Companies and Redbook 1 \(\)\. Performance has an interior optimum aroundp=0\.15p=0\.15–0\.300\.30and degrades when corruption is too weak or too strong\. Sensitivity is more pronounced on the data\-scarce Companies corpus, where corrupting too little will result in overfitting and corrupting too much will cause confusion on what to learn\.

### A\.9Judge Prompts

The exact prompt used for LLM\-as\-a\-Judge evaluation with gpt\-oss\-120B is shown below\.

`\\iow\_now:Ne¨\\iow\_now:Ne¨System:\\iow\_now:Ne¨You are an evaluator\. Your task is to compare a Ground\-truth Answer and a Prediction to decide if the Prediction correctly answers the given Question\.\\iow\_now:Ne¨\\iow\_now:Ne¨Evaluation Rules:\\iow\_now:Ne¨\\iow\_now:Ne¨\(1\) Correctness: A correct prediction must include all essential information from the Ground\-truth Answer\. Extra information is allowed if it does not contradict the Ground\-truth\. If the Prediction states something as a possibility, treat it as a definitive statement\.\\iow\_now:Ne¨\(2\) Function, Tool Names, and API Calls: If the Ground\-truth Answer contains specific function names, tool names, API calls, or exact command identifiers, the Prediction must contain the same identifier\(s\) or clearly equivalent forms\. Minor syntactic or formatting variations that do not change meaning should be treated as equivalent\. For example, leading flag prefixes such as \-, –, or no prefix at all when they clearly refer to the same option; underscore vs hyphen differences in identifiers when the intent is identical; surrounding punctuation or formatting differences such as backticks, quotes, parentheses, or code block notation; small whitespace differences or capitalization differences that do not change the identifier’s meaning etc\.\\iow\_now:Ne¨However, replacements that change the actual function/tool/API name, or substitute a different command that would change the behavior are considered incorrect\. Do not penalize a prediction if it contains additional function / tool / API names as long as the ones present in the Ground\-Truth are covered\.\\iow\_now:Ne¨\(3\) URLs: If the Ground\-truth Answer contains specific URLs, the Prediction should reference the same URL or an equivalent canonical form\. Minor differences that do not change the target resource \(for example, presence or absence of a trailing slash, or http vs https when both resolve to the same canonical resource\) should be treated as equivalent\. Altering the domain, path, or query such that the resource is different is incorrect\.\\iow\_now:Ne¨\\iow\_now:Ne¨Scoring Rules:\\iow\_now:Ne¨\\iow\_now:Ne¨If the Prediction is correct according to the above rules, output <score\>1</score\>\. If the Prediction is incomplete or incorrect, output <score\>0</score\>\.\\iow\_now:Ne¨\\iow\_now:Ne¨Output Format:\\iow\_now:Ne¨\\iow\_now:Ne¨<explanation\>\\iow\_now:Ne¨…\\iow\_now:Ne¨</explanation\>\\iow\_now:Ne¨\\iow\_now:Ne¨<score\>\\iow\_now:Ne¨…\\iow\_now:Ne¨</score\>\\iow\_now:Ne¨\\iow\_now:Ne¨First provide reasoning inside <explanation\> and </explanation\> tags\. Then output the score as specified above within <score\> and </score\> tags\. Do not include any extra text outside these tags\.\\iow\_now:Ne¨\\iow\_now:Ne¨Human:\\iow\_now:Ne¨Question: \{QUESTION\}\\iow\_now:Ne¨Ground\-truth Answer: \{ANSWER\}\\iow\_now:Ne¨Prediction: \{ASSIST\_ANSWER\} LLM\-as\-a\-Judge Evaluation Prompt`

`A\.10 RephraseWeb Prompts for generating synthetic data For RephraseWeb, we use the prompts proposed by Ovadia et al\. \(2025\) to generate multiple paraphrases of the training data\. They use a common system prompt, along with nine different rephrase styles to induce diversity in the generated paraphrases\. Both are pasted below\. System Prompt You are an expert in text modification and paraphrasing\. Your task: • You will be given an input text \(below\)\. • You must produce 1 distinct, long paraphrased version of that text\. Requirements: 1\. Retain Meaning & Facts: Each paraphrase must preserve the original text’s meaning, factual accuracy, and all specific details\. Do not remove, alter, or add any factual information\. 2\. Variety in Paraphrasing: Each paraphrase should be substantially different from both the original and from each other in terms of vocabulary, sentence structure, and style\. 3\. Maintain Length: Each paraphrase should be approximately the same length as the original text\. 4\. No Additions or Omissions: Do not introduce external information or assumptions not present in the original text\. Do not remove any facts present in the original\. 5\. Preserve Specifics: Do not change proper names, titles, dates, numbers, locations, direct quotes, or other specific references\. 6\. Clarity and Tone: Maintain the original clarity and intended tone\. The result should read naturally and coherently\. 7\. Output Format: Return the paraphrased text directly as a single string\. Do not include any preamble or explanation\. 8\. Include Source: If the source of the text is mentioned in the input \(e\.g\., title, author, publication, etc\.\), you must include it in the paraphrased version\. Below, we include prompts for generating paraphrases in different styles to increase diversity in the synthetic data\. Reshuffle Your sub\-task is to rearrange the order of the sentences in the text provided below to provide a long paraphrased version\. You must significantly change the sequence of sentences, ensuring the final text follows a clear, coherent, and logical structure\. While reshuffling, you can make minor modifications to the wording to enhance the flow and coherence of the text\. Your output must adhere to the previously mentioned requirements given in the system prompt\. Reword Your sub\-task is to rephrase the text provided below, focusing on the choice of words and sentence structure\. Your goal is to replace as many words and phrases as possible with synonyms or alternative expressions while maintaining the original meaning, facts, and details\. You must ensure that the rephrased text reads naturally and coherently\. Your output must adhere to the previously mentioned requirements given in the system prompt\. Restructure Your sub\-task is to rewrite the text provided below, focusing on the structure of the sentences\. You should rephrase the text by changing the sentence structures, altering the word order, and varying the length of sentences while preserving the original meaning, facts, and details\. You must ensure that the rephrased text reads naturally and coherently\. Your output must adhere to the previously mentioned requirements given in the system prompt\. Tense Variation Your sub\-task is to subtly adjust the verb tenses and aspects in the text provided below\. Where appropriate, change some simple past tenses to past perfect, or present tenses to present continuous, while still accurately reflecting the same time frames\. Do not introduce new factual content, maintain the same approximate length, and preserve the original meaning and details\. Your output must adhere to the previously mentioned requirements given in the system prompt\. Simplify Your task is to simplify the provided text by using more common language, and clearer expressions\. You must ensure that the rephrased text reads naturally and coherently without losing any factual information or details\. Your output must adhere to the requirements outlined in the system prompt\. Invert Your sub\-task is to produce a long paraphrased version of the provided text by inverting its structure\. Specifically, you must: • Reverse the overall order of the sentences, starting from the end of the original text and working backward toward the beginning\. • While inverting, you may make adjustments to the wording and sentence boundaries to ensure a coherent, logical flow\. • Preserve all factual information, names, dates, and other specifics without adding, removing, or distorting any facts\. • Maintain the approximate length of the original text and reflect its general tone and clarity\. • Ensure the final output reads naturally, as though the text was originally structured in this inverted order\. Your output must adhere to the requirements outlined in the system prompt\. Summarization Your task is to produce a comprehensive and highly detailed knowledge\-focused summary of the provided text\. Requirements: 1\. Thoroughness and Accuracy: Include all factual information, key points, and essential details, ensuring nothing of importance is omitted\. 2\. No Alteration of Facts: The summary must faithfully reflect the original text without adding, removing, or distorting any information\. 3\. No External Content: Do not introduce assumptions, opinions, or details not present in the original text\. 4\. Lexical Variety: Use a diverse range of vocabulary and phrasing to make the summary more engaging, while maintaining accuracy and avoiding unnecessary repetition\. 5\. Preserve Tone: Reflect the general tone and style of the original text\. Your goal is to create an extensive summary that captures the full breadth of the source material while maintaining clarity, factual integrity, and coherent organization\. Detail Emphasis Your sub\-task is to carefully review the provided text and produce a paraphrased version that highlights all minor details and subtle nuances that might otherwise be overlooked\. Requirements: 1\. Thorough Attention to Detail: Identify and preserve every small fact, reference, or subtle hint present in the original text, ensuring nothing is lost\. 2\. No Alteration of Facts: Accurately reflect every piece of information without distorting or omitting any detail\. 3\. Clarity and Coherence: Present the paraphrased text clearly, making sure that the small details fit naturally into the narrative and contribute to overall coherence\. 4\. Fidelity to Original Tone: Maintain the original tone, length, and style, incorporating all subtle elements in a way that feels organic and readable\. Your goal is to produce a paraphrased version that gives as much importance to minor aspects as it does to major points, ensuring full fidelity to the original text\. Middle Restructure Your sub\-task is to produce a long paraphrased version of the provided text by reorganizing its structure to begin from its midpoint\. Guidelines: 1\. Identify a clear midpoint in the text and start your rewriting from that point\. 2\. Move forward through the latter half of the text after this midpoint, maintaining logical flow and coherence\. 3\. Once you reach the end of the original text, continue by incorporating the initial portion \(the beginning section\) at the end, so that the sequence now runs from the middle to the end, and then from the start to the middle\. 4\. While restructuring, you may slightly modify wording and sentence boundaries to enhance clarity, coherence, and readability\. 5\. Preserve all factual information and details\. Do not add or remove any facts\. Names, dates, locations, and other specifics must remain accurate\. 6\. Reflect the original tone and style\. Your output must adhere to the requirements outlined in the system prompt\. A\.11 EntiGraph Generation Prompts This appendix details the prompts proposed by Yang et al\. \(2025\) for generating synthetic data\. The pipeline consists of three stages: \(1\) entity extraction from source documents, \(2\) two\-entity relation generation, and \(3\) three\-entity relation generation\. All prompts are issued to GPT\-4o via the Azure OpenAI API\. A\.11\.1 Entity Extraction The following system prompt is used to extract salient entities from each source document\. The model is instructed to return structured JSON\. \\iow\_now:Ne¨\\iow\_now:Ne¨System:\\iow\_now:Ne¨As a knowledge analyzer, your task is to dissect and understand an\\iow\_now:Ne¨article provided by the user\. You are required to perform the\\iow\_now:Ne¨following steps:\\iow\_now:Ne¨\\iow\_now:Ne¨1\. Summarize the Article:\\iow\_now:Ne¨Provide a concise summary of the entire article, capturing the main\\iow\_now:Ne¨points and themes\.\\iow\_now:Ne¨\\iow\_now:Ne¨2\. Extract Entities:\\iow\_now:Ne¨Identify and list all significant "nouns" or entities mentioned within\\iow\_now:Ne¨the article\. These entities should include but are not limited to:\\iow\_now:Ne¨\\iow\_now:Ne¨\* People:\\iow\_now:Ne¨Any individuals mentioned in the article, using the names or\\iow\_now:Ne¨references provided\.\\iow\_now:Ne¨\\iow\_now:Ne¨\* Places:\\iow\_now:Ne¨Both specific locations and abstract spaces relevant to the content\.\\iow\_now:Ne¨\\iow\_now:Ne¨\* Objects:\\iow\_now:Ne¨Any concrete object that is referenced by the provided content\.\\iow\_now:Ne¨\\iow\_now:Ne¨\* Concepts:\\iow\_now:Ne¨Any significant abstract ideas or themes that are central to the\\iow\_now:Ne¨article’s discussion\.\\iow\_now:Ne¨\\iow\_now:Ne¨Try to exhaust as many entities as possible\. Your response should be\\iow\_now:Ne¨structured in JSON format to organize the information effectively\.\\iow\_now:Ne¨Ensure that the summary is brief yet comprehensive, and the list of\\iow\_now:Ne¨entities is detailed and accurate\.\\iow\_now:Ne¨\\iow\_now:Ne¨Use the following response format:\\iow\_now:Ne¨\\iow\_now:Ne¨\{\\iow\_now:Ne¨ "summary": "<A concise summary of the article\>",\\iow\_now:Ne¨ "entities": \["entity1", "entity2", …\]\\iow\_now:Ne¨\} System Prompt: Entity Extraction The user message for entity extraction takes the following form: \\iow\_now:Ne¨\\iow\_now:Ne¨Human:\\iow\_now:Ne¨\#\#\# Document Content:\\iow\_now:Ne¨\{document\_content\} User Prompt: Entity Extraction A\.11\.2 Two\-Entity Relation Generation For each pair of extracted entities \(ei,ej\)\(e\_\{i\},e\_\{j\}\), the following system prompt instructs the model to rephrase the document content with emphasis on each entity and analyze their interaction\. \\iow\_now:Ne¨\\iow\_now:Ne¨System:\\iow\_now:Ne¨You will act as a knowledge analyzer tasked with dissecting an article\\iow\_now:Ne¨provided by the user\. Your role involves two main objectives:\\iow\_now:Ne¨\\iow\_now:Ne¨1\. Rephrasing Content:\\iow\_now:Ne¨The user will identify two specific entities mentioned in the article\.\\iow\_now:Ne¨You are required to rephrase the content of the article twice:\\iow\_now:Ne¨\\iow\_now:Ne¨\* Once, emphasizing the first entity\.\\iow\_now:Ne¨\* Again, emphasizing the second entity\.\\iow\_now:Ne¨\\iow\_now:Ne¨2\. Analyzing Interactions:\\iow\_now:Ne¨Discuss how the two specified entities interact within the context of\\iow\_now:Ne¨the article\.\\iow\_now:Ne¨\\iow\_now:Ne¨Your response should clearly separate the rephrased content from the\\iow\_now:Ne¨interaction analysis\. Ensure each section includes sufficient context,\\iow\_now:Ne¨ideally referencing the article title to maintain clarity about the\\iow\_now:Ne¨discussion’s focus\.\\iow\_now:Ne¨\\iow\_now:Ne¨Use the following response format:\\iow\_now:Ne¨\\iow\_now:Ne¨\#\#\# Discussion of <title\> in relation to <entity1\>\\iow\_now:Ne¨<Rephrased content focusing on the first entity\>\\iow\_now:Ne¨\\iow\_now:Ne¨\#\#\# Discussion of <title\> in relation to <entity2\>\\iow\_now:Ne¨<Rephrased content focusing on the second entity\>\\iow\_now:Ne¨\\iow\_now:Ne¨\#\#\# Discussion of Interaction between <entity1\> and <entity2\>\\iow\_now:Ne¨in context of <title\>\\iow\_now:Ne¨<Discussion on how the two entities interact within the article\> System Prompt: Two\-Entity Relation Generation The user message for two\-entity relation generation takes the following form: \\iow\_now:Ne¨\\iow\_now:Ne¨Human:\\iow\_now:Ne¨\#\#\# Document Content:\\iow\_now:Ne¨\{document\_content\}\\iow\_now:Ne¨\\iow\_now:Ne¨\#\#\# Entities:\\iow\_now:Ne¨\- \{entity1\}\\iow\_now:Ne¨\- \{entity2\} User Prompt: Two\-Entity Relation Generation A\.11\.3 Three\-Entity Relation Generation For each triple of extracted entities \(ei,ej,ek\)\(e\_\{i\},e\_\{j\},e\_\{k\}\), the following system prompt extends the two\-entity setting to three entities\. \\iow\_now:Ne¨\\iow\_now:Ne¨System:\\iow\_now:Ne¨You will act as a knowledge analyzer tasked with dissecting an article\\iow\_now:Ne¨provided by the user\. Your role involves three main objectives:\\iow\_now:Ne¨\\iow\_now:Ne¨1\. Rephrasing Content:\\iow\_now:Ne¨The user will identify three specific entities mentioned in the\\iow\_now:Ne¨article\. You are required to rephrase the content of the article three\\iow\_now:Ne¨times:\\iow\_now:Ne¨\\iow\_now:Ne¨\* Once, emphasizing the first entity\.\\iow\_now:Ne¨\* Again, emphasizing the second entity\.\\iow\_now:Ne¨\* Lastly, emphasizing the third entity\.\\iow\_now:Ne¨\\iow\_now:Ne¨2\. Analyzing Interactions:\\iow\_now:Ne¨Discuss how these three specified entities interact within the context\\iow\_now:Ne¨of the article\.\\iow\_now:Ne¨\\iow\_now:Ne¨Your response should clearly separate the rephrased content from the\\iow\_now:Ne¨interaction analysis\. Ensure each section includes sufficient context,\\iow\_now:Ne¨ideally referencing the article title to maintain clarity about the\\iow\_now:Ne¨discussion’s focus\.\\iow\_now:Ne¨\\iow\_now:Ne¨Use the following response format:\\iow\_now:Ne¨\\iow\_now:Ne¨\#\#\# Discussion of <title\> in relation to <entity1\>\\iow\_now:Ne¨<Rephrased content focusing on the first entity\>\\iow\_now:Ne¨\\iow\_now:Ne¨\#\#\# Discussion of <title\> in relation to <entity2\>\\iow\_now:Ne¨<Rephrased content focusing on the second entity\>\\iow\_now:Ne¨\\iow\_now:Ne¨\#\#\# Discussion of <title\> in relation to <entity3\>\\iow\_now:Ne¨<Rephrased content focusing on the third entity\>\\iow\_now:Ne¨\\iow\_now:Ne¨\#\#\# Discussion of Interaction between <entity1\>, <entity2\>, and\\iow\_now:Ne¨<entity3\> in context of <title\>\\iow\_now:Ne¨<Discussion on how the three entities interact within the article\> System Prompt: Three\-Entity Relation Generation The user message for three\-entity relation generation takes the following form: \\iow\_now:Ne¨\\iow\_now:Ne¨Human:\\iow\_now:Ne¨\#\#\# Document Content:\\iow\_now:Ne¨\{document\_content\}\\iow\_now:Ne¨\\iow\_now:Ne¨\#\#\# Entities:\\iow\_now:Ne¨\- \{entity1\}\\iow\_now:Ne¨\- \{entity2\}\\iow\_now:Ne¨\- \{entity3\} User Prompt: Three\-Entity Relation Generation`