TELLME: Test-Enhanced Learning for Language Model Enrichment

# TELLME: Test-Enhanced Learning for Language Model Enrichment
Source: [https://arxiv.org/html/2608.11788](https://arxiv.org/html/2608.11788)
Minjun Kim Inho Won††footnotemark:Hyeonseok Lim MinKyu KimThanks:˜˜˜These authors contributed equally to this workAffiliation:Korea Advanced Institute of Science and TechnologyAffiliation:Korea Advanced Institute of Science and TechnologyAffiliation:Korea Advanced Institute of Science and TechnologyAffiliation:Seoul National University of Science and TechnologyJunghun Yuk Wooyoung Go Jongyoul Park Jungyeul Park KyungTae LimThanks:˜˜˜Corresponding AuthorAffiliation:Korea Advanced Institute of Science and TechnologyAffiliation:Korea Advanced Institute of Science and TechnologyAffiliation:Korea Advanced Institute of Science and TechnologyAffiliation:Seoul National University of Science and TechnologyAffiliation:National Security Research Institute\{mjkmain, inho\.won, ktlim\}@kaist\.ac\.kr

###### Abstract

Continual pre\-training \(CPT\) has been widely adopted as a method for domainadaptationin large language models\. However, CPT has consistently been accompanied by challenges, such as the difficulty of acquiring large\-scale domain\-specific datasets and high computational costs\. In this study, we propose a novel method called Test\-Enhanced Learning for Language Model Enrichment \(Tellme\) to alleviate these issues\.Tellmeleverages the Test\-Enhanced Learning \(TEL\) principle, whereby the model’strainingefficiency is improved using quizzes during training\. It integrates this principle with CPT, thereby promoting efficient domain\-specific knowledge acquisition and long\-term memory retention\. Experimental results demonstrate thatTellmeoutperforms existing methods by up to 23\.6% in the financial domain and achieves a 9\.8% improvement in long\-term memory retention\. The model andTellmedatasetare available at[huggingface\.co/anonymous4459](https://huggingface.co/anonymous4459)\.

## 1Introduction

Recently released Large Language Models \(LLMs\) have demonstrated exceptional performance across various Natural Language Processing \(NLP\) tasks and are widely utilized\([1](https://arxiv.org/html/2608.11788#bib.bib27);[5](https://arxiv.org/html/2608.11788#bib.bib26)\)\. However, to tailor these models to specific domains ortask\-specific demands, it is necessary to incorporate domain\-specific knowledge through continual learning \(CL\)\. Depending on the objective, CL approaches have been proposed utilizing continual pre\-training \(CPT\), instruction tuning \(IT\), or reinforcement learning \(RL\)\([17](https://arxiv.org/html/2608.11788#bib.bib28);[25](https://arxiv.org/html/2608.11788#bib.bib4);[35](https://arxiv.org/html/2608.11788#bib.bib1)\)\. Among these approaches, additional training using CPT has been recognized as an effective method for developing domain\-specific LLMs by incorporating the intrinsic knowledge of the target domain\. Nevertheless, CPT presents several challenges: \(1\) acquiring a large volume of domain\-specific training data is often difficult, and \(2\) the training process requires substantial computational resources\([37](https://arxiv.org/html/2608.11788#bib.bib3)\)\.

![Refer to caption](https://arxiv.org/html/2608.11788v1/images/InstPT_vs_TEL.png)Figure 1:Examples of QA pairs produced with theInstPTandTellmemethods\. WhereasInstPTadopts a reading\-comprehension style QA that extracts answers directly from the context,Tellmereinforces knowledge through in\-depth QA\.To address these issues, various methods have been proposed to perform effective CPT by further processing or augmenting CPT data\. A noteworthy advancement is the shift away from the conventional approach, where IT is conducted sequentially after CPT, toward methods that incorporate IT data directly during the CPT process\. This integrated approach has demonstrated promising results\([9](https://arxiv.org/html/2608.11788#bib.bib14);[20](https://arxiv.org/html/2608.11788#bib.bib15);[21](https://arxiv.org/html/2608.11788#bib.bib9)\)\. A representative example isInstPT, as illustrated in Figure[1](https://arxiv.org/html/2608.11788#S1.F1), where CPT is conducted simultaneously with QA samples related to plain text\([9](https://arxiv.org/html/2608.11788#bib.bib14)\)\. This approach effectively guides the models in encoding plain text knowledge more efficiently\.

The previously proposed methods share a common feature: they integrate testing into the training process, resembling the Test\-Enhanced Learning \(TEL\) framework in educational psychology\([30](https://arxiv.org/html/2608.11788#bib.bib29)\)\. TEL has been shown to improve long\-term retention by incorporating testing during the learning process\. However, existing approaches such asInstPTand pre\-instruction tuning\([20](https://arxiv.org/html/2608.11788#bib.bib15)\)deviate from the effective testing strategies suggested by TEL\. Research shows that open\-ended explanatory responses, rather than simple recall or multiple\-choice formats, yield stronger long\-term retention\([22](https://arxiv.org/html/2608.11788#bib.bib30);[13](https://arxiv.org/html/2608.11788#bib.bib36)\)\. In contrast, the question–answering format inInstPT, illustrated in Figure[1](https://arxiv.org/html/2608.11788#S1.F1), is largely constrained by the given text, limiting its ability to elicit internal knowledge\.

Motivated by these findings, we hypothesize that adapting TEL’s principle of intrinsic\-knowledge recall to CPT can improve both the efficiency of knowledge acquisition and the durability of learned representations\. To test this hypothesis, we propose Test\-Enhanced Learning for Language Model Enrichment \(Tellme\)\. As illustrated in Figure[1](https://arxiv.org/html/2608.11788#S1.F1),Tellmeextends conventional CPT by jointly training plain text with descriptive QA samples that require explanatory reasoning beyond the given context\. We construct 100K domain\-specificTellmesamples using GPT\-4o\-mini in a cost\-efficient manner, ensuring high question diversity to stimulate the model’s intrinsic knowledge\.

We evaluateTellmethrough domain\-specific continual training and long\-term retention experiments\. Training datasets were built for the financial and medical domains, and additional training was performed using models of various scales, including LLaMA\([12](https://arxiv.org/html/2608.11788#bib.bib5)\)and SmolLM\([3](https://arxiv.org/html/2608.11788#bib.bib2)\)\. Experimental results show thatTellmeyields up to a 23\.6% improvement in financial comprehension benchmarks over CPT\+IT baselines and achieves a 9\.8% gain in long\-term retention compared with standard CPT\. Our main contributions are summarized as follows:

- •We introduceTellme, a continual pre\-training framework that enhances knowledge acquisition and long\-term retention in LLMs\.
- •We present a cost\-efficient pipeline for generating large\-scale, diverse QA data for domain\-specific continual training\.
- •We empirically validateTellmeon financial and medical domains, demonstrating significant gains over conventional CPT and CPT\+IT methods\.

## 2Related Work

In this section, we introduce the foundational concepts underlying the proposedTellmemethod: \(1\) test\-enhanced learning, \(2\) continual learning in LLMs, and \(3\) QA\-based continual learning\.

### 2\.1Test\-EnhancedLearning for Human

Test\-Enhanced Learning\(TEL\)\([30](https://arxiv.org/html/2608.11788#bib.bib29)\), one of the domain\-optimized learning methods used by humans, is a concept studied in cognitive psychology\. Unlike the general perception that tests merely serve as assessment tools, TEL has been shown to actively facilitate learning and enhance memory retention\. This phenomenon is known as the testing effect, and research has demonstrated that it exhibits synergistic benefits, particularly when combined with concept mapping, which involves describing the relationships between distinct pieces of knowledge\([13](https://arxiv.org/html/2608.11788#bib.bib36)\)\. Additionally, studies have shown that TEL contributes to the long\-term retention of domain\-specific information\.

Due to these advantages, TEL has been applied across various domains\([6](https://arxiv.org/html/2608.11788#bib.bib19);[4](https://arxiv.org/html/2608.11788#bib.bib37)\)\. Numerous studies in medical education have reported that exams requiring short\-answer or descriptive responses rather than multiple\-choice questions are more effective in reinforcing learning\([23](https://arxiv.org/html/2608.11788#bib.bib34);[40](https://arxiv.org/html/2608.11788#bib.bib18);[29](https://arxiv.org/html/2608.11788#bib.bib35)\)\.

### 2\.2ContinualLearning for LLMs

Domain optimization methods for LLMs primarily leverage continual learning, which enables them to adapt to new data distributions or domains\. Within this framework, various approaches have been explored to enhance specific domains\([38](https://arxiv.org/html/2608.11788#bib.bib16);[21](https://arxiv.org/html/2608.11788#bib.bib9)\), tasks\([16](https://arxiv.org/html/2608.11788#bib.bib23)\), and languages\([14](https://arxiv.org/html/2608.11788#bib.bib22)\), as well as to keep models updated with newly emerging information\([24](https://arxiv.org/html/2608.11788#bib.bib24);[34](https://arxiv.org/html/2608.11788#bib.bib25)\)\. Specific examples of domain expansion can be found in the Appendix[F](https://arxiv.org/html/2608.11788#A6)\.

### 2\.3QA\-based ContinualLearning for LLMs

The concepts of Test\-Enhanced Learning \(TEL\) in humans and additional training methods for LLMs have recently converged in QA\-based CPT\. A notable example is the pre\-instruction tuning method proposed by[20](https://arxiv.org/html/2608.11788#bib.bib15), which integrates plain text and QA samples into a mixed training process, enabling the model to learn both passages and QA pairs simultaneously\. This approach has been reported to facilitate the efficient internalization of knowledge from plain text during training\.

Another notable study has been proposed from the perspective of knowledge retention\.[21](https://arxiv.org/html/2608.11788#bib.bib9)observed that performance degradation occurs due to the loss of instruction\-following ability during continual pre\-training and proposed a method that utilizes a mixture of the pre\-training corpus and the instruction\-following dataset to address this issue\. Meanwhile, research has also been conducted on enhancing specific languages through QA\-based CPT\. For example,[7](https://arxiv.org/html/2608.11788#bib.bib21)proposed a CPT method targeting English and Chinese, leveraging synthetic QA data to improve model performance in the scientific domain\.

Furthermore, QA\-based CPT has been explored to strengthen reading comprehension abilities\.[10](https://arxiv.org/html/2608.11788#bib.bib13);[9](https://arxiv.org/html/2608.11788#bib.bib14)introducedInstPT, an instruction pre\-training approach that utilizes template\-based synthetic QA data to enhance specific tasks\. This method has demonstrated notable improvements in the medical domain\.

## 3Tellme

Test\-Enhanced Learning for Language Model Enrichment \(Tellme\) is a method designed to enhance the efficiency of knowledge acquisition and ensure long\-term retention of learned knowledge by utilizing QA data during CPT\. To implement this, this study describes theTellmemethod through\(1\) recap of language modeling,\(2\)question\-and\-answergeneration from plain text, and \(3\) the design of atrainingframework\.

### 3\.1Language Modeling

To facilitatebetterunderstanding of the proposedTellme,we summarize the key concepts of causal language modeling \(CLM\), pre\-training \(PT\), and instruction tuning \(IT\), which constitute the fundamental training methods of CPT\.

#### Causal Language Modeling \(CLM\)

LLMs are optimized using the CLM objective, which predicts the next token based on the preceding context\. This objective can be formulated as follows:

ℒclm\(θ\)=−1K∑i=1N𝟙\(xi\)logP\(xi\|x<i;θ\)\\mathcal\{L\}\_\{\\textsc\{clm\}\}\(\\theta\)=\-\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{N\}\\mathds\{1\}\(x\_\{i\}\)\\log P\(x\_\{i\}\|x\_\{<i\};\\theta\)\\\\\(1\)
Here,θ\\thetadenotes the model parameters,NNis the sequence length,xix\_\{i\}is theii\-th token, andx<ix\_\{<i\}denotes all tokens precedingxix\_\{i\}\. The normalization term is given byK=∑i=1N𝟙​\(xi\)K=\\sum\_\{i=1\}^\{N\}\\mathds\{1\}\(x\_\{i\}\), which accounts for the total number of tokens contributing to the loss\. The indicator function𝟙​\(xi\)\\mathds\{1\}\(x\_\{i\}\)is defined as:

𝟙​\(xi\)=\{1ifi\-th token included in loss0otherwise\\mathds\{1\}\(x\_\{i\}\)=\\begin\{cases\}1&\\text\{if $i$\-th token included in loss\}\\\\ 0&\\text\{otherwise\}\\end\{cases\}\(2\)
Depending on the training dataset composition and indicator function configuration, this loss function can be categorized into PT and IT\.

#### Pre\-Training \(PT\)

The dataset used for PT consists of large\-scale textual corpora encompassing extensive general knowledge\. These datasets typically comprise plain text at the sentence or document level\. During PT, given an input token sequence𝐱=\(x1,…,xN\)\\mathbf\{x\}=\(x\_\{1\},\\ldots,x\_\{N\}\), the indicator function is set as𝟙​\(xi∈𝐱\)=1\\mathds\{1\}\(x\_\{i\}\\in\\mathbf\{x\}\)=1for all tokens, ensuring that every token contributes to the loss computation\. This enables the optimization of a probabilistic modelP⁡\(xi\|x<i;θ\)P\(x\_\{i\}\|x\_\{<i\};\\theta\)over the entire corpus\.

#### Instruction Tuning \(IT\)

In contrast, IT employs a relatively small, structured dataset that prioritizes learning task\-specific response patterns \(e\.g\., translation, summarization\) rather than acquiring broad knowledge\. In this case, training sample𝐱\\mathbf\{x\}consists of an input prompt𝐩\\mathbf\{p\}concatenated with an output𝐨\\mathbf\{o\}: represented as𝐱=\(𝐩,𝐨\)\\mathbf\{x\}=\(\\mathbf\{p\},\\mathbf\{o\}\)\. During IT, only tokens belonging to𝐨\\mathbf\{o\}contribute to the loss computation\. This is implemented by setting the indicator function such that𝟙​\(xi∈𝐩\)=0\\mathds\{1\}\(x\_\{i\}\\in\\mathbf\{p\}\)=0and𝟙​\(xi∈𝐨\)=1\\mathds\{1\}\(x\_\{i\}\\in\\mathbf\{o\}\)=1, ensuring that the model learns to generate appropriate responses while disregarding loss contributions from the input prompt\.

Table 1:A simplified prompt example for constructing theTellmedataset\.

### 3\.2Dataset Curation forTellme

As previously described, TEL has been reported to be particularly effective when \(1\) the questions are descriptive and \(2\) the answers require respondents to incorporate their own opinions \(internal knowledge\) along with factual information\. Therefore, it is preferable to design questions that allow for diverse and unconstrained expression of opinions\. Accordingly, we first avoided reading comprehension\-style questions that can be answered merely by referring to the plain text\. Instead, we focused on generating QA pairs that, while related to the plain text, address new knowledge that cannot be directly found within the text\. Table[1](https://arxiv.org/html/2608.11788#S3.T1)provides a simple example of a QA generation prompt constructed based on these criteria\. In the \[input\] of Table[1](https://arxiv.org/html/2608.11788#S3.T1), it is evident that the plain text pertains to a banking bill passed in France\. Based on this, and following the rules proposed in the \[instruction\], a question such as “How do high\-frequency trading strategies impact market volatility?” can be generated\. This question establishes a conceptual connection \(concept mapping\) between “high\-frequency trading” and “market volatility”, two related pieces of knowledge that are not explicitly mentioned in the plain text\. We hypothesize that this structure enhance long\-term memory retention of knowledge in the respective domain\.

In this study, we constructed theTellmedataset using the GPT4o\-mini based on the proposed prompt\. Generating 100K samples with GPT4o\-mini cost approximately $12 in total\. The training dataspans medical and financial domains, with each sample containing plain text andMMassociated question\-answer pairs\.A concrete example of theTellmedataset is shown in Figure[1](https://arxiv.org/html/2608.11788#S1.F1), with detailed data samples and generation prompts provided in Appendices[D](https://arxiv.org/html/2608.11788#A4)and[E](https://arxiv.org/html/2608.11788#A5)\. Finally, the generated data achieved an average score of 4\.03 out of 5 in an LLM\-as\-a\-judge quality evaluation based on relevance, clarity, and completeness\. The detailed evaluation prompts and results for data quality assessment are provided in Appendix[G\.3](https://arxiv.org/html/2608.11788#A7.SS3)\.

### 3\.3Adapting TEL to Continual Learning

The previously constructedTellmedataset samples follow the structure𝐗=\(𝐭,𝐪,𝐚\)\{\\color\[rgb\]\{0,0,0\}\\mathbf\{X\}\}=\(\\mathbf\{t\},\\mathbf\{q\},\\mathbf\{a\}\), where𝐭\\mathbf\{t\}represents the token sequence of the plain text, and𝐪\\mathbf\{q\}and𝐚\\mathbf\{a\}correspond to the token sequences of the questions and their respective answers\. For simplicity, this structure assumesM=1M=1\. WhenM\>1M\>1, the structure can be extended through QA concatenation as𝐗=\(𝐭,𝐪1,𝐚1,…,𝐪M,𝐚M\)\{\\color\[rgb\]\{0,0,0\}\\mathbf\{X\}\}=\(\\mathbf\{t\},\\mathbf\{q\}\_\{1\},\\mathbf\{a\}\_\{1\},\\ldots,\\mathbf\{q\}\_\{M\},\\mathbf\{a\}\_\{M\}\)\.

To explicitly reflect the testing effect, we utilize theTellmedataset, which includes both plain text and QA within a single sample\. Specifically, in Equation[1](https://arxiv.org/html/2608.11788#S3.E1), we configure the indicator function as𝟙​\(xi∈𝐭∪𝐚\)=1\\mathds\{1\}\(x\_\{i\}\\in\\mathbf\{t\}\\cup\\mathbf\{a\}\)=1,𝟙​\(xi∈𝐪\)=0\\mathds\{1\}\(x\_\{i\}\\in\\mathbf\{q\}\)=0\. This ensures that the model is trained to predict only the plain text and answer components while excluding the question component𝐪\\mathbf\{q\}from the loss computation\.

From the perspective of mixed training using both plain text and QA samples, this method serves as a natural extension of the conventional CLM approach, considering the CPT and IT training paradigms\. Consequently, it incorporates theTellmeframework\.

Table 2:Comparison of performance in the finance and medicine domains under different training methods\.

## 4Experiment

In this section, we present the quantitative evaluation procedure and criteria for the proposedTellmemethod, and analyze the experimental results based on the following research questions: \(1\) DoesTellmemethod acquire domain knowledge more efficiently than existing approaches? and \(2\) Is it effective for long\-term memory retention?

### 4\.1Experimental settings

In this study, we focused on the financial and medical domains, constructing datasets based on the method proposed in Section[3\.2](https://arxiv.org/html/2608.11788#S3.SS2)using 100k PubMed abstracts and 100k Bloomberg financial news articles\. The experiments were conducted with the number of QA pairs per data sample toM=3M=3\. Both the medical and financial domains require specialized knowledge and have been primarily used in previous studies for performance validation based on CPT\([27](https://arxiv.org/html/2608.11788#bib.bib38);[28](https://arxiv.org/html/2608.11788#bib.bib39)\)\. The evaluation benchmarks for finance include FOMC\([32](https://arxiv.org/html/2608.11788#bib.bib20)\), NIFTY\([31](https://arxiv.org/html/2608.11788#bib.bib17)\), and MMLU\-F\(inance\)\. For the medical domain, evaluations were conducted using HeadQA\([36](https://arxiv.org/html/2608.11788#bib.bib10)\), MedMCQA\([26](https://arxiv.org/html/2608.11788#bib.bib8)\), and MMLU\-C\(linic\)\([33](https://arxiv.org/html/2608.11788#bib.bib11)\)\. All evaluations were conducted using the lm\-evaluation\-harness\([15](https://arxiv.org/html/2608.11788#bib.bib7)\)for reproducibility\. Appendix[A](https://arxiv.org/html/2608.11788#A1)provides a detailed description of the benchmark datasets used for evaluation\.

Table 3:Comparison of performance in the finance and medicine domains under different training methods\.The evaluations were conducted using state\-of\-the\-art open\-source LLMs with varying capabilities as the base models\. Specifically, experiments were performed using Llama\-\{3\.2\-1B, 3\.2\-3B, 3\.1\-8B\}\([2](https://arxiv.org/html/2608.11788#bib.bib6)\)and SmolLM2\-1\.7B\([3](https://arxiv.org/html/2608.11788#bib.bib2)\)\.To assess the effectiveness of the proposed method, we compared the performance of theTellmemethod with existing approaches based on baseline models and four variations of the training methods:

- •\+cpt: Refers to the model that has undergone continual pre\-training on domain\-specific texts\.
- •\+cpt\+it: The model instruction\-tuned on a QA dataset based on the \+cptmodel\([39](https://arxiv.org/html/2608.11788#bib.bib33);[8](https://arxiv.org/html/2608.11788#bib.bib32);[11](https://arxiv.org/html/2608.11788#bib.bib31)\)\.
- •\+InstPT: This model is trained based on the template\-based QA generation approach proposed by[9](https://arxiv.org/html/2608.11788#bib.bib14)\. Specifically, an average of 5\.8 short\-form QA pairs is generated for the finance domain, while an average of 1\.25 long\-form QA pairs is generated for the medical domain\. During the subsequent CPT, the plain text and QA datasets are concatenated, and the loss is computed over all the tokens\. In this case, the indicator function in Equation[1](https://arxiv.org/html/2608.11788#S3.E1),𝟙​\(xi∈𝐗\)=1\\mathds\{1\}\(x\_\{i\}\\in\\mathbf\{X\}\)=1\. Further implementation details regardingInstPTcan be found in Appendix[C](https://arxiv.org/html/2608.11788#A3)\.
- •\+Tellme: The model trained using the proposedTellmemethod\. In this approach, each sample includes both plain text and QA pairs; however, only the plain text and answer tokens are used when computing the loss\.

Here, the CPT and IT stages of thecpt\+itmodel utilized the plain text and QA samples from theTellmedataset, respectively\.Consequently, both theTellmeandcpt\+itmodels see the same total number of tokens from the plain\-text and QA data\. However,cpt\+itperforms two seperated forward\-backward passes, whereasTellmeprocessed each mixed sample in a single pass\.Detailed information on the models and the training hyperparameters can be found in Appendix[A](https://arxiv.org/html/2608.11788#A1)\.

### 4\.2Experiment Results

#### Overall

Table[3](https://arxiv.org/html/2608.11788#S4.T3)presents the performance of the baseline,cpt,cpt\+it,InstPT, andTellmemodels in the financial and medical domains\. Overall, theTellmeapproach achieves the highest average performance\. Notably, it outperformed the commonly usedcpt\+itmethod by 10\.0%, demonstrating a significant improvement\. Given that bothTellmeandcpt\+itare trained on the same number of tokens, this result suggests thatTellmeenables more efficient learning of domain\-specific knowledge compared to existing methods\. Furthermore,TellmesurpassesInstPTby 6\.3% overall across both the financial and medical domains, indicating that the incorporation of open\-ended, free\-form QA has a positive impact\.

#### Tellmefor Domain Adaptation

How doesTellmeperform across distinct domains? The experimental results show that theTellmemethod achieves strong performance in both the finance and medical domains, with particularly notable improvements in the finance domain\. In the finance domain,Tellmeconsistently outperformed the baseline model across all tested models, achieving an average performance gain of approximately 9\.8%\. Moreover, despite theInstPTmethod training on nearly twice as much QA data asTellme, theTellmeapproach still achieved higher scores in all models, except for SmolLM2\-1\.7B\. Overall,TellmeoutperformedInstPTby an average of 5\.1% across all models\. In the medical domain,Tellmedemonstrated an average improvement of 0\.09 points over the baseline model and outperformedInstPTby an average of 2\.58 points\.

Figure 2:Perplexity comparison on the medical dataset for different training approaches\. The baseline model shows the highest perplexity, while the proposedTellmeachieves the lowest perplexity across all lengths, indicating improved model performance\.
#### Perplexity in Domain Adaptation

Figure[2](https://arxiv.org/html/2608.11788#S4.F2)presents a comparison of perplexity\(PPL\) for five training strategies, including the proposedTellmemethod, in the medical domain\. For both training and evaluation, plain text from the PubMed dataset was employed, with 100k and 4k disjoint data samples, respectively, to ensure a fair evaluation\. The results indicated that the pre\-trained baseline model \(SmolLM2\-1\.7B\), which was not subjected to domain adaptation, exhibited relatively high PPL across all sequence lengths\. In contrast, the model trained solely on plain text \(\+cpt\) tended to demonstrate lower PPL, suggesting a positive effect on domain adaptation\. However, the model that underwent additional IT with QA data after plain text training \(\+cpt\+it\) unexpectedly exhibited the highest PPL\. This observation is interpreted as the QA\-focused fine\-tuning phase conducted at the end, diluting the plain text representational capacity\. On the other hand, the \+Tellmemodel, which combines plain text and QA within a single data sample achieved the lowest PPL across all sequence length intervals\. This suggests that, compared to the \+cptmodel, the additional QA component in the \+Tellmemodel exerts a positive impact on domain adaptation, and that TEL efficiently acquires domain\-relevant representations\.

### 4\.3The Impact of TEL on Long\-Term Retention

We conducted two experiments using SmolLM2\-1\.7B to assess the effectiveness of the TEL technique in retaining long\-termdomainknowledge\. Figure[3](https://arxiv.org/html/2608.11788#S4.F3)displays a comparison between two models \(CPT\(F\)→\\toCPT\(M\) and TEL\(F\)→\\toCPT\(M\)\) in terms of training\-step PPLmeasured on the Bloomberg corpus\(top\) and performance based on a financial benchmark \(bottom\)\. Our experimental setup compared models trained in the following two phases:

- 1\.First, we trained two initial models using finance domain data: one withcptand the other withTellme\(CPT\(F\) and TEL\(F\)\)\.
- 2\.Subsequently, we further trained both models on medical domain data withcptfor 3 epochs, yielding the final models: CPT\(F\)→\\toCPT\(M\) and TEL\(F\)→\\toCPT\(M\)\.

In this experiment, we analyzed the long\-term retention of previouslyacquiredknowledge \(Finance\) by comparing the evaluation results on the target domain \(Finance\) between the CPT\(F\)→\\toCPT\(M\) model and the TEL\(F\)→\\toCPT\(M\) model\.

Figure 3:Performance of the finance domain after overwriting with medicine data\.#### Perplexity Results on Finance

The top graph in Figure[3](https://arxiv.org/html/2608.11788#S4.F3)compares the PPL of the two models\. The evaluation was conducted on 4K samples from the Bloomberg corpus, each with a sequence length of 4K, using a portion of the corpus that was not included in the training data\. The experimental results show that the TEL\-based model achieved a lower PPL than the CPT\-based model, indicating superior performance\. Specifically, in terms of the PPL increase rate, the TEL\-based model exhibited a2\.5% increase, whereas the CPT\-based model showed a2\.65% increase\. Given that lower PPL indicates greater confidence in predicting the next token, these results suggest that the TEL\-based model maintains a higher probability of generating content from its previously trained domain, despite additional training in an unrelated domain\. This implies that compared to conventionalCPTmethods, the TEL approach better preserves prior domain knowledge even after cross\-domain adaptation\.

#### BenchmarkResults on Finance

The bottom graph in Figure[3](https://arxiv.org/html/2608.11788#S4.F3)compares the average performance of the TEL\(F\)→\\toCPT\(M\) and CPT\(F\)→\\toCPT\(M\) models on the finance benchmark\. Observing the range between 0 and 250 training steps, the CPT\(F\)→\\toCPT\(M\) model exhibits a notable decline in finance domain comprehension early in the training process on the medical dataset\. In contrast, the TEL\(F\)→\\toCPT\(M\) model shows more stable retention of finance domain knowledge, even after additional training on medical data\. As a result, while the CPT\-based approach suffered a5\.72% decline in performance relative to its initial state, the TEL\-based approach showed only a0\.94% reduction, indicating superior knowledge retention\. Additionally, despite starting with an initial performance 1\.53 points higher than that of the CPT\-based model, the TEL\-based model maintained its knowledge more effectively throughout the training, ultimately achieving a 9\.8% higher final performance \(equivalent to a 3\.15\-point increase\) compared to the CPT\-based approach\. These findings suggest that, even when trained on an out\-of\-domain dataset, the TEL method preserves the knowledge of the target domain more effectively than conventional pre\-training methods\. This indicates that TEL has a positive impact on long\-term retention, contributing to greater stability in learned domain knowledge\.

## 5Ablation Study and Analysis

Table 4:A performance comparison for various training methods utilizing QA data in the financial domain\.This section provides an in\-depth analysis ofTellme, focusing on performance variations across different TEL application strategies\. Table[4](https://arxiv.org/html/2608.11788#S5.T4)presents a comparison of model performance across various training techniques in the finance domain\. In this table,‘TEL’ refers to the proposedTellmemethod, and ‘Inv\-TEL’ denotes a training strategy in which the input data is structured as𝐗=\(𝐪,𝐚,𝐭\)\\mathbf\{X\}=\(\\mathbf\{q\},\\mathbf\{a\},\\mathbf\{t\}\), where QA samples precede plain text\.In this case, the loss function is computed in the same manner as in theTellmeapproach, where all tokens, except for the question, are treated as targets during CPT\. Additionally, ‘PIT’ included as a comparative method, follows the approach proposed by[20](https://arxiv.org/html/2608.11788#bib.bib15)\. It consists of a two\-stage training process: first, IT is conducted using a QA dataset, and then CPT is performed on the trained model by treating the QA dataset and plain text data as independent samples\. During this process, the loss calculation for the questions is excluded\. Lastly, ‘TEL\-Q/L’ represents a variation of theTellmemethod in which the loss calculation is applied to all tokens, including questions, during CPT\. See Appendix[D\.4](https://arxiv.org/html/2608.11788#A4.SS4)for experimental details\.

#### Performance on Test QA Placement

Motivated by PIT’s approach, we tested Inv\-TEL to examine if positioning QA before plain text impacts performance\.As shown in Table[4](https://arxiv.org/html/2608.11788#S5.T4), the model trained using the Inv\-TEL method exhibited a performance that was approximately 2\.82 and 0\.02 points lower than those of the TEL and CPT methods, respectively\. These results suggest that even when TEL is applied, the positioning of the QA pair can significantly affect model performance, thereby demonstrating that the proposedTellmemethod effectively leverages this strategy\.

#### Performance on QA Dataset Utilization Method

The QA data proposed in this study can be utilized as independent samples in CPT alongside plain text\. Alternatively, QA data can be incorporated within a single sample, along with plain text, for training purposes\. How does performance differ when QA and plain text are treated as separate samples? As described earlier, the PIT in Table[4](https://arxiv.org/html/2608.11788#S5.T4)represents a model trained with CPT by combining QA and plain text as independent samples\. Compared to TEL, this model exhibited approximately 0\.8 points lower performance\. Ultimately, the results indicate that integrating QA and plain text within a single training sample yields higher efficiency than treating them as independent samples\.

#### The Impact of Question Prediction Loss

Would excluding loss calculation for questions improve model performance? As shown in Table[4](https://arxiv.org/html/2608.11788#S5.T4), TEL\-Q/L exhibited approximately 0\.42 points lower performance compared to TEL\. TEL\-Q/L computes the loss for all tokens, including the questions, whereas TEL calculates the loss only for tokens excluding the questions\. These results suggest that the proposedTellmetraining method efficiently learns and utilizes key information, ultimately leading to improved performance\.

#### Efficiency ofTellme

The CPT has a limitation in which its training process requires substantial computational resources\. To investigate whetherTellmecan alleviate this issue, we conducted an experiment measuring PPL with respect to training steps\. The experimental results showed thatTellmeachieved the same PPL with a 1\.4 times faster compared to CPT\. Further details and results of the experiment are provided in the Appendix[G](https://arxiv.org/html/2608.11788#A7)\.

Table 5:Performance ofTellmein financial domain across synthesizers\. \(M\) and \(S\) indicate datasets generated by Mistral and self\-generated datasets, respectively\.
#### Impact of the Synthesizer in theTellme

TheTellmedataset was primarily constructed using the GPT4o\-mini model\. However, employing such a sophisticated model in dataset construction raises questions about whether performance improvements genuinely stem from the efficiency of the proposedTellmeapproach or merely reflect knowledge distillation from a superior model\.To investigate this perspective, we conducted additional experiments using an alternative synthesizer, specifically the Mistral\-7B model utilized in the InstPT approach, to generate theTellmedataset\. As shown in Table[5](https://arxiv.org/html/2608.11788#S5.T5), the Llama\-3\.2\-3B model trained on the Mistral\-generated dataset \(Tellme\-\(M\)\) achieved an average score of 33\.30, outperforming theInstPTbaseline by 2\.27 points in the financial domain\.Furthermore, we explored the effectiveness of self\-generated datasets, where theTellmedataset was generated by the target model itself \(Tellme\-\(S\)\)\. Results indicate that the self\-generated dataset notably improved performance, 2\.61 points higher than theInstPTbaseline\.These results underscore that while employing a powerful synthesizer like GPT4o\-mini yields superior performance, theTellmemethodology remains robust and effective even when less advanced synthesizers or self\-generated datasets are utilized\.

#### Multilingual Generalization to Korean

Table 6:Benchmark results on KoBEST for the OLMo\-1B model and its variants fine\-tuned withtellme\-ko\.We further investigated whether the proposedTellmeframework generalizes beyond English\. To this end, we generated KoreanTellmedata \(tellme\-ko\) following the English setting\. We evaluated Korean performance using the OLMo2\-1B model on the KoBEST[18](https://arxiv.org/html/2608.11788#bib.bib40)benchmark, aiming to assess how effectivelyTellmecan enhance Korean proficiency in models that originally lack any Korean capability\. As shown in Table[6](https://arxiv.org/html/2608.11788#S5.T6),tellme\-koachieves a remarkable \+8\.4\-point improvement in average accuracy, with over \+20\-point gains on the sentence understanding \(SENT\) task\. These results highlight that the proposedTellmeframework can augment knowledge in a language\-agnostic manner\. Additional experiments across different Korean models, scales, and data generation methods are presented in Appendix[G\.6](https://arxiv.org/html/2608.11788#A7.SS6)\.

## 6Conclusion

In this study, we propose theTellme\(Test\-Enhanced Learning for Language Model Enrichment\) technique, which offers an effective method for continual pre\-training of large language models \(LLMs\)\. This approach applies the TEL \(Test\-Enhanced Learning\) principle to mitigate the limitations of the conventional CPT\+IT method, particularly in learning target domain knowledge and maintaining long\-term memory\. We introduce a CPT method utilizing descriptive QA and a strategy for efficiently constructing training data, which have demonstrated positive experimental results from a domain adaptation perspective\. In the finance domain,Tellmeachieved up to a 23\.6% performance improvement on the finance benchmarks compared to existing methods\.

## Limitations

TheTellmemethod proposed in this study has the following possible limitations\.

#### Model Size\.

Second, although we extended our study to a 70B\-parameter model \([G\.5](https://arxiv.org/html/2608.11788#A7.SS5)\), the available computational budget required parameter\-efficient fine\-tuning, namely Low\-Rank Adaptation \(LoRA\) and 4\-bit quantization\. These techniques reduce memory footprint and training time, but they also introduce additional variables, such as rank selection and quantization noise, that may interact withTellme\. While the preliminary gains at this scale are encouraging, they may not faithfully representTellme’s effect on a fully dense 70B model\. A systematic investigation without compression, and across even larger architectures, remains an important direction for future work\.

#### Domain Diversity\.

Finally, this study focuses on finance and medicine, two domains known for their specialized and complex content\. However, this scope does not cover the full range of real\-world applications\. ExpandingTellmeto additional domains would require reliable benchmark datasets and evaluation metrics, which are not always publicly available\. We acknowledge this limitation and encourage further research to extendTellmeto a broader range of domains, ideally alongside the development of standardized benchmarks in those areas\.

## Acknowledgement

This work was supported by the affiliated institute of ETRI\[2025\-050\] and Institute of Information & communications Technology Planning & Evaluation \(IITP\) grant, funded by the Korea government \(MSIT\) \(No\.RS\-2024\-00456709\)\. We have used GPUs from High\-Performance Research AI Computing Infrastructure Support at the 2 PFLOPS Scale \(RS\-2025\-02653113\)

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2608.11788#S1.p1.1)\.
- AI@Meta \(2024\)AI@MetaLlama 3 model card\.External Links:[Link](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md)Cited by:[§4\.1](https://arxiv.org/html/2608.11788#S4.SS1.p2.1)\.
- Allalet al\.\(2025\)L\. B\. Allal, A\. Lozhkov, E\. Bakouch, G\. M\. Blázquez, G\. Penedo, L\. Tunstall, A\. Marafioti, H\. Kydlíček, A\. P\. Lajarín, V\. Srivastav,et al\.SmolLM2: when smol goes big–data\-centric training of a small language model\.arXiv preprint arXiv:2502\.02737\.Cited by:[§1](https://arxiv.org/html/2608.11788#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.11788#S4.SS1.p2.1)\.
- Brame and Biel \(2015\)C\. J\. Brame and R\. BielTest\-enhanced learning: the potential for testing to promote greater learning in undergraduate science courses\.CBE—Life Sciences Education14\(2\),pp\. es4\.Cited by:[§2\.1](https://arxiv.org/html/2608.11788#S2.SS1.p2.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2608.11788#S1.p1.1)\.
- Butler and Roediger III \(2007\)A\. C\. Butler and H\. L\. Roediger IIITesting improves long\-term retention in a simulated classroom setting\.European Journal of Cognitive Psychology19\(4\-5\),pp\. 514–527\.Cited by:[§2\.1](https://arxiv.org/html/2608.11788#S2.SS1.p2.1)\.
- Chenet al\.\(2024\)J\. Chen, Z\. Chen, J\. Wang, K\. Zhou, Y\. Zhu, J\. Jiang, Y\. Min, W\. X\. Zhao, Z\. Dou, J\. Mao,et al\.Towards effective and efficient continual pre\-training of large language models\.arXiv preprint arXiv:2407\.18743\.Cited by:[§2\.3](https://arxiv.org/html/2608.11788#S2.SS3.p2.1)\.
- Chenet al\.\(2023\)Z\. Chen, A\. H\. Cano, A\. Romanou, A\. Bonnet, K\. Matoba, F\. Salvi, M\. Pagliardini, S\. Fan, A\. Köpf, A\. Mohtashami, A\. Sallinen, A\. Sakhaeirad, V\. Swamy, I\. Krawczuk, D\. Bayazit, A\. Marmet, S\. Montariol, M\. Hartley, M\. Jaggi, and A\. BosselutMEDITRON\-70b: scaling medical pretraining for large language models\.External Links:2311\.16079,[Link](https://arxiv.org/abs/2311.16079)Cited by:[2nd item](https://arxiv.org/html/2608.11788#S4.I1.i2.p1.1)\.
- Chenget al\.\(2024\)D\. Cheng, Y\. Gu, S\. Huang, J\. Bi, M\. Huang, and F\. WeiInstruction pre\-training: language models are supervised multitask learners\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 2529–2550\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.148/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.148)Cited by:[Appendix C](https://arxiv.org/html/2608.11788#A3.p1.1),[§1](https://arxiv.org/html/2608.11788#S1.p2.1),[§2\.3](https://arxiv.org/html/2608.11788#S2.SS3.p3.1),[3rd item](https://arxiv.org/html/2608.11788#S4.I1.i3.p1.1)\.
- Chenget al\.\(2023\)D\. Cheng, S\. Huang, and F\. WeiAdapting large language models via reading comprehension\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2\.3](https://arxiv.org/html/2608.11788#S2.SS3.p3.1)\.
- Colomboet al\.\(2024\)P\. Colombo, T\. P\. Pires, M\. Boudiaf, D\. Culver, R\. Melo, C\. Corro, A\. F\. T\. Martins, F\. Esposito, V\. L\. Raposo, S\. Morgado, and M\. DesaSaulLM\-7b: a pioneering large language model for law\.External Links:2403\.03883,[Link](https://arxiv.org/abs/2403.03883)Cited by:[2nd item](https://arxiv.org/html/2608.11788#S4.I1.i2.p1.1)\.
- Dubeyet al\.\(2024\)A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2608.11788#S1.p5.1)\.
- Franciset al\.\(2020\)A\. P\. Francis, M\. B\. Wieth, K\. L\. Zabel, and T\. H\. CarrA classroom study on the role of prior knowledge and retrieval tool in the testing effect\.Psychology Learning & Teaching19\(3\),pp\. 258–274\.Cited by:[§1](https://arxiv.org/html/2608.11788#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.11788#S2.SS1.p1.1)\.
- Fujiiet al\.\(2024\)K\. Fujii, T\. Nakamura, M\. Loem, H\. Iida, M\. Ohi, K\. Hattori, H\. Shota, S\. Mizuki, R\. Yokota, and N\. OkazakiContinual pre\-training for cross\-lingual llm adaptation: enhancing japanese language capabilities\.arXiv preprint arXiv:2404\.17790\.Cited by:[Appendix F](https://arxiv.org/html/2608.11788#A6.SS0.SSS0.Px2),[§2\.2](https://arxiv.org/html/2608.11788#S2.SS2.p1.1)\.
- Gaoet al\.\(2024\)L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. ZouA framework for few\-shot language model evaluation\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§4\.1](https://arxiv.org/html/2608.11788#S4.SS1.p1.1)\.
- Gururanganet al\.\(2020a\)S\. Gururangan, A\. Marasović, S\. Swayamdipta, K\. Lo, I\. Beltagy, D\. Downey, and N\. A\. SmithDon‘t stop pretraining: adapt language models to domains and tasks\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 8342–8360\.External Links:[Link](https://aclanthology.org/2020.acl-main.740/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.740)Cited by:[Appendix F](https://arxiv.org/html/2608.11788#A6.SS0.SSS0.Px3),[§2\.2](https://arxiv.org/html/2608.11788#S2.SS2.p1.1)\.
- Gururanganet al\.\(2020b\)S\. Gururangan, A\. Marasović, S\. Swayamdipta, K\. Lo, I\. Beltagy, D\. Downey, and N\. A\. SmithDon’t stop pretraining: adapt language models to domains and tasks\.arXiv preprint arXiv:2004\.10964\.Cited by:[§1](https://arxiv.org/html/2608.11788#S1.p1.1)\.
- Janget al\.\(2022\)M\. Jang, D\. Kim, D\. S\. Kwon, and E\. DavisKobest: korean balanced evaluation of significant tasks\.InProceedings of the 29th International Conference on Computational Linguistics,pp\. 3697–3708\.Cited by:[§G\.6](https://arxiv.org/html/2608.11788#A7.SS6.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2608.11788#S5.SS0.SSS0.Px6.p1.1)\.
- Jianget al\.\(2023\)A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier,et al\.Mistral 7b\.arXiv preprint arXiv:2310\.06825\.Cited by:[Appendix C](https://arxiv.org/html/2608.11788#A3.p1.1)\.
- Jianget al\.\(2024\)Z\. Jiang, Z\. Sun, W\. Shi, P\. Rodriguez, C\. Zhou, G\. Neubig, X\. V\. Lin, W\. Yih, and S\. IyerInstruction\-tuned language models are better knowledge learners\.arXiv preprint arXiv:2402\.12847\.Cited by:[§1](https://arxiv.org/html/2608.11788#S1.p2.1),[§1](https://arxiv.org/html/2608.11788#S1.p3.1),[§2\.3](https://arxiv.org/html/2608.11788#S2.SS3.p1.1),[§5](https://arxiv.org/html/2608.11788#S5.p1.1)\.
- Keet al\.\(2025\)Z\. Ke, Y\. Ming, X\. Nguyen, C\. Xiong, and S\. JotyDemystifying domain\-adaptive post\-training for financial llms\.arXiv preprint arXiv:2501\.04961\.Cited by:[Appendix F](https://arxiv.org/html/2608.11788#A6.SS0.SSS0.Px1),[§1](https://arxiv.org/html/2608.11788#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.11788#S2.SS2.p1.1),[§2\.3](https://arxiv.org/html/2608.11788#S2.SS3.p2.1)\.
- Larsenet al\.\(2008\)D\. P\. Larsen, A\. C\. Butler, and H\. L\. Roediger IIITest\-enhanced learning in medical education\.Medical education42\(10\),pp\. 959–966\.Cited by:[§1](https://arxiv.org/html/2608.11788#S1.p3.1)\.
- Larsenet al\.\(2013\)D\. P\. Larsen, A\. C\. Butler, and H\. L\. Roediger IIIComparative effects of test\-enhanced learning and self\-explanation on long\-term retention\.Medical education47\(7\),pp\. 674–682\.Cited by:[§2\.1](https://arxiv.org/html/2608.11788#S2.SS1.p2.1)\.
- Lazaridouet al\.\(2021\)A\. Lazaridou, A\. Kuncoro, E\. Gribovskaya, D\. Agrawal, A\. Liska, T\. Terzi, M\. Gimenez, C\. de Masson d’Autume, T\. Kocisky, S\. Ruder,et al\.Mind the gap: assessing temporal generalization in neural language models\.Advances in Neural Information Processing Systems34,pp\. 29348–29363\.Cited by:[§2\.2](https://arxiv.org/html/2608.11788#S2.SS2.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2608.11788#S1.p1.1)\.
- Palet al\.\(2022\)A\. Pal, L\. K\. Umapathi, and M\. SankarasubbuMedMCQA: a large\-scale multi\-subject multi\-choice dataset for medical domain question answering\.InProceedings of the Conference on Health, Inference, and Learning,G\. Flores, G\. H\. Chen, T\. Pollard, J\. C\. Ho, and T\. Naumann \(Eds\.\),Proceedings of Machine Learning Research, Vol\.174,pp\. 248–260\.External Links:[Link](https://proceedings.mlr.press/v174/pal22a.html)Cited by:[§4\.1](https://arxiv.org/html/2608.11788#S4.SS1.p1.1)\.
- Pezeshkpour and Hruschka \(2025\)P\. Pezeshkpour and E\. HruschkaLearning beyond the surface: how far can continual pre\-training with lora enhance llms’ domain\-specific insight learning?\.arXiv preprint arXiv:2501\.17840\.Cited by:[§B\.1](https://arxiv.org/html/2608.11788#A2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.11788#S4.SS1.p1.1)\.
- Phasooket al\.\(2024\)P\. Phasook, J\. Pranee, C\. Limcharoen, K\. Sukhantharat, A\. Saeoueng, K\. Kerdthaisong, C\. Damrongrat, and S\. KongyoungThaiBKD: effective of continual pre\-training llm in thai language based on knowledge dataset\.In2024 19th International Joint Symposium on Artificial Intelligence and Natural Language Processing \(iSAI\-NLP\),pp\. 1–7\.Cited by:[§B\.1](https://arxiv.org/html/2608.11788#A2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.11788#S4.SS1.p1.1)\.
- Raksakietisaket al\.\(2024\)M\. Raksakietisak, V\. Lertsiripatarajit, N\. Aroonpruksakul, N\. Plailaharn, and K\. RaksamaniTest\-enhanced learning in neuroanesthesia for the first year anesthetic residents: a randomized controlled trial\.BMC Medical Education24\(1\),pp\. 905\.Cited by:[§2\.1](https://arxiv.org/html/2608.11788#S2.SS1.p2.1)\.
- Roediger III and Karpicke \(2006\)H\. L\. Roediger III and J\. D\. KarpickeTest\-enhanced learning: taking memory tests improves long\-term retention\.Psychological science17\(3\),pp\. 249–255\.Cited by:[§1](https://arxiv.org/html/2608.11788#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.11788#S2.SS1.p1.1)\.
- Saquret al\.\(2024\)R\. Saqur, K\. Kato, N\. Vinden, and F\. RudziczNIFTY financial news headlines dataset\.arXiv preprint arXiv:2405\.09747\.Cited by:[§4\.1](https://arxiv.org/html/2608.11788#S4.SS1.p1.1)\.
- Shahet al\.\(2023\)A\. Shah, S\. Paturi, and S\. ChavaTrillion dollar words: a new financial dataset, task & market analysis\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Toronto, Canada,pp\. 6664–6679\.External Links:[Link](https://aclanthology.org/2023.acl-long.368),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.368)Cited by:[§4\.1](https://arxiv.org/html/2608.11788#S4.SS1.p1.1)\.
- Singhalet al\.\(2025\)K\. Singhal, T\. Tu, J\. Gottweis, R\. Sayres, E\. Wulczyn, M\. Amin, L\. Hou, K\. Clark, S\. R\. Pfohl, H\. Cole\-Lewis,et al\.Toward expert\-level medical question answering with large language models\.Nature Medicine,pp\. 1–8\.Cited by:[§4\.1](https://arxiv.org/html/2608.11788#S4.SS1.p1.1)\.
- Suet al\.\(2023\)Z\. Su, J\. Li, Z\. Zhang, Z\. Zhou, and M\. ZhangEfficient continue training of temporal language model with structural information\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 6315–6329\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.418/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.418)Cited by:[Appendix F](https://arxiv.org/html/2608.11788#A6.SS0.SSS0.Px4),[§2\.2](https://arxiv.org/html/2608.11788#S2.SS2.p1.1)\.
- Taoriet al\.\(2023\)R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, X\. Li, C\. Guestrin, P\. Liang, and T\. B\. HashimotoStanford alpaca: an instruction\-following llama model\.Cited by:[§1](https://arxiv.org/html/2608.11788#S1.p1.1)\.
- Vilares and Gómez\-Rodríguez \(2019\)D\. Vilares and C\. Gómez\-RodríguezHEAD\-QA: a healthcare dataset for complex reasoning\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Florence, Italy,pp\. 960–966\.External Links:[Link](https://www.aclweb.org/anthology/P19-1092),[Document](https://dx.doi.org/10.18653/v1/P19-1092)Cited by:[§4\.1](https://arxiv.org/html/2608.11788#S4.SS1.p1.1)\.
- Wuet al\.\(2024\)T\. Wu, L\. Luo, Y\. Li, S\. Pan, T\. Vu, and G\. HaffariContinual learning for large language models: a survey\.arXiv preprint arXiv:2402\.01364\.Cited by:[§1](https://arxiv.org/html/2608.11788#S1.p1.1)\.
- Xieet al\.\(2024\)Y\. Xie, K\. Aggarwal, and A\. AhmadEfficient continual pre\-training for building domain specific large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 10184–10201\.External Links:[Link](https://aclanthology.org/2024.findings-acl.606/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.606)Cited by:[§2\.2](https://arxiv.org/html/2608.11788#S2.SS2.p1.1)\.
- Yanget al\.\(2024\)X\. Yang, J\. Gao, W\. Xue, and E\. AlexanderssonPLLaMa: an open\-source large language model for plant science\.External Links:2401\.01600,[Link](https://arxiv.org/abs/2401.01600)Cited by:[2nd item](https://arxiv.org/html/2608.11788#S4.I1.i2.p1.1)\.
- Zhenget al\.\(2022\)M\. Zheng, K\. O’Brien, K\. Cuenin, C\. Lyon, and D\. BenderImpact of test\-enhanced learning as a study strategy: an exploratory study with first\-year dental students\.Journal of Dental Education86\(12\),pp\. 1611–1619\.Cited by:[§2\.1](https://arxiv.org/html/2608.11788#S2.SS1.p2.1)\.

## Appendix ATraining Details and Hyperparameters

### A\.1Training Setup

We use PyTorch as the primary deep learning framework, along with the HuggingFace Transformers library for efficient model training\. The model is trained on a system equipped with eight NVIDIA A100 GPUs \(80GB VRAM\)\. Mixed\-precision training with bfloat16 is enabled to reduce memory usage and improve computational efficiency\.

The training process follows a single stage fine\-tuning approach, where the model is initialized with a pre\-trained checkpoint and adapted to the target domain using task\-specific data\. A cosine learning rate scheduler with a warm\-up ratio of 0\.03 is applied to prevent unstable updates in the early training phase\.

### A\.2Training Datasets

In this section, we summarize the characteristics of the two main domain datasets additionally utilized in this paper\.

#### Bloomberg

This dataset, extracted from Bloomberg News, focuses on content that is relevant to the financial community\. It provides documents of various lengths, offering domain specific terminology and real market trend information from the finance sector\.

#### PubMed

Constructed from open\-access data containing a large\-scale collection of abstracts from the fields of medicine and life sciences, it enriches the medical domain with specialized knowledge, such as disease names, drug names, and clinical research terminologies that is often lacking in general language models, thereby promoting performance improvements in the respective field\.

### A\.3Model Description

#### Llama\-3\.2\-1B

An open\-source model released by Meta, with 1 billion \(1B\) parameters\. This lightweight model is designed for efficient performance with low computational cost, providing fundamental natural language understanding and task reasoning capabilities\.

#### Llama\-3\.2\-3B

An open\-source model released by Meta, containing 3 billion \(3B\) parameters\. It offers stronger contextual understanding and better generalization compared to the 1B model, making it more suitable for a variety of natural language processing \(NLP\) tasks\.

#### Llama\-3\.1\-8B

An open\-source model released by Meta, equipped with 8 billion \(8B\) parameters\. It is trained on diverse datasets, enabling strong natural language understanding and generation\. It also excels in long\-context understanding and domain adaptation\.

#### SmolLM2\-1\.7B

An open\-source model, developed by HuggingFaceTB, featuring 1\.7 billion \(1\.7B\) parameters\. It is highly computationally efficient and, despite its smaller size, is designed to deliver strong performance in various natural language understanding tasks\.

### A\.4Optimization and Training Strategy

We optimize the model using AdamW\-8bit, a memory\-efficient variant of AdamW, with a weight decay of 0\.01 to prevent overfitting\. The learning rate is set to 5e\-5 and is gradually reduced following a cosine schedule\. Training is performed with a batch size of 1, and gradient accumulation steps of 16 are used to achieve an effective batch size of 16\.

Since our focus is on continual pre\-training, we limit the training process to 1 epoch to prevent catastrophic forgetting while allowing the model to adapt effectively to the target domain\. In continual learning settings, over\-training on new data can lead to the erosion of previously learned knowledge\. By training for only one epoch, we ensure that the model retains its general knowledge while gradually adapting to domain\-specific nuances\. This approach aligns with previous findings in continual pre\-training literature, where limited exposure to new data helps maintain a balance between adaptation and retention\.

### A\.5Hyperparameter settings

Table 7:Hyperparameter settings used for training\. This table summarizes the key hyperparameters, including learning rate, optimizer, batch size, and training schedule\.

## Appendix BEvaluation Setup

### B\.1Experimental Domains and Selection of the QA Generation Model

In this study, we constructed theTellmedataset using the GPT4o\-mini model based on our proposed prompt\. Following prior research\([27](https://arxiv.org/html/2608.11788#bib.bib38);[28](https://arxiv.org/html/2608.11788#bib.bib39)\), we chose medicine and finance as the target domains for the training data\. We found that an open\-source model \(Llama3\.3\-70B\) could also generate data of comparable quality\. However, in terms of efficiency, renting GPUs to use the open\-source language model was both more time\-consuming and more expensive than employing GPT4o\-mini\. Consequently, we opted to use GPT for data generation\.

### B\.2Evaluation Settings

Essentially, all benchmarks were evaluated using accuracy as the primary metric in a 4\-shot in\-context setting\. However,due to significant class imbalance in the FOMC and NIFTY datasets, the F1 score was employed as the evaluation metric in a zero\-shot setting\.

### B\.3Finance Benchmarks

#### FOMC \(Federal Open Market Committee\)

This dataset consists of documents related to the Federal Open Market Committee \(FOMC\)\. It includes FOMC meeting minutes, press conferences, and speeches, and is used to evaluate the performance of models analyzing texts related to monetary policy and financial markets\.

#### NIFTY \(News\-Informed Financial Trend Yield\)

This is a benchmark dataset constructed based on news and analytical materials related to the U\.S\. financial market, including news headlines from February 2019 to September 2020\. It is used to assess models’ domain knowledge in areas such as the stock market, economic indicators, and corporate finance, as well as to measure their understanding and reasoning capabilities with respect to financial texts\.

#### Massive Multitask Language Understanding \- Finance \(MMLU\-F\)

MMLU for the finance domain is a benchmark extracted from a subset of the Massive Multitask Language Understanding \(MMLU\) dataset, specifically focused on finance\-related disciplines\. This benchmark was extracted by selecting a diverse set of subjects relevant to financial studies, including business ethics, econometrics, high school macroeconomics, high school microeconomics, management, marketing, and professional accounting\.

### B\.4Medicine Benchmarks

#### HEAD\-QA \(HEAlthcare Dataset\)

HeadQA is a multiple\-choice question\-answering benchmark designed to advance research in complex reasoning\. The dataset consists of questions taken from exams required for specialized roles in the Spanish healthcare system, posing significant challenges even for experts in the field\.

#### MedMCQA

A large\-scale Multiple\-Choice Question Answering \(MCQA\) dataset created to tackle real\-world medical entrance exam questions\. The MedMCQA task can be defined asX=\{Q,O\}X=\\\{Q,O\\\}, whereQQdenotes the textual questions andOOrepresents the set of possible answer choices\. Each question is accompanied by multiple candidate answers,O=\{O​1,O​2,…,O​n\}O=\\\{O1,O2,…,On\\\}, and the objective is to identify the correct single or multiple answers from the given options\.

#### MMLU\-C \(Massive Multitask Language Understanding \- Clinical\)

MMLU\-Clinic is a part of the MMLU benchmark that includes multiple\-choice questions related to the medical field, designed to assess the medical knowledge understanding of large language models\. This dataset covers various medical subfields such as anatomy, genetics, and clinical knowledge, and is used to evaluate models like Med\-PaLM 2\. Additionally, it is utilized alongside MedQA, MedMCQA, and PubMedQA to assess the precision of LLMs in reasoning and answering questions in the medical domain\. Notably, it can also be applied to professional evaluations like medical licensing exams, making it a significant benchmark in AI research for the healthcare sector\.

## Appendix CDetailed Description forInstPTDataset Generation

For the construction ofInstPTdata, we built the training dataset based on the code provided by[9](https://arxiv.org/html/2608.11788#bib.bib14)111https://github\.com/microsoft/LMOps\. Specifically, forInstPTQA generation, we utilized the open\-source synthesizer based on Mistral 7B\([19](https://arxiv.org/html/2608.11788#bib.bib12)\)released by the authors222https://huggingface\.co/instruction\-pretrain/instruction\-synthesizer\. The parameters used for data generation were set with max\_new\_tokens as 2048, conducted in a 3\-shot setting\.

## Appendix DTellmeDataset Examples and Clarification

### D\.1TellmeDataset Examples

Table[9](https://arxiv.org/html/2608.11788#A4.T9),[10](https://arxiv.org/html/2608.11788#A4.T10)illustrate examples of datasets generated using theTellmeapproach in the Medicine and Finance domains\. The QA pairs are related to the text but are designed to introduce new knowledge that cannot be directly found within the given text\.

### D\.2Comparison example with theInstPTdataset

Table[11](https://arxiv.org/html/2608.11788#A4.T11)presents examples of QA datasets generated using theTellmeapproach and theInstPTapproach for the same plain text\.

### D\.3Plain text dependency of the TELLME QA dataset

To assess the extent to which QA datasets rely on plain text, we define the Coverage Ratio \(CR\)\. Equation[3](https://arxiv.org/html/2608.11788#A4.E3)presents the formula used to compute CR, which is calculated as the proportion of words\(ww\) in the answer that also appear in the corpus, relative to the total number of words in the answer\.

CR=length​\(\{w∣w∈𝐚∩w∈𝐭\}\)length​\(\{w∣w∈𝐚\}\)∗100\\textbf\{CR\}=\\frac\{\\text\{length\}\(\\\{w\\mid w\\in\\mathbf\{a\}\\cap w\\in\\mathbf\{t\}\\\}\)\}\{\\text\{length\}\(\\\{w\\mid w\\in\\mathbf\{a\}\\\}\)\}\*100\(3\)The experiment was conducted using a training corpus from the finance domain, comparing QA datasets generated using theTellmeapproach and theInstPTapproach\.

Table 8:‘Basic’ refers to the model\-generated answer, while ‘w/o stop’ refers to the answer with stopwords removed\.Table[8](https://arxiv.org/html/2608.11788#A4.T8)shows that the dataset constructed using theTellmeapproach has a significantly lower CR compared to theInstPTapproach\. This suggests that theTellmedataset does not rely solely on plain text but also requires external knowledge beyond the given text\.

### D\.4Data Composition and Loss Function Design

Figure[4](https://arxiv.org/html/2608.11788#A4.F4)provides an overview of the data composition and loss calculation strategies employed in the training of various methods, including CPT, IT, PIT, InstPT,Tellme, Inv\-TEL, and TEL\-Q/L\.

In this figure, plain\-text refers to the corpus typically used for domain adaptation, while question and answer denote the components of QA datasets\. These data types can be treated either as independent samples or concatenated into a single sequence during training\. For instance, PIT adopts the former strategy, whereas methods such asTellmeand InstPT follow the latter\. In concatenated settings, the autoregressive nature of language models makes the relative ordering between plain\-text and QA data a critical factor influencing learning outcomes\.

In Figure[4](https://arxiv.org/html/2608.11788#A4.F4), green check marks indicate the positions where the loss is applied, whereas crossmark denote regions excluded from the loss calculation\. It is common in instruction tuning setups to exclude the question portion of QA data from loss computation\. However, several CPT\-integrated QA approaches \(e\.g\., InstPT\) include the question in the prediction objective\. To investigate the impact of this design choice, we additionally evaluate a variant, TEL\-Q/L, in which the question loss is explicitly excluded during training\.

![Refer to caption](https://arxiv.org/html/2608.11788v1/appendix_various_loss.png)Figure 4:Comparison of data composition and token\-level loss masking across training paradigms\. Dashed boxes denote individual mini\-batches; within each, the vertically stacked panels constitute a single training sequence\. \(Continual\) Pre\-training consumes plain text only, applying the objective to every token\. Instruction tuning feeds question–answer \(QA\) pairs but back\-propagates loss exclusively on answer tokens\. Pre\-Instruction Tuning \(PIT\) interleaves plain text with QA pairs, computing loss on plain\-text and answer tokens while masking question tokens\. InstPT activates the objective for all tokens in both plain\-text and QA examples\. Our proposed TELLME keeps PIT’s masking strategy while doubling the proportion of plain\-text sequences to reinforce language modeling; its counterpart Inv\-TEL inverts the ordering of QA and plain\-text segments within each sample\. TEL\-Q/L serves as a fully supervised upper bound, applying loss to all tokens in both modalities\. Green check marks identify tokens whose losses are calculated, whereas red Crossmark denote tokens excluded from loss computation\.Plain TextU\.K\. Pound Weakens Versus Dollar Before BOE Bond\-Purchase, Rate Decision\. \(…\) The Monetary Policy Committee will leave its key rate unchanged at 0\.5 percent, according to all 53 economists surveyed by Bloomberg\. Eleven of 32 economists surveyed separately predict at least a 50 billion\-pound \($77 billion\) increase in its so\- called quantitative easing program\. “There might be a short relief rally in sterling on a lack of further BOE action but it’s likely to be short\- lived as the market is still focused on the likelihood of further QE by the November meeting,” said Ian Stannard , London\-based head of European currency strategy at Morgan Stanley\. “The BOE will move towards further QE in coming months and ultimately sterling will come under pressure\.” The pound weakened 0\.1 percent to $1\.5439 at 8:54 a\.m\. in London\. It declined 0\.2 percent to 118\.47 yen and was little changed against the euro at 86\.36 pence\. Morgan Stanley recommends selling the pound on any “relief rallies” that are likely to occur should the central bank refrain from announcing further monetary stimulus for the U\.K\. economy\. The Bank of England has faced pressure to embark on further quantitative easing to help revive an economy battling the steepest government spending cuts since World War II and a worsening euro\-area debt crisis\. The U\.K\.’s economyQA fromTellmeQ1 : What is the purpose of a central bank’s quantitative easing program?A1 : The purpose of a central bank’s quantitative easing program is to stimulate the economy by increasing the money supply, lowering interest rates, and encouraging lending and investment during periods of economic downturn\.Q2 : How can currency exchange rates be affected by a country’s monetary policy decisions?A2 : Currency exchange rates can be affected by a country’s monetary policy decisions as changes in interest rates or quantitative easing can influence investor sentiment, capital flows, and overall demand for the currency, leading to fluctuations in its value against other currencies\.Q3 : What are the potential risks of investing in government bonds during times of economic uncertainty?A3 : The potential risks of investing in government bonds during times of economic uncertainty include interest rate risk, where rising rates can decrease bond prices, inflation risk, where the purchasing power of returns may be eroded, and credit risk, where concerns about the government’s ability to meet its obligations could lead to defaults\.Table 9:An example of a QA dataset generated using theTellmeapproach in the finance domain\.Plain TextLow\-Energy Hawkins Type III Talar Neck Fracture\-Dislocation With Neurovascular and Tendon Entrapment in a Pediatric Patient\. Several serious complications can occur after talar neck fractures\. However, these fractures are extremely rare in children\. We present a pediatric low\-energy Hawkins type III fracture\-dislocation that had excessive displacement accompanied by neurovascular and tendon entrapment\. A 9\-year\-old male patient referred to our hospital 5 hours after jumping off a swing in a children’s playground\.An excessively displaced talar neck fracture\-dislocation was observed at the initial evaluation\. The patient underwent urgent surgery\. The tibialis posterior flexor digitorum longus tendons, posterior tibial artery, and tibial nerve were entrapped at the fracture site\. The talar neck fracture was reduced using open reduction\. The neurovascular structures and tendons were removed from the fracture site\. The fracture was fixed using two 4\.5\-mm cannulated screws\. The patient was able to bear full weight at 10 weeks postoperatively\. At 6 months, the patient was able to walk unassisted with full ankle range of motion\. However, at 2 years, his American Orthopaedic Foot and Ankle Society Ankle\-Hindfoot scale score had decreased to 72 points, and we observed avascular necrosis in the talar head \(…\) In the pediatric population, even low\-energy trauma, such as had occurred in our patient, can result in severe displaced fracture dislocations\. After severe displaced fracture\-dislocations, important soft tissue structures can become entrapped between fracture fragments, and surgeons should be aware of this situation when considering using closed reduction\.QA fromTellmeQ1 : What are some common complications associated with talar neck fractures?A1 : Common complications include avascular necrosis, neurovascular injury, and tendon entrapment\.Q2 : Why are talar neck fractures considered rare in the pediatric population?A2 : Talar neck fractures are rare in children due to the relative strength and flexibility of pediatric bones compared to adults\.Q3 : What is the typical treatment approach for severely displaced talar neck fractures?A3 : The typical treatment involves surgical intervention, often requiring open reduction and internal fixationTable 10:An example of a QA dataset generated using theTellmeapproach in the medicine domain\.Plain TextChina’s Faster Inflation Fuels Speculation Rate\-Rise Near\.China’s central bank mayraise interest rateswithin weeks afterinflationaccelerated to the fastest pace in 25 months in October, a Bloomberg News survey of economists showed\. The benchmark one\-year lending rate will rise to5\.81 percentby year\-end from5\.56 percent,according to the median forecast of 11 analysts polled after yesterday’s price data\. The deposit rate may climb to2\.75 percentfrom2\.5 percent,the survey showed\. China’s benchmark Shanghai Composite Index slid 2\.6 percent as of 1:01 p\.m\. local time on speculation that officials may move as early as today or this weekend after increasing banks’reserve requirementson Nov\. 10\. Higher rates could complicate government efforts to limit gains in consumer and property prices by luring more money to the fastest\-growing major economy\. “We know there’ll be more tightening given how inflation has accelerated and home prices haven’t come down, but the sudden talk that there may be an interest\-rate hike as early as the end of today really spooked the markets,” said Mark Tan , who helps oversee $12 billion at UOB Asset Management Ltd\. Goldman Sachs Group Inc\. said yesterday that October’s economic data indicated Chinese growth was “firm” and more “policy tightening” is needed\. Industrial & Commercial Bank of China Ltd\. and China Vanke Co\. led lenders and property developers lower as the benchmark index fell the most in three months\. Reserve Requirements Price pressures in China’s economy may be exacerbated by the nation’s currency curbs and imbalances in trade and capital flows that Group of 20 leaders are meeting in Seoul to tackle\. The central bank announced a 0\.5 percentage point increase in lenders’ reserve requirements this week after the customs bureau reported that October’s trade surplus surged to $27\.1 billion\. Some lenders including Bank of Communications Co\., had an additional increase, meaning their requirements rose a full percentage point, two people with direct knowledge of the situation said yesterday\. Brian Jackson , an emerging markets strategist at Royal Bank of Canada in Hong Kong, said yesterday that more interest\-rate increases are “clearly on the way\.” The central bank raised rates last month for the first time since 2007 as part of an exit from crisis policies that included scrapping in June the yuan’s peg to the dollar\. A record expansion in lending has added to concern that China’s inflation, now centered on food costs, will broaden\. In October, new lending was a more\-than\-forecast587\.7 billion yuan\($89 billion\), a central bank report showed yesterday\. While the latest inflation number was above all 28 estimates in a Bloomberg News survey of economists, it matched rumors cited before the data by firms including Guoyuan Securities Co\. in Shanghai\.“There’s talk of an interest\-rate hike over the weekend,”saidWu Kan, a Shanghai\-based fund manager at Dazhong Insurance Co\., which oversees $285 million\. “It’s quite possible given how inflation has accelerated\.”QA fromTellmeQA fromInstPTQ1: What are the potential effects ofraising interest rateson consumer spending and borrowing?A1: Raising interest rates typically leads to higher borrowing costs, which can reduce consumer spending and borrowing\. As loans become more expensive, consumers may delay or reduce purchases, particularly for big\-ticket items that often require financing, such as homes and cars\.Q2: How caninflation impactthe overall economy?A2: Inflation can erode purchasing power, leading to decreased consumer confidence and spending\. If inflation rises too quickly, it may prompt central banks to increase interest rates to stabilize prices, which can further slow economic growth\. Additionally, high inflation can disproportionately affect lower\-income households, as they spend a larger portion of their income on essential goods and services\.Q3: What is the role ofreserve requirementsin monetary policy?A3: Reserve requirements are regulations set by central banks that dictate the minimum amount of reserves a bank must hold against deposits\. By increasing reserve requirements, a central bank can reduce the amount of money available for lending, thus tightening monetary policy\. This can help control inflation but may also slow down economic growth by limiting credit availability\.Q1: What is the name of the fund manager?A1:Wu KanQ2: What did he say?A2:“There’s talk of an interest\-rate hike over the weekend,”Q3: What is the rate expected to rise to by the end of the year?A3:5\.81 percentQ4: What is the current rate?A4:5\.56 percent,Q5: What is the amount expected to be lent?A5:587\.7 billion yuanQ6: What is that converted to the thousands?A6:89 billionQ7: What is the deposit rate expected to rise to?A7:2\.75 percentQ8: From what?A8:2\.5 percent,Q9: What is the net change?A9:0\.25 percent,Q10: What is the one year lending rate expected to be?A10:5\.81 percentQ11: What was it last year?A11:5\.56 percentTable 11:An example of QA datasets generated using theTellmeapproach and the InstPT approach for the same plain text\. InstPT follows a reading\-comprehension format, where answers are typically extractive\. In contrast,Tellmegenerates open\-ended questions that require a deeper understanding of the text beyond surface\-level extraction\.Table 12:Prompt forTellmeDataset Construction\. In this study, \{domain\} refers to either “medicine” or “finance,” depending on the context\. The system message defines the task, while the user message provides the article excerpt to guide question generation\. The \{Input text\} corresponds to a PubMed article excerpt for the “medicine” domain and a Bloomberg article excerpt for the “finance” domain\.

## Appendix EDetailed Methodology for TEL Dataset Construction

In this study, we constructed a TEL dataset using GPT\-generated content tailored to different domains\. Specifically, we designed structured prompts to generate high\-quality question\-answer \(QA\) pairs in the medical and finance fields\. These prompts were crafted to ensure the generated questions were independent of specific article excerpts while remaining relevant to the broader domain knowledge\.

### E\.1Description of the Prompt Design for Domain Specific Dataset

As shown in Table[12](https://arxiv.org/html/2608.11788#A4.T12), the prompt design for TEL dataset construction includes a system message and a user message\. The system message defines the task for generating domain\-specific Q&A pairs, while the user message provides the article excerpt as input\. The \{Input text\} in the user message is sourced from PubMed for the “medicine” domain and Bloomberg for the “finance” domain\. This structured prompt ensures the generation of high\-quality Q&A pairs that are independent of specific details in the provided excerpts\.

### E\.2Ensuring Context Isolation: Filtering for QA Dataset

To construct theTellmedataset, we designed a prompt using GPT\-4o\-mini to generate Question & Answer pairs that can be solved without plain text\. However, after reviewing 100k samples, we found that approximately 80 samples \(0\.008%\) contained keywords such as “in this context” and “described,” indicating that some questions and answers were generated in a way that required context\. Although the number of such samples was small, this issue could compromise the fair evaluation of thecpt\+itandTellmemethods\. Therefore, we filtered out these samples before finalizing theTellmedataset\.

## Appendix FExpanding LLM Domains through Continual Learning

#### FINDAP: A Structured Approach for Financial LLM Adaptation\([21](https://arxiv.org/html/2608.11788#bib.bib9)\)\.

FINDAP applied a training methodology consisting of Financial\-based Continual Pre\-training, Instruction Tuning, and Preference Alignment to train a finance\-specialized LLM\. The PA \(Preference Alignment\) stage incorporates techniques proposed in the paper to enhance financial reasoning performance by introducing two methods: Stepwise Corrective Preference \(SCP\) and Final Answer Preference \(FAP\)\. SCP provides feedback by comparing the model’s reasoning process at each intermediate step with the correct answer, ensuring accurate step\-by\-step inference in financial problem\-solving\. Meanwhile, FAP guides the model to prefer more reliable answers when selecting the final response\.

#### Swallow: Cross\-Lingual Continual Pre\-Training for Japanese LLMs\([14](https://arxiv.org/html/2608.11788#bib.bib22)\)\.

Swallow is a study that applied Cross\-Lingual Continual Pre\-Training to enhance Japanese language performance\. This research analyzes the impact of vocabulary expansion and the use of parallel corpora in the process of adapting an English\-centric LLM to Japanese\.

The training process of the Swallow model followed three stages: \(1\) Continual Pre\-Training using a Japanese corpus, \(2\) Additional training with a Japanese\-English parallel corpus, \(3\) Application of Japanese\-specific vocabulary expansion\.

Through this approach, the model effectively improved English\-Japanese machine translation performance\.

#### Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks\([16](https://arxiv.org/html/2608.11788#bib.bib23)\)\.

This study proposes Domain\-Adaptive Pretraining \(DAPT\) and Task\-Adaptive Pretraining \(TAPT\) to enhance the performance of large language models \(LLMs\)\. DAPT strengthens domain adaptation by further training the model on large\-scale data from a specific domain, while TAPT improves task performance by additional training on task\-specific data\.

Experimental results show that applying both DAPT and TAPT together yields the highest performance, while in certain tasks, TAPT alone is sufficient for significant improvement\. This suggests that an appropriate additional training strategy is more effective than merely increasing model size\. Therefore, the study emphasizes the importance of tailored training strategies for domain\- and task\-specific optimization in NLP models\.

#### Efficient Continual Pre\-training for Building Domain\-Specific Large Language Models\.\([34](https://arxiv.org/html/2608.11788#bib.bib25)\)

This study proposes Continual Pre\-training \(CPT\) as a cost\-effective way to build domain\-specialized Large Language Models \(LLMs\)\. By developing the FinPythia model in finance and applying DACP and TACP, performance improved by up to 8\.3%\.

Furthermore, selecting only key data \(ETS\-DACP, ETA\-DACP\) instead of full dataset training cut costs by 90% while maintaining performance\. Despite domain\-specific gains, open\-domain performance remained stable, proving the method’s broad applicability\.

The study emphasizes that CPT is a more practical and economical alternative to training LLMs from scratch\.

## Appendix GFurther Analysis of theTellme

### G\.1Cost\-efficient Training ofTellme

Figure 5:Perplexity of CPT andTellmemethods based on training steps\.Figure[5](https://arxiv.org/html/2608.11788#A7.F5)illustrates the PPL scores of the CPT andTellmemethods over the training steps in the finance domain based on SmolLM2\-1\.7B\. TheTellme\-based model achieved a PPL of approximately 6\.935 after 1,650 steps, whereas the CPT\-based model reached a similar level only after 2,300 steps\. This indicates that the CPT model requires approximately 1\.4 times more training time to achieve the same performance as the TEL model\. Consequently, theTellmemethod demonstrates its cost efficiency by achieving superior performance within a shorter training duration\.

### G\.2Analysis of theTellmeIndicator on Different Datasets

Table 13:Comparison of model performance on theInstPTdataset when optimized with and without the indicator proposed byTellme\. Here,InstPTrefers to the method using both the dataset and the indicator proposed byInstPTwhile𝟙​\(xi∈𝐪\)=0\\mathds\{1\}\(x\_\{i\}\\in\\mathbf\{q\}\)=0denotes the method that utilizes the dataset proposed byInstPTbut applies the indicator proposed in this study\.Table[13](https://arxiv.org/html/2608.11788#A7.T13)presents a performance comparison when optimizing theInstPTdataset using the indicator proposed byTellme\. The experimental results show that employingTellme’s indicator consistently led to superior performance across all models\. Furthermore, considering the results in Table[4](https://arxiv.org/html/2608.11788#S5.T4), whereTellmeoutperformed TEL\-Q/L, these findings suggest that incorporating the proposed indicator in the optimization process for QA\-based Continual Learning dataset can be more effective\.

### G\.3Performance Variation Based on the Proportion of theTellmeDataset\.

Figure 6:Performance chart illustrating the effect of different ratios of general plain text and theTellmedataset, using a model trained on finance data\. The evaluation is the same financial benchmark used in Table[3](https://arxiv.org/html/2608.11788#S4.T3)Figure 7:Distribution of averaged human evaluation scores for the finance domain\. Each data point represents the mean of human\-evaluated quality scores across three question–answer pairs associated with a given financial text\. The scores are aggregated on a 1–5 scale, where higher values indicate better factuality, coherence, and domain correctness\. Notably, the overall distribution is concentrated above a score of 3, reflecting the high linguistic and conceptual quality of the curated financial dataset\.Figure[6](https://arxiv.org/html/2608.11788#A7.F6)illustrates the impact of the dataset composition ratio between CPT and TEL datasets on model performance when training the SmolLM2\-1\.7B model in the Finance domain using a 100k dataset\. As shown in the figure, model performance tends to improve as the proportion of the TEL dataset increases\. When the TEL dataset comprises 20% of the total data, the model’s performance is comparable to that of a model trained exclusively on the CPT dataset\. However, when the TEL dataset ratio increases to 40%, the model achieves approximately 0\.18 points higher performance than the CPT\-based model\. Additionally, when TEL accounts for 60% or 80% of the dataset, the model’s performance remains nearly identical at around 34\.9 points, marking an improvement of approximately 0\.62 points over the CPT\-based model\. At a TEL dataset ratio of 100%, the model achieves its highest performance, reaching a score of 35\.82\. Notably, even with only 60% TEL data, the model exhibits significant performance gains while maintaining a relatively low training cost of approximately $7\.2, making it a cost\-effective choice\.

In addition, we evaluated the quality of the generated data\. Figure[7](https://arxiv.org/html/2608.11788#A7.F7)presents the results of the LLM\-as\-a\-judge evaluation conducted using GPT\-4 mini, based on the following automatic evaluation prompt:Please evaluate the quality of the following question and answer pair based on relevance, clarity, and completeness\. Provide a single quality score between 1 \(poor\) and 5 \(excellent\)\.We performed the evaluation on 1,000 samples, and the dataset achieved an average score of 4\.03, confirming the high quality of the constructed corpus\.

### G\.4Scalability ofTellmeAcross Models

Table 14:Performance comparison of various models optimized withTellmeon medicine benchmarks\.Table[14](https://arxiv.org/html/2608.11788#A7.T14)presents the performance of various models optimized withTellme\. The results indicate that all models incorporatingTellmeexhibit improved average performance on medicine benchmark datasets\. This suggests thatTellmecan be applied to a wide range of language models and effectively enhances performance\.

### G\.5Scalability ofTellmeAcross Model size

Table 15:Performance ofTellmeon a Large\-Scale ModelTable[15](https://arxiv.org/html/2608.11788#A7.T15)reports the performance ofTellmeon a 70B\-parameter model trained with 4\-bit quantization and Low\-Rank Adaptation \(LoRA, rank=16\) to fit within our computational budget\. The method still achieves an improvement of about 1\.08 points, indicating thatTellmeremains effective even at a much larger scale\.

### G\.6Cross\-Lingual Transfer and Korean Adaptation

#### Cross\-lingual Transfer viaTellmeFramework\.

We further examined whether the proposedTellmeframework generalizes beyond English\. To this end, we selected Korean, a language that is linguistically and typologically distinct from English in both grammar and character system, to evaluate its cross\-lingual scalability\. Due to licensing constraints on Korean financial corpora, we could not directly use domain\-specific financial datasets\. Instead, we generated KoreanTellmedata by translating and adapting the English seed corpus\(Bloomberg\)\.

Specifically, we created two bilingual variants to explore different degrees of linguistic transfer:

- •tellme\-bi: which preserves English passages but provides Korean question–answer pairs, and
- •tellme\-ko: a fully translated version where both passages and QA pairs are in Korean\.

The detailed prompt design and example outputs used for this data construction are provided in Table[18](https://arxiv.org/html/2608.11788#A7.T18)and Table[19](https://arxiv.org/html/2608.11788#A7.T19), respectively\.

#### Experimental Setup and Evaluation\.

Following the same procedure used for the English data construction, we generated a total of 100,000 samples\. The evaluation was conducted using the KoBEST[18](https://arxiv.org/html/2608.11788#bib.bib40)benchmark, which enables a comprehensive assessment of both Korean linguistic competence and reasoning ability across multiple subtasks\. We evaluated Korean performance based on the OLMo2\-1B and OLMo2\-7B models, aiming to measure how effectively the proposedTellmeframework can enhance Korean proficiency in models that originally lack any Korean capability\.

#### Overall Improvement on KoBEST Benchmarks\.

Table 15 summarizes the results on the KoBEST benchmark suite, covering BoolQ, COPA, Hellaswag \(HLSW\.\), Sentineg \(SENT\.\), and WiC\. Across all tasks,tellme\-koconsistently enhances the base model’s accuracy\. For the smaller OLMo2\-1B model,tellme\-koimproves the average accuracy from 0\.477 \(Base\) to 0\.522 in the 0\-shot setting, a relative gain of \+4\.5 %, and from 0\.495 → 0\.579 \(\+8\.4 %\) in the 5\-shot setting\. The larger OLMo2\-7B model shows a similar trend, achieving 0\.508 → 0\.568 \(\+5\.9 %\) in 0\-shot and 0\.543 → 0\.598 \(\+5\.5 %\) in 5\-shot evaluation, demonstrating thatTellmeeffectively scales across model sizes\.

Table 16:Benchmark results on KoBEST for the Olmo\-1B and Olmo\-7B base models, as well as for models withtellme\-koapplied\. \(N\) denotes the number of in\-context samples \(shots\) used for evaluation\. The results show the performance of the OLMo2\-1B and \-7B models, along with the performance when thetellme\-komethod is applied\. The evaluation metric is accuracy, and the sub\-tasks include BoolQ, COPA, Hellaswag \(HLSW\.\), Sentineg \(SENT\.\), and WIC\.
#### Task\-wise Analysis\.

Performance gains vary across task categories\. For BoolQ \(yes/no comprehension\),tellme\-biandtellme\-koexhibit the largest improvement, reaching 0\.632 and 0\.610 \(vs\. base 0\.502\) in the OLMo2\-1B 0\-shot setting—an absolute increase of over \+0\.10\. This suggests strong transferability in sentence\-level reasoning\. For COPA, a causal reasoning task, accuracy improves from 0\.492 → 0\.534 \(tellme\-bi\) and 0\.585 \(tellme\-ko\), highlighting enhanced inferential ability after bilingual exposure\. In contrast, HellaSwag \(commonsense completion\) shows minor or negligible gains, implying that narrative completion may require richer Korean pretraining\. Notably, Sentineg—a sentiment polarity classification task—benefits substantially fromtellme\-ko, rising from 0\.486 → 0\.511 \(0\-shot\) and up to 0\.730 \(5\-shot\), showing that cross\-lingual alignment improves affective understanding in Korean\. Finally, WiC, which tests semantic consistency of word senses, exhibits moderate but stable improvements \(\+0\.02–0\.04 absolute\)\.

#### Effect of Bilingual vs\. Fully Translated Data\.

Comparingtellme\-biandtellme\-koprovides insight into the nature of cross\-lingual transfer\. The bilingual setup \(English passages with Korean QA\) yields strong improvements in sentence understanding \(BoolQ, COPA\), indicating that exposure to mixed\-language contexts suffices for semantic alignment\. However,tellme\-ko, which offers fully localized Korean data, surpassestellme\-biin most settings, especially in the 5\-shot Sentineg and COPA tasks, demonstrating that full translation amplifies Korean adaptation while maintaining English\-aligned reasoning ability\. This suggests that bilingual and translated data jointly facilitate smoother cross\-lingual transfer\.

#### Observation on LLama\-3\.2 in Korean\.

As shown in Table[17](https://arxiv.org/html/2608.11788#A7.T17), applying thetellme\-koframework to LLaMA\-3\.2\-1B results in modest yet consistent improvements in Korean performance on KoBEST\. The overall average increases from 0\.474 to 0\.481, with notable gains in BoolQ \(\+0\.027\) and Sentineg \(\+0\.030\), which test sentence comprehension and sentiment reasoning, respectively\. These improvements indicate that cross\-lingual exposure through TELLME allows the model to internalize Korean sentence\-level semantics without explicit Korean pretraining\.

Table 17:Comparison of KoBEST performance betweenLLaMA\-3\.2\-1B\(Original\) andtellme\-ko\. The evaluation metric is accuracy\.tellme\-koshows consistent gains in BoolQ and Sentineg, indicating improved comprehension and sentiment reasoning ability in Korean\.System messageTask: Read the text, askKoreanquestions and answer them\.
Follow these instructions:
1\. Ask diverse questions
2\. Focus on asking questions about factual information, important knowledge
3\. Write questions and answers inKorean
4\. Tag the question with “질문:" and the answer with “답변:"\.
Task:
After reading the above text, ask up to 8 questions and provide the correct answers\.
Format:
\- 질문: \[first question\] 답변: \[first answer\]Table 18:Prompt for generatingtellme\-bidata using English seed passages\. This dataset connects English knowledge with Korean question–answer pairs, enabling the model to jointly learn information across both languages\.Table 19:Example of data generated based on the Korean version oftellme\-bi\. Each sample consists of an English passage and a Korean Q&A pair\. An additional version,tellme\-ko, was produced by translating the English passages into Korean so that the entire sample \(passage–question–answer\) is in Korean\.