COFFEQA：用于大语言模型问答的反事实引导少样本蒸馏

# Counterfactual-Guided Few-Shot Distillation for LLM Question Answering
Source: [https://arxiv.org/html/2610.12837](https://arxiv.org/html/2610.12837)
Sanghamitra DuttaAffiliation:University of Maryland, College ParkEmail:[sanghamd@umd\.edu](mailto:)

###### Abstract

Knowledge distillation aims to train a smaller student model to mimic a larger teacher model for resource\-constrained environments\. However, the process of distillation is typically data\-intensive and computationally expensive, particularly for generative tasks\. Few\-shot distillation offers an alternative by transferring teacher knowledge using limited training samples\. However, we observe that few\-shot distillation performance is heavily dependent on the choice of these training samples\. In fact, we demonstrate that using more uncertain samples near the teacher’s decision boundary can significantly deteriorate distillation performance\. Motivated by this observation, we proposeCounterfactual\-guidedFew\-shot Distillation in open\-bookQuestionAnswering\(CofFeQA\), a novel strategy that aptly uses counterfactuals to provide additional supervision around challenging factual samples\. In the generative QA setting, we define a counterfactual as a minimally modified context that induces the teacher to change its response to an alternative answer\. We provide geometric insights on how factual\-counterfactual pairs can constrain the deviation between the teacher’s and student’s decision boundaries\. We further introduce a new metric,ContextFollowingDifficulty\(CFD\) that further validates counterfactuals prior to distillation based on agreement with the intended alternative answer and dependence on the modified context\.CofFeQAuses CFD\-validated factual\-counterfactual pairs for few\-shot distillation\. Experiments across five in\-domain and three out\-of\-domain QA benchmarks using T5 and Qwen2\.5 families show thatCofFeQAoutperforms factual\-only distillation baselines under the same few\-shot budget\. For instance, on TriviaQA,CofFeQAimproves over the factual\-only baseline by8\.778\.77average EM points and9\.989\.98average F1 points under the T5 setting with 32 few\-shot pairs\.

## 1Introduction

Large language models \(LLMs\) have achieved remarkable success across a wide range of natural language processing tasks\([Achiam et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib7);[Chung et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib84);[Gou et al\., 2021](https://arxiv.org/html/2610.12837#bib.bib42);[Raffel et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib9);[Fang et al\., 2026](https://arxiv.org/html/2610.12837#bib.bib37)\)\. However, their increasing size and computational requirements make deployment in resource\-constrained environments particularly challenging\([Kaplan et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib10);[Brown et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib8);[Strubell et al\., 2019](https://arxiv.org/html/2610.12837#bib.bib41);[Hoffmann et al\., 2022](https://arxiv.org/html/2610.12837#bib.bib38)\)\. This motivates the need for compact models with low training and inference costs that preserve task\-relevant capabilities of larger models\([Liang et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib16);[Saadi and Wang, 2025](https://arxiv.org/html/2610.12837#bib.bib17);[Liu et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib43);[Dissanayake et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib67)\)\. Knowledge distillation\([Hinton et al\., 2015](https://arxiv.org/html/2610.12837#bib.bib11)\)\(see surveys[Gou et al\. \(2021\)](https://arxiv.org/html/2610.12837#bib.bib42);[Sanh et al\. \(2019\)](https://arxiv.org/html/2610.12837#bib.bib75);[Xu et al\. \(2024b\)](https://arxiv.org/html/2610.12837#bib.bib12)\) seeks to address this challenge by transferring knowledge from a large teacher model to a smaller student model\. However, most distillation approaches typically rely on large\-scale datasets or extensive supervision from the teacher, thus incurring prohibitive training costs for already resource\-intensive language models\([Sanh et al\., 2019](https://arxiv.org/html/2610.12837#bib.bib75);[Sun et al\., 2019](https://arxiv.org/html/2610.12837#bib.bib21);[Jiao et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib2);[Wang et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib20);[Dissanayake and Dutta, 2025](https://arxiv.org/html/2610.12837#bib.bib55);[Hamman et al\., 2025](https://arxiv.org/html/2610.12837#bib.bib19)\)\. This leads to our primary question:*How can we perform data\-efficient distillation in LLMs?*

Towards addressing this question, our work studies few\-shot distillation for generative tasks, with a focus on open\-book question answering\. We find that the performance of few\-shot distillation is heavily dependent on the choice of the distillation dataset, consistent with the broader importance of data selection for efficient LLM training\([Fu et al\., 2026](https://arxiv.org/html/2610.12837#bib.bib58);[Zhou et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib1)\)\. In particular, samples near the teacher’s decision boundary can significantly deteriorate distillation performance \(see Fig\.[6](https://arxiv.org/html/2610.12837#A3.F6)in Section[C](https://arxiv.org/html/2610.12837#A3)\), also consistent with prior findings on knowledge transfer around decision boundaries\([Chen et al\., 2025](https://arxiv.org/html/2610.12837#bib.bib13)\)\. This motivates us to provide additional supervision around these challenging regions while maintaining a limited few\-shot budget\. To achieve this, we leverage counterfactual examples\. Counterfactual\-based data augmentation has improved classification performance\([Kaushik et al\., 2019](https://arxiv.org/html/2610.12837#bib.bib31);[Wu et al\., 2021](https://arxiv.org/html/2610.12837#bib.bib62);[Ross et al\., 2021](https://arxiv.org/html/2610.12837#bib.bib76);[Gardner et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib26);[McAleese and Keane, 2024](https://arxiv.org/html/2610.12837#bib.bib46);[Bhattacharjee et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib61);[Egea et al\., 2025](https://arxiv.org/html/2610.12837#bib.bib57)\)by encouraging models to capture causal dependencies and avoid shortcut learning\([Guo et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib29);[Chen et al\., 2023b](https://arxiv.org/html/2610.12837#bib.bib22)\)\. However, its applicability to few\-shot distillation for generative tasks remains largely unexplored\. Unlike classification, generative tasks produce free\-form outputs, making counterfactuals non\-trivial to define\. In generative tasks, we define a counterfactual to be a minimally modified input that changes the expected output while preserving overall semantic consistency, such as modifying supporting context in a question to induce a different answer\([Guo et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib29);[Longpre et al\., 2021](https://arxiv.org/html/2610.12837#bib.bib23);[Paranjape et al\., 2022](https://arxiv.org/html/2610.12837#bib.bib74);[Köksal et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib30);[Yu et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib28)\)\.

In this work, we proposeCounterfactual\-guidedFew\-shot distillation in open\-bookQuestionAnswering\(CofFeQA\), a novel strategy for few\-shot distillation that aptly uses counterfactuals to provide additional supervision around challenging factual samples near the teacher’s decision boundary\. We demonstrate how selecting high\-quality counterfactuals can guide knowledge transfer more effectively using extremely limited few\-shot budget\. Our main contributions are as follows:

![Refer to caption](https://arxiv.org/html/2610.12837v1/framework_diagram.png)Figure 1:Counterfactual\-guided few\-shot distillation framework\.We begin with \(1\) a few\-shot factual QA dataset𝒟f\\mathcal\{D\}\_\{f\}and \(2\) generate a counterfactual candidate for a factual sample\. \(3\) The generated counterfactual is evaluated using the teacher model with \(4\) our proposed Context Following Difficulty \(CFD\) metric and accepted if it satisfies the CFD thresholdτ\\tau\. \(5\) The CFD\-validated counterfactuals are then paired with their corresponding factuals and used to distill the student model\.Counterfactual\-guided few\-shot distillation for generative tasks\.We show that few\-shot distillation becomes particularly challenging for factuals near the teacher’s decision boundary, where the teacher has a smaller margin between its preferred answer and competing alternatives\. Motivated by this observation, we proposeCofFeQA, a novel framework which leverages counterfactuals for few\-shot distillation in open\-book QA\. Rather than relying solely on factual supervision,CofFeQApairs challenging factuals with their counterfactuals that expose changes in the teacher’s response under minimal modifications to the input context, providing additional supervision around the teacher’s decision boundary\. This reduces dependence on large factual training corpora and improves training efficiency under limited supervision \(see Figure[1](https://arxiv.org/html/2610.12837#S1.F1)for an overview\)\.

Geometric analysis of counterfactual\-guided knowledge transfer\.We first introduce a notion of decision boundaries for the teacher and student in the context\-embedding space, motivated by the fact that variations in the input context can lead to different answers\. Our formulation enables us to analyze distillation in open\-book QA from a geometric perspective\. Specifically, we show that factual\-counterfactual pairs can act as probes for the student to approximate the local geometry of the teacher’s decision boundary\. To quantify the deviation between the teacher and student, we use theHausdorff distance\([Rockafellar and Wets, 1998](https://arxiv.org/html/2610.12837#bib.bib32)\), which measures the worst\-case deviation between their decision boundaries\. We show in Proposition[1](https://arxiv.org/html/2610.12837#Thmproposition1)that this deviation is bounded by the magnitude of the counterfactual perturbations and their coverage of the teacher’s decision boundary\. Importantly, factuals near the teacher’s decision boundary can require smaller perturbations to induce a change in the teacher’s response, which can lead to tighter teacher\-student boundary alignment\.

Context Following Difficulty \(CFD\) for counterfactual validation\.Although we generate counterfactuals by perturbing the original input context to induce a targeted alternative answer, not all generated examples are truly valid with respect to the teacher\. The teacher may not produce same target answers using the modified context or may still be able to infer them without actually relying on the context\. Such cases weaken the intended causal relationship that the student was intended to learn from the teacher and may encourage shortcut learning\. To address this issue, we introduce a new metricContext Following Difficulty \(CFD\)to evaluate generated counterfactuals before distillation\. CFD jointly considers whether the teacher produces the intended alternative answer and whether that answer becomes more predictable when conditioned on the modified context\.CofFeQAuses CFD\-validated counterfactuals, ensuring that the resulting supervision better captures the teacher’s context\-dependent behavior\.

Empirical evaluation\.We conduct comprehensive experiments across five in\-domain and three out\-of\-domain benchmarks using the T5\([Raffel et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib9)\)and Qwen2\.5\([Bai et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib4)\)model families\. The in\-domain benchmarks include NaturalQuestions, SQuAD, NewsQA, TriviaQA, and HotpotQA, while the out\-of\-domain benchmarks include DROP, DuoRC, and RelationExtraction\([Fisch et al\., 2019](https://arxiv.org/html/2610.12837#bib.bib77)\)\. We evaluateCofFeQAwith standard knowledge distillation \(KD\), sequence\-level knowledge distillation \(SeqKD\), and layer\-wise distillation \(LWD\) baselines, under the same few\-shot training budget\. Across these settings,CofFeQAoutperforms factual\-only baselines\. For instance, on TriviaQA,CofFeQAimproves over the KD baseline by8\.778\.77average EM points and9\.989\.98average F1 points under the T5 setting with 32 few\-shot pairs\.

### 1\.1Related Work

Counterfactual Data Augmentation\.Counterfactual data augmentation has been widely studied for improving robustness in neural networks\([Kaushik et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib59);[Sachdeva et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib47);[Calderon et al\., 2022](https://arxiv.org/html/2610.12837#bib.bib53);[Egea et al\., 2025](https://arxiv.org/html/2610.12837#bib.bib57);[Mishra et al\., 2021](https://arxiv.org/html/2610.12837#bib.bib69)\)\. Early approaches relied on human\-generated counterfactuals\([Kaushik et al\., 2019](https://arxiv.org/html/2610.12837#bib.bib31);[Gardner et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib26)\), while later works automated counterfactual generation through trained generators and editors\([Wu et al\., 2021](https://arxiv.org/html/2610.12837#bib.bib62);[Ross et al\., 2021](https://arxiv.org/html/2610.12837#bib.bib76);[Chen et al\., 2021](https://arxiv.org/html/2610.12837#bib.bib24);[Tokpo and Calders, 2024](https://arxiv.org/html/2610.12837#bib.bib25)\)and more recently through prompting LLMs\([Bhattacharjee et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib61)\), alongside efforts to evaluate the quality of generated counterfactuals\([Youssef et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib45);[McAleese and Keane, 2024](https://arxiv.org/html/2610.12837#bib.bib46)\)\. For example, VISION\([Egea et al\., 2025](https://arxiv.org/html/2610.12837#bib.bib57)\)uses LLM\-generated counterfactuals to train graph neural networks for vulnerability classification\. Other works study counterfactual augmentation through active learning\([Balashankar et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib27)\)and fairness\-aware pipelines\([Tokpo and Calders, 2024](https://arxiv.org/html/2610.12837#bib.bib25)\)\. Counterfactual explanations have been studied for model extraction with theoretical guarantees\([Dissanayake and Dutta, 2024](https://arxiv.org/html/2610.12837#bib.bib44);[Dissanayake and Dutta, 2026](https://arxiv.org/html/2610.12837#bib.bib6)\)and for robustness to model changes\([Noorani et al\., 2025](https://arxiv.org/html/2610.12837#bib.bib68);[Hamman et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib66)\)\. Counterfactual applications also extend to generative tasks, including machine translation\([Liu et al\., 2021](https://arxiv.org/html/2610.12837#bib.bib70)\)and dialogue\([Ou et al\., 2022](https://arxiv.org/html/2610.12837#bib.bib71)\)\. However, most of these works focus on neural network models\. Counterfactual augmentation to fine\-tune LLMs has received limited attention\. CATfOOD\([Sachdeva et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib47)\)uses LLM\-generated counterfactuals to train smaller language models, while[Rajagopal et al\. \(2022\)](https://arxiv.org/html/2610.12837#bib.bib72)use counterfactually augmented training data for pretrained summarization models\. In contrast, we use factual\-counterfactual pairs as structured supervision for few\-shot distillation in generative open\-book QA, where the teacher produces free\-form answers rather than predictions over a predefined labels\.

Counterfactual Question Answering\.Counterfactual reasoning has recently gained traction in question answering \(QA\)\([Ross et al\., 2022](https://arxiv.org/html/2610.12837#bib.bib56);[Khashabi et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib48);[Huang et al\., 2025](https://arxiv.org/html/2610.12837#bib.bib49)\)\. DisentQA\([Neeman et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib73)\)uses counterfactual supervision to separate parametric and contextual knowledge\([Xu et al\., 2024a](https://arxiv.org/html/2610.12837#bib.bib50);[Wang et al\., 2023a](https://arxiv.org/html/2610.12837#bib.bib52)\), while IfQA\([Yu et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib28)\)provides a benchmark for QA under hypothetical scenarios that require reasoning beyond factual retrieval\. Counterfactual reasoning has also been studied in multi\-hop QA\([Guo et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib29)\)and CofCA\([Wu et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib65)\), where counterfactual inference is used to mitigate shortcut reasoning\. Retrieval\-guided approaches further construct counterfactuals through context and answer modifications\([Paranjape et al\., 2022](https://arxiv.org/html/2610.12837#bib.bib74);[Dixit et al\., 2022](https://arxiv.org/html/2610.12837#bib.bib51)\)\. While these works demonstrate the value of counterfactuals for improving robustness and reasoning in QA, they do not study how they can facilitate knowledge distillation\.

Distillation in LLMs\.Knowledge Distillation\([Hinton et al\., 2015](https://arxiv.org/html/2610.12837#bib.bib11)\)aims to train smaller student models to mimic larger teacher models, with extensions to sequence\-level distillation\([Xu et al\., 2024b](https://arxiv.org/html/2610.12837#bib.bib12);[Kim and Rush, 2016](https://arxiv.org/html/2610.12837#bib.bib40);[Gu et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib33);[Yang et al\., 2025](https://arxiv.org/html/2610.12837#bib.bib35)\), task\-specific distillation\([Dissanayake et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib67);[Sanh et al\., 2019](https://arxiv.org/html/2610.12837#bib.bib75);[Team et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib34);[Wang et al\., 2025](https://arxiv.org/html/2610.12837#bib.bib36)\), and layer\-wise distillation\([Jiao et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib2)\)\. Few\-shot distillation has been studied for transferring pretrained transformer knowledge in tabular classification\([Dissanayake and Dutta, 2025](https://arxiv.org/html/2610.12837#bib.bib55)\)and multiple\-choice question answering\([Sutanto et al\., 2025](https://arxiv.org/html/2610.12837#bib.bib60)\)\. In particular,[Sutanto et al\. \(2025\)](https://arxiv.org/html/2610.12837#bib.bib60)use an LLM to generate multiple\-choice questions and answer options, together with probability scores that are distilled into an encoder\-only student model\. However, their approach relies on a predefined answer\-choice space, making it less directly applicable to generative open\-book QA, where the student produces free\-form answers\. While generative and instruction\-level distillation often rely on direct supervision or synthetic data\([Hsieh et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib63);[Wang et al\., 2023b](https://arxiv.org/html/2610.12837#bib.bib64);[Gu et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib33)\), only a few recent works have investigated counterfactuals for knowledge transfer\. DISCO\([Chen et al\., 2023a](https://arxiv.org/html/2610.12837#bib.bib54)\)generates counterfactual examples using an LLM and employs a task\-specific teacher to filter the generated examples, which are then used as an augmented dataset to train a smaller student model\. Notably, DISCO is different from standard distillation via aligning representations since the teacher primarily serves as a data\-filtering mechanism rather than directly transferring its predictive behavior to the student through a distillation loss\. Moreover, its empirical evaluation focuses on natural language inference, a classification task with a predefined label space\. Another related work[Hamman et al\. \(2025\)](https://arxiv.org/html/2610.12837#bib.bib19)study counterfactual supervision for few\-shot distillation, but also focus on classification settings\. These approaches do not address generative tasks, where the outputs are open\-ended\. In contrast, we specifically focus on*few\-shot*knowledge distillation for generative open\-book QA\. Our approach is motivated by the difficulty of distilling with factuals near the teacher’s decision boundary and uses counterfactuals to provide additional supervision around these challenging regions\. We also provide new geometric insights on counterfactual\-guided knowledge transfer and a new metric CFD to further check counterfactual validity in open\-book QA\.

## 2CofFeQA: Counterfactual\-Guided Few\-Shot Distillation

Our strategy is motivated by the observation that factual samples near the teacher’s decision boundary are particularly challenging to distill under limited supervision\. We use margin sampling\([Scheffer et al\., 2001](https://arxiv.org/html/2610.12837#bib.bib5)\)as an operational proxy for boundary proximity, where smaller teacher prediction margins indicate boundary near samples\. Empirically, we observe that distillation performance degrades as this margin decreases; a detailed analysis is provided in Section[C](https://arxiv.org/html/2610.12837#A3)\. Motivated by this observation,CofFeQApairs boundary\-near factuals with their counterfactuals that expose how the teacher’s response changes under minimal variations of the input context\. We first define and describe the counterfactual generation process, and then geometrically analyze how factual\-counterfactual pairs facilitate teacher\-student boundary alignment\. Finally, we introduce Context Following Difficulty \(CFD\) to validate generated counterfactuals and present the overallCofFeQAdistillation framework\.

### 2\.1Counterfactual Definition and Generation

We define counterfactual generation for open\-book QA as constructing minimally modified training examples that preserve the original question while changing the evidence to support a different answer\. Given a triplet\(Q,C,A\)\(Q,C,A\), whereQQis the question,CCis the context, andAAis the answer, the counterfactual generation framework constructs a counterfactual by modifying the context using an alternative answer\. The resulting counterfactual triplet is denoted as\(Q,C′,A′\)\(Q,C^\{\\prime\},A^\{\\prime\}\)\. We generate counterfactuals using GPT\-4o\-mini\([OpenAI, 2024](https://arxiv.org/html/2610.12837#bib.bib15)\)in three stages\. First, the answerAAis classified into one of five entity types:person, location, organization, date, ornumeric; samples outside these categories are excluded\. Second, GPT\-4o\-mini generates an alternative answerA′A^\{\\prime\}of the same entity type that is similar in category but distinct fromAA\. Previously generated alternatives are excluded to encourage diversity across generation attempts\. Finally, GPT\-4o\-mini is provided with the original triplet\(Q,C,A\)\(Q,C,A\)andA′A^\{\\prime\}to generate the counterfactual contextC′C^\{\\prime\}by minimally editingCC\. Specifically, occurrences ofAAare replaced withA′A^\{\\prime\}, together with any related references that must be changed to keep the passage grammatically consistent\. As a result, the modified contextC′C^\{\\prime\}supportsA′A^\{\\prime\}as the answer to the same questionQQ, yielding the counterfactual triplet\(Q,C′,A′\)\(Q,C^\{\\prime\},A^\{\\prime\}\)\. The system and user prompts used are shown in Figure[2](https://arxiv.org/html/2610.12837#S2.F2)\. Additional prompt details are provided in Appendix[D](https://arxiv.org/html/2610.12837#A4)\.

Counterfactual Context Generation: System PromptYou are an expert at creating counterfactual reading\-comprehension passages for QA training data\. Given an original context that supports a specific answer, you rewrite it MINIMALLY so that it instead unambiguously supports a different target answer, by replacing occurrences of the original answer \(and grammatically\-dependent references to it\) with the target answer, while leaving everything else in the passage unchanged\. Do not add explanations or extra content\.

Counterfactual Context Generation: User PromptQuestion:\{question\}, Original context \(supports answer “\{original\_answer\}”\):\{original\_context\}Target answer \(the new counterfactual answer\): “\{target\_answer\}”Rewrite the context above so that it minimally changes to unambiguously support “\{target\_answer\}” as the answer to the question, instead of “\{original\_answer\}”\. Replace occurrences of “\{original\_answer\}” \(and any grammatically\-dependent references to it\) with “\{target\_answer\}”, keeping everything else in the passage exactly the same\. Do not add commentary\. Respond with ONLY:\{"reasoning": "<1\-2 sentences on what you changed\>","counterfactual\_context": "<the full rewritten context\>"\}

Figure 2:Counterfactual context generation prompts\.System and user prompts used to minimally modify the factual contextCCinto a counterfactual contextC′C^\{\\prime\}that supports the target answerA′A^\{\\prime\}\.
### 2\.2Geometric analysis of counterfactual\-guided knowledge transfer

We provide a geometric perspective on how factual\-counterfactual pairs can facilitate knowledge transfer\. We first introduce a notion of a decision boundary for open\-book QA\. While classification tasks typically consider alabel\-flipping boundary, we define acontext\-response boundaryin the context\-embedding space\. In classification, the boundary separates regions where two classes have equal probabilities\. In contrast, in open\-book QA, it represents the transition region in context embedding space where the model shifts from generating one answer to another answer\. In open\-book QA, the context space may contain regions corresponding to the factual answerAA, the intended counterfactual answerA′A^\{\\prime\}, and other answer \(A′′A^\{\\prime\\prime\}\) regions, different from bothAAandA′A^\{\\prime\}corresponding to out\-of\-context or unrelated outputs\. Our analysis focuses specifically on the boundary between theAAandA′A^\{\\prime\}response regions, as this is the boundary relevant to factual\-counterfactual supervision\. Our CFD\-based validation helps identify and reject counterfactual candidates that induce responses corresponding to the other answer \(A′′A^\{\\prime\\prime\}\) regions before distillation\. We therefore restrict the geometric analysis to the boundary between factual answer space \(AA\) and intended counterfactual answer \(A′A^\{\\prime\}\) space\. We use the Hausdorff distance\([Rockafellar and Wets, 1998](https://arxiv.org/html/2610.12837#bib.bib32)\)to quantify the worst\-case deviation between the teacher context\-response boundary and the student context\-response boundary and derive an upper bound for this deviation in Proposition[1](https://arxiv.org/html/2610.12837#Thmproposition1)\(The proof can be found in Appendix[E](https://arxiv.org/html/2610.12837#A5)\)\.

Figure 3:Visualizing the Hausdorff distance betweenℳt\\mathcal\{M\}\_\{t\}andℳs\\mathcal\{M\}\_\{s\}\.Definition of Hausdorff Distance\([Rockafellar and Wets, 1998](https://arxiv.org/html/2610.12837#bib.bib32)\)\.Let\(ℳ,d\)\(\\mathcal\{M\},d\)be a metric space\. For each pair of non\-empty subsetsℳt⊂ℳ\\mathcal\{M\}\_\{t\}\\subset\\mathcal\{M\}andℳs⊂ℳ\\mathcal\{M\}\_\{s\}\\subset\\mathcal\{M\}, the Hausdorff distance betweenℳt\\mathcal\{M\}\_\{t\}andℳs\\mathcal\{M\}\_\{s\}is defined as:

dH​\(ℳt,ℳs\):=max⁡\{supct∈ℳtinfcs∈ℳsd⁡\(ct,cs\),supcs∈ℳsinfct∈ℳtd⁡\(ct,cs\)\}\.d\_\{H\}\(\\mathcal\{M\}\_\{t\},\\mathcal\{M\}\_\{s\}\):=\\max\\left\\\{\\sup\_\{c\_\{t\}\\in\\mathcal\{M\}\_\{t\}\}\\inf\_\{c\_\{s\}\\in\\mathcal\{M\}\_\{s\}\}d\(c\_\{t\},c\_\{s\}\),\\sup\_\{c\_\{s\}\\in\\mathcal\{M\}\_\{s\}\}\\inf\_\{c\_\{t\}\\in\\mathcal\{M\}\_\{t\}\}d\(c\_\{t\},c\_\{s\}\)\\right\\\}\.
This measures the maximum of the minimum distances between two sets, capturing the worst\-case geometric deviation\. \(see Figure[3](https://arxiv.org/html/2610.12837#S2.F3)\)\.

###### Proposition 1

Let𝒞\\mathcal\{C\}be the space of context embeddings,c∈𝒞c\\in\\mathcal\{C\}arbitrary input context,Q∈𝒬Q\\in\\mathcal\{Q\}input question,A∈𝒜A\\in\\mathcal\{A\}factual answer, andA′∈𝒜′A^\{\\prime\}\\in\\mathcal\{A^\{\\prime\}\}counterfactual answer\. Consider a teacher and a student with response functionsGt​\(Q,c\)=arg⁡maxA​Pt​\(A\|Q,c\)G\_\{t\}\(Q,c\)=\\arg\\max\_\{A\}P\_\{t\}\(A\|Q,c\)andGs​\(Q,c\)=arg⁡maxA​Ps​\(A\|Q,c\)G\_\{s\}\(Q,c\)=\\arg\\max\_\{A\}P\_\{s\}\(A\|Q,c\)\. The respective context\-flipping boundaries can be defined as the sets:

ℳt=\{c∈𝒞∣Pt​\(A\|Q,c\)=Pt​\(A′\|Q,c\)\},ℳs=\{c∈𝒞∣Ps​\(A\|Q,c\)=Ps​\(A′\|Q,c\)\}\.\\mathcal\{M\}\_\{t\}=\\\{c\\in\\mathcal\{C\}\\mid P\_\{t\}\(A\|Q,c\)=P\_\{t\}\(A^\{\\prime\}\|Q,c\)\\\},\\quad\\mathcal\{M\}\_\{s\}=\\\{c\\in\\mathcal\{C\}\\mid P\_\{s\}\(A\|Q,c\)=P\_\{s\}\(A^\{\\prime\}\|Q,c\)\\\}\.
Suppose there exists a factual\-counterfactual paired dataset𝒟f\+c​f=\{\(ci,ci′\)\}i=1k\\mathcal\{D\}\_\{f\+cf\}=\\\{\(c\_\{i\},c\_\{i\}^\{\\prime\}\)\\\}\_\{i=1\}^\{k\}for the aboveℳt\\mathcal\{M\}\_\{t\}that satisfies three conditions:\(1\)\(1\)Minimal context perturbation in context embedding space is bounded byα\\alpha\(i\.e\.,‖ci−ci′‖≤α\|\|c\_\{i\}\-c\_\{i\}^\{\\prime\}\|\|\\leq\\alpha\),\(2\)\(2\)Exact answer generation:Gt​\(Q,c\)=Gs​\(Q,c\)G\_\{t\}\(Q,c\)=G\_\{s\}\(Q,c\)wherec=\{ci,ci′\}c=\\\{c\_\{i\},c^\{\\prime\}\_\{i\}\\\},\(3\)\(3\)ϵ\\epsilon\-spread boundary coverage: Counterfactuals are well\-distributed across the context\-response boundaries with spreadϵ\\epsilon, i\.e\., ift∈ℳtt\\in\\mathcal\{M\}\_\{t\}, then∃ti∗​such that​‖t−ti∗‖≤ϵ\\exists t\_\{i\}^\{\*\}\\;\\text\{such that\}\\;\\\|t\-t\_\{i\}^\{\*\}\\\|\\leq\\epsilon\. This applies forℳs\\mathcal\{M\}\_\{s\}as well\. i\.e\., ifs∈ℳss\\in\\mathcal\{M\}\_\{s\}, then∃si∗​such that​‖s−si∗‖≤ϵ\\exists s\_\{i\}^\{\*\}\\;\\text\{such that\}\\;\\\|s\-s\_\{i\}^\{\*\}\\\|\\leq\\epsilon\.

Then, the Hausdorff distance between the teacher boundary and student boundary satisfies:dH​\(ℳt,ℳs\)≤α\+ϵ\.d\_\{H\}\(\\mathcal\{M\}\_\{t\},\\mathcal\{M\}\_\{s\}\)\\leq\\alpha\+\\epsilon\.

Intuition behind proposition and assumptions\.Proposition[1](https://arxiv.org/html/2610.12837#Thmproposition1)highlights two properties of factual\-counterfactual supervision that govern teacher\-student boundary alignment: the magnitude of the counterfactual perturbations \(α\\alpha\) and the coverage of the context\-response boundary \(ϵ\\epsilon\)\. First, each factual\-counterfactual pair provides supervision across a local change in the teacher’s response\. Importantly, when the factual sample is already close to the teacher’s decision boundary, a smaller contextual perturbation is sufficient to cross the boundary and induce the alternative responseA′A^\{\\prime\}\. Therefore, starting from boundary\-near factual samples leads to a smaller upper bound on the perturbationα\\alpha, tightening the bound\. Although those samples are difficult under factual\-only supervision, their proximity to the boundary makes them useful starting points for constructing local counterfactual supervision\. Second, the boundary coverage condition captures how well the induced crossing points cover both the context\-response boundaries\. Better coverage corresponds to a smallerϵ\\epsilon, tightening the bound\. Thus, we provide a geometric characterization of how local counterfactual perturbations and boundary coverage relate to teacher\-student boundary deviation\.

The analysis above also assumes that the student model agrees with the teacher model on the paired factual and counterfactual samples \(Assumption \(2\) Exact answer generation\)\. This idealized condition captures the objective of distillation, where the student is trained to reproduce the teacher’s behavior on the available supervision\. More importantly, the analysis also relies on the generated counterfactual contextC′C^\{\\prime\}to induce the intended alternative answerA′A^\{\\prime\}\. In practice, this may not be guaranteed due to inherently stochastic and unpredictable generative behavior of LLMs[Holtzman et al\. \(2019\)](https://arxiv.org/html/2610.12837#bib.bib18)\. The teacher may fail to produceA′A^\{\\prime\}, or conditioning onC′C^\{\\prime\}may provide only limited additional predictive benefit forA′A^\{\\prime\}\. For realization of this objective in our implementation, we therefore introduce a new metricContext Following Difficulty \(CFD\)in the following section to validate generated counterfactuals before they are used for knowledge distillation\.

Figure 4:Illustrative examples of CFD\-based counterfactual evaluation from the NQ dataset\.The four cases demonstrate how Answer Agreement \(AA\) and Contextual Loss Ratio \(CLR\) complement each other in evaluating generated counterfactual candidates\.
### 2\.3Context Following Difficulty \(CFD\) for counterfactual validation

Generated counterfactual samples should be evaluated using the teacher to determine whether they induce the intended alternative answerA′A^\{\\prime\}rather than another response\. Since exact matching can be unreliable for generative tasks\([Holtzman et al\., 2019](https://arxiv.org/html/2610.12837#bib.bib18)\), we use token\-level F1\([Rajpurkar et al\., 2016](https://arxiv.org/html/2610.12837#bib.bib3)\)to measure agreement between the teacher output andA′A^\{\\prime\}\. However, answer agreement alone is insufficient, as generating the intended answer does not necessarily indicate that theC′C^\{\\prime\}provides strong evidence for that prediction\. A high\-quality counterfactual should therefore not only induce the intended answerA′A^\{\\prime\}, but also makeA′A^\{\\prime\}substantially more predictable when conditioned onC′C^\{\\prime\}than from theQQalone\. To capture these two properties, we introduce a new metricContext Following Difficulty \(CFD\), which combinesAnswer Agreement \(AA\)with theContextual Loss Ratio \(CLR\)\. First,AAθ\\mathrm\{AA\}\_\{\\theta\}measures whether the teacher generates the intended alternative answerA′A^\{\\prime\}under the counterfactual contextC′C^\{\\prime\}:

AAθ​\(Q,C′,A′\)=F1⁡\(A^θ​\(Q,C′\),A′\),\\mathrm\{AA\}\_\{\\theta\}\(Q,C^\{\\prime\},A^\{\\prime\}\)=\\mathrm\{F1\}\\big\(\\hat\{A\}\_\{\\theta\}\(Q,C^\{\\prime\}\),A^\{\\prime\}\\big\),\(1\)whereA^θ​\(Q,C′\)\\hat\{A\}\_\{\\theta\}\(Q,C^\{\\prime\}\)is the answer generated by the teacher givenQQandC′C^\{\\prime\}\. A higherAAθ\\mathrm\{AA\}\_\{\\theta\}indicates stronger agreement with the intended answerA′A^\{\\prime\}\. However, high answer agreement alone does not establish that the counterfactual context plays a role in predictingA′A^\{\\prime\}\. Inspired by the Instruction Following Difficulty \(IFD\) metric\([Li et al\., 2024](https://arxiv.org/html/2610.12837#bib.bib39)\), we define theContextual Loss Ratio \(CLR\)as:

CLRθ​\(Q,C′,A′\)=Lθ​\(A′∣Q,C′\)Lθ​\(A′∣Q\)\.\\mathrm\{CLR\}\_\{\\theta\}\(Q,C^\{\\prime\},A^\{\\prime\}\)=\\frac\{L\_\{\\theta\}\(A^\{\\prime\}\\mid Q,C^\{\\prime\}\)\}\{L\_\{\\theta\}\(A^\{\\prime\}\\mid Q\)\}\.\(2\)A lowCLRθ\\mathrm\{CLR\}\_\{\\theta\}indicates that conditioning onC′C^\{\\prime\}substantially reduces the teacher’s loss forA′A^\{\\prime\}relative to conditioning onQQalone, i\.e\.,C′C^\{\\prime\}plays a strong role in predicting the intended alternative answer\. Conversely, a higherCLRθ\\mathrm\{CLR\}\_\{\\theta\}indicates that providingC′C^\{\\prime\}yields only a limited reduction in the loss ofA′A^\{\\prime\}, suggesting weaker dependence of the target prediction onC′C^\{\\prime\}\. Thus,CLRθ\\mathrm\{CLR\}\_\{\\theta\}measures how stronglyC′C^\{\\prime\}contributes to the prediction of the teacher modelA′A^\{\\prime\}\.

Combining these two measures, we finally define ourContext Following Difficulty \(CFD\)score as:

CFDθ​\(Q,C′,A′\)=CLRθ​\(Q,C′,A′\)AAθ​\(Q,C′,A′\)\+δ,\\mathrm\{CFD\}\_\{\\theta\}\(Q,C^\{\\prime\},A^\{\\prime\}\)=\\frac\{\\mathrm\{CLR\}\_\{\\theta\}\(Q,C^\{\\prime\},A^\{\\prime\}\)\}\{\\mathrm\{AA\}\_\{\\theta\}\(Q,C^\{\\prime\},A^\{\\prime\}\)\{\+\}\\delta\},\(3\)
Algorithm 1CofFeQA0:

𝒟\\mathcal\{D\}, teacher

θt\\theta\_\{t\}, student

θs\\theta\_\{s\}, epochs

EE, learning rate

η\\eta, weight

α\\alpha, CFD threshold

τ\\tau, few\-shot budget

kk,

𝒟c​f=ϕ,𝒟f=ϕ\\mathcal\{D\}\_\{cf\}\{=\}\\phi,\\mathcal\{D\}\_\{f\}\{=\}\\phi
1:while

\|𝒟c​f\|<k/2\|\\mathcal\{D\}\_\{cf\}\|<k/2do

2:Pick

\(Q,C,A\)∈𝒟\(Q,C,A\)\\in\\mathcal\{D\}
3:

\(Q,C′,A′\)←GenCF​\(Q,C,A\)\(Q,C^\{\\prime\},A^\{\\prime\}\)\\leftarrow\\text\{GenCF\}\(Q,C,A\)
4:if

CFDθt​\(Q,C′,A′\)≤τ\\text\{CFD\}\_\{\\theta\_\{t\}\}\(Q,C^\{\\prime\},A^\{\\prime\}\)\\leq\\tauthen

5:

𝒟c​f←𝒟c​f∪\{\(Q,C′,A′\)\}\\mathcal\{D\}\_\{cf\}\\leftarrow\\mathcal\{D\}\_\{cf\}\\cup\\\{\(Q,C^\{\\prime\},A^\{\\prime\}\)\\\}
6:

𝒟f←𝒟f∪\{\(Q,C,A\)\}\\mathcal\{D\}\_\{f\}\\leftarrow\\mathcal\{D\}\_\{f\}\\cup\\\{\(Q,C,A\)\\\}
7:endif

8:endwhile

9:for

e=1e=1to

EEdo

10:for all

\(Q,C,A\)∈𝒟f∪𝒟c​f\(Q,C,A\)\\in\\mathcal\{D\}\_\{f\}\\cup\\mathcal\{D\}\_\{cf\}do

11:

ℒCE←CE​\(θs​\(Q,C\),A\)\\mathcal\{L\}\_\{\\text\{CE\}\}\\leftarrow\\text\{CE\}\(\\theta\_\{s\}\(Q,C\),A\)
12:

ℒKD←KL\(θt\(Q,C\)∥θs\(Q,C\)\)\\mathcal\{L\}\_\{\\text\{KD\}\}\\leftarrow\\text\{KL\}\(\\theta\_\{t\}\(Q,C\)\{\\parallel\}\\theta\_\{s\}\(Q,C\)\)
13:

ℒ←α​ℒCE\+\(1−α\)​ℒKD\\mathcal\{L\}\\leftarrow\\alpha\\mathcal\{L\}\_\{\\text\{CE\}\}\+\(1\-\\alpha\)\\mathcal\{L\}\_\{\\text\{KD\}\}
14:Update

θs←θs−η​∇θsℒ\\theta\_\{s\}\\leftarrow\\theta\_\{s\}\-\\eta\\nabla\_\{\\theta\_\{s\}\}\\mathcal\{L\}
15:endfor

16:endfor

17:returndistilled student

θs\\theta\_\{s\}

whereδ\>0\\delta\>0is a small constant for numerical stability\. We prefer lowCLRθ\\mathrm\{CLR\}\_\{\\theta\}and highAAθ\\mathrm\{AA\}\_\{\\theta\}\. A lowerCFDθ\\mathrm\{CFD\}\_\{\\theta\}score favors counterfactuals for which the teacher both generates an answer close to the intended targetA′A^\{\\prime\}and relies more strongly on the counterfactual context when predictingA′A^\{\\prime\}\. Figure[4](https://arxiv.org/html/2610.12837#S2.F4)illustrates the complementary roles ofAAθ\\mathrm\{AA\}\_\{\\theta\}andCLRθ\\mathrm\{CLR\}\_\{\\theta\}in four representative cases\. Cases \(a\) and \(b\) show that a counterfactual may fail to induce the intended answer even when the modified context contains or supports it\. Case \(c\) has highAAθ\\mathrm\{AA\}\_\{\\theta\}but highCLRθ\\mathrm\{CLR\}\_\{\\theta\}, indicating limited additional predictive benefit fromC′C^\{\\prime\}\. In contrast, case \(c\) represents a valid counterfactual with a lowCFDθ\\mathrm\{CFD\}\_\{\\theta\}\. Together, these examples show why bothAAθ\\mathrm\{AA\}\_\{\\theta\}andCLRθ\\mathrm\{CLR\}\_\{\\theta\}are necessary for validation\. The two components ofCFDθ\\mathrm\{CFD\}\_\{\\theta\}therefore capture complementary properties of a valid and informative counterfactual\.AAθ\\mathrm\{AA\}\_\{\\theta\}verifies the intended answerA′A^\{\\prime\}, whileCLRθ\\mathrm\{CLR\}\_\{\\theta\}measures how strongly the modified context contributes to predictingA′A^\{\\prime\}\. Our algorithm is provided in Algorithm[1](https://arxiv.org/html/2610.12837#alg1)\(also in Figure[1](https://arxiv.org/html/2610.12837#S1.F1)\)\.

## 3Experiments

Datasets\.We evaluateCofFeQAon eight open\-domain question answering benchmarks\. The five in\-domain benchmarks are Natural Questions \(NQ\)\([Kwiatkowski et al\., 2019](https://arxiv.org/html/2610.12837#bib.bib78)\), SQuAD\([Rajpurkar et al\., 2016](https://arxiv.org/html/2610.12837#bib.bib3)\), NewsQA\([Trischler et al\., 2017](https://arxiv.org/html/2610.12837#bib.bib79)\), TriviaQA\([Joshi et al\., 2017](https://arxiv.org/html/2610.12837#bib.bib80)\), and HotpotQA\([Yang et al\., 2018](https://arxiv.org/html/2610.12837#bib.bib81)\)\. The three out\-of\-domain benchmarks are DROP\([Dua et al\., 2019](https://arxiv.org/html/2610.12837#bib.bib82)\), DuoRC\([Saha et al\., 2018](https://arxiv.org/html/2610.12837#bib.bib83)\), and RelationExtraction\([Levy et al\., 2017](https://arxiv.org/html/2610.12837#bib.bib14)\)\. Additional details about the datasets used in our experimental evaluations are provided in Appendix[F\.1](https://arxiv.org/html/2610.12837#A6.SS1)

Models\.We evaluate our method using two model families with different architectures: the encoder\-decoder model T5\([Raffel et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib9)\)and the decoder\-only model Qwen2\.5\([Bai et al\., 2023](https://arxiv.org/html/2610.12837#bib.bib4)\)\. In the T5 setting, T5\-Large \(770M parameters\) serves as the teacher, while T5\-Base \(220M parameters\) serves as the student\. In the Qwen2\.5 setting, Qwen2\.5\-7B\-IT serves as the teacher, while Qwen2\.5\-1\.5B\-IT serves as the student\. Full\-training details are in Appendix[F\.2](https://arxiv.org/html/2610.12837#A6.SS2)\.

Table 1:In\-domain F1 score comparison acrosskk\-shots for T5 and Qwen2\.5 models using baselines\. Mean F1 scores are reported with standard deviations as subscripts \(M​e​a​ns​t​dMean\_\{std\}\)\. Bold values indicate the better\-performing method \(CofFeQAvs\. baseline\)\. Blue shading highlights aggregate results\.Baselines\.We compareCofFeQAagainst three distillation baselines: \(i\) Standard Knowledge Distillation \(KD\)\([Hinton et al\., 2015](https://arxiv.org/html/2610.12837#bib.bib11)\), \(ii\) Sequence\-Level Knowledge Distillation \(seqKD\)\([Kim and Rush, 2016](https://arxiv.org/html/2610.12837#bib.bib40)\), and \(iii\) Layer\-Wise Distillation \(LWD\)\([Jiao et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib2)\)\. All methods use the same few\-shot splits for fair comparison\. Additional details of the baselines and implementation details are provided in Appendix[F\.3](https://arxiv.org/html/2610.12837#A6.SS3)\.

Setup\.Following prior work\([Hamman et al\., 2025](https://arxiv.org/html/2610.12837#bib.bib19)\), we first fine\-tune the teacher model on the full training dataset\. For each few\-shot budgetk∈\{16,32,64,128,256\}k\\in\\\{16,32,64,128,256\\\}, we rank factual samples by their teacher prediction margins, using margin sampling\([Scheffer et al\., 2001](https://arxiv.org/html/2610.12837#bib.bib5)\)\. The factual\-only distillation baseline uses thekksamples with the smallest margins\.CofFeQAinstead uses thek/2k/2factual samples with the smallest margins together with their correspondingk/2k/2counterfactuals, maintaining the same total few\-shot budgetkk\. We evaluate accuracy using EM and F1 against the gold answers and teacher\-student agreement using EM and F1 between their generated answers\. All results are reported as the mean and standard deviation over five random seeds\.

### 3\.1Main Experimental Results

CofFeQAoutperforms the baselines\.As shown in Table[1](https://arxiv.org/html/2610.12837#S3.T1),CofFeQAimproves performance across all three distillation baselines under the T5 setting\. When averaged across all datasets andkk\-shot settings,CofFeQAimproves the F1 score from66\.4266\.42to69\.2769\.27with standard KD, from66\.3766\.37to69\.2869\.28with LWD, and from68\.6368\.63to70\.5270\.52with SeqKD\. Moreover,CofFeQAexhibits lower standard deviations than the corresponding baselines in many settings, indicating more stable performance across seeds\. These results demonstrate that counterfactual supervision provides consistent benefits over the baselines\. Detailed EM scores are provided in Table[8](https://arxiv.org/html/2610.12837#A8.T8)in Appendix[H\.1](https://arxiv.org/html/2610.12837#A8.SS1)\.

CofFeQAgeneralizes across different model families\.CofFeQAimproves standard KD across both the encoder\-decoder T5 and decoder\-only Qwen2\.5 model families, where the teachers are approximately3\.5×3\.5\\timesand4\.7×4\.7\\timeslarger than their respective students\. Under Qwen2\.5,CofFeQAimproves the average F1 score from66\.2166\.21to67\.7267\.72with standard KD, from66\.7766\.77to68\.0068\.00with LWD, and from67\.16to67\.88with SeqKD\. These results indicate that the benefits ofCofFeQAextend across different model architectures and scales\.

Students mimic the teacher more closely underCofFeQA\.To examine how closely the student follows the teacher, we measure teacher\-student agreement using EM and F1\. As shown in Figure[5](https://arxiv.org/html/2610.12837#S3.F5),CofFeQAgenerally achieves higher teacher\-student agreement than standard KD across datasets andkk\-shot settings\. Averaged across all benchmarks and settings,CofFeQAimproves F1 agreement under standard KD from68\.5168\.51to71\.5071\.50under T5, with a similar trend for Qwen2\.5\. These results suggest that counterfactual supervision helps the student more closely reproduce the teacher’s output behavior\. Detailed results and additional radar plots are provided in Appendix[H\.2](https://arxiv.org/html/2610.12837#A8.SS2)\.

Figure 5:Teacher\-student F1 agreement of standard KD andCofFeQAacross datasets andkk\-shot settings using T5\.CofFeQAgenerally achieves higher agreement than the baseline\. Spokes denotekk\-shot settings, and radial axis F1 score ranges are adjusted independently by dataset for visualization\.CofFeQAgeneralizes well to out\-of\-domain datasets\.To evaluate out\-of\-domain generalization, we use Natural Questions for distillation and evaluate the resulting students on three out\-of\-domain test sets: DROP, DuoRC, and RelationExtraction\. As shown in Table[2](https://arxiv.org/html/2610.12837#S3.T2),CofFeQAimproves over standard KD when averaged across datasets andkk\-shot settings for both model families\. For T5 and Qwen2\.5,CofFeQAimproves the overall F1 score over standard KD from53\.3553\.35to54\.5254\.52and from57\.0957\.09to57\.7757\.77, respectively\. These results indicate that the benefits of counterfactual supervision extend beyond the in\-domain datasets used for distillation\. Detailed EM scores are in Appendix[H\.3](https://arxiv.org/html/2610.12837#A8.SS3)\.

Table 2:Out\-of\-domain F1 score comparison acrosskk\-shots for T5 and Qwen2\.5 models using KD\. Mean F1 scores are reported with standard deviations as subscripts \(M​e​a​ns​t​dMean\_\{std\}\)\. Bold values indicate the better\-performing method \(CofFeQAvs\. KD\)\. Blue shading highlights aggregate results\.CofFeQAimproves performance under modified contextual evidence\.Following[Longpre et al\. \(2021\)](https://arxiv.org/html/2610.12837#bib.bib23), we replace each test answer in the context with a randomly selected answer of the same entity type and evaluate against the modified target\. As shown in Table[13](https://arxiv.org/html/2610.12837#A8.T13),CofFeQAimproves average F1 agreement over standard KD from66\.7266\.72to69\.5569\.55for T5 and from49\.2349\.23to53\.5553\.55for Qwen2\.5\. This suggests thatCofFeQAbetter adapts predictions to modified contextual evidence when it differs from pre\-trained knowledge\. Corresponding EM results are provided in Appendix[H\.4](https://arxiv.org/html/2610.12837#A8.SS4)\.

### 3\.2Ablation Studies

We conduct three ablation studies to examine different components ofCofFeQA: \(i\) We compareCofFeQAwith a variant that uses generated counterfactuals without CFD verification to isolate the contribution of CFD; \(ii\) We vary the CFD thresholdτ\\tauto analyze its effect on downstream performance and teacher\-student agreement; and \(iii\) We measure counterfactual generation costs, training time and energy consumption across baselines to analyze the computational cost ofCofFeQA\. Full ablation results and experimental details are in Appendix[G](https://arxiv.org/html/2610.12837#A7)\.

## 4Conclusion

In this paper, we present a counterfactual\-guided knowledge distillation method for few\-shot open\-book question answering\. Motivated by the difficulty of distilling with factual samples near the teacher’s decision boundary, our approach uses factual\-counterfactual pairs to provide additional supervision around these challenging regions\. We further introduce CFD to validate counterfactuals before distillation\. Experiments across multiple datasets and LLM architectures demonstrate improved knowledge transfer under limited supervision\. Limitations and future directions are discussed in Appendix[B](https://arxiv.org/html/2610.12837#A2)\. Overall, our findings highlight the potential of counterfactual supervision for improving knowledge transfer in generative QA under limited supervision in the few\-shot setting\.

### Reproducibility statement

To support reproducibility, we provide detailed descriptions of the datasets, training procedures, evaluation settings, and baseline implementations in Appendix[F](https://arxiv.org/html/2610.12837#A6)\. Specifically, Appendix[F\.1](https://arxiv.org/html/2610.12837#A6.SS1)describes the datasets and experimental splits, Appendix[F\.2](https://arxiv.org/html/2610.12837#A6.SS2)provides details of teacher model fine\-tuning and evaluation, and Appendix[F\.3](https://arxiv.org/html/2610.12837#A6.SS3)describes the baseline formulations and implementation details\. We also provide the assumptions and complete proof of our geometric analysis in Appendix[E](https://arxiv.org/html/2610.12837#A5)\.

## References

- Achiamet al\.\(2023\)J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Baiet al\.\(2023\)J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p7.1),[§3](https://arxiv.org/html/2610.12837#S3.p2.1)\.
- Balashankaret al\.\(2023\)A\. Balashankar, X\. Wang, Y\. Qin, B\. Packer, N\. Thain, E\. H\. Chi, J\. Chen, and A\. BeutelImproving classifier robustness through active generative counterfactual data augmentation\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 127–139\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Bhattacharjeeet al\.\(2024\)A\. Bhattacharjee, R\. Moraffah, J\. Garland, and H\. LiuZero\-shot llm\-guided counterfactual generation: a case study on nlp model evaluation\.In2024 IEEE International Conference on Big Data \(BigData\),pp\. 1243–1248\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1),[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Calderonet al\.\(2022\)N\. Calderon, E\. Ben\-David, A\. Feder, and R\. ReichartDoCoGen: domain counterfactual generation for low resource domain adaptation\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7727–7746\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Chenet al\.\(2021\)H\. Chen, R\. Xia, and J\. YuReinforced counterfactual data augmentation for dual sentiment classification\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 269–278\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Chenet al\.\(2025\)Y\. Chen, X\. Xu, F\. de Hoog, J\. Liu, and S\. WangMedium\-difficulty samples constitute smoothed decision boundary for knowledge distillation on pruned datasets\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 54106–54123\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Chenet al\.\(2023a\)Z\. Chen, Q\. Gao, A\. Bosselut, A\. Sabharwal, and K\. RichardsonDISCO: distilling counterfactuals with large language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5514–5528\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1)\.
- Chenet al\.\(2023b\)Z\. Chen, L\. Hu, W\. Li, Y\. Shao, and L\. NieCausal intervention and counterfactual reasoning for multi\-modal fake news detection\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 627–638\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Chunget al\.\(2024\)H\. W\. Chung, L\. Hou, S\. Longpre, B\. Zoph, Y\. Tay, W\. Fedus, Y\. Li, X\. Wang, M\. Dehghani, S\. Brahma,et al\.Scaling instruction\-finetuned language models\.Journal of Machine Learning Research25\(70\),pp\. 1–53\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Dissanayake and Dutta \(2024\)P\. Dissanayake and S\. DuttaModel reconstruction using counterfactual explanations: a perspective from polytope theory\.Advances in Neural Information Processing Systems37,pp\. 83397–83429\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Dissanayake and Dutta \(2025\)P\. Dissanayake and S\. DuttaTabdistill: distilling transformers into neural nets for few\-shot tabular classification\.arXiv preprint arXiv:2511\.05704\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1),[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Dissanayake and Dutta \(2026\)P\. Dissanayake and S\. DuttaFoundations of efficient model reconstruction using counterfactual explanations\.IEEE Journal on Selected Areas in Information Theory\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Dissanayakeet al\.\(2024\)P\. Dissanayake, F\. Hamman, B\. Halder, I\. Sucholutsky, Q\. Zhang, and S\. DuttaQuantifying knowledge distillation using partial information decomposition\.arXiv preprint arXiv:2411\.07483\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1),[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Dixitet al\.\(2022\)T\. Dixit, B\. Paranjape, H\. Hajishirzi, and L\. ZettlemoyerCore: a retrieve\-then\-edit framework for counterfactual data generation\.InFindings of the Association for Computational Linguistics: EMNLP 2022,pp\. 2964–2984\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p2.1)\.
- Duaet al\.\(2019\)D\. Dua, Y\. Wang, P\. Dasigi, G\. Stanovsky, S\. Singh, and M\. GardnerDROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 2368–2378\.Cited by:[§F\.1](https://arxiv.org/html/2610.12837#A6.SS1.p1.1),[§3](https://arxiv.org/html/2610.12837#S3.p1.1)\.
- Egeaet al\.\(2025\)D\. Egea, B\. Halder, and S\. DuttaVision: robust and interpretable code vulnerability detection leveraging counterfactual augmentation\.InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society,Vol\.8,pp\. 812–823\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1),[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Fanget al\.\(2026\)L\. Fang, X\. Yu, J\. Cai, Y\. Chen, S\. Wu, Z\. Liu, Z\. Yang, H\. Lu, X\. Gong, Y\. Liu,et al\.Knowledge distillation and dataset distillation of large language models: emerging trends, challenges, and future directions\.Artificial Intelligence Review59\(1\),pp\. 17\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Fischet al\.\(2019\)A\. Fisch, A\. Talmor, R\. Jia, M\. Seo, E\. Choi, and D\. ChenMRQA 2019 shared task: evaluating generalization in reading comprehension\.InProceedings of the 2nd workshop on machine reading for question answering,pp\. 1–13\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p7.1)\.
- Fuet al\.\(2026\)Y\. Fu, F\. Hamman, and S\. DuttaT\-shirt: token\-selective hierarchical data selection for instruction tuning\.Advances in Neural Information Processing Systems38,pp\. 113932–113958\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Gardneret al\.\(2020\)M\. Gardner, Y\. Artzi, V\. Basmov, J\. Berant, B\. Bogin, S\. Chen, P\. Dasigi, D\. Dua, Y\. Elazar, A\. Gottumukkala,et al\.Evaluating models’ local decision boundaries via contrast sets\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 1307–1323\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1),[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Gouet al\.\(2021\)J\. Gou, B\. Yu, S\. J\. Maybank, and D\. TaoKnowledge distillation: a survey\.International Journal of Computer Vision129\(6\),pp\. 1789–1819\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Guet al\.\(2024\)Y\. Gu, L\. Dong, F\. Wei, and M\. HuangMinillm: knowledge distillation of large language models\.InThe twelfth international conference on learning representations,Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1)\.
- Guoet al\.\(2023\)W\. Guo, Q\. Gong, Y\. Rao, and H\. LaiCounterfactual multihop qa: a cause\-effect approach for reducing disconnected reasoning\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4214–4226\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p2.1),[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Hammanet al\.\(2025\)F\. Hamman, P\. Dissanayake, Y\. Fu, and S\. DuttaFew\-shot knowledge distillation of llms with counterfactual explanations\.arXiv preprint arXiv:2510\.21631\.Cited by:[Appendix E](https://arxiv.org/html/2610.12837#A5.p2.2),[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1),[§1](https://arxiv.org/html/2610.12837#S1.p1.1),[§3](https://arxiv.org/html/2610.12837#S3.p4.1)\.
- Hammanet al\.\(2024\)F\. Hamman, E\. Noorani, S\. Mishra, D\. Magazzeni, and S\. DuttaRobust algorithmic recourse under model multiplicity with probabilistic guarantees\.IEEE Journal on Selected Areas in Information Theory5,pp\. 357–368\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[1st item](https://arxiv.org/html/2610.12837#A6.I2.i1.p1.1),[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1),[§1](https://arxiv.org/html/2610.12837#S1.p1.1),[§3](https://arxiv.org/html/2610.12837#S3.p3.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.Training compute\-optimal large language models\.arXiv preprint arXiv:2203\.1555610\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Holtzmanet al\.\(2019\)A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. ChoiThe curious case of neural text degeneration\.arXiv preprint arXiv:1904\.09751\.Cited by:[§2\.2](https://arxiv.org/html/2610.12837#S2.SS2.p6.1),[§2\.3](https://arxiv.org/html/2610.12837#S2.SS3.p1.1)\.
- Hsiehet al\.\(2023\)C\. Hsieh, C\. Li, C\. Yeh, H\. Nakhost, Y\. Fujii, A\. Ratner, R\. Krishna, C\. Lee, and T\. PfisterDistilling step\-by\-step\! outperforming larger language models with less training data and smaller model sizes\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 8003–8017\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1)\.
- Huanget al\.\(2025\)L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin,et al\.A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems43\(2\),pp\. 1–55\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p2.1)\.
- Jiaoet al\.\(2020\)X\. Jiao, Y\. Yin, L\. Shang, X\. Jiang, X\. Chen, L\. Li, F\. Wang, and Q\. LiuTinybert: distilling bert for natural language understanding\.InFindings of the association for computational linguistics: EMNLP 2020,pp\. 4163–4174\.Cited by:[2nd item](https://arxiv.org/html/2610.12837#A6.I2.i2.p1.1),[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1),[§1](https://arxiv.org/html/2610.12837#S1.p1.1),[§3](https://arxiv.org/html/2610.12837#S3.p3.1)\.
- Joshiet al\.\(2017\)M\. Joshi, E\. Choi, D\. S\. Weld, and L\. ZettlemoyerTriviaqa: a large scale distantly supervised challenge dataset for reading comprehension\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1601–1611\.Cited by:[§F\.1](https://arxiv.org/html/2610.12837#A6.SS1.p1.1),[§3](https://arxiv.org/html/2610.12837#S3.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Kaushiket al\.\(2019\)D\. Kaushik, E\. Hovy, and Z\. C\. LiptonLearning the difference that makes a difference with counterfactually\-augmented data\.arXiv preprint arXiv:1909\.12434\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1),[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Kaushiket al\.\(2020\)D\. Kaushik, A\. Setlur, E\. Hovy, and Z\. C\. LiptonExplaining the efficacy of counterfactually augmented data\.arXiv preprint arXiv:2010\.02114\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Khashabiet al\.\(2020\)D\. Khashabi, T\. Khot, and A\. SabharwalMore bang for your buck: natural perturbation for robust question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 163–170\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p2.1)\.
- Kim and Rush \(2016\)Y\. Kim and A\. M\. RushSequence\-level knowledge distillation\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 1317–1327\.Cited by:[3rd item](https://arxiv.org/html/2610.12837#A6.I2.i3.p1.1),[§F\.3](https://arxiv.org/html/2610.12837#A6.SS3.p5.1),[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1),[§3](https://arxiv.org/html/2610.12837#S3.p3.1)\.
- Köksalet al\.\(2023\)A\. Köksal, R\. Aksitov, and C\. ChangHallucination augmented recitations for language models\.arXiv preprint arXiv:2311\.07424\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Kwiatkowskiet al\.\(2019\)T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee,et al\.Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 453–466\.Cited by:[§F\.1](https://arxiv.org/html/2610.12837#A6.SS1.p1.1),[§3](https://arxiv.org/html/2610.12837#S3.p1.1)\.
- Levyet al\.\(2017\)O\. Levy, M\. Seo, E\. Choi, and L\. ZettlemoyerZero\-shot relation extraction via reading comprehension\.InProceedings of the 21st conference on computational natural language learning \(CoNLL 2017\),pp\. 333–342\.Cited by:[§F\.1](https://arxiv.org/html/2610.12837#A6.SS1.p1.1),[§3](https://arxiv.org/html/2610.12837#S3.p1.1)\.
- Liet al\.\(2024\)M\. Li, Y\. Zhang, Z\. Li, J\. Chen, L\. Chen, N\. Cheng, J\. Wang, T\. Zhou, and J\. XiaoFrom quantity to quality: boosting llm performance with self\-guided data selection for instruction tuning\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 7602–7635\.Cited by:[§2\.3](https://arxiv.org/html/2610.12837#S2.SS3.p1.2)\.
- Lianget al\.\(2023\)C\. Liang, S\. Zuo, Q\. Zhang, P\. He, W\. Chen, and T\. ZhaoLess is more: task\-aware layer\-wise distillation for language model compression\.InInternational Conference on Machine Learning,pp\. 20852–20867\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Liuet al\.\(2024\)J\. Liu, C\. Zhang, J\. Guo, Y\. Zhang, H\. Que, K\. Deng, Z\. Bai, J\. Liu, G\. Zhang, J\. Wang,et al\.Ddk: distilling domain knowledge for efficient large language models\.Advances in Neural Information Processing Systems37,pp\. 98297–98319\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Liuet al\.\(2021\)Q\. Liu, M\. Kusner, and P\. BlunsomCounterfactual data augmentation for neural machine translation\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 187–197\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Longpreet al\.\(2021\)S\. Longpre, K\. Perisetla, A\. Chen, N\. Ramesh, C\. DuBois, and S\. SinghEntity\-based knowledge conflicts in question answering\.InProceedings of the 2021 conference on empirical methods in natural language processing,pp\. 7052–7063\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p2.1),[§3\.1](https://arxiv.org/html/2610.12837#S3.SS1.p5.1)\.
- McAleese and Keane \(2024\)S\. McAleese and M\. KeaneA comparative analysis of counterfactual explanation methods for text classifiers\.arXiv preprint arXiv:2411\.02643\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1),[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Mishraet al\.\(2021\)S\. Mishra, S\. Dutta, J\. Long, and D\. MagazzeniA survey on the robustness of feature importance and counterfactual explanations\.arXiv preprint arXiv:2111\.00358\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Neemanet al\.\(2023\)E\. Neeman, R\. Aharoni, O\. Honovich, L\. Choshen, I\. Szpektor, and O\. AbendDisentqa: disentangling parametric and contextual knowledge with counterfactual question answering\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10056–10070\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p2.1)\.
- Nooraniet al\.\(2025\)E\. Noorani, P\. Dissanayake, F\. Hamman, and S\. DuttaCounterfactual explanations for model ensembles using entropic risk measures\.arXiv preprint arXiv:2503\.07934\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- OpenAI \(2024\)OpenAIGPT\-4o mini: Advancing Cost\-Efficient Intelligence\.Note:[https://openai\.com/index/gpt\-4o\-mini\-advancing\-cost\-efficient\-intelligence/](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/)Accessed: 2026\-09\-20Cited by:[Appendix D](https://arxiv.org/html/2610.12837#A4.p1.1),[§2\.1](https://arxiv.org/html/2610.12837#S2.SS1.p1.1)\.
- Ouet al\.\(2022\)J\. Ou, J\. Zhang, Y\. Feng, and J\. ZhouCounterfactual data augmentation via perspective transition for open\-domain dialogues\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 1635–1648\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Paranjapeet al\.\(2022\)B\. Paranjape, M\. Lamm, and I\. TenneyRetrieval\-guided counterfactual generation for qa\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1670–1686\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p2.1),[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Raffelet al\.\(2020\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. LiuExploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1),[§1](https://arxiv.org/html/2610.12837#S1.p7.1),[§3](https://arxiv.org/html/2610.12837#S3.p2.1)\.
- Rajagopalet al\.\(2022\)D\. Rajagopal, S\. Shakeri, C\. N\. d\. Santos, E\. Hovy, and C\. ChangCounterfactual data augmentation improves factuality of abstractive summarization\.arXiv preprint arXiv:2205\.12416\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Rajpurkaret al\.\(2016\)P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. LiangSquad: 100,000\+ questions for machine comprehension of text\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 2383–2392\.Cited by:[§F\.1](https://arxiv.org/html/2610.12837#A6.SS1.p1.1),[§2\.3](https://arxiv.org/html/2610.12837#S2.SS3.p1.1),[§3](https://arxiv.org/html/2610.12837#S3.p1.1)\.
- Rockafellar and Wets \(1998\)R\. T\. Rockafellar and R\. J\. WetsVariational analysis\.Springer\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p5.1),[§2\.2](https://arxiv.org/html/2610.12837#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2610.12837#S2.SS2.p2.1.1)\.
- Rosset al\.\(2021\)A\. Ross, A\. Marasović, and M\. E\. PetersExplaining nlp models via minimal contrastive editing \(mice\)\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,pp\. 3840–3852\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1),[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Rosset al\.\(2022\)A\. Ross, T\. Wu, H\. Peng, M\. E\. Peters, and M\. GardnerTailor: generating and perturbing text with semantic controls\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3194–3213\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p2.1)\.
- Saadi and Wang \(2025\)K\. Saadi and D\. WangTASKD\-llm: task\-aware selective knowledge distillation for llms\.InICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy,Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Sachdevaet al\.\(2024\)R\. Sachdeva, M\. Tutek, and I\. GurevychCATfOOD: counterfactual augmented training for improving out\-of\-domain performance and calibration\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1876–1898\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Sahaet al\.\(2018\)A\. Saha, R\. Aralikatte, M\. M\. Khapra, and K\. SankaranarayananDuoRC: towards complex language understanding with paraphrased reading comprehension\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1683–1693\.Cited by:[§F\.1](https://arxiv.org/html/2610.12837#A6.SS1.p1.1),[§3](https://arxiv.org/html/2610.12837#S3.p1.1)\.
- Sanhet al\.\(2019\)V\. Sanh, L\. Debut, J\. Chaumond, and T\. WolfDistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.arXiv preprint arXiv:1910\.01108\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1),[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Schefferet al\.\(2001\)T\. Scheffer, C\. Decomain, and S\. WrobelActive hidden markov models for information extraction\.InInternational symposium on intelligent data analysis,pp\. 309–318\.Cited by:[Appendix C](https://arxiv.org/html/2610.12837#A3.p1.1),[§2](https://arxiv.org/html/2610.12837#S2.p1.1),[§3](https://arxiv.org/html/2610.12837#S3.p4.1)\.
- Strubellet al\.\(2019\)E\. Strubell, A\. Ganesh, and A\. McCallumEnergy and policy considerations for deep learning in nlp\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 3645–3650\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Sunet al\.\(2019\)S\. Sun, Y\. Cheng, Z\. Gan, and J\. LiuPatient knowledge distillation for bert model compression\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\),pp\. 4323–4332\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Sutantoet al\.\(2025\)P\. Sutanto, J\. Santoso, E\. I\. Setiawan, and A\. P\. WibawaLLM distillation for efficient few\-shot multiple choice question answering\.\.InEMNLP \(Findings\),pp\. 8502–8530\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1)\.
- Teamet al\.\(2024\)G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé,et al\.Gemma 2: improving open language models at a practical size\.arXiv preprint arXiv:2408\.00118\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1)\.
- Tokpo and Calders \(2024\)E\. K\. Tokpo and T\. CaldersFairflow: an automated approach to model\-based counterfactual data augmentation for nlp\.InJoint European Conference on Machine Learning and Knowledge Discovery in Databases,pp\. 160–176\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Trischleret al\.\(2017\)A\. Trischler, T\. Wang, X\. Yuan, J\. Harris, A\. Sordoni, P\. Bachman, and K\. SulemanNewsqa: a machine comprehension dataset\.InProceedings of the 2nd Workshop on Representation Learning for NLP,pp\. 191–200\.Cited by:[§F\.1](https://arxiv.org/html/2610.12837#A6.SS1.p1.1),[§3](https://arxiv.org/html/2610.12837#S3.p1.1)\.
- Wanget al\.\(2025\)F\. Wang, Z\. Zhang, X\. Zhang, Z\. Wu, T\. Mo, Q\. Lu, W\. Wang, R\. Li, J\. Xu, X\. Tang,et al\.A comprehensive survey of small language models in the era of large language models: techniques, enhancements, applications, collaboration with llms, and trustworthiness\.ACM Transactions on Intelligent Systems and Technology16\(6\),pp\. 1–87\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1)\.
- Wanget al\.\(2020\)W\. Wang, F\. Wei, L\. Dong, H\. Bao, N\. Yang, and M\. ZhouMinilm: deep self\-attention distillation for task\-agnostic compression of pre\-trained transformers\.Advances in neural information processing systems33,pp\. 5776–5788\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Wanget al\.\(2023a\)Y\. Wang, S\. Feng, H\. Wang, W\. Shi, V\. Balachandran, T\. He, and Y\. TsvetkovResolving knowledge conflicts in large language models\.arXiv preprint arXiv:2310\.00935\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p2.1)\.
- Wanget al\.\(2023b\)Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. HajishirziSelf\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 13484–13508\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1)\.
- Wuet al\.\(2024\)J\. Wu, L\. Yang, Z\. Wang, M\. Okumura, and Y\. ZhangCofca: a step\-wise counterfactual multi\-hop qa benchmark\.arXiv preprint arXiv:2402\.11924\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p2.1)\.
- Wuet al\.\(2021\)T\. Wu, M\. T\. Ribeiro, J\. Heer, and D\. S\. WeldPolyjuice: generating counterfactuals for explaining, evaluating, and improving models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 6707–6723\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1),[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Xuet al\.\(2024a\)R\. Xu, Z\. Qi, Z\. Guo, C\. Wang, H\. Wang, Y\. Zhang, and W\. XuKnowledge conflicts for llms: a survey\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8541–8565\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p2.1)\.
- Xuet al\.\(2024b\)X\. Xu, M\. Li, C\. Tao, T\. Shen, R\. Cheng, J\. Li, C\. Xu, D\. Tao, and T\. ZhouA survey on knowledge distillation of large language models\.arXiv preprint arXiv:2402\.13116\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1),[§1](https://arxiv.org/html/2610.12837#S1.p1.1)\.
- Yanget al\.\(2025\)C\. Yang, Y\. Zhu, W\. Lu, Y\. Wang, Q\. Chen, C\. Gao, B\. Yan, and Y\. ChenSurvey on knowledge distillation for large language models: methods, evaluation, and application\.ACM Transactions on Intelligent Systems and Technology16\(6\),pp\. 1–27\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p3.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2369–2380\.Cited by:[§F\.1](https://arxiv.org/html/2610.12837#A6.SS1.p1.1),[§3](https://arxiv.org/html/2610.12837#S3.p1.1)\.
- Youssefet al\.\(2024\)P\. Youssef, C\. Seifert, J\. Schlötterer,et al\.LLMs for generating and evaluating counterfactuals: a comprehensive study\.InFindings of the association for computational linguistics: EMNLP 2024,pp\. 14809–14824\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p1.1)\.
- Yuet al\.\(2023\)W\. Yu, M\. Jiang, P\. Clark, and A\. SabharwalIfqa: a dataset for open\-domain question answering under counterfactual presuppositions\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 8276–8288\.Cited by:[§1\.1](https://arxiv.org/html/2610.12837#S1.SS1.p2.1),[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.
- Zhouet al\.\(2023\)C\. Zhou, P\. Liu, P\. Xu, S\. Iyer, J\. Sun, Y\. Mao, X\. Ma, A\. Efrat, P\. Yu, L\. Yu,et al\.Lima: less is more for alignment\.Advances in Neural Information Processing Systems36,pp\. 55006–55021\.Cited by:[§1](https://arxiv.org/html/2610.12837#S1.p2.1)\.

## Appendix AAppendix

### Contents

- •Section[B](https://arxiv.org/html/2610.12837#A2)Limitations and Future Work
- •Section[C](https://arxiv.org/html/2610.12837#A3)KD Performance on Boundary\-Near Factual Samples
- •Section[D](https://arxiv.org/html/2610.12837#A4)Counterfactual Generation Prompts
- •Section[E](https://arxiv.org/html/2610.12837#A5)Proof of Proposition[1](https://arxiv.org/html/2610.12837#Thmproposition1)
- •Section[F](https://arxiv.org/html/2610.12837#A6)Experimental Setup Details - –Section[F\.1](https://arxiv.org/html/2610.12837#A6.SS1)Datasets - –Section[F\.2](https://arxiv.org/html/2610.12837#A6.SS2)Teacher Model Fine\-Tuning and Evaluation - –Section[F\.3](https://arxiv.org/html/2610.12837#A6.SS3)Additional Details of the Baselines
- •Section[G](https://arxiv.org/html/2610.12837#A7)Ablation Studies - –Section[G\.1](https://arxiv.org/html/2610.12837#A7.SS1)Effect of CFD Validation - –Section[G\.2](https://arxiv.org/html/2610.12837#A7.SS2)CFD Threshold Ablation - –Section[G\.3](https://arxiv.org/html/2610.12837#A7.SS3)Cost Analysis
- •Section[H](https://arxiv.org/html/2610.12837#A8)Additional Experimental Results - –Section[H\.1](https://arxiv.org/html/2610.12837#A8.SS1)In\-Domain EM Results - –Section[H\.2](https://arxiv.org/html/2610.12837#A8.SS2)In\-Domain Agreement Results - –Section[H\.3](https://arxiv.org/html/2610.12837#A8.SS3)Out\-of\-Domain EM Results - –Section[H\.4](https://arxiv.org/html/2610.12837#A8.SS4)Performance under Modified Contextual Evidence

## Appendix BLimitations and Future Work

Our current counterfactual generation framework is limited to QA samples with entity\-based answers\. Extending counterfactual generation beyond these could broaden the applicability ofCofFeQA\. Future work could explore more general counterfactual generation strategies that can accommodate diverse answer types while preserving the intended contextual changes\.

The current formulation ofCofFeQAis also designed specifically for open\-book question answering, where counterfactuals can be constructed by modifying the supporting context while preserving the question\. Extending the definition, generation, and validation of counterfactuals to other generative tasks is an important direction for investigating whether counterfactual\-guided few\-shot distillation generalizes beyond open\-book QA\.

## Appendix CKD Performance on Boundary\-Near Factual Samples

To investigate how proximity to the teacher’s decision boundary affects knowledge distillation, we group factual samples according to their prediction margins under the teacher model\. Intuitively, a small margin indicates that the teacher assigns similar probabilities to its two most likely token predictions, reflecting greater uncertainty in its output\. We use margin sampling\([Scheffer et al\., 2001](https://arxiv.org/html/2610.12837#bib.bib5)\)as a proxy for proximity to the teacher’s decision boundary\.

Measuring proximity to the decision boundary\.For a factual samplex=\(Q,C,A\)x=\(Q,C,A\)with target answer sequencey=\(y1,…,yT\)y=\(y\_\{1\},\\ldots,y\_\{T\}\), the teacher produces a probability distribution over the vocabulary at each answer positiontt\. Letv\(1\)tv\_\{\(1\)\}^\{t\}andv\(2\)tv\_\{\(2\)\}^\{t\}denote the teacher’s most likely and next most likely tokens, respectively, at positiontt, conditioned on the inputxxand the preceding target tokensy<ty\_\{<t\}\. We define the token\-level prediction margin as

mt​\(x,y\)=log⁡Pθ​\(v\(1\)t∣x,y<t\)−log⁡Pθ​\(v\(2\)t∣x,y<t\)\.m\_\{t\}\(x,y\)=\\log P\_\{\\theta\}\\\!\\left\(v\_\{\(1\)\}^\{t\}\\mid x,y\_\{<t\}\\right\)\-\\log P\_\{\\theta\}\\\!\\left\(v\_\{\(2\)\}^\{t\}\\mid x,y\_\{<t\}\\right\)\.\(4\)The prediction margin for the complete answer sequence is then computed as the mean of the token\-level margins:

margin⁡\(x,y\)=1T​∑t=1Tmt​\(x,y\)\.\\operatorname\{margin\}\(x,y\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}m\_\{t\}\(x,y\)\.\(5\)Here,v\(1\)tv\_\{\(1\)\}^\{t\}andv\(2\)tv\_\{\(2\)\}^\{t\}are the teacher’s own top\-1 and top\-2 predictions at each position and are not defined according to whether they match the ground\-truth token\. A smaller average margin indicates greater ambiguity between the teacher’s two most likely token predictions\. We therefore use this quantity as a proxy for proximity to the teacher’s decision boundary\.

For each dataset, we sort the factual samples in ascending order of their teacher prediction margins and divide them into three margin bins, where Bin 0 contains samples with the smallest margins, Bin 1 contains samples with intermediate margins, and Bin 2 contains samples with the largest margins\. To evaluate the effect of teacher prediction margin on knowledge distillation under a controlled few\-shot budget, we sample 100 factual examples from each bin and perform knowledge distillation separately using the samples from each bin\. Thus, each setting uses the same distillation budget of 100 factual samples, with the primary difference being their teacher prediction margins\.

Figure[6](https://arxiv.org/html/2610.12837#A3.F6)reports the resulting KD performance when distillation is performed using 100 factual samples from each of the three margin bins\. Figures[6](https://arxiv.org/html/2610.12837#A3.F6)\(a\) and[6](https://arxiv.org/html/2610.12837#A3.F6)\(b\) show EM and F1 averaged across the five in\-domain datasets, respectively\. For both metrics, distillation using the smallest\-margin samples in Bin 0 results in substantially lower performance than using samples from Bins 1 and 2\. The mean EM increases from56\.94%56\.94\\%in Bin 0 to67%67\\%in Bins 1–2, while the mean F1 increases from65\.99%65\.99\\%to72​–​73%72–73\\%in Bins 1–2\. In contrast, the difference between Bins 1 and 2 is relatively small for both metrics\. This suggests that the main degradation in distillation performance occurs when the few\-shot factual supervision consists of samples with the smallest teacher prediction margins\.

\(a\) Aggregated EM

\(b\) Aggregated F1

\(c\) Dataset\-wise EM

\(d\) Dataset\-wise F1

Figure 6:KD performance as a function of the teacher prediction margin\.Factual samples are grouped into three bins according to their average token\-level prediction margins, with Bin 0 containing the smallest margin samples and Bin 2 containing the largest margin samples\. Smaller margins are used as a proxy for greater proximity to the teacher’s decision boundary\. Knowledge distillation is performed separately using 100 factual samples from each bin, maintaining the same distillation budget across all three settings\. Panels \(a\) and \(b\) report EM and F1 averaged across the five in\-domain datasets, respectively, while panels \(c\) and \(d\) report the corresponding dataset\-wise results\. Distillation using samples from the smallest margin bin generally results in lower performance than using samples from the larger margin bins\.Figures[6](https://arxiv.org/html/2610.12837#A3.F6)\(c\) and[6](https://arxiv.org/html/2610.12837#A3.F6)\(d\) show the corresponding dataset\-wise EM and F1 results for HotpotQA, NewsQA, Natural Questions \(NQ\), SQuAD, and TriviaQA\. The same general pattern is observed across the five datasets: distillation using samples from Bin 0 results in lower EM and F1 than using samples from the larger\-margin bins\. The difference is particularly pronounced for HotpotQA, NewsQA, TriviaQA and NQ, whereas SQuAD exhibits a smaller difference across the three bins\. Thus, the lower distillation performance associated with the smallest\-margin samples is observed across datasets rather than being driven by a single benchmark\.

Overall, these results support our motivating observation that factual samples with small teacher prediction margins are more challenging to distill under limited factual supervision\. Since a small margin reflects greater ambiguity between the teacher’s most likely token predictions, we use these samples as an operational approximation of boundary\-near factual examples\. Importantly, this degradation is observed while maintaining the same 100 sample distillation budget across all three margin bins\. This observation motivatesCofFeQAto complement boundary\-near factual samples with counterfactual supervision, providing additional information about how the teacher’s response changes under modified contextual evidence\.

## Appendix DCounterfactual Generation Prompts

We use GPT\-4o\-mini\([OpenAI, 2024](https://arxiv.org/html/2610.12837#bib.bib15)\)for counterfactual generation\. The same prompts are used for both the T5 and Qwen2\.5 experiments\. Counterfactual generation consists of three stages: \(i\) identifying the entity type of the original answer, \(ii\) generating a plausible alternative entity of the same type, and \(iii\) minimally modifying the original context to support the alternative answer\. The prompts used in these stages are shown in Figure[7](https://arxiv.org/html/2610.12837#A4.F7)\.

Entity Type ClassificationWhat type of named entity is this answer: “\{answer\_text\}”?Classify as PERSON, LOCATION, ORGANIZATION, DATE, NUMERIC, or OTHER if none apply\.

Alternative Entity GenerationThe original answer is “\{original\_answer\}”, which is a\{entity\_type\}\.Propose ONE plausible alternative\{entity\_type\}that is CLOSE to “\{original\_answer\}” \(similar in scale, category, or context\) but clearly DIFFERENT from it\.Do NOT propose any of these already\-tried alternatives:\{avoid\_list\}\.Respond with ONLY a JSON object:\{"alternative\_entity": "<your proposed alternative\>"\}

Counterfactual Context Generation: System PromptYou are an expert at creating counterfactual reading\-comprehension passages for QA training data\. Given an original context that supports a specific answer, you rewrite it MINIMALLY so that it instead unambiguously supports a different target answer, by replacing occurrences of the original answer \(and grammatically\-dependent references to it\) with the target answer, while leaving everything else in the passage unchanged\. Do not add explanations or extra content\.

Counterfactual Context Generation: User PromptQuestion:\{question\}Original context \(supports answer “\{original\_answer\}”\):\{original\_context\}Target answer \(the new counterfactual answer\): “\{target\_answer\}”Rewrite the context above so that it minimally changes to unambiguously support “\{target\_answer\}” as the answer to the question, instead of “\{original\_answer\}”\. Replace occurrences of “\{original\_answer\}” \(and any grammatically\-dependent references to it\) with “\{target\_answer\}”, keeping everything else in the passage exactly the same\. Do not add commentary\.Respond with ONLY a JSON object:\{"reasoning": "<1\-2 sentences on what you changed\>", "counterfactual\_context": "<the full rewritten context\>"\}

Figure 7:Prompts used for counterfactual generation\.The generation process consists of entity type classification, alternative entity generation, and counterfactual context generation\. The final stage uses a system prompt and a user prompt to minimally modify the original context to support the alternative answer\.Generated counterfactuals are subsequently evaluated using CFD before being used for distillation\.

## Appendix EProof of Proposition[1](https://arxiv.org/html/2610.12837#Thmproposition1)

In this section, we provide the proof of the upper bound on the Hausdorff distance between the teacher’s context\-response boundaryℳt\\mathcal\{M\}\_\{t\}and the student’s context\-response boundaryℳs\\mathcal\{M\}\_\{s\}\.

By definition, the Hausdorff distance betweenℳt\\mathcal\{M\}\_\{t\}andℳs\\mathcal\{M\}\_\{s\}is

dH​\(ℳt,ℳs\)=max⁡\{supt∈ℳtinfs∈ℳs‖t−s‖,sups∈ℳsinft∈ℳt‖s−t‖\}\.d\_\{H\}\(\\mathcal\{M\}\_\{t\},\\mathcal\{M\}\_\{s\}\)=\\max\\left\\\{\\sup\_\{t\\in\\mathcal\{M\}\_\{t\}\}\\inf\_\{s\\in\\mathcal\{M\}\_\{s\}\}\\\|t\-s\\\|,\\;\\sup\_\{s\\in\\mathcal\{M\}\_\{s\}\}\\inf\_\{t\\in\\mathcal\{M\}\_\{t\}\}\\\|s\-t\\\|\\right\\\}\.\(6\)We bound the two directed distances separately\([Hamman et al\., 2025](https://arxiv.org/html/2610.12837#bib.bib19)\)\.

### Part 1: Bounding the deviation fromℳt\\mathcal\{M\}\_\{t\}toℳs\\mathcal\{M\}\_\{s\}

Lett∈ℳtt\\in\\mathcal\{M\}\_\{t\}be an arbitrary point on the teacher’s boundary\. By the boundary\-coverage assumption \(Assumption 3\), there exists a teacher boundary crossing pointti∗∈ℳtt\_\{i\}^\{\*\}\\in\\mathcal\{M\}\_\{t\}associated with a factual\-counterfactual pair\(ci,ci′\)\(c\_\{i\},c\_\{i\}^\{\\prime\}\)such that \(see Figure[8](https://arxiv.org/html/2610.12837#A5.F8)\):

‖t−ti∗‖≤ϵ\.\\\|t\-t\_\{i\}^\{\*\}\\\|\\leq\\epsilon\.\(7\)
Figure 8:Visualization of the distances betweenℳt\\mathcal\{M\}\_\{t\}andℳs\\mathcal\{M\}\_\{s\}\.cic\_\{i\}andci′c^\{\\prime\}\_\{i\}are factual and counterfactual pairs with‖ci−ci′‖=α′≤α\\\|c\_\{i\}\-c\_\{i\}^\{\\prime\}\\\|=\\alpha^\{\\prime\}\\leq\\alpha\)\.ttis an arbitrary point inℳt\\mathcal\{M\}\_\{t\}that finds someti∗t\_\{i\}^\{\*\}cut point inℳt\\mathcal\{M\}\_\{t\}withinϵ′\\epsilon^\{\\prime\}distance \(ϵ′≤ϵ\\epsilon^\{\\prime\}\\leq\\epsilon\)\.Under Assumption 1, the factual\-counterfactual perturbation is bounded as

‖ci−ci′‖≤α\.\\\|c\_\{i\}\-c\_\{i\}^\{\\prime\}\\\|\\leq\\alpha\.\(8\)Under the teacher\-student agreement assumption \(Assumption 2\), the teacher and student agree on the responses associated with both endpointscic\_\{i\}andci′c\_\{i\}^\{\\prime\}\. Since the factual and counterfactual contexts correspond to different responses, both the teacher and student boundaries cross the segment\[ci,ci′\]\[c\_\{i\},c\_\{i\}^\{\\prime\}\]\. Letti∗∈ℳtt\_\{i\}^\{\*\}\\in\\mathcal\{M\}\_\{t\}andsi∗∈ℳss\_\{i\}^\{\*\}\\in\\mathcal\{M\}\_\{s\}denote their respective boundary crossing points on this segment\.

Because bothti∗t\_\{i\}^\{\*\}andsi∗s\_\{i\}^\{\*\}lie on the same segment\[ci,ci′\]\[c\_\{i\},c\_\{i\}^\{\\prime\}\],

‖ti∗−si∗‖≤‖ci−ci′‖≤α\.\\\|t\_\{i\}^\{\*\}\-s\_\{i\}^\{\*\}\\\|\\leq\\\|c\_\{i\}\-c\_\{i\}^\{\\prime\}\\\|\\leq\\alpha\.\(9\)
Combining Equations equation[7](https://arxiv.org/html/2610.12837#A5.E7)and equation[9](https://arxiv.org/html/2610.12837#A5.E9)and applying the triangle inequality, we obtain

‖t−si∗‖≤‖t−ti∗‖\+‖ti∗−si∗‖≤ϵ\+α\.\\\|t\-s\_\{i\}^\{\*\}\\\|\\leq\\\|t\-t\_\{i\}^\{\*\}\\\|\+\\\|t\_\{i\}^\{\*\}\-s\_\{i\}^\{\*\}\\\|\\leq\\epsilon\+\\alpha\.\(10\)
Since there exists at least one pointsi∗∈ℳss\_\{i\}^\{\*\}\\in\\mathcal\{M\}\_\{s\}within distanceϵ\+α\\epsilon\+\\alphaoftt, the infimum distance satisfies:

infs∈ℳs‖t−s‖≤ϵ\+α\.\\inf\_\{s\\in\\mathcal\{M\}\_\{s\}\}\\\|t\-s\\\|\\leq\\epsilon\+\\alpha\.\(11\)Taking the supremum over all pointst∈ℳtt\\in\\mathcal\{M\}\_\{t\}yields:

supt∈ℳtinfs∈ℳs‖t−s‖≤ϵ\+α\.\\sup\_\{t\\in\\mathcal\{M\}\_\{t\}\}\\inf\_\{s\\in\\mathcal\{M\}\_\{s\}\}\\\|t\-s\\\|\\leq\\epsilon\+\\alpha\.\(12\)

### Part 2: Bounding the deviation fromℳs\\mathcal\{M\}\_\{s\}toℳt\\mathcal\{M\}\_\{t\}

We follow a symmetric logic for the student boundary\. For any points∈ℳss\\in\\mathcal\{M\}\_\{s\}, Assumption 3 guarantees the existence of a student boundary cutpointsi∗s\_\{i\}^\{\*\}such that‖s−si∗‖≤ϵ\\\|s\-s\_\{i\}^\{\*\}\\\|\\leq\\epsilon\.

From the relationship established in Part 1, the distance between this student cutpoint and the corresponding teacher cutpointti∗t\_\{i\}^\{\*\}is‖si∗−ti∗‖≤α\\\|s\_\{i\}^\{\*\}\-t\_\{i\}^\{\*\}\\\|\\leq\\alpha\. Applying the triangle inequality:

‖s−ti∗‖≤‖s−si∗‖\+‖si∗−ti∗‖≤ϵ\+α\\\|s\-t\_\{i\}^\{\*\}\\\|\\leq\\\|s\-s\_\{i\}^\{\*\}\\\|\+\\\|s\_\{i\}^\{\*\}\-t\_\{i\}^\{\*\}\\\|\\leq\\epsilon\+\\alpha\(13\)
Sinceti∗∈ℳtt\_\{i\}^\{\*\}\\in\\mathcal\{M\}\_\{t\}, it follows thatinft∈ℳt‖s−t‖≤ϵ\+α\\inf\_\{t\\in\\mathcal\{M\}\_\{t\}\}\\\|s\-t\\\|\\leq\\epsilon\+\\alpha\. Taking the supremum over alls∈ℳss\\in\\mathcal\{M\}\_\{s\}:

sups∈ℳsinft∈ℳt‖s−t‖≤ϵ\+α\\sup\_\{s\\in\\mathcal\{M\}\_\{s\}\}\\inf\_\{t\\in\\mathcal\{M\}\_\{t\}\}\\\|s\-t\\\|\\leq\\epsilon\+\\alpha\(14\)

### Final Result

Combining Equations equation[12](https://arxiv.org/html/2610.12837#A5.E12)and equation[14](https://arxiv.org/html/2610.12837#A5.E14), we obtain

dH​\(ℳt,ℳs\)≤max⁡\{ϵ\+α,ϵ\+α\}=α\+ϵ\.d\_\{H\}\(\\mathcal\{M\}\_\{t\},\\mathcal\{M\}\_\{s\}\)\\leq\\max\\\{\\epsilon\+\\alpha,\\epsilon\+\\alpha\\\}=\\alpha\+\\epsilon\.\(15\)
This confirms that the student boundary remains within an\(α\+ϵ\)\(\\alpha\+\\epsilon\)\-tube of the teacher boundary\.

## Appendix FExperimental Setup Details

### F\.1Datasets

We evaluateCofFeQAon eight open\-domain question answering benchmarks, comprising five in\-domain and three out\-of\-domain datasets\. The in\-domain benchmarks include Natural Questions \(NQ\), SQuAD, NewsQA, TriviaQA, and HotpotQA\. NQ consists of real user queries collected from Google search logs\([Kwiatkowski et al\., 2019](https://arxiv.org/html/2610.12837#bib.bib78)\)\. SQuAD \(Stanford Question Answering Dataset\) contains questions authored by crowdworkers based on Wikipedia articles\([Rajpurkar et al\., 2016](https://arxiv.org/html/2610.12837#bib.bib3)\)\. NewsQA is constructed from CNN news articles with questions generated crowdworkers\([Trischler et al\., 2017](https://arxiv.org/html/2610.12837#bib.bib79)\)\. TriviaQA consists QA pairs sourced from trivia and quiz\-league websites\([Joshi et al\., 2017](https://arxiv.org/html/2610.12837#bib.bib80)\)\. HotpotQA focuses on multi\-hop reasoning, requiring models to combine information from multiple Wikipedia articles\([Yang et al\., 2018](https://arxiv.org/html/2610.12837#bib.bib81)\)\. The out\-of\-domain benchmarks include DROP, DuoRC, and RelationExtraction\. DROP contains quantitative QA pairs which has numerical and text answers\([Dua et al\., 2019](https://arxiv.org/html/2610.12837#bib.bib82)\)\. DuoRC contains QA pairs derived from movie plots, constructed by crowdworkers using IMDb and Wikipedia\([Saha et al\., 2018](https://arxiv.org/html/2610.12837#bib.bib83)\)\. RelationExtraction reformulates relation extraction as a reading comprehension task using Wikipedia text\([Levy et al\., 2017](https://arxiv.org/html/2610.12837#bib.bib14)\)\.

The datasets are designed for generalization in open\-book question answering, where samples are drawn from diverse distributions\. The datasets have following properties in common\.

- •Question distribution:The test sets contain different styles of questions, such as entity\-based and relational questions, collected from diverse sources, including crowdworkers, domain experts, and exam writers\.
- •Context distribution:Passages are drawn from heterogeneous sources such as science articles, novels, medical abstracts, news articles\.
- •Answerability:The answer can be generated using the provided context, and the answer span appears within the first 800 tokens of the passage\.

The training and validation sets for the teacher are drawn from the original training splits after removing duplicate entries\. Dataset statistics are reported in Table[3](https://arxiv.org/html/2610.12837#A6.T3)\. We generate counterfactuals only for the samples with entity\-based answers\. Statistics of the knowledge distillation datasets are also shown in Table[3](https://arxiv.org/html/2610.12837#A6.T3), where studenttrainandvaldenote available factual training samples with entity\-based answers for distillation\. The in\-domain test sets for knowledge distillation are taken from the original development splits after filtering for entity\-based answer samples\. The out\-of\-domain test set counts are as follows: DROP \(496\), and DuoRC \(450\), RelationExtraction \(912\)\.

Table 3:Statistics of teacher and student datasets used in our experiments\. Teacher datasets contain the original training/validation splits, while student datasets contain available factual instances for distillation and evaluation\.
### F\.2Teacher Model Fine\-Tuning and Evaluation

All fine\-tuning experiments were conducted on a machine equipped with four NVIDIA RTX 6000 Ada GPUs\. For T5\-Large fine\-tuning experiments, we used a batch size of 4, a learning rate of1×10−41\\times 10^\{\-4\}, and a maximum of 5 training epochs\. Training was stopped early based on validation loss using an early stopping criterion with a patience of 3 validation checks\. For Qwen2\.5\-7B\-IT fine\-tuning experiments, we used a batch size of 1, a learning rate of2×10−52\\times 10^\{\-5\}, and a maximum of 5 training epochs\. Training was stopped early based on validation loss using an early stopping criterion with a patience of 3 validation checks\. On average, all models converged within 1 to 2 epochs\.

We evaluated the fine\-tuned teacher models using the same test sets for each dataset\. The results can be found in Table[4](https://arxiv.org/html/2610.12837#A6.T4)

Table 4:Teacher model performance \(EM and F1\) on factual and counterfactual test sets across five in\-domain QA benchmarks for T5 and Qwen2\.5 models\.
### F\.3Additional details of the baselines

We compareCofFeQAagainst three knowledge distillation baselines\. To ensure a fair comparison, all methods use the samekk\-shot data splits, and evaluation settings\.

- •Standard Knowledge Distillation \(KD\)\.Standard knowledge distillation \(KD\)\([Hinton et al\., 2015](https://arxiv.org/html/2610.12837#bib.bib11)\)transfers knowledge from a larger teacher model to a smaller student model by aligning their output distributions\. Letxxdenote the input,y=\(y1,…,yT\)y=\(y\_\{1\},\\dots,y\_\{T\}\)the ground\-truth output sequence, andpsp\_\{s\}andptp\_\{t\}the student and teacher model distributions, respectively\. The student is first trained using the standard cross\-entropy objective ℒCE=−∑t=1Tlogps\(yt∣x,y<t\)\.\\mathcal\{L\}\_\{\\mathrm\{CE\}\}=\-\\sum\_\{t=1\}^\{T\}\\log p\_\{s\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\.\(16\) In addition, KD encourages the student to match the teacher’s token\-level output distribution at each decoding step using ℒKD=∑t=1TKL\(pt\(⋅∣x,y<t\)∥ps\(⋅∣x,y<t\)\)\.\\mathcal\{L\}\_\{\\mathrm\{KD\}\}=\\sum\_\{t=1\}^\{T\}\\mathrm\{KL\}\\left\(p\_\{t\}\(\\cdot\\mid x,y\_\{<t\}\)\\,\\\|\\,p\_\{s\}\(\\cdot\\mid x,y\_\{<t\}\)\\right\)\.\(17\) The final training objective combines the supervised and distillation losses: ℒ=α​ℒCE\+\(1−α\)​ℒKD,\\mathcal\{L\}=\\alpha\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\+\(1\-\\alpha\)\\mathcal\{L\}\_\{\\mathrm\{KD\}\},\(18\)whereα∈\[0,1\]\\alpha\\in\[0,1\]controls the trade\-off between fitting the ground\-truth sequence and matching the teacher’s output distribution\. Thus, standard KD transfers knowledge by aligning the teacher and student token\-level distributions while retaining supervision from the ground\-truth sequence\.
- •Layer\-wise Distillation \(LWD\)\.Layer\-wise distillation \(LWD\)\([Jiao et al\., 2020](https://arxiv.org/html/2610.12837#bib.bib2)\)extends output\-level knowledge distillation by additionally transferring information from the intermediate representations of the teacher\. Representations from selected teacher layers are aligned with the corresponding representations of the student through an auxiliary layer\-wise distillation objective, providing supervision at both the intermediate representation and output levels\. Let𝐡t\(lt\)\\mathbf\{h\}\_\{t\}^\{\(l\_\{t\}\)\}and𝐡s\(ls\)\\mathbf\{h\}\_\{s\}^\{\(l\_\{s\}\)\}denote the hidden representations from a paired teacher layerltl\_\{t\}and student layerlsl\_\{s\}, respectively\. Since the teacher and student may have different hidden dimensions, a learnable projectionWlsW\_\{l\_\{s\}\}is applied to the student representation before alignment\. The layer\-wise distillation loss is defined as ℒLWD=1\|𝒫\|​∑\(ls,lt\)∈𝒫‖Wls​𝐡s\(ls\)−𝐡t\(lt\)‖22,\\mathcal\{L\}\_\{\\mathrm\{LWD\}\}=\\frac\{1\}\{\|\\mathcal\{P\}\|\}\\sum\_\{\(l\_\{s\},l\_\{t\}\)\\in\\mathcal\{P\}\}\\left\\\|W\_\{l\_\{s\}\}\\mathbf\{h\}\_\{s\}^\{\(l\_\{s\}\)\}\-\\mathbf\{h\}\_\{t\}^\{\(l\_\{t\}\)\}\\right\\\|\_\{2\}^\{2\},\(19\)where𝒫\\mathcal\{P\}denotes the set of paired student and teacher layers\. The final objective combines the standard KD objective with the layer\-wise representation loss: ℒ=α​ℒCE\+\(1−α\)​ℒKD\+β​ℒLWD,\\mathcal\{L\}=\\alpha\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\+\(1\-\\alpha\)\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\+\\beta\\mathcal\{L\}\_\{\\mathrm\{LWD\}\},\(20\)whereβ\\betacontrols the contribution of intermediate representation alignment\. Thus, unlike standard KD, which transfers knowledge through the output distributions, LWD additionally encourages the student to reproduce the teacher’s intermediate representations\.
- •Sequence\-Level Knowledge Distillation \(SeqKD\)\.Unlike standard KD, which aligns the teacher and student token\-level distributions at each decoding step, sequence\-level knowledge distillation \(seqKD\) transfers knowledge through complete sequences generated by the teacher[Kim and Rush \(2016\)](https://arxiv.org/html/2610.12837#bib.bib40)\. Letxxdenote the input andy=\(y1,…,yT\)y=\(y\_\{1\},\\dots,y\_\{T\}\)denote the output sequence, and letpsp\_\{s\}andptp\_\{t\}denote the student and teacher model distributions, respectively\. The sequence\-level distillation objective is defined as ℒSeqKD=−∑y∈𝒴pt\(y∣x\)logps\(y∣x\),\\mathcal\{L\}\_\{\\mathrm\{SeqKD\}\}=\-\\sum\_\{y\\in\\mathcal\{Y\}\}p\_\{t\}\(y\\mid x\)\\log p\_\{s\}\(y\\mid x\),\(21\) where𝒴\\mathcal\{Y\}denotes the set of all possible output sequences\. Since directly optimizing over all possible output sequences is intractable, seqKD approximates the teacher’s sequence\-level distribution using its highest\-probability output sequence\. Specifically, the teacher\-generated sequence is obtained as y^≈arg⁡maxy∈𝒴​pt​\(y∣x\),\\hat\{y\}\\approx\\arg\\max\_\{y\\in\\mathcal\{Y\}\}p\_\{t\}\(y\\mid x\),\(22\) where beam search is used to approximate the highest probability sequence under the teacher distribution\. The resulting sequence\-level distillation objective is then ℒSeqKD≈−logps\(y^∣x\)=−∑t=1\|y^\|logps\(y^t∣x,y^<t\)\.\\mathcal\{L\}\_\{\\mathrm\{SeqKD\}\}\\approx\-\\log p\_\{s\}\(\\hat\{y\}\\mid x\)=\-\\sum\_\{t=1\}^\{\|\\hat\{y\}\|\}\\log p\_\{s\}\\left\(\\hat\{y\}\_\{t\}\\mid x,\\hat\{y\}\_\{<t\}\\right\)\.\(23\) Thus, while standard KD transfers knowledge by matching the teacher’s token\-level probability distributions, seqKD uses the complete sequence generated by the teacher as the training target for the student\.

For standard KD, the student was trained using a weighted combination of the MLE loss and the token\-level distillation loss\. The temperature was set to 1, and theα\\alphawas set to0\.50\.5\.

For LWD, student decoder layers were uniformly mapped to corresponding teacher decoder layers across the model depth\. Layer\-wise knowledge transfer was performed by aligning the decoder self\-attention distributions of the mapped teacher and student layers using KL divergence\. Attention distributions were averaged across attention heads before computing the layer\-wise loss\. The layer\-wise distillation loss was added to the standard KD objective with a weight of5\.05\.0\.

For seqKD, the teacher generated a complete output sequence for each training input using beam search with a beam size of 4\. The original ground\-truth answer was replaced by the corresponding teacher\-generated sequence, which was then used as the training target for the student\. Following the original seqKD formulation\([Kim and Rush, 2016](https://arxiv.org/html/2610.12837#bib.bib40)\), the student was trained using cross\-entropy loss on the teacher\-generated sequences without an additional token\-level distillation loss\.

For all three baselines, we used a batch size of 4 and a learning rate of1×10−41\\times 10^\{\-4\}\. The maximum number of training epochs was set to 100, with early stopping based on validation loss using a patience of 3 validation checks\.

## Appendix GAblation Study Results

### G\.1Effect of CFD Validation

To isolate the contribution of CFD verification, we compareCofFeQAwith a variant that generates counterfactuals without CFD verification\. Both variants use identical factual data splits and training settings, with the only difference being whether the generated counterfactuals are verified using CFD\. ForCofFeQAwith CFD verification, we use the selected thresholdτ=0\.3\\tau=0\.3\. Experiments are conducted using T5 across all datasets with five random seeds for eachkk\-shot setting, and the results are averaged across datasets\. Figure[9](https://arxiv.org/html/2610.12837#A7.F9)reports the improvement of each variant over standard KD\. CFD verification generally provides larger gains in both downstream QA performance and teacher\-student agreement, with particularly clear improvements atk=32k=32,6464, and128128\. These results indicate that the improvements ofCofFeQAare not solely due to the inclusion of counterfactual examples; verifying the generated counterfactuals using CFD provides additional benefits for knowledge transfer\.

\(a\) Accuracy EM

\(b\) Accuracy F1

\(c\) Agreement EM

\(d\) Agreement F1

Figure 9:Effect of CFD verification onCofFeQA\.Performance gains over standard KD \(Δ=CofFeQA−KD\\Delta=\\mathrm\{\\textsc\{CofFeQA\}\}\-\\mathrm\{KD\}\) are shown across different few\-shot budgetskk\. We compareCofFeQAwithout CFD verification against the fullCofFeQAframework with CFD verification \(τ=0\.3\\tau=0\.3\)\. Panels \(a\) and \(b\) report accuracy EM and F1 gains, while panels \(c\) and \(d\) report teacher\-student agreement EM and F1 gains\. CFD verification generally yields larger improvements, demonstrating the benefit of validating generated counterfactuals before distillation\.
### G\.2CFD Threshold Ablation

We analyze the sensitivity ofCofFeQAto the CFD thresholdτ\\tauusing T5 on TriviaQA\. We evaluateτ∈\{0\.1,0\.2,0\.3,0\.4,0\.5\}\\tau\\in\\\{0\.1,0\.2,0\.3,0\.4,0\.5\\\}acrosskk, with five random seeds for eachkk\-shot setting\. For each threshold, we report the averaged improvement ofCofFeQAover standard KD \(Δ=CofFeQA−KD\\Delta=\\mathrm\{\\textsc\{CofFeQA\}\}\-\\mathrm\{KD\}\)\. As shown in Figure[10](https://arxiv.org/html/2610.12837#A7.F10), performance is sensitive to the CFD threshold\. The thresholdτ=0\.1\\tau=0\.1yields negligible improvements, whereas moderate thresholds produce substantially larger gains in both accuracy EM/F1 and teacher\-student agreement\. The gains peak aroundτ=0\.2\\tau=0\.2–0\.30\.3and gradually decrease as the threshold becomes more permissive\. Overall, these results demonstrate the importance of controlling counterfactual quality through the CFD threshold\.

Figure 10:CFD threshold ablation\.Effect of the CFD thresholdτ\\tauonCofFeQAusing T5 on TriviaQA\. The bars show the improvement ofCofFeQAover standard KD \(Δ=CofFeQA−KD\\Delta=\\mathrm\{\\textsc\{CofFeQA\}\}\-\\mathrm\{KD\}\), averaged acrossk∈\{16,32,64,128,256\}k\\in\\\{16,32,64,128,256\\\}and five random seeds\. The left plot shows the improvement in accuracy measured by EM and F1, while the right plot shows the improvement in teacher\-student agreement measured by EM and F1\.
### G\.3Cost Analysis

We analyze the computational cost ofCofFeQAduring both counterfactual generation and distillation\. Table[5](https://arxiv.org/html/2610.12837#A7.T5)reports the per\-sample cost of counterfactual generation, averaged over all generation attempts\. Counterfactual generation requires approximately2\.882\.88seconds per candidate, with relatively small CPU and GPU energy consumption for both model settings\.

Table[6](https://arxiv.org/html/2610.12837#A7.T6)and Table[7](https://arxiv.org/html/2610.12837#A7.T7)reports the training duration and energy consumption during distillation under T5 and Qwen2\.5, respectively, measured using CodeCarbon\. In both models, for standard KD and LWD,CofFeQAachieves higher accuracy while requiring less training time and energy across allkk\-shot settings\. For SeqKD,CofFeQAalso improves accuracy across all settings, but incurs higher computational costs at the smaller budgets \(k=16k=16and3232\)\. Fromk=64k=64onward, its computational cost becomes comparable to the baseline\. Overall, these results show thatCofFeQAintroduces an additional counterfactual\-generation cost, while maintaining favorable computational efficiency during the subsequent distillation stage\.

Table 5:Average duration and CPU/GPU energy consumption per counterfactual generation candidate under the T5 and Qwen2\.5 settings\. Results are averaged across all generation attempts\.Table 6:Comparison of EM score, distillation duration, and CPU/GPU energy consumption between baseline methods and COFFEQA across different few\-shot budgets under the T5 setting\.Table 7:Comparison of EM score, distillation duration, and CPU/GPU energy consumption between baseline methods and COFFEQA across different few\-shot budgets under the Qwen2\.5\.

## Appendix HAdditional Experimental Results

### H\.1In\-Domain EM Results

Table[8](https://arxiv.org/html/2610.12837#A8.T8)reports the in\-domain EM results across all the baselines\.

Table 8:In\-Domain EM score comparison acrosskk\-shots for T5 and Qwen2\.5 models using baselines\. Mean EM scores are reported with standard deviations as subscripts \(M​e​a​ns​t​dMean\_\{std\}\)\. Bold values indicate the better\-performing method \(CofFeQAvs\. baseline\)\. Blue shading highlights aggregate results\.
### H\.2In\-Domain Agreement Results

Tables[9](https://arxiv.org/html/2610.12837#A8.T9)and[10](https://arxiv.org/html/2610.12837#A8.T10)report the in\-domain teacher\-student agreement results using EM and F1, respectively\. For better visualization, Figures[11](https://arxiv.org/html/2610.12837#A8.F11)and[12](https://arxiv.org/html/2610.12837#A8.F12)illustrate the F1 agreement across datasets andkk\-shot settings for all baselines using T5 and Qwen2\.5, respectively\.

Table 9:In\-domain teacher\-student agreement EM score comparison acrosskk\-shots for T5 and Qwen2\.5 models using baselines\. Mean agreement EM scores are reported with standard deviations as subscripts \(M​e​a​ns​t​dMean\_\{std\}\)\. Bold values indicate the better\-performing method \(CofFeQAvs\. baseline\)\. Blue shading highlights aggregate results\.Table 10:In\-domain teacher\-student agreement F1 score comparison acrosskk\-shots for T5 and Qwen2\.5 models using baselines\. Mean agreement F1 scores are reported with standard deviations as subscripts \(M​e​a​ns​t​dMean\_\{std\}\)\. Bold values indicate the better\-performing method \(CofFeQAvs\. baseline\)\. Blue shading highlights aggregate results\.Figure 11:Teacher\-student F1 agreement of KD, LWD, and SeqKD with and withoutCofFeQAacross datasets andkk\-shot settings using T5\.CofFeQAgenerally achieves higher agreement across the distillation baselines\. Spokes denotekk\-shot settings, and radial axis F1 score ranges are adjusted independently by dataset for visualization\.Figure 12:Teacher\-student F1 agreement of KD, LWD, and SeqKD with and withoutCofFeQAacross datasets andkk\-shot settings using Qwen2\.5\.CofFeQAgenerally achieves higher agreement across the distillation baselines\. Spokes denotekk\-shot settings, and radial axis F1 score ranges are adjusted independently by dataset for visualization\.
### H\.3Out\-of\-Domain EM Results

Table[11](https://arxiv.org/html/2610.12837#A8.T11)reports the out\-of\-domain EM results for T5 and Qwen2\.5\.

Table 11:Out\-of\-domain EM score comparison acrosskk\-shots for T5 and Qwen2\.5 models using KD\. Mean EM scores are reported with standard deviations as subscripts \(M​e​a​ns​t​dMean\_\{std\}\)\. Bold values indicate the better\-performing method \(CofFeQAvs\. KD\)\. Blue shading highlights aggregate results\.
### H\.4Performance under Modified Contextual Evidence

Tables[12](https://arxiv.org/html/2610.12837#A8.T12)and[13](https://arxiv.org/html/2610.12837#A8.T13)report the EM and F1 scores on modified\-context test sets, respectively\. Tables[14](https://arxiv.org/html/2610.12837#A8.T14)and[15](https://arxiv.org/html/2610.12837#A8.T15)report the corresponding teacher\-student agreement scores using EM and F1, respectively\.

Table 12:EM score comparison on modified\-context test sets acrosskk\-shot settings for T5 and Qwen2\.5 models using KD\. Mean EM scores are reported with standard deviations as subscripts \(M​e​a​ns​t​dMean\_\{std\}\)\. Bold values indicate the better\-performing method \(CofFeQAvs\. KD\)\. Blue shading highlights aggregate results\.Table 13:F1 score comparison on modified\-context test sets acrosskk\-shot settings for T5 and Qwen2\.5 models using KD\. Mean F1 scores are reported with standard deviations as subscripts \(M​e​a​ns​t​dMean\_\{std\}\)\. Bold values indicate the better\-performing method \(CofFeQAvs\. KD\)\. Blue shading highlights aggregate results\.Table 14:Teacher\-student agreement EM score comparison on modified\-context test sets acrosskk\-shot settings for T5 and Qwen2\.5 models using KD\. Mean agreement EM scores are reported with standard deviations as subscripts \(M​e​a​ns​t​dMean\_\{std\}\)\. Bold values indicate the better\-performing method \(CofFeQAvs\. KD\)\. Blue shading highlights aggregate results\.Table 15:Teacher\-student agreement F1 score comparison on modified\-context test sets acrosskk\-shot settings for T5 and Qwen2\.5 models using KD\. Mean agreement F1 scores are reported with standard deviations as subscripts \(M​e​a​ns​t​dMean\_\{std\}\)\. Bold values indicate the better\-performing method \(CofFeQAvs\. KD\)\. Blue shading highlights aggregate results\.