From Observation to Intervention: Distilling Self-Generated In-Context Demonstrations via Executable Causal Intervention

# From Observation to Intervention: Distilling Self-Generated In-Context Demonstrations via Executable Causal Intervention
Source: [https://arxiv.org/html/2610.21149](https://arxiv.org/html/2610.21149)
Min LiuAffiliation:Academy of Military ScienceHaoang ChiAffiliation:National University of Defense Technology†Corresponding authorCiheng WuAffiliation:Academy of Military ScienceNaiyang GuanAffiliation:Academy of Military Science

###### Abstract

Self\-generated In\-Context Learning \(ICL\) enables LLMs to synthesize their own few\-shot demonstrations, often outperforming manual prompts\. However, existing methods filter demonstrations by observation, assuming a rationale is high\-quality if it leads to the correct output\. We show such filtering suffers fromobservational equivalence, whereby both faithful causal rationales and spurious post\-hoc rationales can yield same outputs and are statistically indistinguishable from observation alone\. Inspired by independent causal mechanisms, we argue that faithful rationales implytransfer invariance, in which causal rationales should remain effective across logically similar samples\. We instantiate this criterion asExecutableCausalIntervention forDemonstrationsDistillation \(ECI2D\), which abstracts language rationales into executable programs and applies the causal interventions \(d​odo\-operator\) on logically similar samples, retaining only those demonstrations that imply transfer invariance\. Experiments on math, logical and commonsense reasoning benchmarks demonstrate that ECI2D consistently outperforms observation\-based ICL while retaining far fewer demonstrations, with gains exceeding 7 points as the test distribution shifts\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2610.21149v1/post-hoc_v3_cropped.png)Figure 1:Illustration of post\-hoc rationalization usingLlama3\-8Bon the GSM8K[Cobbe et al\. \(2021\)](https://arxiv.org/html/2610.21149#bib.bib25)\.Early Answeringdenotes the original CoT is truncated \(denoted bygray\) to test if the answer is predetermined, andAdding Mistakesdenotes a logical error is injected \(highlighted inred\) before answering the question\.Few\-shot in\-context learning \(ICL\) with chain\-of\-thought \(rationale\) has become a cornerstone for eliciting reasoning in large language models \(LLMs\)[Dong et al\. \(2024\)](https://arxiv.org/html/2610.21149#bib.bib1);[Brown et al\. \(2020\)](https://arxiv.org/html/2610.21149#bib.bib2);[Wei et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib3);[Min et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib20);[Liu et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib12)\. To reduce manual annotation, recent works focus onself\-generated ICL, which leverages the generative capacity of LLMs to synthesize in\-context demonstrations[Wan et al\. \(2025\)](https://arxiv.org/html/2610.21149#bib.bib6);[Agarwal et al\. \(2024\)](https://arxiv.org/html/2610.21149#bib.bib5);[Yang et al\. \(2024\)](https://arxiv.org/html/2610.21149#bib.bib41);[Li et al\. \(2024b\)](https://arxiv.org/html/2610.21149#bib.bib8);[Chen et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib40);[Kim et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib7);[Zhang et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib4)\. Among them, reinforced ICL[Agarwal et al\. \(2024\)](https://arxiv.org/html/2610.21149#bib.bib5);[Singh et al\. \(2024\)](https://arxiv.org/html/2610.21149#bib.bib21)samples multiple rationales per training query and retains only those yielding the correct output, reporting gains over human annotation[Li et al\. \(2024b\)](https://arxiv.org/html/2610.21149#bib.bib8);[Kim et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib7)\. We categorize them asSelf\-Generated Reinforced ICL\.

\(a\)XXRoR\_\{o\}YYinputobservedoutput\(b\)XXRcR\_\{c\}YYRsR\_\{s\}inputoutputObservational equivalence:P⁡\(Y∣X,R\)P\(Y\\mid X,R\)identical \(Section[3](https://arxiv.org/html/2610.21149#S3)\)Figure 2:Structural Causal Models \(SCMs\) in LLM inference\. \(a\)Observed SCM:It treats the observed rationale as the direct causal mediator, forming a linear chainX→Ro→YX\\to R\_\{o\}\\to Y\. \(b\)True SCM:The actual mechanism comprises a faithful causal pathX→Rc→YX\\to R\_\{c\}\\to Yand a spurious post\-hoc path whereRsR\_\{s\}is dependent on bothXXandYY\(X⇢Rs⇠YX\\dashrightarrow R\_\{s\}\\dashleftarrow Y\)\. The two SCMs areobservational equivalenceas discussed in Section[3](https://arxiv.org/html/2610.21149#S3)\.However, the demonstration distillation criterion of self\-generated reinforced ICL simply relies onobservation, which is a outcome\-oriented criterion that equates answer correctness with rationale quality\. Rationales are not always faithful, as shown in Figure[1](https://arxiv.org/html/2610.21149#S1.F1), LLMs often engage in post\-hoc rationalization[Holzinger et al\. \(2017\)](https://arxiv.org/html/2610.21149#bib.bib11), generating plausible explanations after answering[Turpin et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib9)\. The faithfulness of post\-hoc rationales cannot be guaranteed\. We argue that observational distillation suffers fromobservational equivalence\. As shown in Figure[2](https://arxiv.org/html/2610.21149#S1.F2), based on outcome correctness, the true inference process \(comprising both faithfulX→Rc→YX\\to R\_\{c\}\\to Yand spuriousY⇢RsY\\dashrightarrow R\_\{s\}\) is statistically indistinguishable from the observed causal chainX→Ro→YX\\to R\_\{o\}\\to Y\. Self\-generated reinforced ICL fails to disentangleRcR\_\{c\}fromRsR\_\{s\}, treating them as a single observed variableRoR\_\{o\}, which leads to the problem ofobservational equivalence\. Critically, our experiments confirm that these unreliable rationales actively misguide the model \(Table[1](https://arxiv.org/html/2610.21149#S4.T1)\), degrading ICL performance\. This naturally raises a question that needs attention:

How can we reliably distinguish faithful causal rationales from spurious post\-hoc justifications?

Inspired by the causal invariance[Peters et al\. \(2016\)](https://arxiv.org/html/2610.21149#bib.bib16), transportability[Pearl and Bareinboim \(2011\)](https://arxiv.org/html/2610.21149#bib.bib17), and independent causal mechanisms[Schölkopf et al\. \(2012\)](https://arxiv.org/html/2610.21149#bib.bib36), we adopt a practical notion that faithful rationales should exhibittransfer invariance, remaining effective across similar samples\. To this end, we proposeExecutableCausalIntervention forDemonstrationDistillation \(ECI2D\), a novel framework to distillRcR\_\{c\}fromRoR\_\{o\}via executable intervention\. ECI2D first abstracts language rationales into executable programs\. It then generates rationales and intervenes on logically similar samples by replacing the original rationale with these alternatives\. We define an intervention\-invariant scoring function to quantify transfer invariance, and only those rationales that consistently yield correct results across interventions are identified as the faithful causal pathRcR\_\{c\}, while those that fail are discarded as spuriousRsR\_\{s\}\. Our contributions are fourfold:

- •We provide a formal causal analysis of self\-generated reinforced ICL, identifying the prevalence ofobservational equivalenceand showing that faithful causal rationales implytransfer invariance\(Section[3](https://arxiv.org/html/2610.21149#S3)\)\.
- •We propose ECI2D, a framework that utilizes executable programs to perform causal interventions, effectively distilling faithful rationales \(Section[4](https://arxiv.org/html/2610.21149#S4)\)\.
- •We implement two abstraction forms, showing ECI2D can generalize from deterministic to non\-deterministic tasks \(Section[4](https://arxiv.org/html/2610.21149#S4)\)\.
- •Extensive experiments demonstrate that ECI2D significantly improves performance and OOD generalization compared to both manual and self\-generated reinforced ICL baselines \(Section[5](https://arxiv.org/html/2610.21149#S5)and[6](https://arxiv.org/html/2610.21149#S6)\)\.

## 2Related Works

### 2\.1Self\-Generated In\-Context Learning

Early works[Kim et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib7)synthesize demonstrations conditioned on test inputs\. Auto\-ICL[Yang et al\. \(2024\)](https://arxiv.org/html/2610.21149#bib.bib41), Self\-ICL[Chen et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib40), and AutoCoT[Zhang et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib4)extend this into fully automated pipelines that bootstrap demonstrations from zero\-shot inference, surpassing human\-crafted counterparts in complex reasoning[Li et al\. \(2024b\)](https://arxiv.org/html/2610.21149#bib.bib8)\. Reinforced ICL[Agarwal et al\. \(2024\)](https://arxiv.org/html/2610.21149#bib.bib5)instead applies an outcome\-oriented criterion, retaining only the rationale traces that yield the correct final answer\.

This criterion cannot guarantee that a retained rationale reflects the process producing the answer, raising the concern that unfaithful reasoning enters the prompt and degrades generalization[Turpin et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib9);[Lanham et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib10);[Sharma et al\. \(2024\)](https://arxiv.org/html/2610.21149#bib.bib18)\. Prior studies diagnose this through observational analysis or input perturbations[Jacovi and Goldberg \(2020\)](https://arxiv.org/html/2610.21149#bib.bib13);[Wei et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib3)\. We instead propose a novel framework that utilizes interventional transfer invariance to rigorously quantify and verify the faithfulness of rationales\.

### 2\.2Causal Abstraction & Externally Verified Reasoning

Causal abstraction interprets LLM behaviour by mapping natural language to structured causal models[Geiger et al\. \(2020\)](https://arxiv.org/html/2610.21149#bib.bib15);[Wu et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib23);[Yamakoshi et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib24)\. Separately, program\-aided prompting executes generated code[Gao et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib27);[Chen et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib26);[Li et al\. \(2024a\)](https://arxiv.org/html/2610.21149#bib.bib28)and symbolic approaches delegate a logical form to a solver[Pan et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib44);[Ye et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib46);[Olausson et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib45), both requiring an external engine at inference time\. We combine the two by applying causal abstraction to the rationale itself, whose logical steps correspond to causal nodes[Tan \(2023\)](https://arxiv.org/html/2610.21149#bib.bib14), and converting it into an executable program that enables precise intervention during demonstration distillation rather than inference\.

## 3Problem Analysis

The objective of self\-generated reinforced ICL is to construct a high\-quality demonstration setDDfrom a training set defined over input space𝒳\\mathcal\{X\}, output space𝒴\\mathcal\{Y\}, and rationale spaceℛ\\mathcal\{R\}\. Unlike manual curation, the LLM with parametersΘ\\Thetaautonomously synthesizes demonstrations in two phases\. The Generation phase samples a candidate pair\(r^,y^\)∼Θ\(⋅∣x\)\(\\hat\{r\},\\hat\{y\}\)\\sim\\Theta\(\\cdot\\mid x\)givenx∈𝒳x\\in\\mathcal\{X\}, wherer^∈ℛ\\hat\{r\}\\in\\mathcal\{R\}andy^∈𝒴\\hat\{y\}\\in\\mathcal\{Y\}\. The Distillation phase filters candidates using a scoring functionS⁡\(x,r^,y^\)S\(x,\\hat\{r\},\\hat\{y\}\)\. Existing methods predominantly adoptS⁡\(x,r^,y^\)=𝕀⁡\(y^=y\)S\(x,\\hat\{r\},\\hat\{y\}\)=\\mathbb\{I\}\(\\hat\{y\}=y\), relying solely on observational correctness relative to ground truthyy\.

![Refer to caption](https://arxiv.org/html/2610.21149v1/main_v4.png)Figure 3:Invariant Reasoning Distillation phase in ECI2D\. ECI2D distills faithful causal demonstrations through anAbstract\-Intervene\-Verifypipeline\.Step 1: Causal Abstraction\. For each sample\(xi,yi\)\(x\_\{i\},y\_\{i\}\), the LLM generatorΘGen\\Theta\_\{\\text\{Gen\}\}produces a natural language rationaler^i\\hat\{r\}\_\{i\}, which is then abstracted by LLM AbstractorΘΠ\\Theta\_\{\\Pi\}into an executable programp^i\\hat\{p\}\_\{i\}\.Step 2: Executable Causal Intervention\. For all logically similar samples\(xj,yj\)∈𝒩⁡\(xi\)\(x\_\{j\},y\_\{j\}\)\\in\\mathcal\{N\}\(x\_\{i\}\), LLM adapterΘAdapter\\Theta\_\{\\text\{Adapter\}\}transplants the logical skeleton ofp^i\\hat\{p\}\_\{i\}to the logically similar inputs to generatep^i→j\\hat\{p\}\_\{i\\to j\}, performing an executable interventiond​o​\(R=r^i\)do\(R=\\hat\{r\}\_\{i\}\)on\(xj,yj\)\(x\_\{j\},y\_\{j\}\)via execution\.Step 3: Transfer Invariance Verification\. The intervention output is scored against the ground truthyjy\_\{j\}\. Rationales that fail to generalize are identified as spurious and discarded, while those demonstrating transfer invariance are retained in the causal poolDcausalD\_\{\\text\{causal\}\}\.### 3\.1The Limitation of Self\-Generated Reinforced ICL

The criterion of observational distillation assumes that answer correctness guarantees rationale quality\. However, it overlooks post\-hoc rationalization in LLM inference\. LLMs may determine answers via pattern matching or memorized shortcuts, then retroactively generate plausible\-sounding rationales to justify the conclusion[Lanham et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib10);[Holzinger et al\. \(2017\)](https://arxiv.org/html/2610.21149#bib.bib11)\. Since these rationales are downstream of the prediction, they lack genuine influence on output\.

To further analyze this limitation, we model the inference process using SCM𝒢=⟨𝐕,𝐄⟩\\mathcal\{G\}=\\langle\\mathbf\{V\},\\mathbf\{E\}\\rangle[Pearl \(2010\)](https://arxiv.org/html/2610.21149#bib.bib19), where𝐕=\{X,R,Y\}\\mathbf\{V\}=\\\{X,R,Y\\\}represents the input, rationale, and output, respectively\. As shown in Figure[2](https://arxiv.org/html/2610.21149#S1.F2), current distillation methods implicitly assume an observed causal graphX→Ro→YX\\to R\_\{o\}\\to Y, treating the rationale as a direct mediator ofYY\. We identify the fundamental limitation of existing distillation methods as the inability to differentiate between faithful causalRcR\_\{c\}and spurious post\-hoc rationalesRsR\_\{s\}based solely on observation, which suffers fromobservational equivalence\.

###### Definition 1\(Faithful Causal Rationale\)\.

A faithful causal rationale is one that mediates the inference and has a genuine causal effect on the output\.

###### Definition 2\(Spurious Post\-hoc Rationale\)\.

A post\-hoc rationale is an explanation that is generated after the prediction has been determined\. A post\-hoc rationale is considered spurious if it is retrospectively constructed to fit the observed input and output, while failing to reflect the causal factors that actually produced the prediction\.

###### Definition 3\(Observational Equivalence\)\.

Two causal structures are observationally equivalent if they generate identical joint probability distributions over the observed variables\.

As derived in Appendix[E](https://arxiv.org/html/2610.21149#A5), given a demonstration\(x,r^,y\)\(x,\\hat\{r\},y\), both the faithful and the spurious factorization produce identical observational likelihoodsP⁡\(Y=y∣X=x,R=r^\)P\(Y=y\\mid X=x,R=\\hat\{r\}\)\. Consequently, outcome\-oriented filtering suffers fromobservational equivalence, rendering the true causal structure unidentifiable[Verma and Pearl \(2022\)](https://arxiv.org/html/2610.21149#bib.bib39);[Bareinboim and Pearl \(2012\)](https://arxiv.org/html/2610.21149#bib.bib22);[Spirtes et al\. \(2000\)](https://arxiv.org/html/2610.21149#bib.bib35)\.

### 3\.2Transfer Invariance as a Proxy for Faithfulness

To resolve theobservational equivalence, it is imperative to distinguish between the faithful causal rationaleRcR\_\{c\}and the spurious post\-hoc rationaleRsR\_\{s\}\. As analyzed in Appendix[F](https://arxiv.org/html/2610.21149#A6), inspired by the Independent Causal Mechanisms[Schölkopf et al\. \(2012\)](https://arxiv.org/html/2610.21149#bib.bib36), we adopt the practical notion that a faithful causal relation should remain effective across similar samples[Peters et al\. \(2016\)](https://arxiv.org/html/2610.21149#bib.bib16);[Pearl and Bareinboim \(2011\)](https://arxiv.org/html/2610.21149#bib.bib17), which impliestransfer invariance\.

Consider a faithful rationaler^i∈ℛ\\hat\{r\}\_\{i\}\\in\\mathcal\{R\}derived from an inputxi∈𝒳x\_\{i\}\\in\\mathcal\{X\}\. Ifr^i\\hat\{r\}\_\{i\}represents the true causal rationale of outputyiy\_\{i\}, it should be applicable to a logically similar samplexj∈𝒩⁡\(xi\)x\_\{j\}\\in\\mathcal\{N\}\(x\_\{i\}\)without losing its validity, where𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)denotes the logically similar sample space ofxix\_\{i\}\. Formally, SampleQQcan be characterized by logical featuresA=\(P,S,T\)A=\(P,S,T\), wherePP,SS, andTTrepresent premises, subjects, and result type, respectively\. Discrete structural distance is defined asθ⁡\(Ai,Aj\)∈\{0,1\}\\theta\(A\_\{i\},A\_\{j\}\)\\in\\\{0,1\\\}\. Two samplesQiQ\_\{i\}andQjQ\_\{j\}are logically similar ifθ⁡\(Ai,Aj\)=0\\theta\(A\_\{i\},A\_\{j\}\)=0\. The logically similar samples are thus𝒩⁡\(xi\)=\{xj∣θ⁡\(Ai,Aj\)=0\}\\mathcal\{N\}\(x\_\{i\}\)=\\\{x\_\{j\}\\mid\\theta\(A\_\{i\},A\_\{j\}\)=0\\\}\. Faithfulness implies that applyingr^i\\hat\{r\}\_\{i\}to𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)preserves predictive correctness underd​odointervention:

P⁡\(Y=yi∣d​o​\(R=r^i\),X=xi\)≈P⁡\(Y=yj∣d​o​\(R=r^i\),X=xj\)\.\\begin\{split\}&P\(Y=y\_\{i\}\\mid do\(R=\\hat\{r\}\_\{i\}\),X=x\_\{i\}\)\\\\ &\\approx P\(Y=y\_\{j\}\\mid do\(R=\\hat\{r\}\_\{i\}\),X=x\_\{j\}\)\.\\end\{split\}\(1\)This formulation rests on two complementary grounds\. Faithfulness implies invariance, since any rationale that fails to remain valid across logically similar samples cannot be the underlying causal mechanism that mediates the inference\. A rationale also derives from being a procedure connecting inputs to outputs, so a rationale that fails to transfer beyond its source input may be a post\-hoc label rather than a causal explanation\. Equation[1](https://arxiv.org/html/2610.21149#S3.E1)shifts distillation from passive observation to active intervention, directly motivating ECI2D \(Section[4](https://arxiv.org/html/2610.21149#S4)\)\.

Additionally, the inference of LLMs is inevitably susceptible to latent confounders[Chi et al\. \(2024\)](https://arxiv.org/html/2610.21149#bib.bib43)\. However, transfer invariance circumvents most latent confounders, such as shortcuts in LLM memory, which fail to satisfy Equation[1](https://arxiv.org/html/2610.21149#S3.E1)due to their limited generalizability across similar samples\.

## 4The ECI2D framework

ECI2D shifts the paradigm from observation to intervention and consists of two phases: \(1\) Invariant Reasoning Distillation, where we construct a pool of causal rationales by verifying transfer invariance, and \(2\) Causal Steering Inference, where validated demonstrations guide the model during inference\.

As shown in Figure[3](https://arxiv.org/html/2610.21149#S3.F3)and Algorithm[1](https://arxiv.org/html/2610.21149#alg1), during the Invariant Reasoning Distillation phase, we introduce anAbstract\-Intervene\-Verifypipeline whose abstraction step maps a natural language rationale to a program, implemented as a Python program in the main experiments \(Section[4\.1](https://arxiv.org/html/2610.21149#S4.SS1)\) and as a hybrid program that extends the intervention principle to semantic reasoning \(Section[4\.2](https://arxiv.org/html/2610.21149#S4.SS2)\)\.

### 4\.1Invariant Reasoning Distillation with Python Program

#### Step 1: Causal Abstraction\.

Previous work has argued that rationales can be interpreted as implicit causal models[Tan \(2023\)](https://arxiv.org/html/2610.21149#bib.bib14); however, rationales expressed in natural language are high\-dimensional, ambiguous, and entangled with specific contexts\. To enable precise intervention, we convert these rationales into deterministic Python programs[Li et al\. \(2024a\)](https://arxiv.org/html/2610.21149#bib.bib28);[Gao et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib27);[Chen et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib26)via causal abstraction mapping functionΠ:ℛ→𝒫\\Pi:\\mathcal\{R\}\\to\\mathcal\{P\}\.

For training sample\(xi,yi\)\(x\_\{i\},y\_\{i\}\), an LLM GeneratorΘGen\\Theta\_\{\\text\{Gen\}\}generates a natural language rationaler^i\\hat\{r\}\_\{i\}:

r^i,y^i=ΘGen​\(xi\)\.\\hat\{r\}\_\{i\},\\hat\{y\}\_\{i\}=\\Theta\_\{\\text\{Gen\}\}\(x\_\{i\}\)\.\(2\)Then an LLM AbstractorΘΠ\\Theta\_\{\\Pi\}is utilized to convert rationaler^i\\hat\{r\}\_\{i\}into a Python programp^i∈𝒫\\hat\{p\}\_\{i\}\\in\\mathcal\{P\}:

p^i=ΘΠ​\(r^i\)\.\\hat\{p\}\_\{i\}=\\Theta\_\{\\Pi\}\(\\hat\{r\}\_\{i\}\)\.\(3\)Following self\-generated reinforced ICL, we initially filter rationales using the outcome\-oriented criterion\. The programp^i\\hat\{p\}\_\{i\}serves as a causal abstraction of rationaler^i\\hat\{r\}\_\{i\}, which allows us to perform precise causal interventions\. In our experiment, generation and abstraction were implemented in a single LLM prompt shown in Appendix[I\.1](https://arxiv.org/html/2610.21149#A9.SS1)\.

#### Step 2: Executable Causal Intervention\.

Next, we utilize the executable programp^i\\hat\{p\}\_\{i\}to perform the causal intervention\. However, sincexix\_\{i\}andxjx\_\{j\}contain different parameters, it is infeasible to directly executep^i\\hat\{p\}\_\{i\}onxjx\_\{j\}\. To this end, we decompose programp^i\\hat\{p\}\_\{i\}into a tuple⟨si,vi⟩\\langle\\mathrm\{s\}\_\{i\},\\mathrm\{v\}\_\{i\}\\rangle, wheresi\\mathrm\{s\}\_\{i\}denotes the logical skeleton, capturing the reasoning structure independent of specific parametersvi\\mathrm\{v\}\_\{i\}from inputxix\_\{i\}\. We prompt an LLM Adapter to generate an adapted programp^i→j\\hat\{p\}\_\{i\\to j\}by modifying only the parametersvi\\mathrm\{v\}\_\{i\}, while preserving the skeletonsi\\mathrm\{s\}\_\{i\}:

p^i→j=ΘAdapter​\(p^i,xj\)=⟨si,vj⟩\.\\hat\{p\}\_\{i\\to j\}=\\Theta\_\{\\text\{Adapter\}\}\(\\hat\{p\}\_\{i\},x\_\{j\}\)=\\langle\\mathrm\{s\}\_\{i\},\\mathrm\{v\}\_\{j\}\\rangle\.\(4\)Consequently, we can apply the interventiond​o​\(R=r^i\)do\(R=\\hat\{r\}\_\{i\}\)to\(xj,yj\)\(x\_\{j\},y\_\{j\}\):

y^j=Exec​\(p^i→j\),\\hat\{y\}\_\{j\}=\\text\{Exec\}\(\\hat\{p\}\_\{i\\to j\}\),\(5\)wherey^j\\hat\{y\}\_\{j\}is the output after applying interventiond​o​\(R=r^i\)do\(R=\\hat\{r\}\_\{i\}\)to the logically similar samplexjx\_\{j\}\. The detailed prompts used to adapt and intervene are in Appendix[I\.2](https://arxiv.org/html/2610.21149#A9.SS2)\. Crucially, intervention targets are drawn from the logically similar samples𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)\. We construct𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)via a two\-stage retrieval: \(1\) coarse semantic filtering using Sentence\-Transformer111https://huggingface\.co/sentence\-transformers/all\-MiniLM\-L6\-v2and \(2\) fine\-grained structural verification via an LLM that outputs a binary indicatorθ∈\{0,1\}\\theta\\in\\\{0,1\\\}to judge equivalence ofA=\(P,S,T\)A=\(P,S,T\)\. Detailed prompts are in Appendix[I\.7](https://arxiv.org/html/2610.21149#A9.SS7)\.

Table 1:Performance comparison on math and logical reasoning benchmarks\. Following standard CoT[Wei et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib3), we use 8\-shot demonstrations for math and 3\-shot for logical tasks\. We report accuracy \(%\) for all benchmarks except DROP \(F1 %\)\.\+ Codedenotes the self\-generated reinforced ICL with program\-augmented demonstrations, which is the same rationale\-and\-program format as ECI2D\. Retention rates after distillation are included\. Best results per benchmark arebold, second\-best areunderlined\. GSM\-S = GSM\-Symbolic, P1/P2 = Plus\-1/Plus\-2, Date U\. = Date Understanding, Obj\. C\. = Object Counting, Word S\. = Word Sorting, Ret\. = Retention
#### Step 3: Verification of Transfer Invariance\.

As discussed in Section[3](https://arxiv.org/html/2610.21149#S3), a faithful rationale is expected to generalize across𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)\. SinceP⁡\(Y=yj∣d​o​\(R=r^i\),X=xj\)P\(Y=y\_\{j\}\\mid do\(R=\\hat\{r\}\_\{i\}\),X=x\_\{j\}\)in Equation[1](https://arxiv.org/html/2610.21149#S3.E1)cannot be computed in closed form for an LLM, we estimate it through Monte Carlo sampling over𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)\. Each binary outcome𝕀⁡\(y^j=yj\)\\mathbb\{I\}\(\\hat\{y\}\_\{j\}=y\_\{j\}\)is an unbiased single\-sample estimator of this probability at neighborxjx\_\{j\}, and their average overNNneighbors estimates how reliablyr^i\\hat\{r\}\_\{i\}preserves correctness across𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)\. The continuous tolerance≈\\approxin Equation[1](https://arxiv.org/html/2610.21149#S3.E1)is realized as a discrete thresholdδ\\deltathat determines whetherr^i\\hat\{r\}\_\{i\}is retained\. Formally, we define an intervention\-invariant scoring functionSinvS\_\{\\text\{inv\}\}that evaluates whether a rationale remains valid under causal intervention across related inputs:

Sinv​\(xj,d​o​\(r^i\),y^j\)=𝕀⁡\(y^j=yj\),S\_\{\\text\{inv\}\}\(x\_\{j\},do\(\\hat\{r\}\_\{i\}\),\\hat\{y\}\_\{j\}\)=\\mathbb\{I\}\\\!\\left\(\\hat\{y\}\_\{j\}=y\_\{j\}\\right\),\(6\)wherey^j\\hat\{y\}\_\{j\}is calculated via Equation[5](https://arxiv.org/html/2610.21149#S4.E5)and𝕀⁡\(⋅\)\\mathbb\{I\}\(\\cdot\)denotes the indicator function\. For a sample\(xi,yi\)\(x\_\{i\},y\_\{i\}\)and a set of logically similar samples\{x1,…,xN\}⊂𝒩⁡\(xi\)\\\{x\_\{1\},\.\.\.,x\_\{N\}\\\}\\subset\\mathcal\{N\}\(x\_\{i\}\), the set of invariant causal demonstrationsDcausalD\_\{\\text\{causal\}\}is constructed as:

\{\(xi,p^i,r^i,y^i\)\|1N​∑j=1NSinv​\(xj,d​o​\(r^i\),y^j\)≥δ\},\\left\\\{\(x\_\{i\},\\hat\{p\}\_\{i\},\\hat\{r\}\_\{i\},\\hat\{y\}\_\{i\}\)\\;\\middle\|\\;\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}S\_\{\\text\{inv\}\}\(x\_\{j\},do\(\\hat\{r\}\_\{i\}\),\\hat\{y\}\_\{j\}\)\\geq\\delta\\right\\\},\(7\)where\(xj,yj\)\(x\_\{j\},y\_\{j\}\)denotes thejj\-th logically similar sample from𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)andδ\\deltadenotes the threshold\. Consequently, we explicitly store the tuple\(xi,p^i,r^i,y^i\)\(x\_\{i\},\\hat\{p\}\_\{i\},\\hat\{r\}\_\{i\},\\hat\{y\}\_\{i\}\)to preserve both the executable programp^i\\hat\{p\}\_\{i\}and natural language rationaler^i\\hat\{r\}\_\{i\}\.

### 4\.2Invariant Reasoning Distillation with Hybrid Program

A Python program cannot express steps that require world knowledge, so the LLM AbstractorΘΠ\\Theta\_\{\\Pi\}of Section[4\.1](https://arxiv.org/html/2610.21149#S4.SS1)cannot abstract such rationales\. Following Chain\-of\-Code[Li et al\. \(2024a\)](https://arxiv.org/html/2610.21149#bib.bib28), we widen ECI2D to hybrid programs, in which each such step is mapped to a function declaration answered by an LLM call\. This requires generalizing onlyΘΠ\\Theta\_\{\\Pi\}in Step 1 and the intervention in Step 2\.

#### Hybrid Causal Abstraction ofΘΠ\\Theta\_\{\\Pi\}\.

In Equation[3](https://arxiv.org/html/2610.21149#S4.E3),ΘΠ\\Theta\_\{\\Pi\}mapsr^i\\hat\{r\}\_\{i\}into three parts,

p^i=ΘΠ​\(r^i\)=⟨di,si,vi⟩,\\hat\{p\}\_\{i\}=\\Theta\_\{\\Pi\}\(\\hat\{r\}\_\{i\}\)=\\langle\\mathrm\{d\}\_\{i\},\\mathrm\{s\}\_\{i\},\\mathrm\{v\}\_\{i\}\\rangle,\(8\)wheredi\\mathrm\{d\}\_\{i\}are the declarations of the steps requiring world knowledge, each carrying a type signature and a docstring,si\\mathrm\{s\}\_\{i\}is the Python logical skeleton that calls them, andvi\\mathrm\{v\}\_\{i\}are the parameters ofsi\\mathrm\{s\}\_\{i\}\. Bothdi\\mathrm\{d\}\_\{i\}andsi\\mathrm\{s\}\_\{i\}carry the reasoning structure, so the LLM Adapter in Equation[4](https://arxiv.org/html/2610.21149#S4.E4)still modifiesvi\\mathrm\{v\}\_\{i\}alone to obtainp^i→j=ΘAdapter​\(p^i,xj\)=⟨di,si,vj⟩\\hat\{p\}\_\{i\\to j\}=\\Theta\_\{\\text\{Adapter\}\}\(\\hat\{p\}\_\{i\},x\_\{j\}\)=\\langle\\mathrm\{d\}\_\{i\},\\mathrm\{s\}\_\{i\},\\mathrm\{v\}\_\{j\}\\rangle\. Detailed prompts are in Appendix[I\.4](https://arxiv.org/html/2610.21149#A9.SS4)\.

#### Intervention for Hybrid Program\.

Sincedi\\mathrm\{d\}\_\{i\}carries no function body,p^i→j\\hat\{p\}\_\{i\\to j\}is not executable\. Each declaration is therefore resolved by an LLM EvaluatorΘEval\\Theta\_\{\\text\{Eval\}\}at the point wheresi\\mathrm\{s\}\_\{i\}calls it, which generalises Equation[5](https://arxiv.org/html/2610.21149#S4.E5)to

y^j=Exec⁡\(⟨ΘEval​\(di,aj\),si,vj⟩\),\\hat\{y\}\_\{j\}=\\mathrm\{Exec\}\\big\(\\langle\\Theta\_\{\\text\{Eval\}\}\(\\mathrm\{d\}\_\{i\},\\mathrm\{a\}\_\{j\}\),\\mathrm\{s\}\_\{i\},\\mathrm\{v\}\_\{j\}\\rangle\\big\),\(9\)whereaj\\mathrm\{a\}\_\{j\}denotes the argumentssi\\mathrm\{s\}\_\{i\}passes at that call site\.ΘEval\\Theta\_\{\\text\{Eval\}\}is givendi\\mathrm\{d\}\_\{i\}andaj\\mathrm\{a\}\_\{j\}alone, and its answer is returned tosi\\mathrm\{s\}\_\{i\}, which resumes execution and producesy^j\\hat\{y\}\_\{j\}\. The prompt forΘEval\\Theta\_\{\\text\{Eval\}\}is given in Appendix[I\.6](https://arxiv.org/html/2610.21149#A9.SS6)\. The interventiond​o​\(R=r^i\)do\(R=\\hat\{r\}\_\{i\}\)on\(xj,yj\)\(x\_\{j\},y\_\{j\}\), the retrieval of logically similar samples, and the comparison ofSinvS\_\{\\text\{inv\}\}againstδ\\deltaare the same as Section[4\.1](https://arxiv.org/html/2610.21149#S4.SS1)\. Detailed prompts are in Appendix[I\.5](https://arxiv.org/html/2610.21149#A9.SS5)\.

### 4\.3Causal Steering Inference

During the inference phase, we leverage the distilled invariant causal demonstrationsDcausalD\_\{\\text\{causal\}\}to steer the reasoning process for test sets\. Given a test queryxtestx\_\{\\text\{test\}\}, we retrieve the top\-kkdemonstrations using Sentence\-Transformer fromDcausalD\_\{\\text\{causal\}\}to construct the few\-shot ICL\. Crucially, each retrieved demonstration consists of a tuple\(x,p^,r^,y\)\(x,\\hat\{p\},\\hat\{r\},y\), wherep^\\hat\{p\}represents the causal program andr^\\hat\{r\}represents the corresponding natural language rationale\. We prompt the LLM to internalize the algorithmic precision of the programp^\\hat\{p\}while articulating the reasoning steps in the natural language format ofr^\\hat\{r\}\. Detailed inference prompts are in Appendix[I\.3](https://arxiv.org/html/2610.21149#A9.SS3)\.

## 5Main Experiments of Python Program

### 5\.1Experimental Setup

#### Benchmarks\.

We evaluate ECI2D on Math Reasoning: GSM8K[Cobbe et al\. \(2021\)](https://arxiv.org/html/2610.21149#bib.bib25), SVAMP[Patel et al\. \(2021\)](https://arxiv.org/html/2610.21149#bib.bib29), GSM\-Symbolic[Mirzadeh et al\. \(2024\)](https://arxiv.org/html/2610.21149#bib.bib30)and Logical Reasoning: DROP[Dua et al\. \(2019\)](https://arxiv.org/html/2610.21149#bib.bib31), Big\-Bench Hard \(BBH\) logical subset \(Date Understanding, Object Counting, and Word Sorting\)[Suzgun et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib32)\. Details of benchmarks are in Appendix[B\.1](https://arxiv.org/html/2610.21149#A2.SS1)and[B\.2](https://arxiv.org/html/2610.21149#A2.SS2)\. We report more results on other logical subsets of BBH in Appendix[A\.2](https://arxiv.org/html/2610.21149#A1.SS2)\.

#### Baselines and Implementation Details\.

We compare ECI2D against five baselines: Direct, Zero\-shot CoT, Manual Few\-shot CoT, Self\-Generated Reinforced ICL, and Self\-Generated Reinforced ICL with program\-augmented demonstrations\. We mainly employ two instruction models,Meta\-Llama\-3\-8B\-Instruct222https://huggingface\.co/meta\-llama/Meta\-Llama\-3\-8BandQwen2\.5\-7B\-Instruct333https://huggingface\.co/Qwen/Qwen2\.5\-7B\-Instruct\. Detailed settings for baselines and implementation can be found in Appendix[C](https://arxiv.org/html/2610.21149#A3)\. We report more results of reasoning models in Appendix[A\.1](https://arxiv.org/html/2610.21149#A1.SS1), and compare against PoT[Chen et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib26)in Appendix[A\.5](https://arxiv.org/html/2610.21149#A1.SS5)\.

Table 2:Ablation study analyzing different rationale components\. We compare performance across three configurations: Program\-onlyp^\\hat\{p\}\(Prog\.\), Language rationaler^\\hat\{r\}alone \(Lang\.\), and Both \(p^\\hat\{p\}andr^\\hat\{r\}\)\. Best per model and benchmark inbold\. GSM\-S = GSM\-Symbolic, P1/P2 = Plus\-1/Plus\-2\.

### 5\.2Main Results

#### Performance Comparison\.

Table[1](https://arxiv.org/html/2610.21149#S4.T1)shows that while self\-generated reinforced ICL marginally improves upon manual prompts in some math datasets, it suffers from degradation in logical reasoning scenarios\. By contrast, ECI2D improves over it on every benchmark, and its advantages are most pronounced in OOD scenarios, boosting accuracy by over 7 points on the hardest GSM\-Symbolic\-P2 with Qwen2\.5\-7B where other methods falter\. Although manual Few\-shot CoT retains the lead on parts of BBH, owing to the domain gap where DROP centers on historical events unlike the explicit date calculation in Date Understanding, ECI2D still consistently outperforms Self\-Generated Reinforced ICL there\. These results indicate that our interventional filtering distills faithful rationales invariant across distribution shifts: the GSM8K margin falls within the seed\-to\-seed variation of distillation \(Appendix[A\.3](https://arxiv.org/html/2610.21149#A1.SS3)\), whereas the DROP and largest OOD gains far exceed it\.

ECI2D also maintains high data efficiency, reducing the final demonstration counts by 39% to 57% compared to the outcome\-oriented baseline\. This reduction implies ECI2D effectively discards a large number of observationally correct but causally spurious post\-hoc examples\. Subsampling that baseline to the same pool size degrades it on every benchmark tested \(Appendix[A\.4](https://arxiv.org/html/2610.21149#A1.SS4)\), so the compactness ofDcausalD\_\{\\text\{causal\}\}is a cost rather than a source of the advantage\. Transfer invariance is therefore a more reliable standard for demonstration distillation than simple observational correctness\.

#### Ablation Study\.

The ablation results in Table[2](https://arxiv.org/html/2610.21149#S5.T2)provide empirical evidence that combining executable programsp^\\hat\{p\}with natural language rationalesr^\\hat\{r\}yields a consistent performance benefit\. The hybrid configuration utilizing bothp^\\hat\{p\}andr^\\hat\{r\}outperforms single\-component setups on all ten model–benchmark pairs, and the ordering is identical throughout\. The gap highlights the complementary roles of the two components\. The Program\-only setting performs worse than the Language\-only setting, reflecting a semantic mismatch between code and natural language reasoning\. Since the model answers in natural language without executing the program, a demonstration containing onlyp^\\hat\{p\}carries the computation but not the form in which the answer must be produced\. Nevertheless, the program remains necessary, as it supplies a deterministic algorithmic scaffold that constrains the reasoning search to invariant trajectories, while the language rationale provides the linguistic mapping that translates those trajectories into expressive and interpretable explanations\. The\+ Coderow of Table[1](https://arxiv.org/html/2610.21149#S4.T1)separates this format from the filtering criterion, using the identical prompt as ECI2D but retaining candidates by observational correctness alone\. ECI2D outperforms it on all nine benchmarks under both models, and adding code to the observational pipeline does not improve it consistently, so the gains stem from interventional verification rather than from the presence of code\.

Table 3:Impact of sample numberNNon performance\. We report accuracy \(%\) across benchmarks\. The column ‘Num’ denotes the size of the causal pool\|Dcausal\|\|D\_\{\\text\{causal\}\}\|\.
#### Hyperparameter Sensitivity\.

We investigate how the countNNof logically similar samples affects the distilled demonstrations\. As shown in Table[3](https://arxiv.org/html/2610.21149#S5.T3), performance does not increase monotonically withNN; the degradation is mild forLlama3but reaches8\.968\.96points on GSM\-Symbolic\-P2 forQwen2\.5whenNNrises from33to55\. This trend highlights a critical trade\-off between verification rigor and demonstration diversity\. AsNNincreases, the retrieved samples inevitably include those of greater semantic distance, where minor structural deviations cause valid causal skeletons to fail the strict execution check\.

#### Robustness\.

ECI2D relies on LLMs at three stages upstream of the filter, raising concerns about cascading error propagation\. Retrieval may return a neighbor that does not share the reasoning structure, abstraction may formalize the rationale incorrectly, and adaptation may alter the skeleton or misbind the parameters of the target instance\. Appendix[H](https://arxiv.org/html/2610.21149#A8)quantifies the first two by human annotation, reporting high agreement, and illustrates each adaptation failure\. To probe the third we deliberately degrade the Abstractor and Adapter on GSM8K \(Qwen2\.5\-7B\), relaxing the prompt constraints and substitutingQwen2\.5\-1\.5B\. As shown in Table[4](https://arxiv.org/html/2610.21149#S5.T4), retention falls from 57\.3% to 24\.6% while downstream accuracy remains above 87 points\. Upstream errors therefore cost coverage rather than pool quality, since any failed check yieldsSinv=0S\_\{\\text\{inv\}\}=0and discards the candidate\.

Table 4:Impact of Abstractor\(Abs\.\)/Adapter\(Adpt\.\) degradation on error propagation and downstream performance \(GSM8K, Qwen2\.5\-7B\)\. Ret\. = Retention

## 6Additional Experiments of Hybrid Program

### 6\.1Experimental Setup

#### Benchmarks\.

For the hybrid program abstraction of Section[4\.2](https://arxiv.org/html/2610.21149#S4.SS2), we use CommonsenseQA[Talmor et al\. \(2019\)](https://arxiv.org/html/2610.21149#bib.bib47)and BBH semantic subset \(Sports Understanding and Ruin Names\)[Suzgun et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib32)\. Details of benchmarks are in Appendix[B\.3](https://arxiv.org/html/2610.21149#A2.SS3)\.

#### Baselines and Implementation Details\.

We evaluate ECI2D with hybrid program withQwen2\.5\-7B\-Instructagainst the same baselines in Section[5](https://arxiv.org/html/2610.21149#S5)\. By default, we distill from the CommonsenseQA training set withN=3N=3\. More detailed settings for baselines and implementation can be found in Appendix[C](https://arxiv.org/html/2610.21149#A3)\.

### 6\.2Results

Table[5](https://arxiv.org/html/2610.21149#S6.T5)reports the hybrid abstraction on three benchmarks that admit no deterministic program\. Requiring one is harmful here, as the\+ Codebaseline falls below Self\-Generated Reinforced ICL on two of the three and below direct answering on CommonsenseQA\. The hybrid abstraction improves over both on all three and achieves the best accuracy on CommonsenseQA, exceeding Self\-Generated Reinforced ICL by2\.302\.30points there and by4\.404\.40and3\.603\.60points on the two BBH subsets, while retaining 58\.8% of the training set against 79\.1% for the observational baseline\. Manual Few\-shot CoT retains the lead on BBH, as in the main experiments, owing to the domain gap where the pool is distilled on CommonsenseQA alone and transferred across domains\. On Ruin Names this transfer also falls below zero\-shot inference\. Of the interventions that score zero, 95% return a wrong option and the remainder fail to execute, so the criterion registers whether the reasoning transfers rather than whether the program runs\. Appendix[D](https://arxiv.org/html/2610.21149#A4)reports the full decomposition with the constraints enforced on the abstraction and case study\. The criterion therefore extends beyond the domains where a deterministic program can be written\.

Table 5:Performance of the hybrid abstraction\. The pool is distilled on CommonsenseQA alone, so the two BBH subsets measure cross\-domain transfer\. Following standard CoT[Wei et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib3), we use 7\-shot demonstrations on CommonsenseQA and 3\-shot on BBH, and report accuracy \(%\)\.\+ Coderestricts the abstraction to a deterministic Python program and retains candidates by observational correctness alone\. Best per benchmark inbold, second\-bestunderlined\. Sports = Sports Understanding, Ruin = Ruin Names, Ret\. = Retention\.

## 7Conclusion

In this paper, we addressed the critical challenge ofobservational equivalencein self\-generated In\-Context Learning \(ICL\), where standard observation\-based filtering fails to distinguish between faithful causal rationales and spurious post\-hoc rationales\. We argued that a faithful relation should remain effective across similar environments, implyingtransfer invariance\. To instantiate this, we introduced the Executable Causal Intervention for Demonstrations Distillation \(ECI2D\) framework\. By abstracting natural language rationales into executable programs and applying thed​odo\-operator via causal interventions, ECI2D shifts the demonstration selection paradigm from passive observation to active intervention\. ECI2D outperforms outcome\-oriented reinforced ICL by over 7 points while retaining fewer demonstrations, with gains scaling under increasing distribution shift\. These results establish intervention\-based invariance as a sound criterion for distilling faithful rationales in self\-generated In\-Context Learning\.

## Limitations

Computational cost\.ECI2D adds overhead during offline distillation: iterative rationale sampling, program abstraction and adaptation, neighbor retrieval, and repeated sandbox execution\. ConstructingDcausalD\_\{\\text\{causal\}\}therefore costs more than outcome\-oriented filtering, although the cost is amortized offline and inference is unaffected \(Appendix[G](https://arxiv.org/html/2610.21149#A7)\)\.

Availability of logically similar samples\.Verification requiresNNinstances sharing a candidate’s logical skeleton, which ECI2D obtains by retrieval from the training set\. The criterion is thus applicable only where such instances exist, and degrades when the data are scarce or problem structures diverse\. Section[5\.2](https://arxiv.org/html/2610.21149#S5.SS2)exhibits the dependency: the pool shrinks from3,9133\{,\}913to1,3871\{,\}387candidates on GSM8K asNNrises from33to88, so genuinely same\-structure instances are not abundant even at this scale\. Constructing the neighborhood from the source instance rather than retrieving it would remove the dependency, which we leave to future work\.

Incomplete identification\.Transfer invariance is a necessary but not sufficient condition on faithfulness \(Appendix[F](https://arxiv.org/html/2610.21149#A6)\)\. It rules out latent factors that vary across𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\), such as memorized\(x,y\)\(x,y\)associations and surface\-level shortcuts, but not factors that act uniformly within the verified neighborhood\. ECI2D should therefore be read as a conservative local filter rather than an identifier of the causal mechanism\. Its residual error is asymmetric in direction\. Abstraction and adaptation failures cost coverage \(Section[5\.2](https://arxiv.org/html/2610.21149#S5.SS2.SSS0.Px4)\), whereas silent repair by the abstractor can admit an unfaithful rationale and is not measured here \(Appendix[H](https://arxiv.org/html/2610.21149#A8)\)\.

## References

- Agarwalet al\.\(2024\)R\. Agarwal, A\. Singh, L\. Zhang, B\. Bohnet, L\. Rosias, S\. Chan, B\. Zhang, A\. Anand, Z\. Abbas, A\. Nova, J\. D\. Co\-Reyes, E\. Chu, F\. Behbahani, A\. Faust, and H\. LarochelleMany\-shot in\-context learning\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 76930–76966\.External Links:[Document](https://dx.doi.org/10.52202/079017-2447),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/8cb564df771e9eacbfe9d72bd46a24a9-Paper-Conference.pdf)Cited by:[§C\.1](https://arxiv.org/html/2610.21149#A3.SS1.p5.1),[§1](https://arxiv.org/html/2610.21149#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21149#S2.SS1.p1.1)\.
- Bareinboim and Pearl \(2012\)E\. Bareinboim and J\. PearlControlling selection bias in causal inference\.InProceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics,N\. D\. Lawrence and M\. Girolami \(Eds\.\),Proceedings of Machine Learning Research, Vol\.22,La Palma, Canary Islands,pp\. 100–108\.External Links:[Link](https://proceedings.mlr.press/v22/bareinboim12.html)Cited by:[§3\.1](https://arxiv.org/html/2610.21149#S3.SS1.p3.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2610.21149#S1.p1.1)\.
- Chenet al\.\(2023\)W\. Chen, C\. Wu, Y\. Chen, and H\. ChenSelf\-ICL: zero\-shot in\-context learning with self\-generated demonstrations\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 15651–15662\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.968/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.968)Cited by:[§1](https://arxiv.org/html/2610.21149#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21149#S2.SS1.p1.1)\.
- Chenet al\.\(2022\)W\. Chen, X\. Ma, X\. Wang, and W\. W\. CohenProgram of thoughts prompting: disentangling computation from reasoning for numerical reasoning tasks\.arXiv preprint arXiv:2211\.12588\.Cited by:[§A\.5](https://arxiv.org/html/2610.21149#A1.SS5.p1.1),[§C\.1](https://arxiv.org/html/2610.21149#A3.SS1.p7.1),[§2\.2](https://arxiv.org/html/2610.21149#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.21149#S4.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.21149#S5.SS1.SSS0.Px2.p1.1)\.
- Chiet al\.\(2024\)H\. Chi, H\. Li, W\. Yang, F\. Liu, L\. Lan, X\. Ren, T\. Liu, and B\. HanUnveiling causal reasoning in large language models: reality or mirage?\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 96640–96670\.External Links:[Document](https://dx.doi.org/10.52202/079017-3064),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/af2bb2b2280d36f8842e440b4e275152-Paper-Conference.pdf)Cited by:[§3\.2](https://arxiv.org/html/2610.21149#S3.SS2.p3.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§B\.1](https://arxiv.org/html/2610.21149#A2.SS1.p1.1),[Figure 1](https://arxiv.org/html/2610.21149#S1.F1),[§5\.1](https://arxiv.org/html/2610.21149#S5.SS1.SSS0.Px1.p1.1)\.
- Donget al\.\(2024\)Q\. Dong, L\. Li, D\. Dai, C\. Zheng, J\. Ma, R\. Li, H\. Xia, J\. Xu, Z\. Wu, B\. Chang, X\. Sun, L\. Li, and Z\. SuiA survey on in\-context learning\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 1107–1128\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.64/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.64)Cited by:[§1](https://arxiv.org/html/2610.21149#S1.p1.1)\.
- Duaet al\.\(2019\)D\. Dua, Y\. Wang, P\. Dasigi, G\. Stanovsky, S\. Singh, and M\. GardnerDROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 2368–2378\.External Links:[Link](https://aclanthology.org/N19-1246/),[Document](https://dx.doi.org/10.18653/v1/N19-1246)Cited by:[§B\.2](https://arxiv.org/html/2610.21149#A2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.21149#S5.SS1.SSS0.Px1.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. NeubigPAL: program\-aided language models\.InProceedings of the 40th International Conference on Machine Learning,ICML’23\.Cited by:[§2\.2](https://arxiv.org/html/2610.21149#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.21149#S4.SS1.SSS0.Px1.p1.1)\.
- Geigeret al\.\(2020\)A\. Geiger, K\. Richardson, and C\. PottsNeural natural language inference models partially embed theories of lexical entailment and negation\.InProceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP,A\. Alishahi, Y\. Belinkov, G\. Chrupała, D\. Hupkes, Y\. Pinter, and H\. Sajjad \(Eds\.\),Online,pp\. 163–173\.External Links:[Link](https://aclanthology.org/2020.blackboxnlp-1.16/),[Document](https://dx.doi.org/10.18653/v1/2020.blackboxnlp-1.16)Cited by:[§2\.2](https://arxiv.org/html/2610.21149#S2.SS2.p1.1)\.
- Holzingeret al\.\(2017\)A\. Holzinger, C\. Biemann, C\. S\. Pattichis, and D\. B\. KellWhat do we need to build explainable ai systems for the medical domain?\.arXiv preprint arXiv:1712\.09923\.Cited by:[§1](https://arxiv.org/html/2610.21149#S1.p2.1),[§3\.1](https://arxiv.org/html/2610.21149#S3.SS1.p1.1)\.
- Jacovi and Goldberg \(2020\)A\. Jacovi and Y\. GoldbergTowards faithfully interpretable NLP systems: how should we define and evaluate faithfulness?\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 4198–4205\.External Links:[Link](https://aclanthology.org/2020.acl-main.386/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.386)Cited by:[§2\.1](https://arxiv.org/html/2610.21149#S2.SS1.p2.1)\.
- Kimet al\.\(2022\)H\. J\. Kim, H\. Cho, J\. Kim, T\. Kim, K\. M\. Yoo, and S\. LeeSelf\-generated in\-context learning: leveraging auto\-regressive language models as a demonstration generator\.arXiv preprint arXiv:2206\.08082\.Cited by:[§C\.1](https://arxiv.org/html/2610.21149#A3.SS1.p5.1),[§1](https://arxiv.org/html/2610.21149#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21149#S2.SS1.p1.1)\.
- Kojimaet al\.\(2022\)T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. IwasawaLarge language models are zero\-shot reasoners\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§C\.1](https://arxiv.org/html/2610.21149#A3.SS1.p3.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the 29th Symposium on Operating Systems Principles,SOSP ’23,New York, NY, USA,pp\. 611–626\.External Links:ISBN 9798400702297,[Link](https://doi.org/10.1145/3600006.3613165),[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[§C\.2](https://arxiv.org/html/2610.21149#A3.SS2.p6.1)\.
- Lanhamet al\.\(2023\)T\. Lanham, A\. Chen, A\. Radhakrishnan, B\. Steiner, C\. Denison, D\. Hernandez, D\. Li, E\. Durmus, E\. Hubinger, J\. Kernion,et al\.Measuring faithfulness in chain\-of\-thought reasoning\.arXiv preprint arXiv:2307\.13702\.Cited by:[§2\.1](https://arxiv.org/html/2610.21149#S2.SS1.p2.1),[§3\.1](https://arxiv.org/html/2610.21149#S3.SS1.p1.1)\.
- Liet al\.\(2024a\)C\. Li, J\. Liang, A\. Zeng, X\. Chen, K\. Hausman, D\. Sadigh, S\. Levine, L\. Fei\-Fei, F\. Xia, and B\. IchterChain of code: reasoning with a language model\-augmented code emulator\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 28259–28277\.External Links:[Link](https://proceedings.mlr.press/v235/li24ar.html)Cited by:[§2\.2](https://arxiv.org/html/2610.21149#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.21149#S4.SS1.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.21149#S4.SS2.p1.1)\.
- Liet al\.\(2024b\)R\. Li, G\. Wang, and J\. LiAre human\-generated demonstrations necessary for in\-context learning?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=frRDT6EOhg)Cited by:[§C\.1](https://arxiv.org/html/2610.21149#A3.SS1.p5.1),[§1](https://arxiv.org/html/2610.21149#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21149#S2.SS1.p1.1)\.
- Liuet al\.\(2022\)J\. Liu, D\. Shen, Y\. Zhang, B\. Dolan, L\. Carin, and W\. ChenWhat makes good in\-context examples for GPT\-3?\.InProceedings of Deep Learning Inside Out \(DeeLIO 2022\): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures,E\. Agirre, M\. Apidianaki, and I\. Vulić \(Eds\.\),Dublin, Ireland and Online,pp\. 100–114\.External Links:[Link](https://aclanthology.org/2022.deelio-1.10/),[Document](https://dx.doi.org/10.18653/v1/2022.deelio-1.10)Cited by:[§1](https://arxiv.org/html/2610.21149#S1.p1.1)\.
- Minet al\.\(2022\)S\. Min, X\. Lyu, A\. Holtzman, M\. Artetxe, M\. Lewis, H\. Hajishirzi, and L\. ZettlemoyerRethinking the role of demonstrations: what makes in\-context learning work?\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 11048–11064\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.759/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.759)Cited by:[§1](https://arxiv.org/html/2610.21149#S1.p1.1)\.
- Mirzadehet al\.\(2024\)I\. Mirzadeh, K\. Alizadeh, H\. Shahrokhi, O\. Tuzel, S\. Bengio, and M\. FarajtabarGSM\-symbolic: understanding the limitations of mathematical reasoning in large language models\.External Links:[Link](https://arxiv.org/abs/2410.05229)Cited by:[§B\.1](https://arxiv.org/html/2610.21149#A2.SS1.p3.1),[§5\.1](https://arxiv.org/html/2610.21149#S5.SS1.SSS0.Px1.p1.1)\.
- Olaussonet al\.\(2023\)T\. Olausson, A\. Gu, B\. Lipkin, C\. Zhang, A\. Solar\-Lezama, J\. Tenenbaum, and R\. LevyLINC: a neurosymbolic approach for logical reasoning by combining language models with first\-order logic provers\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 5153–5176\.External Links:[Link](http://dx.doi.org/10.18653/v1/2023.emnlp-main.313),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.313)Cited by:[§2\.2](https://arxiv.org/html/2610.21149#S2.SS2.p1.1)\.
- Panet al\.\(2023\)L\. Pan, A\. Albalak, X\. Wang, and W\. WangLogic\-LM: empowering large language models with symbolic solvers for faithful logical reasoning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 3806–3824\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.248/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.248)Cited by:[§2\.2](https://arxiv.org/html/2610.21149#S2.SS2.p1.1)\.
- Patelet al\.\(2021\)A\. Patel, S\. Bhattamishra, and N\. GoyalAre NLP models really able to solve simple math word problems?\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 2080–2094\.External Links:[Link](https://aclanthology.org/2021.naacl-main.168/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.168)Cited by:[§B\.1](https://arxiv.org/html/2610.21149#A2.SS1.p2.1),[§5\.1](https://arxiv.org/html/2610.21149#S5.SS1.SSS0.Px1.p1.1)\.
- Pearl and Bareinboim \(2011\)J\. Pearl and E\. BareinboimTransportability of causal and statistical relations: a formal approach\.In2011 IEEE 11th International Conference on Data Mining Workshops,Vol\.,pp\. 540–547\.External Links:[Document](https://dx.doi.org/10.1109/ICDMW.2011.169)Cited by:[§1](https://arxiv.org/html/2610.21149#S1.p4.1),[§3\.2](https://arxiv.org/html/2610.21149#S3.SS2.p1.1)\.
- Pearl \(2010\)J\. PearlCausal inference\.Causality: objectives and assessment,pp\. 39–58\.Cited by:[Appendix E](https://arxiv.org/html/2610.21149#A5.p1.3),[§3\.1](https://arxiv.org/html/2610.21149#S3.SS1.p2.1)\.
- Peterset al\.\(2016\)J\. Peters, P\. Bühlmann, and N\. MeinshausenCausal inference by using invariant prediction: identification and confidence intervals\.Journal of the Royal Statistical Society Series B: Statistical Methodology78\(5\),pp\. 947–1012\.Cited by:[§1](https://arxiv.org/html/2610.21149#S1.p4.1),[§3\.2](https://arxiv.org/html/2610.21149#S3.SS2.p1.1)\.
- Peterset al\.\(2017\)J\. Peters, D\. Janzing, and B\. SchlkopfElements of causal inference: foundations and learning algorithms\.The MIT Press\.External Links:ISBN 0262037319Cited by:[Appendix E](https://arxiv.org/html/2610.21149#A5.p1.3)\.
- Schölkopfet al\.\(2012\)B\. Schölkopf, D\. Janzing, J\. Peters, E\. Sgouritsa, K\. Zhang, and J\. MooijOn causal and anticausal learning\.InProceedings of the 29th International Conference on International Conference on Machine Learning,ICML’12,Madison, WI, USA,pp\. 459–466\.External Links:ISBN 9781450312851Cited by:[Appendix F](https://arxiv.org/html/2610.21149#A6.p1.1),[§1](https://arxiv.org/html/2610.21149#S1.p4.1),[§3\.2](https://arxiv.org/html/2610.21149#S3.SS2.p1.1)\.
- Sharmaet al\.\(2024\)M\. Sharma, M\. Tong, T\. Korbak, D\. Duvenaud, A\. Askell, S\. R\. Bowman, E\. DURMUS, Z\. Hatfield\-Dodds, S\. R\. Johnston, S\. M\. Kravec, T\. Maxwell, S\. McCandlish, K\. Ndousse, O\. Rausch, N\. Schiefer, D\. Yan, M\. Zhang, and E\. PerezTowards understanding sycophancy in language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tvhaxkMKAn)Cited by:[§2\.1](https://arxiv.org/html/2610.21149#S2.SS1.p2.1)\.
- Singhet al\.\(2024\)A\. Singh, J\. D\. Co\-Reyes, R\. Agarwal, A\. Anand, P\. Patil, X\. Garcia, P\. J\. Liu, J\. Harrison, J\. Lee, K\. Xu, A\. T\. Parisi, A\. Kumar, A\. A\. Alemi, A\. Rizkowsky, A\. Nova, B\. Adlam, B\. Bohnet, G\. F\. Elsayed, H\. Sedghi, I\. Mordatch, I\. Simpson, I\. Gur, J\. Snoek, J\. Pennington, J\. Hron, K\. Kenealy, K\. Swersky, K\. Mahajan, L\. A\. Culp, L\. Xiao, M\. Bileschi, N\. Constant, R\. Novak, R\. Liu, T\. Warkentin, Y\. Bansal, E\. Dyer, B\. Neyshabur, J\. Sohl\-Dickstein, and N\. FiedelBeyond human data: scaling self\-training for problem\-solving with language models\.Transactions on Machine Learning Research\.Note:Expert CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=lNAyUngGFK)Cited by:[§1](https://arxiv.org/html/2610.21149#S1.p1.1)\.
- Spirteset al\.\(2000\)P\. Spirtes, C\. N\. Glymour, and R\. ScheinesCausation, prediction, and search\.MIT press\.Cited by:[Appendix E](https://arxiv.org/html/2610.21149#A5.p1.3),[§3\.1](https://arxiv.org/html/2610.21149#S3.SS1.p3.1)\.
- Suzgunet al\.\(2023\)M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou, and J\. WeiChallenging BIG\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13003–13051\.External Links:[Link](https://aclanthology.org/2023.findings-acl.824/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.824)Cited by:[§B\.2](https://arxiv.org/html/2610.21149#A2.SS2.p2.1),[§B\.3](https://arxiv.org/html/2610.21149#A2.SS3.p2.1),[§C\.1](https://arxiv.org/html/2610.21149#A3.SS1.p4.1),[§5\.1](https://arxiv.org/html/2610.21149#S5.SS1.SSS0.Px1.p1.1),[§6\.1](https://arxiv.org/html/2610.21149#S6.SS1.SSS0.Px1.p1.1)\.
- Talmoret al\.\(2019\)A\. Talmor, J\. Herzig, N\. Lourie, and J\. BerantCommonsenseQA: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4149–4158\.External Links:[Link](https://aclanthology.org/N19-1421/),[Document](https://dx.doi.org/10.18653/v1/N19-1421)Cited by:[§B\.3](https://arxiv.org/html/2610.21149#A2.SS3.p1.1),[§6\.1](https://arxiv.org/html/2610.21149#S6.SS1.SSS0.Px1.p1.1)\.
- Tan \(2023\)J\. \(\. TanCausal abstraction for chain\-of\-thought reasoning in arithmetic word problems\.InProceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,Y\. Belinkov, S\. Hao, J\. Jumelet, N\. Kim, A\. McCarthy, and H\. Mohebbi \(Eds\.\),Singapore,pp\. 155–168\.External Links:[Link](https://aclanthology.org/2023.blackboxnlp-1.12/),[Document](https://dx.doi.org/10.18653/v1/2023.blackboxnlp-1.12)Cited by:[§2\.2](https://arxiv.org/html/2610.21149#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.21149#S4.SS1.SSS0.Px1.p1.1)\.
- Team \(2025\)Q\. TeamQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[Table 7](https://arxiv.org/html/2610.21149#A0.T7)\.
- Turpinet al\.\(2023\)M\. Turpin, J\. Michael, E\. Perez, and S\. R\. BowmanLanguage models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=bzs4uPLXvi)Cited by:[§1](https://arxiv.org/html/2610.21149#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.21149#S2.SS1.p2.1)\.
- Verma and Pearl \(2022\)T\. Verma and J\. PearlEquivalence and synthesis of causal models\.InProbabilistic and Causal Inference: The Works of Judea Pearl,pp\. 221–236\.External Links:ISBN 9781450395861,[Link](https://doi.org/10.1145/3501714.3501732)Cited by:[Appendix E](https://arxiv.org/html/2610.21149#A5.p1.3),[§3\.1](https://arxiv.org/html/2610.21149#S3.SS1.p3.1)\.
- Wanet al\.\(2025\)X\. Wan, H\. Zhou, R\. Sun, and S\. O\. ArikFrom few to many: self\-improving many\-shot reasoners through iterative optimization and generation\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=JBXO05r4AV)Cited by:[§C\.1](https://arxiv.org/html/2610.21149#A3.SS1.p5.1),[§1](https://arxiv.org/html/2610.21149#S1.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, b\. ichter, F\. Xia, E\. Chi, Q\. V\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 24824–24837\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf)Cited by:[§C\.1](https://arxiv.org/html/2610.21149#A3.SS1.p4.1),[§1](https://arxiv.org/html/2610.21149#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21149#S2.SS1.p2.1),[Table 1](https://arxiv.org/html/2610.21149#S4.T1),[Table 5](https://arxiv.org/html/2610.21149#S6.T5)\.
- Wuet al\.\(2023\)Z\. Wu, A\. Geiger, T\. Icard, C\. Potts, and N\. GoodmanInterpretability at scale: identifying causal mechanisms in alpaca\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 78205–78226\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/f6a8b109d4d4fd64c75e94aaf85d9697-Paper-Conference.pdf)Cited by:[§2\.2](https://arxiv.org/html/2610.21149#S2.SS2.p1.1)\.
- Yamakoshiet al\.\(2023\)T\. Yamakoshi, J\. McClelland, A\. Goldberg, and R\. HawkinsCausal interventions expose implicit situation models for commonsense language understanding\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13265–13293\.External Links:[Link](https://aclanthology.org/2023.findings-acl.839/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.839)Cited by:[§2\.2](https://arxiv.org/html/2610.21149#S2.SS2.p1.1)\.
- Yanget al\.\(2024\)J\. Yang, S\. Ma, and F\. WeiAuto\-icl: in\-context learning without human supervision\.External Links:2311\.09263,[Link](https://arxiv.org/abs/2311.09263)Cited by:[§1](https://arxiv.org/html/2610.21149#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21149#S2.SS1.p1.1)\.
- Yeet al\.\(2023\)X\. Ye, Q\. Chen, I\. Dillig, and G\. DurrettSatLM: satisfiability\-aided language models using declarative prompting\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=TqW5PL1Poi)Cited by:[§2\.2](https://arxiv.org/html/2610.21149#S2.SS2.p1.1)\.
- Zhanget al\.\(2023\)Z\. Zhang, A\. Zhang, M\. Li, and A\. SmolaAutomatic chain of thought prompting in large language models\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=5NTt8GFjUHkr)Cited by:[§C\.1](https://arxiv.org/html/2610.21149#A3.SS1.p5.1),[§1](https://arxiv.org/html/2610.21149#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21149#S2.SS1.p1.1)\.
- Zhouet al\.\(2023\)D\. Zhou, N\. Schärli, L\. Hou, J\. Wei, N\. Scales, X\. Wang, D\. Schuurmans, C\. Cui, O\. Bousquet, Q\. V\. Le, and E\. H\. ChiLeast\-to\-most prompting enables complex reasoning in large language models\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=WZH7099tgfM)Cited by:[§C\.1](https://arxiv.org/html/2610.21149#A3.SS1.p4.1)\.

Table 6:More results on math reasoning benchmarks usingDeepSeek\-R1\-Distill\-Qwen\-7B\. We utilize 8\-shot demonstrations\. We reproduce all baselines and report accuracy across all benchmarks\. The demonstration retention rate after distillation is also reported\. The best results for each setting are marked inbold\.Table 7:More results on math reasoning benchmarks usingQwen3\-8B\. Following[Team \(2025\)](https://arxiv.org/html/2610.21149#bib.bib42), we utilize 4\-shot demonstrations\. We reproduce all baselines and report accuracy across all benchmarks\. The demonstration retention rate after distillation is also reported\. The best results for each setting are marked inbold\.## Appendix AMore Experiment Results

### A\.1More models in Math Reasoning

While our primary experiments align with standard self\-generated ICL protocols using instruction\-tuned LLMs, we extend our evaluation to reasoning\-specialized architectures to assess the architectural generalizability of ECI2D\. Specifically, we conduct math reasoning benchmarks onDeepSeek\-R1\-Distill\-Qwen\-7B444https://huggingface\.co/deepseek\-ai/DeepSeek\-R1\-Distill\-Qwen\-7BandQwen3\-8B555https://huggingface\.co/Qwen/Qwen3\-8B\. As detailed in Table[6](https://arxiv.org/html/2610.21149#A0.T6)and Table[7](https://arxiv.org/html/2610.21149#A0.T7), ECI2D surpasses the self\-generated reinforced ICL baseline on all evaluated benchmarks\. Notably, this advantage is achieved despite a lower demonstration retention rate, suggesting that the distilled subset prioritizes fidelity over volume\. Two observations follow\. First, the intervention\-based criterion transfers across the four models we evaluate, including architectures explicitly optimized for chain\-of\-thought generation\. Second, the persistent gap suggests that reasoning\-specialized models also produce rationales which satisfy outcome\-oriented verification without transferring to logically similar samples\.

### A\.2More datasets in Logical Reasoning

We extend our analysis to the Boolean Expressions and Penguins in a Table subsets\. Semantic subsets of BBH are covered by the hybrid program of Section[6](https://arxiv.org/html/2610.21149#S6)and Appendix[D](https://arxiv.org/html/2610.21149#A4)\. As detailed in Table[8](https://arxiv.org/html/2610.21149#A1.T8), ECI2D achieves performance comparable to manual prompts while surpassing the self\-generated reinforced ICL baseline on both subsets and under both models, which indicates that transfer\-invariant distillation extends to logical benchmarks beyond those used in our main experiments\.

Table 8:More results on logical reasoning benchmarks across different models\. The best results for each setting are marked inbold, and the second\-best results areunderlined\.
### A\.3Stability across Distillation Seeds

Inference uses greedy decoding \(τ=0\.0\\tau=0\.0\) and is therefore deterministic, but constructingDcausalD\_\{\\text\{causal\}\}relies on sampling \(τ=0\.7\\tau=0\.7\) and is seed\-dependent\. We rebuild the causal pool from scratch under three independent seeds \(4242,4343,4444\) and report the resulting distribution in Table[9](https://arxiv.org/html/2610.21149#A1.T9); Table[1](https://arxiv.org/html/2610.21149#S4.T1)reports the run with seed4242\. Variation is below0\.70\.7points on six of the nine benchmarks and below2\.52\.5points on all of them, and retention rates vary by less than0\.40\.4points, indicating that the composition of the distilled pool is stable rather than an artifact of a favorable sample\. The comparison also delimits which margins in Table[1](https://arxiv.org/html/2610.21149#S4.T1)are meaningful: the GSM8K margins \(\+0\.23 and \+0\.16\) are of the same order as this variation, whereas the gains on DROP \(\+5\.88 and \+7\.81\) and under distribution shift \(e\.g\., \+7\.08 and \+7\.16 on GSM\-Symbolic\-P1/P2 with Qwen2\.5\) exceed it by one to two orders of magnitude\.

Table 9:Stability of ECI2D across three independent distillation seeds \(4242,4343,4444\), with the causal poolDcausalD\_\{\\text\{causal\}\}rebuilt from scratch for each seed\.Seed 42reproduces the run reported in Table[1](https://arxiv.org/html/2610.21149#S4.T1)\. Ret\. = Retention\.
### A\.4Size\-Matched Observational Pool

The causal pool is smaller than the observational one, which raises the question of whether the improvement follows from pool size rather than from pool composition\. To separate the two, we uniformly subsample the observational pool to\|Dcausal\|\|D\_\{\\text\{causal\}\}\|and repeat inference with identical retrieval, prompt format, shot count and decoding\. The subsample is drawn from the observationally filtered pool and is not constrained to lie withinDcausalD\_\{\\text\{causal\}\}\.

As shown in Table[10](https://arxiv.org/html/2610.21149#A1.T10), reducing the pool degrades the observational baseline on all three benchmarks, by2\.732\.73points on GSM8K,1\.681\.68on GSM\-Symbolic\-P2 and1\.451\.45on DROP\. Pool size therefore acts in the opposite direction to the one the concern presupposes\. The compactness ofDcausalD\_\{\\text\{causal\}\}is a cost that ECI2D absorbs rather than a source of its advantage, and at equal pool size the margin over the observational criterion widens on every benchmark\.

Random subsampling is the appropriate null model here because it preserves the composition of the observational pool in expectation, so any difference from the full pool is attributable to size alone\. It is not, however, the strongest selection heuristic available at that size, and the comparison should be read as isolating the effect of pool size rather than as a comparison against demonstration selection methods in general\.

Table 10:Size\-matched control withMeta\-Llama\-3\-8B\-Instruct\. The observational pool is subsampled from6,4276\{,\}427to3,9133\{,\}913candidates for math reasoning and from2,9812\{,\}981to1,2781\{,\}278for DROP, matching\|Dcausal\|\|D\_\{\\text\{causal\}\}\|in each domain\. Retrieval, shot count, prompt format and decoding are identical throughout\.
### A\.5Comparison with Program\-Aided Prompting

PAL and PoT differ primarily in prompt format rather than in mechanism, and we take PoT as representative of program\-aided prompting\. PoT[Chen et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib26)generates a program for each test query and executes it to obtain the answer, whereas ECI2D uses programs only during offline distillation and issues a standard few\-shot prompt at inference time\. The two operate under different deployment assumptions, and we report the comparison separately for that reason rather than folding it into Table[1](https://arxiv.org/html/2610.21149#S4.T1)\. We restrict it to math reasoning: on the BBH subsets the task reduces to a single library call once a program is emitted, so a comparison there measures interpreter access rather than demonstration quality\.

As shown in Table[11](https://arxiv.org/html/2610.21149#A1.T11), access to program execution improves over human\-written prompts under the training distribution \(81\.6581\.65against79\.0079\.00on GSM8K\) but does not confer robustness to structural complexity: PoT reaches only58\.6258\.62on GSM\-Symbolic\-P1 and32\.2832\.28on P2, below the self\-generated baseline and7\.407\.40and5\.445\.44points below ECI2D\. The bottleneck under distribution shift therefore lies in selecting the reasoning structure rather than in executing it precisely, and interpreter access does not substitute for filtering which rationales enter the demonstration pool\.

Table 11:Comparison with program\-aided prompting on math reasoning \(Meta\-Llama\-3\-8B\-Instruct\)\. PoT†generates and executes a program for each test query and therefore requires interpreter access at inference time; the remaining methods issue a standard few\-shot prompt\. No ranking is marked, since the two regimes operate under different deployment assumptions\. GSM\-S = GSM\-Symbolic, P1/P2 = Plus\-1/Plus\-2\.

## Appendix BBenchmark Details

In this section, we provide detailed specifications of the datasets used in our experiments\. We categorize them into Math Reasoning, Logical Reasoning, and Semantic Reasoning benchmarks\. Specifically, GSM8K, a filtered subset of DROP, and CommonsenseQA serve as In\-Distribution \(ID\) datasets for distilling and evaluation, while SVAMP, GSM\-Symbolic, and Big\-Bench Hard \(BBH\) serve as Out\-Of\-Distribution \(OOD\) benchmarks to evaluate the robustness and effectiveness of our proposed framework\. All the distilled demonstrations are from training set, no test examples are included in𝒩⁡\(x\)\\mathcal\{N\}\(x\)orDcausalD\_\{\\text\{causal\}\}\.

### B\.1Math Reasoning Benchmarks

GSM8K \(In\-Distribution\)[Cobbe et al\. \(2021\)](https://arxiv.org/html/2610.21149#bib.bib25)consists of 8\.5K high\-quality linguistically diverse grade school math word problems\. It is divided into 7,473 training examples and 1,319 test examples\. Each problem takes between 2 to 8 reasoning steps to solve\. We use the training set of GSM8K to construct our invariant causal demonstrations and the test set for standard performance evaluation\.

SVAMP \(Out\-Of\-Distribution\)[Patel et al\. \(2021\)](https://arxiv.org/html/2610.21149#bib.bib29)is a challenge dataset created by applying varying structural modifications to elementary math word problems\. It consists of 1,000 examples designed to test the robustness of models against changes in word order and problem structure\. In our experiments, SVAMP is used as an OOD benchmark to verify whether the distilled rationales can generalize to problems with different linguistic patterns than GSM8K\.

GSM\-Symbolic \(Out\-Of\-Distribution\)[Mirzadeh et al\. \(2024\)](https://arxiv.org/html/2610.21149#bib.bib30)is an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions\. Unlike static datasets, it allows for the generation of massive variations of a single problem logic\. We utilize these three different difficulty level subsets to evaluate robustness:GSM\-Symbolic\(Standard\),GSM\-Symbolic\-Plus\-1, andGSM\-Symbolic\-Plus\-2\. GSM\-Symbolic contains 100 templates generating 5,000 examples\. It mirrors the distribution of the original GSM8K but with refreshed numbers and names to prevent memorization\. One or two clauses can be added to the questions to increase their difficulty, yielding GSM\-Symbolic\-Plus\-1 \(GSM\-Symbolic\-P1\) and GSM\-Symbolic\-Plus\-2 \(GSM\-Symbolic\-P2\), respectively\.

### B\.2Logical Reasoning Benchmarks

Discrete Reasoning Over Paragraphs \(DROP\) \(In\-Distribution\)[Dua et al\. \(2019\)](https://arxiv.org/html/2610.21149#bib.bib31)is a reading comprehension benchmark requiring discrete reasoning over paragraphs\. The original dataset contains 77,409 training and 9,536 validation examples\. DROP has nine reasoning categories, covering discrete reasoning and span information extraction\. To save computational resources and improve distilling efficiency, we pre\-filter the benchmark to retain only questions requiringsorting,counting, andtemporalreasoning, resulting 4,732 training and 737 validation examples\. We developed a heuristic classifier based on question keywords and answer types\. The filtering process is formally described in Algorithm[2](https://arxiv.org/html/2610.21149#alg2)\.

Big\-Bench Hard \(BBH\) \(Out\-Of\-Distribution\)[Suzgun et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib32)is a suite of 23 challenging tasks from BIG\-Bench where language models previously struggled\. To evaluate logical reasoning, we select three specific subsets that require multi\-step logical deduction\.Date Understanding: involves inferring dates from relative time descriptions \(e\.g\., "yesterday", "the first Monday of April"\)\.Object Counting: requires maintaining a mental state to count objects satisfying specific conditions\.Word Sorting: tests the ability to lexicographically sort a list of words\.Boolean Expressions: evaluates formal logical reasoning by requiring models to determine the truth value of nested Boolean expressions containing operators such asand,or, andnot\.Penguins in a Table: assesses structured data comprehension by querying a tabular dataset of penguin attributes, requiring models to perform precise filtering, counting, and comparison operations to derive deterministic answers\. Based on this classification, we extracted a balanced subset focusing on quantitative and temporal logic to serve as the training source for our logical reasoning experiments\.

### B\.3Semantic Reasoning Benchmarks

CommonsenseQA \(In\-Distribution\)[Talmor et al\. \(2019\)](https://arxiv.org/html/2610.21149#bib.bib47)is a five\-way multiple\-choice benchmark built from ConceptNet, in which each question targets a commonsense relation between concepts\. It contains 9,741 training and 1,221 validation examples, and the test labels are withheld for the official leaderboard, so we evaluate on the validation split\. Answering a question requires world knowledge about objects and actions rather than computation, and no deterministic program decides it\. We distil from the full training split and use CommonsenseQA as the ID benchmark for the hybrid abstraction of Section[4\.2](https://arxiv.org/html/2610.21149#S4.SS2)\.

BBH Semantic Subset \(Out\-Of\-Distribution\)[Suzgun et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib32)comprises the two subsets of BBH whose answers likewise rest on world knowledge, each containing 250 examples\.Sports Understanding: judges whether an artificially constructed sentence about sports is plausible, which requires knowing which sport an athlete plays and which actions belong to that sport\.Ruin Names: selects the humorous edit of an artist or movie name among near\-identical alternatives, which requires recognising wordplay\. Neither is expressible as a deterministic program\. Both are evaluated with the pool distilled on CommonsenseQA, so they measure cross\-domain transfer in the same way that BBH does for DROP\.

Algorithm 1Invariant Reasoning Distillation of ECI2D1:Training set

𝒟=\{\(xi,yi\)\}\\mathcal\{D\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}, LLM Generator

ΘGen\\Theta\_\{\\text\{Gen\}\}, LLM Abstractor

ΘΠ\\Theta\_\{\\Pi\}, LLM Adapter

ΘAdapter\\Theta\_\{\\text\{Adapter\}\}, threshold

δ\\delta
2:Invariant Causal Demonstrations

DcausalD\_\{\\text\{causal\}\}
3:Initialize

Dcausal←∅D\_\{\\text\{causal\}\}\\leftarrow\\emptyset
4:foreach sample

\(xi,yi\)∈𝒟\(x\_\{i\},y\_\{i\}\)\\in\\mathcal\{D\}do

5:// Step 1: Causal Abstraction

6:Generate rationale

r^i\\hat\{r\}\_\{i\}for

xix\_\{i\}using

ΘGen\\Theta\_\{\\text\{Gen\}\}
7:Abstract into executable program:

p^i←ΘΠ​\(r^i\)\\hat\{p\}\_\{i\}\\leftarrow\\Theta\_\{\\Pi\}\(\\hat\{r\}\_\{i\}\)
8:Initialize cumulative score

Ssum←0S\_\{\\mathrm\{sum\}\}\\leftarrow 0
9:foreach neighbor

\(xj,yj\)∈𝒩⁡\(xi\)\(x\_\{j\},y\_\{j\}\)\\in\\mathcal\{N\}\(x\_\{i\}\)do

10:// Step 2: Executable Causal Intervention

11:Adapt:

p^i→j←ΘAdapter​\(p^i,xj\)\\hat\{p\}\_\{i\\to j\}\\leftarrow\\Theta\_\{\\text\{Adapter\}\}\(\\hat\{p\}\_\{i\},x\_\{j\}\)
12:Execute:

y^j←Exec​\(p^i→j\)\\hat\{y\}\_\{j\}\\leftarrow\\text\{Exec\}\(\\hat\{p\}\_\{i\\to j\}\)
13:// Step 3: Transfer Invariance Verification

14:Score:

Sj←Sinv​\(xj,d​o​\(r^i\),y^j\)S\_\{j\}\\leftarrow S\_\{\\text\{inv\}\}\(x\_\{j\},do\(\\hat\{r\}\_\{i\}\),\\hat\{y\}\_\{j\}\)
15:

Ssum←Ssum\+SjS\_\{\\mathrm\{sum\}\}\\leftarrow S\_\{\\mathrm\{sum\}\}\+S\_\{j\}
16:endfor

17:Compute average score:

Savg←Ssum\|𝒩⁡\(xi\)\|S\_\{\\mathrm\{avg\}\}\\leftarrow\\frac\{S\_\{\\mathrm\{sum\}\}\}\{\|\\mathcal\{N\}\(x\_\{i\}\)\|\}
18:if

Savg≥δS\_\{\\mathrm\{avg\}\}\\geq\\deltathen

19:Add

\(xi,p^i,r^i,y^i\)\(x\_\{i\},\\hat\{p\}\_\{i\},\\hat\{r\}\_\{i\},\\hat\{y\}\_\{i\}\)to

DcausalD\_\{\\text\{causal\}\}
20:endif

21:endfor

22:return

DcausalD\_\{\\text\{causal\}\}

Algorithm 2Heuristic Filtering for DROP Logical SubsetsQuestion string

QQ, Answer types set

𝒜\\mathcal\{A\}
Logical Category

C∈\{Sorting, Counting, Temporal, Other\}C\\in\\\{\\text\{Sorting, Counting, Temporal, Other\}\\\}
// 1\. Identify Sorting Logic

ifany keyword in \["first", "last", "most", "least", "largest", "smallest", "order", "sort"\] exists in

QQthen

return

C←SortingC\\leftarrow\\text\{Sorting\}
endif

// 2\. Identify Counting Logic

if"how many" is in

QQthen

ifno arithmetic keyword \(e\.g\., "difference", "change", "total", "sum", "more", "less"\) exists in

QQthen

if"number" is in

𝒜\\mathcal\{A\}then

return

C←CountingC\\leftarrow\\text\{Counting\}
endif

endif

endif

// 3\. Identify Temporal Logic

if"date" is in

𝒜\\mathcal\{A\}then

return

C←TemporalC\\leftarrow\\text\{Temporal\}
endif

ifany keyword in \["year", "month", "day", "date", "period", "when"\] exists in

QQthen

return

C←TemporalC\\leftarrow\\text\{Temporal\}
endif

return

C←OtherC\\leftarrow\\text\{Other\}⊳\\trianglerightDiscarded in our subset

## Appendix CBaselines and Implementation Details

In this section, we provide a comprehensive overview of the baseline methods, model specifications, and the specific implementation configurations used in our experiments\. Code and data\-generation scripts will be released upon acceptance\.

### C\.1Baseline Methods

To evaluate the effectiveness and robustness of ECI2D, we compare it against a spectrum of prompting paradigms ranging from standard usage to state\-of\-the\-art self\-generated methods\.

Direct Prompting: The model predicts the answer directly given the question, without any intermediate reasoning steps\. This serves as the lower bound to assess the necessity of rationales\.

Zero\-Shot CoT[Kojima et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib37): We append the trigger promptLet’s think step by stepto the input\. This baseline evaluates the intrinsic reasoning capability of LLMs without the guidance of external demonstrations\.

Manual Few\-Shot CoT[Wei et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib3): We utilize high\-quality human\-annotated Chain\-of\-Thought demonstrations to serve as a strong baseline\. To ensure a fair comparison with established standards, we source the demonstrations from representative works,[Wei et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib3)for all math reasoning datasets \(GSM8K, SVAMP, GSM\-Symbolic\),[Zhou et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib34)for the DROP dataset, and[Suzgun et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib32)for BBH tasks\.

Self\-Generated Reinforced ICL[Wan et al\. \(2025\)](https://arxiv.org/html/2610.21149#bib.bib6);[Agarwal et al\. \(2024\)](https://arxiv.org/html/2610.21149#bib.bib5);[Li et al\. \(2024b\)](https://arxiv.org/html/2610.21149#bib.bib8);[Kim et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib7);[Zhang et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib4): This is the primary baseline to our approach\. It distills demonstrations based on an outcome\-oriented criterion, retaining generated rationales that lead to the correct ground truth answer\. Aside from the difference in the quality of the self\-generated demonstrations, our method maintains consistency with self\-generated reinforced ICL in terms of retrieval ICL, Sentence\-transformer selection, and hyperparameter settings\. Comparing against this baseline highlights the specific contribution of our transfer intervention mechanism over simple correctness distilling\.

Self\-Generated Reinforced ICL \+ Code: A control that separates the filtering criterion from the demonstration format\. It uses the identical generation prompt as ECI2D \(Appendix[I\.1](https://arxiv.org/html/2610.21149#A9.SS1)\), so each candidate contains both a natural\-language rationale and a Python program, but retains candidates by observational correctness on the source question alone, omitting Steps 2 and 3 of Algorithm[1](https://arxiv.org/html/2610.21149#alg1)\. Retrieval, encoder and all hyperparameters are identical to ECI2D\. The same control is applied to the hybrid abstraction of Section[4\.2](https://arxiv.org/html/2610.21149#S4.SS2), which uses the template of Appendix[I\.4](https://arxiv.org/html/2610.21149#A9.SS4)without the declaration block\.

PoT[Chen et al\. \(2022\)](https://arxiv.org/html/2610.21149#bib.bib26): The model emits a Python program for the test query, which an external interpreter executes to obtain the answer\. Unlike all other methods considered here, this requires interpreter access at test time; the comparison is reported in Appendix[A\.5](https://arxiv.org/html/2610.21149#A1.SS5)\.

All the baselines adopt a greedy decoding configuration to ensure consistency and reproducibility\. This hierarchy of baselines allows us to deconstruct the performance gains, in which improving over Zero\-Shot proves the value of demonstrations; improving over Self\-generated Reinforced ICL proves the value of intervention over observation\.

### C\.2Model Selection

We conduct experiments on four open\-source Large Language Models, spanning standard instruction\-tuned LLMs and reasoning\-specialized architectures\.

Meta\-Llama\-3\-8B\-Instruct666https://huggingface\.co/meta\-llama/Meta\-Llama\-3\-8B\-Instruct: As a widely adopted decoder\-only model, Llama\-3 exhibits strong instruction\-following capabilities and serves as a representative benchmark for the 8B parameter scale\.

Qwen2\.5\-7B\-Instruct777https://huggingface\.co/Qwen/Qwen2\.5\-7B\-Instruct: We specifically select Qwen2\.5 due to its specialized strengths in math reasoning and code generation\. Since our ECI2D framework relies on abstracting reasoning into Python programs, Qwen2\.5’s coding proficiency makes it an ideal candidate to test the limits of our framework\.

DeepSeek\-R1\-Distill\-Qwen\-7B888https://huggingface\.co/deepseek\-ai/DeepSeek\-R1\-Distill\-Qwen\-7B: Distilled from DeepSeek\-R1 onto a Qwen2\.5\-7B backbone, this model embeds long chain\-of\-thought reasoning patterns directly into its parameters\. We include it to assess whether ECI2D’s intervention\-based filtering remains effective on architectures explicitly optimized for extended rationale generation, where the risk of post\-hoc rationalization is arguably higher\.

Qwen3\-8B999https://huggingface\.co/Qwen/Qwen3\-8B: The latest 8B model in the Qwen series, featuring further\-strengthened reasoning and coding capabilities relative to Qwen2\.5\. We include it to verify that the gains of ECI2D persist as the underlying base model continues to improve, rather than reflecting weaknesses specific to a particular generation of LLMs\.

All models are deployed using the vLLM[Kwon et al\. \(2023\)](https://arxiv.org/html/2610.21149#bib.bib33)library on a single NVIDIA A6000 \(48GB\) GPU to ensure efficient, high\-throughput inference\.

### C\.3Implementation Details

#### Logically Similar Sample Retrieval for Intervention

To construct the logically similar samples𝒩⁡\(x\)\\mathcal\{N\}\(x\)for the executable causal intervention, we employ a coarse\-to\-fine retrieval strategy\. This two\-stage process is designed to ensure that the retrieved neighborhood𝒩⁡\(x\)\\mathcal\{N\}\(x\)satisfies both feasibility and comparability for the subsequent intervention\. In the initial coarse retrieval stage, we utilize thesentence\-transformers/all\-MiniLM\-L6\-v2101010https://huggingface\.co/sentence\-transformers/all\-MiniLM\-L6\-v2model to encode input queries\. For each training samplexx, we retrieve a broad candidate pool of size5​N5N\. This step serves as a necessary domain constraint, ensuring that the candidate samples reside within the same semantic context as the input\. Subsequently, in the fine\-grained selection stage, we further leverage the LLM to identify logically analogous examples\. Since mere semantic similarity does not guarantee logical compatibility, and applying a rationale to a logically heterogeneous problem would render the verification meaningless, we prompt the LLM to analyze the underlying problem logic\. It selects the final top\-NN\(withN∈\{3,5,8\}N\\in\\\{3,5,8\\\}\) samples that share similar reasoning logic with the inputxx\. This ensures that the intervention is executable and the results are comparable, thereby providing a testbed for verifying the transfer invariance of the distilled rationales\. The prompts for fine\-grained retrieval can be found in Appendix[I\.7](https://arxiv.org/html/2610.21149#A9.SS7)\.

#### Invariance Threshold Settings

The transfer invariance verification relies on a scoring thresholdδ\\deltato determine if a rationale’s logic holds across logically similar samples\. For math reasoning, since math problems have unique, objective solutions, we set the thresholdδ=1\.0\\delta=1\.0\. A rationale is considered invariant only if the executed program yields the exact match of the ground truth for the neighbor problem\. For logical reasoning \(DROP\), the DROP dataset often involves answer sets or text spans where slight formatting variations are permissible\. We use the F1 score as the evaluation metric and set the invariance thresholdδ=0\.9\\delta=0\.9\. This strict yet flexible threshold ensures that the logic is correct while accommodating minor textual discrepancies common in reading comprehension tasks\. For semantic reasoning \(CommonsenseQA\), the answer is a single option from a closed label set, so we again setδ=1\.0\\delta=1\.0and retain a candidate only when allNNinterventions recover the ground truth of their target\.

#### Hyperparameter Configuration

We adopt a distinct temperature strategy for the distillation and inference phases to balance diversity and determinism\. In invariant reasoning distillation phase, we set temperatureT=0\.7T=0\.7\. A higher temperature is crucial here to encourage the model to explore a diverse space of rationales\. This diversity is necessary to uncover potentially faithful causal mechanisms among various spurious possibilities\. In causal steering inference phase, we setT=0\.0T=0\.0\(greedy decoding\)\. This ensures reproducibility and minimizes variance, forcing the model to strictly adhere to the guidance provided by the selected causal demonstrations without introducing stochastic noise\.

#### Timeout in Execution

For the causal abstraction, we limit the generation to a maximum of 512 tokens to encourage concise algorithmic logic\. All generated programs are executed in a sandboxed Python environment with a timeout of 2 seconds to prevent infinite loops\.

#### Hybrid Program Configuration

The hybrid abstraction of Section[4\.2](https://arxiv.org/html/2610.21149#S4.SS2)is evaluated withQwen2\.5\-7B\-Instruct\. We distil from the full CommonsenseQA training set of 9,741 examples and retrieve logically similar samples withall\-mpnet\-base\-v2, settingN=3N=3andδ=1\.0\\delta=1\.0as in the preceding paragraph\. The coarse retrieval pool is5​N5N\.ΘΠ\\Theta\_\{\\Pi\}decodes at temperature0\.70\.7and every other call is greedy, following the main experiments\.ΘΠ\\Theta\_\{\\Pi\}is allowed two retries when a candidate fails the constraints of Appendix[D](https://arxiv.org/html/2610.21149#A4), whileΘAdapter\\Theta\_\{\\text\{Adapter\}\}andΘEval\\Theta\_\{\\text\{Eval\}\}are each called once\. Every execution is bounded by a two second timeout\. At inference time we use 7\-shot demonstrations on CommonsenseQA and 3\-shot on the semantic subset of BBH\.

CommonsenseQA presents five candidate answers, soΘΠ\\Theta\_\{\\Pi\}abstracts a rationale that discriminates among them into a test of the candidates against the properties the rationale names, andsi\\mathrm\{s\}\_\{i\}visits each candidate in turn\. One intervention therefore issues one call toΘEval\\Theta\_\{\\text\{Eval\}\}per candidate per declaration indi\\mathrm\{d\}\_\{i\}, which for Figure[4](https://arxiv.org/html/2610.21149#A4.F4)is five candidates against two declarations\. Answers are cached on the pair of declaration and arguments, so a candidate appearing in the option lists of several neighbors is queried once\. Sports Understanding admits two outcomes and its skeletons contain no such loop, issuing one call per declaration for the whole instance\. The loop follows from the number of candidates the task presents, and both benchmarks pass through the same intervention and execution\.

## Appendix DMore Experiment of Hybrid Program

This section details the hybrid abstraction of Section[4\.2](https://arxiv.org/html/2610.21149#S4.SS2)\. Its settings are given in Section[C](https://arxiv.org/html/2610.21149#A3)\.

### D\.1Static Validation

Three constraints are enforced by inspecting the abstract syntax tree rather than by prompt instruction, and a candidate failing any of them after the retries of Appendix[C](https://arxiv.org/html/2610.21149#A3)is discarded\. The program body contains at least one deterministic construct\. Every call to an undefined function takes at most two arguments, each a single string or number, and no argument is the question, the option list, or any name bound to either along an assignment chain\. Every undefined name called in the body is declared, carries complete type annotations and a non\-empty docstring, and returnsbool,int,floator a shortstr\. The second constraint is what confinesΘEval\\Theta\_\{\\text\{Eval\}\}to its arguments\. Following the outcome\-oriented criterion of Section[4\.1](https://arxiv.org/html/2610.21149#S4.SS1), we additionally require the abstraction to reproduce the answer on its own instance\. The\+ Codecondition of Table[5](https://arxiv.org/html/2610.21149#S6.T5)uses the template of Appendix[I\.4](https://arxiv.org/html/2610.21149#A9.SS4)without the declaration block, so the abstraction is a deterministic program, and candidates are retained by observational correctness alone\.

### D\.2Run\-Time Evaluation

ΘAdapter\\Theta\_\{\\text\{Adapter\}\}returns the call line alone, and the declarations and the body are reused verbatim\. Sincesi\\mathrm\{s\}\_\{i\}returns the text of the selected option rather than its label, we match the returned string against the option list ofxjx\_\{j\}and take the label at the matching position, so a return value absent from that list yieldsSinv=0S\_\{\\text\{inv\}\}=0in the same way as a failed execution\.

### D\.3Diagnostics

Observational filtering retains a similar share whether or not the demonstration carries a program, 79\.1% without and 77\.1% with, so the reduction to 58\.8% under ECI2D follows from the intervention rather than from the difficulty of producing the abstraction\. Table[12](https://arxiv.org/html/2610.21149#A4.T12)decomposes the interventions that score zero\. Wrong options dominate and execution failures account for 5% together, so what the criterion rejects is a rationale that does not transfer rather than a program that does not run\. The retained skeletons combine their undefined functions over the candidate answers rather than sequentially, with one call rarely informing the argument of another, so what Step 2 transplants in this domain is the choice of functions and the way their returned values are combined\.

Table 12:Decomposition of the interventions scoringSinv=0S\_\{\\text\{inv\}\}=0under the hybrid abstraction on CommonsenseQA\.
### D\.4Case Study

Figure[4](https://arxiv.org/html/2610.21149#A4.F4)traces one retained candidate\. The abstraction declares two functions and scores the answer options by the values they return, and the adapter rewrites only the call line\. On the neighbor shown, the ground truth is selected over a competing option that also scores, so the outcome follows from the comparison rather than from a default\.

A second candidate illustrates the separation ofsi\\mathrm\{s\}\_\{i\}fromvi\\mathrm\{v\}\_\{i\}\. Its skeleton declaresis\_pleasant\_person\(person\)alongside two functions over the options, and the adapter rebinds that argument fromreceptionist ladytoJennywhile leaving the body untouched\. Its interventions also draw values from the cache, sinceofficeappears among the options of several neighbors\.

Source instance \(xix\_\{i\}, gold E\)

- •Question:She had to comfort friend that day, she gave her a big ole what?
- •Options:A\) friendly B\) sympathetic C\) handshake D\) cook E\) hug
- •Abstraction \(p^i\\hat\{p\}\_\{i\}\): defis\_physical\_action\(action:str\)\-\>bool: """ReturnsTrueiftheactioninvolvesphysicalcontact\.""" defis\_emotional\_support\(action:str\)\-\>bool: """ReturnsTrueiftheactionprovidesemotionalsupport\.""" defsolution\(param,options\): best,best\_score=options\[0\],\-1\.0 foroptioninoptions: score=0\.0 ifis\_physical\_action\(option\):score\+=1\.0 ifis\_emotional\_support\(option\):score\+=1\.0 ifscore\>best\_score: best,best\_score=option,score returnbest
- •Self\-execution:returnsE\) hug, matching the gold answer\.

Intervention on a neighbor \(xjx\_\{j\}, gold D\)

- •Question:James and Lilly have strong feelings for each other\. To express those feelings they kiss\. What else might they do?
- •Adapted call \(p^i→j\\hat\{p\}\_\{i\\to j\}\): solution\("kiss",\["shakehands","fallinlove","smile", "embrace","fallinginlove"\]\)
- •Returned values:shake handsscores 1\.0,embracescores 2\.0, the remaining options score 0\.0\.
- •Result:returnsembrace, matchingyjy\_\{j\}, soSinv=1S\_\{\\text\{inv\}\}=1\.

Figure 4:One retained candidate under the hybrid abstraction\. The logical skeleton is transplanted unchanged and only the call line is rewritten\.

## Appendix EMore analysis of Observational Equivalence under Observation Distillation

We formally analyze the observational equivalence of the causal structure governing the generation of rationales under the constraint of observational data\. Let𝒟=\{\(xi,ri,yi\)\}i=1\|𝒟\|\\mathcal\{D\}=\\\{\(x\_\{i\},r\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{\|\\mathcal\{D\}\|\}denote a dataset of demonstrations distilled via an outcome\-oriented criterion, whereyiy\_\{i\}matches the ground truthyg​ty\_\{gt\}\. We posit two competing structural causal models \(SCMs\) that could generate this data: the Faithful Causal Modelℳcausal\\mathcal\{M\}\_\{\\text\{causal\}\}and the Spurious Post\-hoc Modelℳspurious\\mathcal\{M\}\_\{\\text\{spurious\}\}\. Under the Faithful Causal Modelℳcausal\\mathcal\{M\}\_\{\\text\{causal\}\}, the rationaleRRfunctions as a necessary mediator that transmits the causal influence of inputXXto outputYY\. The joint probability distribution factorizes according to the directed acyclic graphX→R→YX\\to R\\to Y:

Pℳcausal​\(X,R,Y\)=P⁡\(Y∣R\)​P​\(R∣X\)​P​\(X\)\.P\_\{\\mathcal\{M\}\_\{\\text\{causal\}\}\}\(X,R,Y\)=P\(Y\\mid R\)P\(R\\mid X\)P\(X\)\.\(10\)This factorization implies a conditional independence structure where the output is independent of the input given the rationale, denoted as\(Y⟂X∣R\)\(Y\\perp X\\mid R\)\. Conversely, under theSpurious Post\-hoc Modelℳspurious\\mathcal\{M\}\_\{\\text\{spurious\}\}, the outputYYis determined directly by the inputXX\(potentially via latent shortcuts\), while the rationaleRRis generated retrospectively to justify the observed\(X,Y\)\(X,Y\)pair\. This structure corresponds to the graphX→Y→RX\\to Y\\to Ror a confounding structure whereRRis a colliderX→R←YX\\to R\\leftarrow Y\. The joint probability distribution for the post\-hoc chainX→Y→RX\\to Y\\to Rfactorizes as:

Pℳspurious​\(X,R,Y\)=P⁡\(R∣X,Y\)​P​\(Y∣X\)​P​\(X\)\.P\_\{\\mathcal\{M\}\_\{\\text\{spurious\}\}\}\(X,R,Y\)=P\(R\\mid X,Y\)P\(Y\\mid X\)P\(X\)\.\(11\)The fundamental challenge of observation distillation arises from the fact that for any distributionP⁡\(X,R,Y\)P\(X,R,Y\)generated by a faithful model, there exists a parameterization of a spurious model that can reproduce the exact same observational distribution\. By the chain rule of probability, the joint distribution can generally be decomposed asP⁡\(X,R,Y\)=P⁡\(R∣X,Y\)​P​\(Y∣X\)​P​\(X\)P\(X,R,Y\)=P\(R\\mid X,Y\)P\(Y\\mid X\)P\(X\)\. Without external constraints or interventional data, the faithful factorization in Equation[10](https://arxiv.org/html/2610.21149#A5.E10)is mathematically indistinguishable from the spurious factorization in Equation[11](https://arxiv.org/html/2610.21149#A5.E11)solely based on the observed samples\. This phenomenon corresponds to the concept of Markov Equivalence Classes in causal discovery[Spirtes et al\. \(2000\)](https://arxiv.org/html/2610.21149#bib.bib35);[Verma and Pearl \(2022\)](https://arxiv.org/html/2610.21149#bib.bib39), where multiple causal graphs entail the same set of conditional independencies and thus generate identical observational likelihoods\. Consequently, an outcome\-oriented filter that selects demonstrations maximizingP⁡\(Y=yg​t∣X,R\)P\(Y=y\_\{gt\}\\mid X,R\)cannot differentiate between a rationale thatcausesthe correct answer \(R→YR\\to Y\) and a rationale that iscaused bythe correct answer \(Y→RY\\to R\)\. Although these two models are observationally equivalent, they are interventionally distinct[Pearl \(2010\)](https://arxiv.org/html/2610.21149#bib.bib19);[Peters et al\. \(2017\)](https://arxiv.org/html/2610.21149#bib.bib38)\. Specifically, the interventional distributions differ significantly:

Pℳcausal​\(Y∣d​o​\(R=r\),X\)=P⁡\(Y∣r\),Pℳspurious​\(Y∣d​o​\(R=r\),X\)=P⁡\(Y∣X\)\.\\begin\{split\}P\_\{\\mathcal\{M\}\_\{\\text\{causal\}\}\}\(Y\\mid do\(R=r\),X\)&=P\(Y\\mid r\),\\\\ P\_\{\\mathcal\{M\}\_\{\\text\{spurious\}\}\}\(Y\\mid do\(R=r\),X\)&=P\(Y\\mid X\)\.\\end\{split\}\(12\)
In the faithful case, intervening onRRalters the probability ofYY, whereas in the spurious case, the intervention leaves the distribution ofYYinvariant \(or governed solely byXX\)\. This derivation theoretically justifies our proposed ECI2D framework, which utilizes thed​odo\-operator to break the observational equivalence and identifyRcR\_\{c\}through the invariance of the interventional distribution\.

## Appendix FMore analysis of Transfer Invariance from Independent Causal Mechanisms

In this section, we provide a formal justification for utilizing transfer invariance as a proxy for rationale faithfulness\. Our derivation relies on the Principle of Independent Causal Mechanisms \(ICM\)[Schölkopf et al\. \(2012\)](https://arxiv.org/html/2610.21149#bib.bib36), which posits that the conditional distribution of an effect given its direct causes remains invariant to changes in the marginal distribution of the causes\. We analyze the behavior of faithful and spurious rationales under causal intervention across local distributional shifts\.

LetSSdenote a shift variable representing the transition from a source contextxix\_\{i\}to a neighbor contextxj∈𝒩⁡\(xi\)x\_\{j\}\\in\\mathcal\{N\}\(x\_\{i\}\)\. This shift induces a change in the input distributionP⁡\(X∣S\)P\(X\\mid S\)\. We examine the stability of the conditional probabilityP⁡\(Y∣d​o​\(R\),X\)P\(Y\\mid do\(R\),X\)under this shift for two distinct causal structures\.

#### Rationale as an Operational Procedure\.

Before the case analysis, we make explicit the operational sense in which a rationale acts as a causal mechanism\. We treat a rationaler∈ℛr\\in\\mathcal\{R\}as a functionr:𝒳𝒞→𝒴𝒞r:\\mathcal\{X\}\_\{\\mathcal\{C\}\}\\to\\mathcal\{Y\}\_\{\\mathcal\{C\}\}that maps inputs within a problem class𝒞\\mathcal\{C\}, defined by shared logical featuresA=\(P,S,T\)A=\(P,S,T\), to their predicted outputs\. Under this view, faithfulness with respect to\(xi,yi\)\(x\_\{i\},y\_\{i\}\)requiresr⁡\(xj\)=yjr\(x\_\{j\}\)=y\_\{j\}for every\(xj,yj\)∈𝒞\(x\_\{j\},y\_\{j\}\)\\in\\mathcal\{C\}\. A rationale that satisfies this requirement only on its source input provides no procedural content and reduces to a constant function returningyiy\_\{i\}, which carries no explanatory role\. Equation[1](https://arxiv.org/html/2610.21149#S3.E1)formalizes this requirement in probabilistic form by reducing it to the necessary condition that interventions withrron neighborsxj∈𝒩⁡\(xi\)x\_\{j\}\\in\\mathcal\{N\}\(x\_\{i\}\)preserve correctness in distribution\.

#### Case 1: Faithfulness Implies Invariance\.

Consider the faithful causal rationaleRcR\_\{c\}defined in Definition[1](https://arxiv.org/html/2610.21149#Thmdefinition1), where the causal structure is governed by the directed edgeRc→YR\_\{c\}\\to Y\. According to the ICM principle, the causal mechanismP⁡\(Y∣Rc\)P\(Y\\mid R\_\{c\}\)is autonomous and modular, meaning it remains invariant to interventions or shifts in the input generation processP⁡\(X\)P\(X\)\. Formally, when we perform an interventiond​o​\(R=r\)do\(R=r\)by transplanting a rationale skeleton extracted fromxix\_\{i\}toxjx\_\{j\}, the interventional distribution factorizes as:

P⁡\(Y∣d​o​\(Rc=r\),xj\)=P⁡\(Y∣r\)\.P\(Y\\mid do\(R\_\{c\}=r\),x\_\{j\}\)=P\(Y\\mid r\)\.\(13\)Since the mechanismP⁡\(Y∣R\)P\(Y\\mid R\)captures the underlying functional law \(e\.g\., the arithmetic logic of addition\) rather than statistical correlations, it remains valid regardless of the specific parameterization of the inputxjx\_\{j\}\. Consequently, a faithful rationale demonstrates transfer invariance, as the intervened mechanism consistently induces the correct ground truthyjy\_\{j\}across the local neighborhood𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)\.

#### Case 2: Spuriousness Implies Variance\.

Conversely, consider the spurious post\-hoc rationaleRsR\_\{s\}defined in Definition[2](https://arxiv.org/html/2610.21149#Thmdefinition2), which arises from the reverse causal pathY→RsY\\to R\_\{s\}or confounding byXX\. In this scenario, the rationale is functionally dependent on the specific outcomeyiy\_\{i\}of the source context\. Applying thed​odo\-operator to setRs=rR\_\{s\}=rbreaks the incoming edges toRR, renderingRRstatistically independent of its original parentsXXandYYin the interventional graph\. The distribution ofYYis thus governed solely by the inputXXvia the spurious mechanismℳspurious\\mathcal\{M\}\_\{\\text\{spurious\}\}:

P⁡\(Y∣d​o​\(Rs=r\),xj\)=P⁡\(Y∣xj\)\.P\(Y\\mid do\(R\_\{s\}=r\),x\_\{j\}\)=P\(Y\\mid x\_\{j\}\)\.\(14\)However, the explicit content of the intervened rationalerrcarries the specific context ofxix\_\{i\}\. SinceRsR\_\{s\}does not structurally causeYYbut merely correlates with it in the source distribution, forcing this rationale upon a new contextxjx\_\{j\}creates a mismatch between the logic encoded inrrand the ground truthyjy\_\{j\}\. The observational correlationP⁡\(Y∣R\)P\(Y\\mid R\)observed atxix\_\{i\}fails to transport toxjx\_\{j\}because the underlying joint distributionP⁡\(X,Y,R\)P\(X,Y,R\)changes withSS\. Therefore, spurious rationales violate transfer invariance, exhibiting a performance degradation when subjected to causal intervention\.

#### Sound but Incomplete Identification\.

The two cases above establish the asymmetric identifiability ofRcR\_\{c\}andRsR\_\{s\}under transfer invariance\. LetT⁡\(r^i\)∈\{0,1\}T\(\\hat\{r\}\_\{i\}\)\\in\\\{0,1\\\}denote the invariance verdict, whereT⁡\(r^i\)=1T\(\\hat\{r\}\_\{i\}\)=1ifr^i\\hat\{r\}\_\{i\}satisfies Equation[1](https://arxiv.org/html/2610.21149#S3.E1)across𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)\. Case 1 establishes the implicationr^i=Rc⇒T⁡\(r^i\)=1\\hat\{r\}\_\{i\}=R\_\{c\}\\Rightarrow T\(\\hat\{r\}\_\{i\}\)=1, whose contrapositiveT⁡\(r^i\)=0⇒r^i≠RcT\(\\hat\{r\}\_\{i\}\)=0\\Rightarrow\\hat\{r\}\_\{i\}\\neq R\_\{c\}provides a soundness guarantee for the filter and does not depend on the absence of latent variables\. Case 2 establishes that a generic post\-hocRsR\_\{s\}violates Equation[1](https://arxiv.org/html/2610.21149#S3.E1), yet does not preclude the existence of pathological rationales that share statistical regularities withRcR\_\{c\}across𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)without being causally faithful, for instance through unobserved factors that act uniformly across the neighborhood\. Equation[1](https://arxiv.org/html/2610.21149#S3.E1)therefore provides a necessary but not sufficient condition for faithfulness, and ECI2D should be understood as a sound filter that excludes the most prevalent non\-faithful rationales rather than a complete identifier of the causal mechanism\. The residual configurations are discussed in Limitations\.

#### Empirical Estimation in Finite Samples\.

Direct computation of the probabilities in Equation[1](https://arxiv.org/html/2610.21149#S3.E1)is infeasible for an LLM\-induced distribution, so Section[4](https://arxiv.org/html/2610.21149#S4)replaces them with the empirical estimator

S^N​\(r^i\)=1N​∑j=1N𝕀⁡\(y^j=yj\),\\hat\{S\}\_\{N\}\(\\hat\{r\}\_\{i\}\)=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\\mathbb\{I\}\\\!\\left\(\\hat\{y\}\_\{j\}=y\_\{j\}\\right\),\(15\)where\{xj\}j=1N\\\{x\_\{j\}\\\}\_\{j=1\}^\{N\}are drawn from𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)andy^j=Exec⁡\(p^i→j\)\\hat\{y\}\_\{j\}=\\mathrm\{Exec\}\(\\hat\{p\}\_\{i\\to j\}\)\. Each indicator is an unbiased single\-sample estimator ofP⁡\(Y=yj∣d​o​\(R=r^i\),X=xj\)P\(Y=y\_\{j\}\\mid do\(R=\\hat\{r\}\_\{i\}\),X=x\_\{j\}\)at neighborxjx\_\{j\}\. At the small values ofNNused in practice the estimator is coarse, and we therefore adopt the strictest available threshold, requiring agreement on every neighbor except on DROP, where the F1 metric admits a threshold of0\.90\.9\. Sensitivity toNNis reported in Section[5\.2](https://arxiv.org/html/2610.21149#S5.SS2)\.

## Appendix GComputational Cost Analysis

The ECI2D framework incurs additional computational overhead strictly during the offline demonstration distillation phase\. This overhead originates from iterative rationale sampling, program abstraction and adaptation, logical neighbor retrieval, and repeated sandbox executions for verification\. For a standard benchmark such as GSM8K with a logical neighborhood size ofN=3N=3, the pipeline entails approximately 22,000–30,000 lightweight Python operations\. Execution of these scripts requires less than one minute on a standard CPU, with the dominant cost arising from offline LLM generation calls\.

Crucially, this preprocessing cost is strictly a one\-time investment\. During the online inference phase, ECI2D introduces no additional computational overhead compared to standard self\-generated ICL\. The model receives a standard few\-shot prompt format, and no external execution or intervention steps are performed at test time\.

The offline computational investment yields a compact, causally validated demonstration pool\. This distilled subset reduces online vector retrieval latency and decreases context token consumption\. Consequently, the framework implements a standard offline\-online trade\-off: increased preprocessing cost is exchanged for higher demonstration fidelity and downstream reasoning robustness, while preserving the computational profile of conventional ICL during deployment\.

## Appendix HHuman Validation

All annotation was carried out by three annotators with graduate\-level training in NLP, working independently and blind to the pipeline’s output\.

#### Validity of neighborhood construction\.

The construction of𝒩⁡\(xi\)\\mathcal\{N\}\(x\_\{i\}\)rests on the assumption that the two\-stage retrieval identifies instances sharing a reasoning structure\. We sampled1,0001\{,\}000pairs judged logically similar by the pipeline from each of GSM8K and DROP, and asked three annotators to decide independently, and blind to the pipeline’s verdict, whether the two instances admit a common logical skeleton underA=\(P,S,T\)A=\(P,S,T\)\. Mean agreement with the pipeline is94\.4%94\.4\\%on GSM8K and91\.3%91\.3\\%on DROP \(Table[13](https://arxiv.org/html/2610.21149#A8.T13)\)\. Disagreements concentrate on instances with strong lexical overlap but divergent premise structure, which the semantic retriever ranks highly and the structural verifier does not always reject\. The residual error rate propagates conservatively: a faithful rationale evaluated against a mismatched neighbor fails the execution check and is discarded, which lowers retention without admitting spurious rationales intoDcausalD\_\{\\text\{causal\}\}\.

#### Alignment between rationale and program\.

A second assumption is thatΠ\\Piformalizes the reasoning structure ofr^i\\hat\{r\}\_\{i\}rather than altering it\. Three annotators judged2,0002\{,\}000⟨r^,p^⟩\\langle\\hat\{r\},\\hat\{p\}\\ranglepairs for step\-level correspondence, reporting a mean alignment of95\.5%95\.5\\%\. Sincer^\\hat\{r\}andp^\\hat\{p\}are emitted in a single forward pass \(Appendix[I\.1](https://arxiv.org/html/2610.21149#A9.SS1)\), they are conditioned on the same context, which we expect accounts for the high agreement\. Observed misalignments are of two kinds with opposite consequences\. Alteration of the skeleton causes the adapted program to fail on its neighbors, so the demonstration is discarded\. Silent repair, in whichΠ\\Pimaps a flawedr^\\hat\{r\}to a correctp^\\hat\{p\}, has the opposite effect and is not detectable by alignment annotation, since the annotated program is correct by construction\. We additionally inspected500500demonstrations sampled fromDcausalD\_\{\\text\{causal\}\}for chain\-level correctness and observed no incorrect reasoning chain, which bounds the prevalence of silent repair in the retained pool at0\.76%0\.76\\%with95%95\\%confidence, but a direct measurement of this rate remains future work\.

Table 13:Human validation of the assumptions underlying ECI2D\. Three annotators judged every sample independently and blind to condition\. Ranges give the per\-annotator minimum and maximum\.
#### Case study of adapter errors\.

Figure[5](https://arxiv.org/html/2610.21149#A8.F5)illustrates the two failure modes discussed above with instances drawn from GSM8K\. In Case 1 the Adapter alters the arithmetic structure of the source skeleton, so the adapted program returns a value inconsistent with the ground truth and the source demonstration is discarded\. In Case 2 the skeleton is preserved but the target instance expresses its parameter in a different unit, which the Adapter fails to normalize before substitution\. Both are conservative in effect, since a failed execution yieldsSinv=0S\_\{\\text\{inv\}\}=0and removes the candidate fromDcausalD\_\{\\text\{causal\}\}\.

Case 1: Skeleton Alteration

- •Source Problem \(xix\_\{i\}\):Janet’s ducks lay 16 eggs/day\. She eats 3, uses 4 for baking, and sells the rest for $2 each\. Daily earnings?
- •Source Code \(p^i\\hat\{p\}\_\{i\}\): defsolution\(\): laid,eaten,baked,price=16,3,4,2 remainder=laid\-eaten\-baked returnremainder\*price
- •Logically Similar Target \(xjx\_\{j\}\):A farm produces 40 gallons of milk/day\. The farmer uses 5 for his family, gives 10 to calves, and sells the rest for $3/gallon\. Daily earnings?
- •Adapter Code \(p^i→j\\hat\{p\}\_\{i\\to j\}\): defsolution\(\): produced,family,calves,price=40,5,10,3 returnproduced\-\(family\*price\)\-\(calves\*price\)
- •Error:Adapter alters the causal skeleton and distributes the price incorrectly\.
- •Result:Outputs \-5 instead of the correct answer 75\.

Case 2: Variable Mapping Failure

- •Source Problem \(xix\_\{i\}\):Weng earns $12/hour\. She worked for 50 minutes\. How much did she earn?
- •Source Code \(p^i\\hat\{p\}\_\{i\}\): defsolution\(\): rate,minutes=12,50 returnrate\*\(minutes/60\)
- •Logically Similar Target \(xjx\_\{j\}\):A worker is paid $15/hour\. He worked for 3 hours and 20 minutes\. How much was he paid?
- •Adapter Code \(p^i→j\\hat\{p\}\_\{i\\to j\}\): defsolution\(\): rate=15 minutes=20 returnrate\*\(minutes/60\)
- •Error:Adapter fails to synthesize "3 hours and 20 mins" into a single minute variable to fit the skeleton\.
- •Result:Outputs 5 instead of the correct answer 50\.

Figure 5:Case study of LLM adapter errors\.

## Appendix IPrompts

### I\.1Rationales Generation and Abstraction

Prompt template of Rationales Generation and Abstraction for Math ReasoningInstruction You are a Python Expert\. Write a generic Python function ‘def solution\(\):‘ to solve this math problem\. First, think step\-by\-step about the logic required \(Chain of Thought\)\. Then, write a Python function ‘def solution\(\):‘ that returns the answer\. Question: \[question\] Format your response as: Rationale: \.\.\. content \.\.\. Code: ‘‘‘python \.\.\. code \.\.\. ‘‘‘

Prompt template of Rationales Generation and Abstraction for Logic Reasoning\.Instruction You are a Python Expert\. Solve the problem based on the Context\. First, think step\-by\-step about the logic required \(Chain of Thought\)\. Then, write a Python function ‘def solution\(\):‘ that returns the answer\. Use ‘datetime‘, ‘math‘, ‘sorting‘, or ‘counting‘ logic as needed\. Context: \{passage\} Question: \{question\} Format your response as: CoT: \.\.\. content \.\.\. Code: ‘‘‘python \.\.\. code \.\.\. ‘‘‘

### I\.2Adaptation and Intervention

Prompt template of Adaptation and InterventionInstruction Solve the NEW PROBLEM using the LOGIC of the EXISTING CODE\. 1\. Analyze the NEW PROBLEM\. 2\. Reuse the structure of EXISTING CODE \(do not change the formulas/logic\)\. 3\. Call the function with values from the NEW PROBLEM\. EXISTING CODE: ‘‘‘python \{retrieved\_code\} ‘‘‘ NEW PROBLEM: \{retrieved\_question\} Output Python Code based on the Existing Code’s logic: ‘‘‘python code ‘‘‘

### I\.3Causal Steering Inference

Prompt template of Math ReasoningInstruction You are a Math and Reasoning Expert utilizing a Hybrid Reasoning approach\. Combine natural language intuition with Python precision\. Reference the provided examples for reasoning style\. You should: Natural Language Analysis: Explain the reasoning step\-by\-step \(chain\-of\-thought\)\. Python Implementation: Consider the logic in code\. CRITICAL: Do NOT output python code to execute\. At the very end of your response, strictly output the answer in this format: Final Answer:Demonstration \-\-\- Example i\+1 \-\-\- Question: \{question\} CoT: \{rationale\} Code: \{code\} \-\-\- Current Task \-\-\- Question: \{question\} Think step by step, ensure precision\. Format: Final Answer:

Prompt template of DROP ReasoningInstruction You are a Logic and Reasoning Expert\. Solve the reading comprehension problem based on the Context\. Reference the provided examples for reasoning style\. Do NOT output python code to execute\. At the very end of your response, strictly output the answer in this format: Final Answer:Demonstration \-\-\- Example i\+1 \-\-\- Context: \{passage\} Question: \{question\} CoT: \{rationale\} Code: \{code\} \-\-\- Current Task \-\-\- Context: \{passage\} Question: \{question\} Think step by step, ensure precision\. Format: Final Answer:

Prompt template of BBH logic subset ReasoningInstruction You are an expert in logical reasoning and causal inference\. I will provide you with REFERENCE EXAMPLES that include a Question, Analysis \(CoT\), and Pseudo\-Code\. NOTE: The reference examples might be from a different domain \(e\.g\., reading comprehension\), but you must abstract the \*\*reasoning logic\*\* and \*\*algorithmic thinking\*\* from them\. Apply this logic to solve the CURRENT TASK step\-by\-step\. Do NOT execute python code\. Use the code only as a logical blueprint\. Demonstration \-\-\- Example i\+1 \-\-\- Context: \{passage\} Question: \{question\} CoT: \{rationale\} Code: \{code\} Note: Learn the reasoning logic \(how to calculate/count/sort\), even if the context format is different\. \-\-\- Current Task \-\-\- Context: \{passage\} Question: \{question\} Think step by step, ensure precision\. Format: Final Answer:

Prompt template of Commonsense and BBH semantic subset ReasoningInstruction You are a Commonsense Reasoning Expert utilizing a Hybrid Reasoning approach\. Combine natural language intuition with Python precision\. Reference the provided examples for reasoning style\. The helper functions in the examples are not implemented, so treat them as judgements you make yourself\. Do NOT output python code to execute\. At the very end of your response, strictly output the answer in this format: Final Answer:Demonstration \-\-\- Example i\+1 \-\-\- Question: \{question\} Options: \{options\} CoT: \{rationale\} Code: \{primitives\} \{code\} \-\-\- Current Task \-\-\- Question: \{question\} Options: \{options\} Think step by step, ensure precision\. Format: Final Answer:

### I\.4Hybrid Abstraction

Prompt template of Rationales Generation and Hybrid AbstractionInstruction You are a Python Expert\. Solve the multiple\-choice question by writing a generic Python function ‘def solution\(\):‘\. First, think step\-by\-step about the logic required \(Chain of Thought\)\. Then, declare any step that requires world knowledge rather than computation as an undefined helper function, with a type signature and a docstring\. Finally, write ‘def solution\(\):‘ that scores each option using those helpers and returns the highest scoring one\. Rules: 1\. Each helper takes at most two arguments, each a single string or number\. 2\. Never pass the question or the full option list to a helper\. 3\. Accumulate scores as floats and compare with ‘\>‘, so ties keep the first option\. 4\. Always return an option, never ‘None‘\. Question: \{question\} Options: \{options\} Format your response as: Rationale: \.\.\. content \.\.\. Primitives: ‘‘‘python \.\.\. declarations only \.\.\. ‘‘‘ Code: ‘‘‘python \.\.\. def solution \.\.\. ‘‘‘ Call: ‘‘‘python \.\.\. solution\(\.\.\.\) \.\.\. ‘‘‘

### I\.5Adaptation and Intervention for Hybrid Programs

Prompt template of Adaptation and Intervention for Hybrid ProgramsInstruction Bind the NEW PROBLEM to the EXISTING CODE\. 1\. Analyze the NEW PROBLEM\. 2\. Do NOT change the helper declarations or the function body\. 3\. Output only the call line, with values from the NEW PROBLEM\. EXISTING CODE: ‘‘‘python \{primitives\} \{code\} ‘‘‘ NEW PROBLEM: \{retrieved\_question\} Options: \{retrieved\_options\} Output the call line only: ‘‘‘python solution\(\.\.\.\) ‘‘‘

### I\.6Run\-Time Function Evaluation

Prompt template of Run\-Time Function EvaluationInstruction Answer the following function call\. You are given the declaration of the function and its arguments, and nothing else\. Return a single value that matches the declared return type\. Do not explain\. Declaration: ‘‘‘python \{declaration\} ‘‘‘ Arguments: \{arguments\} Return value:

### I\.7Similar Sample Fine\-grained Retrieval

Prompt Template for Logically Similar Sample RetrievalInstruction You are a structural reasoning evaluator\. Assess logical similarity between a Target Question and each candidate based on three formal components:1\.Premises \(PP\):Initial conditions, constraints, or given facts\.2\.Subjects \(SS\):Core entities or variables under analysis\.3\.Result Type \(TT\):Required output format or objective\.Two questions are logically similar \(θ=0\\theta=0\) if their\(P,S,T\)\(P,S,T\)structures are equivalent\. Ignore surface phrasing, domain context, or specific numerical values\. Computeθ∈\{0,1\}\\theta\\in\\\{0,1\\\}for each candidate\. Return the indices of the top\-NNcandidates whereθ=0\\theta=0as a Python list\.Target Question: \{question\} Related Questions: \{related\_questions\_List\} Output: