预训练能力何时有助于因果学习？从标量预测到反事实后训练

# When Does Pre-Trained Capacity Help Causal Learning?From Scalar Prediction to Counterfactual Post-Training
Source: [https://arxiv.org/html/2610.33311](https://arxiv.org/html/2610.33311)
## When Does Pre\-Trained Capacity Help Causal Learning? From Scalar Prediction to Counterfactual Post\-TrainingThanks:Accepted at the NeurIPS 2026 Workshop: Transitioning from Pre\-Training to Post\-Training\.

Zijie LeiZhifei DengZhigang HuaLuke SimonAffiliation:Meta Platforms, Inc\.Affiliation:Sunnyvale, CA, USA

###### Abstract

How does pre\-trained model capacity relate to performance after adaptation for causal tasks? We study two systems built on the Qwen 2\.5 family: CausalLLM, which estimates scalar potential outcomes with treatment\-specific heads, and CausalGen, which generates treatment\-conditioned text or code using soft prefixes and factual supervised fine\-tuning \(SFT\), counterfactual SFT, and preference alignment\. Increasing backbone size from 0\.5B to 1\.5B yields mixed scalar\-estimation results on IHDP, but improves CodeGen pass@1 from 0\.267 to 0\.575 and treatment\-effect score from 0\.231 to 0\.515 after the complete generative pipeline\. Within 0\.5B CausalLLM, narrative inputs reduce meanPEHE\\sqrt\{\\mathrm\{PEHE\}\}from 1\.842 to 1\.780, a 3\.4% improvement over tabular inputs; the 5\.8% improvement over a no\-LLM baseline combines changes in representation and input format\. These descriptive comparisons motivate a distinction between available and convertible capacity\. They do not isolate the contribution of post\-training: the two systems differ in dataset, architecture, output modality, and supervision, and intermediate\-stage evaluations are unavailable\. With only two small backbones and three replications, the results support a hypothesis about capacity–adaptation alignment rather than a general scaling law or a demonstrated causal mechanism\.

## 1Introduction

Modern language models are not produced by a single training stage\. Pre\-training establishes broad representations and generative capabilities, while supervised fine\-tuning, preference optimization, and reinforcement\-learning\-style objectives are used to elicit and reshape downstream behavior\. A central question is therefore not simply whether post\-training improves a benchmark, but*which capabilities are already present after pre\-training, which are unlocked by task adaptation, and how model scale changes the effectiveness of the transition*\.

Causal learning provides a useful testbed for this question because closely related tasks can require very different forms of adaptation\. Classical individual treatment effect \(ITE\) estimation asks a model to predict scalar potential outcomesY⁡\(0\)Y\(0\)andY⁡\(1\)Y\(1\)\. Generative counterfactual modeling asks the richer question of producing the full outcome under an alternative intervention\. Both require treatment\-conditioned reasoning, but the former primarily tests whether pre\-trained representations can be adapted into a causally useful predictive representation, whereas the latter additionally relies on the generative capabilities of the base model and on post\-training procedures that teach the model how to express intervention\-specific outcomes\.

We study these regimes with two paradigms built on a common LLM family \(Figure[1](https://arxiv.org/html/2610.33311#S1.F1)\)\.CausalLLMadapts a pre\-trained LLM representation with treatment\-specific scalar heads forY^​\(0\)\\hat\{Y\}\(0\)andY^​\(1\)\\hat\{Y\}\(1\); we additionally vary how structured covariates are rendered to test whether semantically meaningful inputs better expose knowledge acquired during pre\-training\.CausalGenreplaces scalar heads with treatment\-specific projections and soft prefixes that steer a shared decoder, and then applies a three\-stage post\-training pipeline: factual supervised fine\-tuning \(SFT\), counterfactual SFT, and causal preference alignment\.

The resulting comparison reveals different scaling patterns in the two final systems\. On IHDP,PEHE\\sqrt\{\\mathrm\{PEHE\}\}changes from 1\.780 to 1\.796, ATE error from 0\.475 to 0\.445, and RMSE from 1\.247 to 1\.259\. On CodeGen, pass@1 and treatment\-effect score increase by 2\.15×\\timesand 2\.23×\\times, respectively\. Separately, narrative inputs improve meanPEHE\\sqrt\{\\mathrm\{PEHE\}\}by 3\.4% relative to tabular CausalLLM\. These observations motivate further controlled experiments; they do not establish that preference alignment or counterfactual SFT causes the larger model’s advantage\.

Our contributions are threefold: \(1\) we describe two related adaptation designs for scalar potential\-outcome prediction and intervention\-conditioned generation; \(2\) we report input\-format and two\-size comparisons within these systems; and \(3\) we formulate the distinction between*available capacity*and*convertible capacity*as a hypothesis for future capacity–adaptation studies\. The common model family supports within\-system comparisons, but the cross\-system contrast is not a matched experiment because dataset, architecture, output modality, supervision, and training procedure change together\.

![Refer to caption](https://arxiv.org/html/2610.33311v1/fig_architecture_new.png)Figure 1:Two adaptation regimes built on pre\-trained LLMs\. \(a\) CausalLLM adapts a shared encoder with lightweight treatment\-specific scalar heads for ITE estimation\. \(b\) CausalGen uses treatment\-specific projections and soft prefixes followed by factual SFT, counterfactual SFT, and causal preference alignment\. Only the 0\.5B and 1\.5B backbones and DPO\-style preference alignment are evaluated in this paper; the additional 3B, GRPO, and DAPO options shown in the schematic are not evaluated\. The systems differ in data, heads, output modality, supervision, and training procedure; their scaling contrast is descriptive\.
## 2Related work

Deep treatment\-effect estimation\.TARNet\[[7](https://arxiv.org/html/2610.33311#bib.bib7)\]learns shared representations with treatment\-specific heads and balanced representation objectives; DragonNet\[[8](https://arxiv.org/html/2610.33311#bib.bib8)\]augments this setup with propensity modeling and targeted regularization; TransTEE\[[17](https://arxiv.org/html/2610.33311#bib.bib17)\]uses transformer\-based treatment–covariate interactions\. These methods primarily evaluate scalar potential outcomes and treatment effects\.

LLMs and causal inference\.Kiciman et al\.\[[16](https://arxiv.org/html/2610.33311#bib.bib16)\]examine language models as sources of causal knowledge and reasoning\. CLadder\[[11](https://arxiv.org/html/2610.33311#bib.bib11)\]evaluates formal associational, interventional, and counterfactual queries, whereas CausalEval\[[13](https://arxiv.org/html/2610.33311#bib.bib13)\]organizes and evaluates approaches that use language models as reasoning engines or as helpers for traditional causal methods\. Liu et al\.\[[12](https://arxiv.org/html/2610.33311#bib.bib12)\]survey this broader interaction, including effect estimation\. Our scalar estimator belongs to representation\-based adaptation, while our generator evaluates intervention\-conditioned outputs; neither setting alone establishes general causal reasoning ability\.

Post\-training for causal tasks\.CausalGym\[[15](https://arxiv.org/html/2610.33311#bib.bib15)\]directly compares SFT, DPO, KTO, PPO, and GRPO for causal inference tasks\. This is closely related to our motivating question, but differs from our treatment\-specific scalar heads and soft\-prefix generation design\. Unlike that systematic comparison of post\-training approaches, our experiments evaluate CausalGen only after the complete pipeline\. We therefore do not claim to isolate which post\-training objective is responsible for its performance\.

Counterfactual language generation\.CausaLM\[[10](https://arxiv.org/html/2610.33311#bib.bib10)\]uses counterfactual language\-model representations for causal explanation, and Polyjuice\[[9](https://arxiv.org/html/2610.33311#bib.bib9)\]produces controlled text perturbations\. Bynum and Cho\[[14](https://arxiv.org/html/2610.33311#bib.bib14)\]use language\-model\-defined mechanisms in sequence\-driven structural causal models to sample observational, interventional, and counterfactual data\. CausalGen instead learns treatment\-specific output generators from paired targets\. Such targets supply supervision for generation, but generated plausibility or reference similarity does not by itself identify real\-world causal effects\.

## 3Method

### 3\.1CausalLLM: Scalar ITE

A pre\-trained LLM \(Qwen 2\.5\-0\.5B\) encodes covariates via mean pooling:𝐫=MeanPool​\(LLM​\(tokenize​\(𝐱\)\)\)∈ℝ896\\mathbf\{r\}=\\text\{MeanPool\}\(\\text\{LLM\}\(\\text\{tokenize\}\(\\mathbf\{x\}\)\)\)\\in\\mathbb\{R\}^\{896\}\. Two 3\-layer MLP heads \(896→256→128→1896\\to 256\\to 128\\to 1, ReLU, LayerNorm, Dropout\) predicty^​\(t\)=ht​\(𝐫\)\\hat\{y\}\(t\)=h\_\{t\}\(\\mathbf\{r\}\), with ITEτ^=h1​\(𝐫\)−h0​\(𝐫\)\\hat\{\\tau\}=h\_\{1\}\(\\mathbf\{r\}\)\-h\_\{0\}\(\\mathbf\{r\}\)\. Training uses factual loss with treatment routing:

ℒ=1n​∑i\[Ti​ℓ​\(h1​\(𝐫i\),yi\)\+\(1−Ti\)​ℓ​\(h0​\(𝐫i\),yi\)\]\\mathcal\{L\}=\\tfrac\{1\}\{n\}\\textstyle\\sum\_\{i\}\\left\[T\_\{i\}\\ell\(h\_\{1\}\(\\mathbf\{r\}\_\{i\}\),y\_\{i\}\)\+\(1\{\-\}T\_\{i\}\)\\ell\(h\_\{0\}\(\\mathbf\{r\}\_\{i\}\),y\_\{i\}\)\\right\]\(1\)with optional MMD regularization \(λMMD=0\.1\\lambda\_\{\\text\{MMD\}\}\{=\}0\.1, Gaussian kernel with median heuristic bandwidth\) to encourage balanced representations between treated and control groups\. The LLM backbone is fine\-tuned via LoRA\[[2](https://arxiv.org/html/2610.33311#bib.bib2)\]\(rank 16,α=32\\alpha\{=\}32,∼\{\\sim\}0\.5% trainable parameters\), preserving pre\-trained knowledge while enabling task\-specific adaptation\.

Text enrichment\.A key design choice is how tabular covariates are presented to the LLM\. We compare three formatting modes of increasing semantic richness:*Tabular*\(raw numbers: “0\.5 1\.0 0\.0…”\),*Structured*\(key\-value pairs: “age: 25, education: college, friends: 342”\), and*Narrative*\(natural language: “A 25\-year\-old college\-educated user with 342 friends and moderate engagement…”\)\. The hypothesis, not directly tested by representation probes or a randomly initialized backbone control, is that narrative mode better exposes the LLM’s pre\-trained understanding of real\-world feature relationships—e\.g\., that education level correlates with employment outcomes—while tabular mode treats the LLM as a generic feature extractor\.

Treatment\-conditioned alternative\.A natural treatment\-conditioned alternative is to concatenate a treatment token to the input and evaluate the same model twice, once witht=0t=0and once witht=1t=1, yieldingτ^​\(x\)=f⁡\(x,1\)−f⁡\(x,0\)\\hat\{\\tau\}\(x\)=f\(x,1\)\-f\(x,0\)\. This single\-head design evaluates the encoder separately for each treatment\. Our dual\-head formulation shares the covariate encoding and applies lightweight treatment\-specific heads or prefixes\. For scalar prediction, both outcomes can be computed from one encoding\. For generation, producing both outcomes still requires two autoregressive decoding sequences; sharing the encoding does not eliminate this cost, and we report no measured latency advantage\.

### 3\.2CausalGen: Counterfactual Generation

CausalGen shares the LLM encoder with CausalLLM but replaces MLP prediction heads withdual projection headsthat transform the shared representation into treatment\-specific representations via residual projection:𝐳t=MLPt​\(𝐫\)\+𝐫\\mathbf\{z\}\_\{t\}=\\text\{MLP\}\_\{t\}\(\\mathbf\{r\}\)\+\\mathbf\{r\}, where each MLP is a 2\-layer network \(dmodel→2​dmodel→dmodeld\_\{\\text\{model\}\}\\to 2d\_\{\\text\{model\}\}\\to d\_\{\\text\{model\}\}\) with GELU activation\. Each𝐳t\\mathbf\{z\}\_\{t\}is expanded intoK=8K\{=\}8soft prefix tokens via learned linear projection:𝐏t=Reshape​\(Wprefix​𝐳t\)∈ℝK×dmodel\\mathbf\{P\}\_\{t\}=\\text\{Reshape\}\(W\_\{\\text\{prefix\}\}\\mathbf\{z\}\_\{t\}\)\\in\\mathbb\{R\}^\{K\\times d\_\{\\text\{model\}\}\}\. These prefix tokens are prepended to the target sequence embeddings, steering a*shared*LLM decoder to generateY⁡\(t\)Y\(t\)as text or code—analogous to prefix tuning\[[4](https://arxiv.org/html/2610.33311#bib.bib4)\]but conditioned on causal representations\. An optionalDeltaHeadgenerates treatment effect descriptions by composing𝐳0\\mathbf\{z\}\_\{0\}and𝐳1\\mathbf\{z\}\_\{1\}\(via gating, cross\-attention, subtraction, or bilinear modes\)\.

Training uses three stages with different supervision\. These stages describe the implemented pipeline; their individual necessity and incremental benefit are not established by the reported experiments:

Stage 1: Factual SFT\(15 epochs\)\. Each head is trained on observed outcomes only, with treatment routing: samples withT=0T\{=\}0trainh0h\_\{0\}, samples withT=1T\{=\}1trainh1h\_\{1\}\. This establishes factual generation quality\.

Stage 2: Counterfactual SFT\(15 epochs\)\. Both heads are trained jointly on paired\(Y⁡\(0\),Y⁡\(1\)\)\(Y\(0\),Y\(1\)\)data, forcing each head to generate the correct potential outcome:ℒ2=12​\[ℒCE​\(h0,Y⁡\(0\)\)\+ℒCE​\(h1,Y⁡\(1\)\)\]\\mathcal\{L\}\_\{2\}=\\frac\{1\}\{2\}\[\\mathcal\{L\}\_\{\\text\{CE\}\}\(h\_\{0\},Y\(0\)\)\+\\mathcal\{L\}\_\{\\text\{CE\}\}\(h\_\{1\},Y\(1\)\)\]\.

Stage 3: Causal preference alignment\(5 epochs\)\. Preference optimization targets treatment\-specific fidelity using treatment\-derived preference pairs: forh1h\_\{1\},Y⁡\(1\)Y\(1\)is “preferred” andY⁡\(0\)Y\(0\)is “rejected”; vice versa forh0h\_\{0\}\. We use DPO\-style preference optimization\[[6](https://arxiv.org/html/2610.33311#bib.bib6)\]\. GRPO and DAPO are not evaluated here\. Opposite\-treatment outcomes serve as rejected targets only when the intervention changes the outcome; unchanged or equally valid outcomes require careful pair filtering\. Appendix[A](https://arxiv.org/html/2610.33311#A1)explains the conditioning and remaining implementation details\.

## 4Experiments

Datasets\.CausalLLM:IHDP\[[1](https://arxiv.org/html/2610.33311#bib.bib1)\]\(747 samples, 25 covariates, semi\-synthetic outcomes\),Twins\[[5](https://arxiv.org/html/2610.33311#bib.bib5)\]\(5K twin pairs, 15 covariates, binary mortality outcome\)\. CausalGen:CodeGen\(340 paired code samples with quality degradation treatment\),Jobs\[[3](https://arxiv.org/html/2610.33311#bib.bib3)\]\(3\.2K samples, job training treatment\)\.

Baselines\.CausalLLM: T\-Learner \(separate models\), TARNet\[[7](https://arxiv.org/html/2610.33311#bib.bib7)\]\(shared encoder \+ dual heads\)\. CausalGen: single\-head \(one head for both outcomes\), treatment\-conditioned \(treatment label in prompt\)\.

Implementation\.Unless otherwise stated, the main experiments use Qwen 2\.5\-0\.5B with LoRA \(rank 16,α=32\\alpha\{=\}32\); the capacity study in Section[4\.4](https://arxiv.org/html/2610.33311#S4.SS4)repeats the matched setup with Qwen 2\.5\-1\.5B\. MLP baselines: Adam, lr=10−310^\{\-3\}, 100 epochs\. LLM models: AdamW, backbone lr=2×10−52\{\\times\}10^\{\-5\}, head lr=10−310^\{\-3\}\. CausalLLM: 20 epochs\. CausalGen: 15\+15\+5 epochs across three stages\. All results are reported as averages over 3 replications\. Only Table[1](https://arxiv.org/html/2610.33311#S4.T1)provides standard deviations for PEHE; the remaining summaries do not establish statistical significance\. Appendix[A](https://arxiv.org/html/2610.33311#A1)consolidates the documented configuration and definitions\.

### 4\.1CausalLLM Results

Table 1:Scalar ITE estimation \(3 reps, mean±\\pmstd\)\. “Dual\-head MLP” is the no\-LLM baseline shown as “MLP \(no LLM\)” in Fig\.[2](https://arxiv.org/html/2610.33311#S4.F2); CausalLLM \(Narrative\) uses the pre\-trained LLM representation with the same treatment\-specific prediction structure\.↓\\downarrow= lower is better\.On IHDP, narrative CausalLLM has lower meanPEHE\\sqrt\{\\mathrm\{PEHE\}\}than the no\-LLM dual\-head MLP \(1\.780 versus 1\.890\), a 5\.8% relative reduction\. This comparison changes both the representation and input rendering and cannot isolate either contribution\. Within CausalLLM, the mean errors for tabular, structured, and narrative inputs are 1\.842, 1\.810, and 1\.780, respectively \(Figure[2](https://arxiv.org/html/2610.33311#S4.F2)\)\. The narrative–tabular reduction is\(1\.842−1\.780\)/1\.842=3\.4%\(1\.842\-1\.780\)/1\.842=3\.4\\%, while structured–tabular and narrative–structured reductions are 1\.7% and 1\.7%\. These mean differences are descriptive: three replications and the reported IHDP variability do not justify a significance claim\.

The metric tradeoff also matters\. On IHDP, narrative CausalLLM’s ATE error \(0\.48\) is worse than each listed baseline \(0\.21, 0\.34, and 0\.31\), despite its lower mean PEHE\. On Twins, its PEHE is numerically close to the no\-LLM baseline \(0\.450 versus 0\.452\), and its rounded ATE error ties TARNet \(0\.012\)\. Thus, the evidence does not support a uniform advantage across datasets or estimands\.

Figure 2:Input\-format comparison \(IHDP, 0\.5B\)\. Narrative inputs reduce meanPEHE\\sqrt\{\\mathrm\{PEHE\}\}by 3\.4% versus tabular CausalLLM\. The 5\.8% reduction versus the no\-LLM MLP is a different, combined comparison; uncertainty for the format differences is not available\.
### 4\.2CausalGen Results

Table 2:Counterfactual generation \(3 reps\)\. The final dual\-head system has 2\.3–2\.5×\\timeshigher reference BLEU for treatment armY⁡\(0\)Y\(0\)\.CausalGen obtains 2\.5×\\timeshigher reference BLEU forY⁡\(0\)Y\(0\)on CodeGen \(0\.374 versus 0\.151\) and 2\.3×\\timeson Jobs \(0\.440 versus 0\.193\)\. These comparisons favor the final dual\-head system, but do not separately identify architecture and training\-stage effects\. Reference similarity is also not equivalent to functional correctness or causal identification\. Although the original experiment description labelsY⁡\(0\)Y\(0\)counterfactual andY⁡\(1\)Y\(1\)factual, this designation is treatment\-dependent: for an example with observedTiT\_\{i\}, the factual outcome isYi​\(Ti\)Y\_\{i\}\(T\_\{i\}\)and the counterfactual isYi​\(1−Ti\)Y\_\{i\}\(1\-T\_\{i\}\)\. Interpretation of the arm\-specific scores therefore requires the evaluation cohort’s treatment assignment and paired\-target construction\.

Generation example\.On CodeGen, givendef calculate\_area\(shape, dims\):, CausalGen generates treatment\-specific counterfactual implementations\.Y\(0\)Y^\{\(0\)\}\(without code review\) contains a minimal implementation with hardcoded constants and limited validation, whileY\(1\)Y^\{\(1\)\}\(with code review\) adds type hints, docstrings, input validation, and replaces hardcoded values withmath\.pi\. The DeltaHead summarizes the inferred treatment effect as: “Added documentation, validation, and numerical precision improvements\.” On Jobs, given age 28, high\-school education, and prior earnings of $4,200,Y\(0\)Y^\{\(0\)\}describes prolonged unemployment and unstable work, whileY\(1\)Y^\{\(1\)\}describes obtaining stable employment after training\.

### 4\.3DeltaHead Ablation

The DeltaHead composes𝐳0\\mathbf\{z\}\_\{0\}and𝐳1\\mathbf\{z\}\_\{1\}to generate treatment effect text\. We compare four composition modes on CodeGen \(Table[3](https://arxiv.org/html/2610.33311#S4.T3)\)\.

Table 3:DeltaHead ablation \(CodeGen, 0\.5B, 3 reps\)\.Cross\-attention has the highest reported pass@1 \(0\.408\), TE Score \(0\.363\), and Swap score \(0\.654\)\. Gate has the highest CF Faithfulness \(0\.429\), using adaptive gating over\[𝐳1;𝐳0;𝐳1−𝐳0;𝐳1⊙𝐳0\]\[\\mathbf\{z\}\_\{1\};\\mathbf\{z\}\_\{0\};\\mathbf\{z\}\_\{1\}\-\\mathbf\{z\}\_\{0\};\\mathbf\{z\}\_\{1\}\\odot\\mathbf\{z\}\_\{0\}\]\. Thus, the table shows a tradeoff rather than a single best composition\. Without uncertainty estimates, the difference between cross\-attention and gating TE scores \(0\.009\) should not be treated as statistically established\. The generation and scaling tables represent separate reported settings; their 0\.5B rows are not identical and cannot be pooled as a single configuration\.

### 4\.4A Task\-Dependent Capacity Effect

Table 4:Effect of increasing backbone size from 0\.5B to 1\.5B\. Scalar ITE estimation on IHDP shows no consistent gain across error metrics, whereas counterfactual generation on CodeGen improves substantially\. We treat this two\-size comparison as a task\-dependent capacity effect, not a scaling law\.Table[4](https://arxiv.org/html/2610.33311#S4.T4)shows a task\-dependent response to additional pre\-trained capacity\. Moving from Qwen 2\.5\-0\.5B to 1\.5B does not consistently improve scalar ITE estimation on IHDP:PEHE\\sqrt\{\\mathrm\{PEHE\}\}changes from 1\.780 to 1\.796, ATE error improves from 0\.475 to 0\.445, and RMSE changes from 1\.247 to 1\.259\. In contrast, the same increase in backbone size produces a large gain for counterfactual generation on CodeGen\.

Scalar ITE shows no consistent capacity benefit in this comparison\.The mixed movement acrossPEHE\\sqrt\{\\mathrm\{PEHE\}\}, ATE error, and RMSE indicates that a larger pre\-trained backbone alone is not sufficient to improve scalar causal estimation in our setting\. One possible explanation is that scalar ITE performance is constrained by causal supervision and treatment\-specific adaptation rather than generic semantic capacity, but our experiments do not isolate the mechanism\.

Counterfactual generation benefits strongly from the larger backbone\.From 0\.5B to 1\.5B, pass@1 rises 2\.15×\\times\(0\.267→\\to0\.575\), TE Score rises 2\.23×\\times\(0\.231→\\to0\.515\), and ATE\-direction accuracy rises from 73\.3% to 93\.3%\. These results describe better final\-system generation metrics under the reported CodeGen pipeline\. They do not distinguish improvements already present in the larger base model from those attributable to individual adaptation stages\.

Interpretation\.The contrast motivates a distinction between*available capacity*\(the backbone’s parameter and representation resources\) and*convertible capacity*\(the portion that an adaptation setup can use for a target task\)\. These are conceptual terms, not independently measured quantities\. Counterfactual SFT and preference alignment are plausible pathways for exploiting capacity, but the present experiments cannot attribute the scaling difference to them\. Dataset, architecture, output modality, and supervision all differ across systems\.

## 5Discussion and Limitations

What the experiments establish\.We observe mixed 0\.5B\-to\-1\.5B scalar metrics on IHDP, improved final\-system generation metrics on CodeGen, and lower mean PEHE with narrative rather than tabular inputs within 0\.5B CausalLLM\. These results suggest that task design and input representation deserve attention alongside parameter count\. They do not demonstrate a universal relation between pre\-training scale and causal capability\.

What remains unisolated\.We do not evaluate an untuned base model or intermediate checkpoints after factual SFT and counterfactual SFT at both sizes\. Consequently, the marginal contributions of paired supervision and preference alignment, and their interaction with capacity, are unknown\. Likewise, the input\-format study lacks additional replications and controls for semantic content, sequence length, and random initialization; it cannot establish that narrative rendering activates particular pre\-trained knowledge\.

Scope and statistical uncertainty\.Only two small Qwen 2\.5 backbones are compared, and the datasets are small, especially CodeGen \(340 paired examples before splitting\)\. Three replications and missing dispersion for most generation metrics limit statistical confidence\. The cross\-system comparison changes dataset, architecture, output type, supervision, and training procedure together\. Neither larger\-model behavior nor transfer across model families is established\. The IHDP ATE tradeoff further prevents a claim of uniformly better causal estimation\.

Causal identification and target provenance\.Potential\-outcome estimation from observational data relies on consistency, overlap, and conditional exchangeability; an LLM representation or balancing penalty does not guarantee these assumptions\. Paired generation targets provide supervised alternative outcomes, but their validity depends on their construction\. Jobs does not ordinarily contain both individual potential outcomes, so reference narratives must not be interpreted as observed individual counterfactual truth\. The construction and evaluation of those paired targets require explicit documentation\. BLEU and qualitative examples measure output agreement or illustrate behavior, rather than proving real\-world causal validity\.

Next controlled tests\.A stronger study would evaluate the base model and every post\-training checkpoint at both sizes on the same held\-out examples; train scalar and generative outputs on one paired dataset with identical information and splits; and add seeds, confidence intervals, larger backbones, and another model family\. Validation\-selected checkpoints are also needed to assess whether the 100\-epoch MLP schedule overfits\. These tests are proposed work, not completed experiments\.

## 6Conclusion

Our two adaptation systems exhibit different reported capacity responses: scalar potential\-outcome estimation has mixed metric changes, whereas the fully adapted counterfactual generator improves at 1\.5B\. Narrative inputs also reduce mean scalar PEHE relative to tabular inputs, with uncertain statistical support\. These observations motivate available versus convertible capacity as a useful hypothesis\. Establishing the mechanism requires matched data and stage\-by\-stage evaluations that separate pre\-training, architecture, supervision, and optimization\.

## References

- \[1\]J\. L\. Hill\. 2011\. Bayesian nonparametric modeling for causal inference\.JCGS20, 1 \(2011\), 217–240\.
- \[2\]E\. J\. Hu et al\. 2022\. LoRA: Low\-rank adaptation of large language models\. InICLR\.
- \[3\]R\. J\. LaLonde\. 1986\. Evaluating the econometric evaluations of training programs\.AER76, 4 \(1986\), 604–620\.
- \[4\]X\. L\. Li and P\. Liang\. 2021\. Prefix\-tuning: Optimizing continuous prompts for generation\. InACL\.
- \[5\]C\. Louizos et al\. 2017\. Causal effect inference with deep latent\-variable models\. InNeurIPS\. 6446–6456\.
- \[6\]R\. Rafailov et al\. 2023\. Direct preference optimization\. InNeurIPS\.
- \[7\]U\. Shalit, F\. D\. Johansson, and D\. Sontag\. 2017\. Estimating individual treatment effects\. InICML\. 3076–3085\.
- \[8\]C\. Shi, D\. M\. Blei, and V\. Veitch\. 2019\. Adapting neural networks for treatment effects\. InNeurIPS\. 2507–2517\.
- \[9\]T\. Wu et al\. 2021\. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models\. InACL\.
- \[10\]A\. Feder et al\. 2021\. CausaLM: Causal model explanation through counterfactual language models\.Computational Linguistics\(2021\)\.
- \[11\]Z\. Jin et al\. 2023\. CLadder: Assessing causal reasoning in language models\. InNeurIPS\.[https://arxiv\.org/abs/2312\.04350](https://arxiv.org/abs/2312.04350)\.
- \[12\]X\. Liu et al\. 2025\. Large language models and causal inference in collaboration: A comprehensive survey\. InFindings of NAACL, 7683–7699\.
- \[13\]L\. Yu et al\. 2025\. CausalEval: Towards better causal reasoning in language models\. InNAACL, 12512–12540\.
- \[14\]L\. E\. J\. Bynum and K\. Cho\. 2025\. Language models as causal effect generators\. InEMNLP, 2096–2115\.
- \[15\]J\. Chen, S\. Chen, and C\. Lu\. 2026\. Can post\-training transform LLMs into causal reasoners? InFindings of ACL, 17020–17038\.
- \[16\]E\. Kiciman et al\. 2023\. Causal reasoning and large language models: Opening a new frontier for causality\.arXiv:2305\.00050\(2023\)\.
- \[17\]H\. Zhang et al\. 2022\. Exploring transformer backbones for heterogeneous treatment effect estimation\. InNeurIPS\.

## Appendix AExperimental Details and Definitions

This appendix consolidates the documented configuration and distinguishes standard definitions from implementation details not available in the experiment record\.

### A\.1Scalar estimation

For covariatesxix\_\{i\}and binary treatmentTiT\_\{i\}, the observed outcome isYi​\(Ti\)Y\_\{i\}\(T\_\{i\}\)and the individual treatment effect isτi=Yi​\(1\)−Yi​\(0\)\\tau\_\{i\}=Y\_\{i\}\(1\)\-Y\_\{i\}\(0\)\. CausalLLM predictsτ^i=Y^i​\(1\)−Y^i​\(0\)\\hat\{\\tau\}\_\{i\}=\\hat\{Y\}\_\{i\}\(1\)\-\\hat\{Y\}\_\{i\}\(0\)\. The standard scalar metrics are

PEHE\\displaystyle\\sqrt\{\\mathrm\{PEHE\}\}=1n​∑i\(τ^i−τi\)2,\\displaystyle=\\sqrt\{\\frac\{1\}\{n\}\\sum\_\{i\}\(\\hat\{\\tau\}\_\{i\}\-\\tau\_\{i\}\)^\{2\}\},\(2\)ATE​error\\displaystyle\\mathrm\{ATE\\ error\}=\|1n​∑iτ^i−1n​∑iτi\|\.\\displaystyle=\\left\|\\frac\{1\}\{n\}\\sum\_\{i\}\\hat\{\\tau\}\_\{i\}\-\\frac\{1\}\{n\}\\sum\_\{i\}\\tau\_\{i\}\\right\|\.\(3\)ATE is an estimand, whereas the downward\-arrow scalar\-table entries refer to its absolute estimation error\. Reference individual effects are required for PEHE\. The standard factual\-outcome RMSE isn−1​∑i\(Y^i​\(Ti\)−Yi​\(Ti\)\)2\\sqrt\{n^\{\-1\}\\sum\_\{i\}\(\\hat\{Y\}\_\{i\}\(T\_\{i\}\)\-Y\_\{i\}\(T\_\{i\}\)\)^\{2\}\}; whether the reported RMSE uses factual outcomes or both potential outcomes remains to be verified against the evaluation code\.

Factual training routes each unit only to its observed\-treatment head\. The optional MMD regularizer compares treated and control representations using a Gaussian kernel with median\-heuristic bandwidth and weight 0\.1\. Which reported runs enable MMD is not specified\. The input\-format examples in the method are schematic, not the exact IHDP or Twins templates\. A format comparison should retain identical pre\-treatment information and exclude outcomes, post\-treatment variables, or inferred facts from narratives\.

### A\.2Soft prefixes and generation supervision

The documented generator computeszt=r\+MLPt​\(r\)z\_\{t\}=r\+\\mathrm\{MLP\}\_\{t\}\(r\)andPt=Reshape⁡\(Wprefix​zt\)∈ℝ8×dP\_\{t\}=\\mathrm\{Reshape\}\(W\_\{\\mathrm\{prefix\}\}z\_\{t\}\)\\in\\mathbb\{R\}^\{8\\times d\}\. Each prefix row is a continuous embedding, not a vocabulary token\. Treatment selects the projection and prefix, which precede the target embeddings and condition a shared autoregressive decoder\. Both requested outcomes require separate decoding sequences despite sharing the covariate encoding\.

The sequence objective is cross\-entropy on target outcomes\. Factual SFT supervises only the observed arm, whereas counterfactual SFT supervises both paired targets with equal weight\. A standard expression of the stated conditioning is

−∑j=1\|Y⁡\(t\)\|logpθ\(Y\(t\)j∣Pt,Y\(t\)<j\)\.\-\\sum\_\{j=1\}^\{\|Y\(t\)\|\}\\log p\_\{\\theta\}\\big\(Y\(t\)\_\{j\}\\mid P\_\{t\},Y\(t\)\_\{<j\}\\big\)\.\(4\)This expresses the design rather than an exact training prompt\. Additional covariate or instruction tokens, attention masks, position handling, truncation, and target\-loss masking are not specified in the supplied experiment record\.

Stage 3 usesY⁡\(t\)Y\(t\)as the preferred outcome andY⁡\(1−t\)Y\(1\-t\)as the rejected outcome for armtt\. Standard DPO regularizes a preference likelihood ratio relative to a reference policy\[[6](https://arxiv.org/html/2610.33311#bib.bib6)\]\. The implemented reference checkpoint, preference strength, pair filtering, and sequence normalization are not specified\. Unchanged or equally valid outcomes need careful filtering\. This offline preference objective does not imply online RL rollouts; no GRPO or DAPO comparison is reported\.

### A\.3Baselines and hyperparameters

The T\-Learner fits independent treatment and control models, TARNet uses a shared representation and treatment\-specific heads, and the dual\-head MLP provides the no\-LLM scalar comparison\. The generation table reports a single\-head baseline\. The treatment\-conditioned alternative is described but has no numerical result in that table\. Exact baseline layer sizes, regularization, and whether generator baselines receive identical training data and stages are not specified, so the comparisons do not isolate architecture alone\.

Table 5:Documented experimental configuration\.These settings do not establish validation\-based hyperparameter selection\. Batch size, dropout rate, weight decay, learning\-rate schedule, LoRA target modules, checkpoint selection, seeds, and split sizes are not available\. A 100\-epoch MLP schedule alone establishes neither overfitting nor its absence; validation curves and checkpoint selection are needed\. The experiment record also does not explain the different 0\.5B generator scores in the composition and size studies\.

### A\.4Generation metrics and target provenance

BLEU measures reference overlap rather than causal validity\. Reproduction requires tokenization, n\-gram order, smoothing, aggregation, and code normalization\. In code generation, standard pass@1 measures the probability that one candidate passes the task’s tests\. The actual tests, decoding settings, evaluation denominator, and use of that standard protocol need confirmation\. TE Score, CF Faithfulness, Swap, and generation ATE\-direction percentage are task\-specific metrics; scoring functions, references, tie handling, and aggregation are not provided in the experiment record\. These quantities cannot be equated with scalar PEHE or ATE error\.

The described generation datasets contain 340 paired CodeGen examples and 3\.2K Jobs examples\. The code illustration uses code review, whereas the dataset description refers to quality degradation; their precise treatment mapping needs documentation\. CodeGen’s paired\-target construction, split, and related\-code leakage controls are not specified\. Jobs ordinarily contains observed earnings rather than both individual potential outcomes\. Its conversion to paired outcome narratives, and whether they are synthetic or model\-generated, therefore needs documentation\. Generated narratives should not be interpreted as observed individual counterfactual truth\. The main\-text examples illustrate behavior rather than providing this provenance\.

## Appendix BA Controlled Follow\-up Design

To test capacity–adaptation interactions, evaluate the untuned base, factual\-SFT, counterfactual\-SFT, and preference\-aligned checkpoints for both sizes on identical held\-out examples\. Fix prompts, decoding, and evaluation\. Stage\-to\-stage changes quantify incremental gains; differences in those changes across model sizes test whether a stage benefits the larger model more\. Multiple seeds and paired uncertainty estimates are needed\. No missing checkpoint results are imputed in this revision\.

A shared paired dataset could provide complete outcomes and pre\-specified scalar summaries, allowing both output tasks to use identical covariates, treatments, splits, and supervised information\. Input\-format controls should preserve information and record token lengths\. Larger backbones and another family would assess external validity\. These are proposed controls, not completed experiments\.