Conditional Hypothesis Generation for LLM-Based Text Analysis with Researcher-Specified Covariates

# Conditional Hypothesis Generation for LLM-Based Text Analysis with Researcher-Specified Covariates
Source: [https://arxiv.org/html/2606.03029](https://arxiv.org/html/2606.03029)
Wei Ai

University of Maryland, College Park \{paiheng,jliu28,aiwei\}@umd\.edu

###### Abstract

A core goal of computational social science is to discover interpretable differences in how language varies across outcomes of interest, such as political affiliation or instructional quality\. Recentllm\-based hypothesis generation methods describe such differences in natural language, but select for globally discriminative patterns without accounting for covariates that shape the data based on researchers’ domain knowledge\. When covariates are ignored, selected patterns can reflect confounds rather than differences of substantive interest\. We introduce*conditional*hypothesis generation, a framework that incorporates researcher\-specified covariates to steer hypothesis discovery toward differences that hold within relevant subgroups\. Two challenges arise: the target subgroup may be underrepresented \(*stratum imbalance*\), and the direction of a difference may reverse across subgroups \(*sign reversal*\)\. We propose two econometrics\-inspired methods: one introduces feature–covariate interactions to detect sign reversals, and the other applies within\-stratum demeaning and inverse\-frequency reweighting to equalize underrepresented strata\. Synthetic experiments show each method outperforms global baselines in its targeted setting, and expert evaluation on two real\-world datasets confirms that covariate\-aware generation surfaces more useful hypotheses within relevant subgroups\.

Conditional Hypothesis Generation for LLM\-Based Text Analysis with Researcher\-Specified Covariates

Paiheng Xu, Jing Liu and Wei AiUniversity of Maryland, College Park\{paiheng,jliu28,aiwei\}@umd\.edu

## 1Introduction

A central goal of computational social science \(css\) is to understand how text relates to variables such as political affiliation, instructional quality, or social media engagement\. Rather than predicting these outcomes, researchers seek interpretable hypotheses—natural\-language descriptions of how textual patterns differ across outcome values—that can guide further investigation\(Grimmer and Stewart,[2013](https://arxiv.org/html/2606.03029#bib.bib27); Card,[2019](https://arxiv.org/html/2606.03029#bib.bib28); Grimmeret al\.,[2022](https://arxiv.org/html/2606.03029#bib.bib26)\)\.

Recentllm\-based methods support this form of analysis by sampling labeled examples and prompting anllmto propose natural\-language hypotheses that characterize textual patterns associated with different outcome values\(Zhonget al\.,[2022](https://arxiv.org/html/2606.03029#bib.bib30),[2023](https://arxiv.org/html/2606.03029#bib.bib31),[2024](https://arxiv.org/html/2606.03029#bib.bib33); Zhouet al\.,[2024](https://arxiv.org/html/2606.03029#bib.bib24); Movvaet al\.,[2025](https://arxiv.org/html/2606.03029#bib.bib32)\)\. These methods typically select hypotheses by how well they discriminate between outcome groups\.

However, global discrimination can be misleading\. A globally discriminative feature may reflect a confound rather than a difference of substantive interest, a persistent concern in text\-as\-data research\(Grimmer and Stewart,[2013](https://arxiv.org/html/2606.03029#bib.bib27); Gentzkowet al\.,[2019](https://arxiv.org/html/2606.03029#bib.bib21); Grimmeret al\.,[2022](https://arxiv.org/html/2606.03029#bib.bib26)\)\. For example,Taddy \([2013](https://arxiv.org/html/2606.03029#bib.bib22)\)shows that national\-park language appears as a party\-predictive feature because public lands are unevenly distributed across states, though national park issues are not inherently partisan\. The challenge is to steer discovery toward differences that hold within conditions researchers care about—and to let researchers specify those conditions\.

We introduce*conditional*hypothesis generation, a framework for generating hypotheses that are discriminative within researcher\-specified covariate strata\. Covariates—such as policy area, time period, or classroom environment—encode the domain knowledge that researchers bring to text analysis: they define the conditions under which differences should be examined, without requiring the hypothesis itself to be known in advance\.

Conditioning on covariates introduces two statistical challenges\(Simpson,[1951](https://arxiv.org/html/2606.03029#bib.bib29); Gail and Simon,[1985](https://arxiv.org/html/2606.03029#bib.bib38)\)\. The target stratum may be underrepresented, so that its signal is dominated by larger strata \(*stratum imbalance*\); or the direction of a difference may reverse across strata \(*sign reversal*\), so that global aggregation cancels the conditional pattern\.

We build onMovvaet al\.\([2025](https://arxiv.org/html/2606.03029#bib.bib32)\), which maps documents to monosemantic Sparse Autoencoder \(sae\) features and selects discriminative features vialasso\. Because thesaefeatures are fixed before statistical selection, covariates can be incorporated directly into the selection step\. Drawing on econometrics, we propose two complementary methods\.*Interaction\-lasso*augments the feature space with feature–covariate interactions, allowing a feature to be selected when it is discriminative within a single stratum, even if its global effect is zero\.*Demeaned\-reweighted\-lasso*residualizes features and outcomes within covariate strata to isolate within\-stratum variation, and applies inverse\-frequency weighting so that underrepresented strata contribute comparably to feature selection\.

In synthetic evaluations with known ground\-truth hypotheses and covariate structure, demeaned\-reweighted\-lasso outperforms global baselines and approaches oracle performance across imbalance levels, while interaction\-lasso is the only method that recovers differences under sign reversal\.

We validate on two real\-world datasets:congress, a longstanding testbed for political language\(Gentzkow and Shapiro,[2010](https://arxiv.org/html/2606.03029#bib.bib15); Grimmeret al\.,[2021](https://arxiv.org/html/2606.03029#bib.bib14)\), andncte, a dataset of math classroom transcripts with rich annotations of instructional quality\(Demszky and Hill,[2023](https://arxiv.org/html/2606.03029#bib.bib13); Hillet al\.,[2008](https://arxiv.org/html/2606.03029#bib.bib9); Piantaet al\.,[2012](https://arxiv.org/html/2606.03029#bib.bib12)\)\. Expert evaluations show that covariate\-aware selection surfaces hypotheses that domain experts rate as more useful than those unique to the global baseline\.

Our contributions are: \(1\) We formalize conditional hypothesis generation for text analysis with researcher\-specified covariates\. \(2\) We introduce two complementary covariate\-aware methods, each targeting a distinct statistical challenge \(i\.e\., stratum imbalance and sign reversal\)\. \(3\) We design controlled synthetic evaluations covering these two challenges\. \(4\) Expert evaluation on two real\-world datasets shows that covariate\-aware methods guide discovery toward hypotheses that domain experts find more useful\.

## 2Preliminaries

### 2\.1Task Formulation

We consider a dataset\{\(xi,yi\)\}i∈\[N\]\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i\\in\[N\]\}, wherexix\_\{i\}is a text document andyiy\_\{i\}is a target variable\. In the simplest case,yiy\_\{i\}indicates group membership of the document\. The goal of*hypothesis generation*is to produce a set of natural\-language statementsℋ\\mathcal\{H\}that characterize howyiy\_\{i\}varies with text content, where\|ℋ\|\|\\mathcal\{H\}\|is a pre\-specified number of hypotheses to generate\. Each text is associated with a set of covariates𝒱\\mathcal\{V\}, which may include metadata \(e\.g\., time period, author demographics\) or attributes inferable from text \(e\.g\., topic\)\. In our setting, a researcher may specify a subset𝐂\\mathbf\{C\}of these covariates based on domain knowledge to guide hypothesis generation within relevant covariate strata\.

### 2\.2llm\-Based Hypothesis Generation

llm\-based hypothesis generation for comparing texts typically follows a sample\-and\-propose pattern: sample a small set of labeled examples, then ask anllmto induce patterns and propose natural\-language hypotheses from those examples\. Existing methods differ in their sampling strategies\.Zhonget al\.\([2022](https://arxiv.org/html/2606.03029#bib.bib30),[2023](https://arxiv.org/html/2606.03029#bib.bib31)\)train a classification model to randomly sample the most discriminative texts\.Zhouet al\.\([2024](https://arxiv.org/html/2606.03029#bib.bib24)\)iteratively generate hypotheses from examples that the current candidate list mis\-classifies\. These methods then filter candidate hypotheses by their discriminative power\.

Zhonget al\.\([2024](https://arxiv.org/html/2606.03029#bib.bib33)\); Movvaet al\.\([2025](https://arxiv.org/html/2606.03029#bib.bib32)\)instead use statistical models to select the most predictive features over allNNsamples, then call anllmto describe each selected feature from a sample of feature\-related texts\.Movvaet al\.\([2025](https://arxiv.org/html/2606.03029#bib.bib32)\), the current state of the art, train a Sparse Autoencoder \(sae\) on text embeddings to obtain a fixed set of interpretable features, then select those most predictive ofyyvialasso, as detailed in §[2\.3](https://arxiv.org/html/2606.03029#S2.SS3)\. We build on this feature\-selection view because it makes incorporating covariates𝐂\\mathbf\{C\}well\-defined\.

### 2\.3Hypothesis Generation withsae

Asaeencourages each input to be reconstructed from only a small active subset of features\(Hubenet al\.,[2024](https://arxiv.org/html/2606.03029#bib.bib35)\)\. This training objective tends to produce monosemantic features, where each dimension captures a coherent, interpretable concept\.Movvaet al\.\([2025](https://arxiv.org/html/2606.03029#bib.bib32)\)exploit this property with three steps:\(1\) SAE encoding: a trainedsaemaps each document embedding to a sparse vector of feature activations\. Each dimension in this vector corresponds to one learnedsaefeature\.\(2\) Feature selection: a statistical model \(e\.g\.,lasso\) selects\|ℋ\|\|\\mathcal\{H\}\|features most predictive ofyy\.\(3\)llminterpretation: anllmdescribes the texts that contrast most strongly activating input documents \(positive examples\) vs\. zero\-activating input documents \(negative examples\) for each selected feature as a natural\-language hypothesis; Table[8](https://arxiv.org/html/2606.03029#A3.T8)shows the prompt\. Step 2 accepts any statistical model, making it straightforward to condition feature selection on covariates𝐂\\mathbf\{C\}, as we show next\.

## 3Method

We define the goal of*conditional*hypothesis generation as discovering group differences that are discriminative within strata of𝐂\\mathbf\{C\}, which brings two statistical challenges\(Simpson,[1951](https://arxiv.org/html/2606.03029#bib.bib29); Blyth,[1972](https://arxiv.org/html/2606.03029#bib.bib37); Gail and Simon,[1985](https://arxiv.org/html/2606.03029#bib.bib38)\): \(1\)stratum imbalance: the relevant covariate stratum may be rare in the corpora, the targeted difference is dominated by irrelevant differences in other strata\. \(2\)sign reversal: the direction of the group difference changes across strata, so the global difference can cancel or misrepresent the targeted differences\. A classic example is Simpson’s paradox\(Simpson,[1951](https://arxiv.org/html/2606.03029#bib.bib29)\)\. The targeted hypothesis can only be reliably generated with within\-stratum comparison\. We adapt two well\-studied econometric tools tosaefeature selection: interaction models and group fixed effects \(implemented via within\-stratum demeaning\)\. Both subsume the global case: features that are discriminative across all strata remain selected alongside stratum\-specific ones\.

##### Setup\.

LetZ∈ℝN×MZ\\in\\mathbb\{R\}^\{N\\times M\}denotesaeactivation matrix, wherezi,mz\_\{i,m\}is the activation of featuremmfor documentiiandMMis the number ofsaefeatures\. Let𝐂=\[C1,…,CP\]∈ℝN×P\\mathbf\{C\}=\[C\_\{1\},\\ldots,C\_\{P\}\]\\in\\mathbb\{R\}^\{N\\times P\}denote the covariate matrix, whereCpC\_\{p\}is theppth covariate column andPPis the number of covariates\. For simplicity, we focus on binary corpus comparison \(y∈\{0,1\}Ny\\in\\\{0,1\\\}^\{N\}, encoding corpusAAorBB\)\. The baselinelassofits L1\-regularized logistic regression:

β^=arg⁡minβ⁡ℓ​\(y,Z​β\)\+λ​‖β‖1,\\hat\{\\beta\}=\\arg\\min\_\{\\beta\}\\ell\(y,Z\\beta\)\+\\lambda\\\|\\beta\\\|\_\{1\},\(1\)and selects the top\|ℋ\|\|\\mathcal\{H\}\|features ranked by\|β^m\|\|\\hat\{\\beta\}\_\{m\}\|, capturing only globally discriminative features\.

### 3\.1Interaction\-lasso

A standard approach for modeling covariate\-specific effects in regression is to include*interaction terms*\(products of two variables\)\(Angrist and Pischke,[2009](https://arxiv.org/html/2606.03029#bib.bib36)\)\. When ansaefeature is interacted with a covariate, the feature’s coefficient can vary with that covariate, capturing within\-stratum differences\. We apply this idea by augmenting thesaeactivation matrix with the covariates𝐂\\mathbf\{C\}and feature–covariate interaction blocksZ⊙CpZ\\odot C\_\{p\}for each covariatep∈\[P\]p\\in\[P\], whereZ⊙CpZ\\odot C\_\{p\}denotes row\-wise multiplication of allMMsaefeatures by covariateCpC\_\{p\}\. Letη​\(β,δ,γ\)=Z​β\+𝐂​δ\+∑p=1P\(Z⊙Cp\)​γp\\eta\(\\beta,\\delta,\\gamma\)=Z\\beta\+\\mathbf\{C\}\\delta\+\\sum\_\{p=1\}^\{P\}\(Z\\odot C\_\{p\}\)\\gamma\_\{p\}\. We fitlassoon the full augmented space:

β^,δ^,γ^=arg⁡minβ,δ,γ⁡ℓ​\(y,η​\(β,δ,γ\)\)\+λ​‖\(β,δ,γ\)‖1\.\\hat\{\\beta\},\\hat\{\\delta\},\\hat\{\\gamma\}=\\arg\\min\_\{\\beta,\\delta,\\gamma\}\\ell\\\!\\left\(y,\\eta\(\\beta,\\delta,\\gamma\)\\right\)\+\\lambda\\\|\(\\beta,\\delta,\\gamma\)\\\|\_\{1\}\.\(2\)Hereβ\\betaaresaefeature main effects,δ\\deltaare covariate main effects, andγp\\gamma\_\{p\}are interaction effects for covariatepp\. We rank features bymax⁡\(\|β^m\|,maxp∈\[P\]⁡\|γ^p,m\|\)\\max\(\|\\hat\{\\beta\}\_\{m\}\|,\\max\_\{p\\in\[P\]\}\|\\hat\{\\gamma\}\_\{p,m\}\|\), and select the top\|ℋ\|\|\\mathcal\{H\}\|features\. A feature qualifies if it is discriminative globally \(large\|β^m\|\|\\hat\{\\beta\}\_\{m\}\|\) or through any covariate\-specific interaction \(large\|γ^p,m\|\|\\hat\{\\gamma\}\_\{p,m\}\|for somepp\), directly implementing the stated goal\. The covariate main effectsδ\\deltaare included as nuisance controls and are not ranked as potential hypotheses\.

However, interaction\-lasso faces two practical limitations\. First, the feature space substantially expands by includingM×PM\{\\times\}Pinteraction features, which increases computational cost and can make feature selection less stable in high dimensions\(Bienet al\.,[2013](https://arxiv.org/html/2606.03029#bib.bib8)\)\. Second,saefeatures are inherently sparse, so each interaction termZ⊙CpZ\{\\odot\}C\_\{p\}is doubly sparse: nonzero only for samples withCi,p≠0C\_\{i,p\}\{\\neq\}0and an active feature\. When a covariate is rare, these terms become nearly all\-zero, making theγ^p,m\\hat\{\\gamma\}\_\{p,m\}estimates noisy\(Crumpet al\.,[2009](https://arxiv.org/html/2606.03029#bib.bib7)\)\. We validate this empirically in §[4\.4](https://arxiv.org/html/2606.03029#S4.SS4)\.

### 3\.2Demeaned\-Reweighted\-Lasso

To overcome the practical limitations, we adapt another standard econometrics technique called demeaning, which subtracts group means from observations to remove between\-group variation and isolate within\-group differences\(Angrist and Pischke,[2009](https://arxiv.org/html/2606.03029#bib.bib36)\)\. By the Frisch–Waugh–Lovell theorem\(Lovell,[1963](https://arxiv.org/html/2606.03029#bib.bib34)\), this is analogous to including group fixed effects in the regression\. Specifically, we group documents by their observed covariate values, residualize bothsaeactivationsZZandyywithin each group, then runlassoon the residuals\. Unlike interaction\-lasso, demeaning operates on the originalMMfeatures without variable expansion, and estimates each feature’s within\-stratum difference from allNNsamples rather than sparse interaction terms, avoiding both limitations above\.

Formally, letgig\_\{i\}denote the covariate stratum of documentii, and letz¯m\(gi\)\\bar\{z\}\_\{m\}^\{\(g\_\{i\}\)\}andy¯\(gi\)\\bar\{y\}^\{\(g\_\{i\}\)\}be the mean feature activation and mean label within that stratum\. We computez~i,m=zi,m−z¯m\(gi\)\\tilde\{z\}\_\{i,m\}=z\_\{i,m\}\-\\bar\{z\}\_\{m\}^\{\(g\_\{i\}\)\}andy~i=yi−y¯\(gi\)\\tilde\{y\}\_\{i\}=y\_\{i\}\-\\bar\{y\}^\{\(g\_\{i\}\)\}\. Thedemeaned\-lassothen selects features by runninglassoon\(Z~,y~\)\(\\tilde\{Z\},\\tilde\{y\}\)\. Theβ~m\\tilde\{\\beta\}\_\{m\}estimates capture the within\-stratum difference associated with featuremm, partialling out per\-stratum mean shifts in both the outcome and the feature activations\.

Additionally, to address stratum imbalance, we proposedemeaned\-reweighted\-lasso: set sample weightswiw\_\{i\}proportional to the inverse frequency of stratumgig\_\{i\}and select features by the weighted lasso on\(Z~,y~,w\)\(\\tilde\{Z\},\\tilde\{y\},w\)\. Reweighting equalizes stratum contributions to feature selection, allowing features from underrepresented target strata to be selected when their within\-stratum differences are strong\.

Demeaning\-based methods assumeno qualitative interaction \(NQI\): the targeted difference has a consistent sign across strata\(Gail and Simon,[1985](https://arxiv.org/html/2606.03029#bib.bib38)\)\. Under NQI, the covariates are treated as nuisance fixed effects rather than variables that change the direction of the text–outcome relationship\. In the sign reversal scenario, NQI is violated and interaction\-lasso should be used instead\. We also includedemeaned\-lassoto isolate the contribution of the reweighting step\.

## 4Experiments on Synthetic Datasets

### 4\.1Synthetic Dataset Construction

We evaluate the methods by recoveringknown ground\-truth hypothesesfrom synthetic datasets\. We choose a dataset containing bill summaries from the 110\-114th U\.S\. Congress\(Hoyleet al\.,[2022](https://arxiv.org/html/2606.03029#bib.bib17)\)\. It has human\-labeled high\-level topics and granular subtopics, and metadata such as bill creation time, allowing us to repurpose it to construct synthetic corpora with*controlled covariate structure*\. We instantiate the two data scenarios demonstrating the two challenges from §[3](https://arxiv.org/html/2606.03029#S3):stratum imbalanceandsign reversal\. Figure[1](https://arxiv.org/html/2606.03029#S4.F1)summarizes their corpus compositions\.

Scenario 1: stratum imbalanceMany global differences exist, but covariates point to𝒯s​o​c\\mathcal\{T\}\_\{soc\}\.r=0\.35r\{=\}0\.35AArrBB1−r1\-rℋ∗\\mathcal\{H\}^\{\*\}: target strata𝒯s​o​c\\mathcal\{T\}\_\{soc\}inAA𝒯s​o​c\\mathcal\{T\}\_\{soc\}inBB𝒯g​o​v\\mathcal\{T\}\_\{gov\}inAA𝒯g​o​v\\mathcal\{T\}\_\{gov\}inBBScenario 2: sign reversalCorpus composition reverses after 2011\.AABBpre\-2011post\-2011𝒯1\\mathcal\{T\}\_\{1\}inAA𝒯1\\mathcal\{T\}\_\{1\}inBB𝒯2\\mathcal\{T\}\_\{2\}inAA𝒯2\\mathcal\{T\}\_\{2\}inBB

Figure 1:Corpus compositions for synthetic datasets\.#### 4\.1\.1Scenario 1: Stratum Imbalance

We investigate whether methods can recover a*targeted subset*of differences when many differences are simultaneously present\. We split the topics into two themes:Government & Economy,𝒯g​o​v\\mathcal\{T\}\_\{gov\}andSocial policy,𝒯s​o​c\\mathcal\{T\}\_\{soc\}\. For each seed run, we randomly select three topics from𝒯g​o​v\\mathcal\{T\}\_\{gov\}and three topics from𝒯s​o​c\\mathcal\{T\}\_\{soc\}\. For each topicTiT\_\{i\}\(e\.g\., “Health”\), we randomly select one subtopic forAAand one subtopic forBB\(e\.g\., “Mental” and “Drug Industry”\)\. The example ground truth is in the form of “is about Health: Mental”\. The targeted difference is in the social policy theme, and we thus evaluate recovery of ground truth under𝒯s​o​c\\mathcal\{T\}\_\{soc\}\.

To simulate the case when relevant strata are rare, an imbalance ratiorrcontrols the frequency of𝒯s​o​c\\mathcal\{T\}\_\{soc\}topics inAA: each𝒯s​o​c\\mathcal\{T\}\_\{soc\}topic contributes at raterrand each𝒯g​o​v\\mathcal\{T\}\_\{gov\}topic at rate1−r1\{\-\}r, withBBreceiving the mirror allocation\. Lowerrrsuppresses the𝒯s​o​c\\mathcal\{T\}\_\{soc\}signal inAAmore, making it harder to recover the targeted difference\. We pass three binary topic covariates to covariate\-aware methods, one indicator per𝒯s​o​c\\mathcal\{T\}\_\{soc\}topic \(more data details in Appendix[A](https://arxiv.org/html/2606.03029#A1)\)\. These covariates should steer the methods toward the relevant strata, but do not reveal which subtopic distinguishesAAfromBBwithin each topic\.

#### 4\.1\.2Scenario 2: Sign Reversal

We then investigate whether methods can recover targeted differences when a covariate masks the aggregate signal\. The covariate is the bill creation time period \(binary: pre/post 2011\)\. For each seed run, we randomly select1010subtopics, each from a distinct topic, and split them into two groups𝒯1\\mathcal\{T\}\_\{1\}and𝒯2\\mathcal\{T\}\_\{2\}of five each\. In the pre\-2011 period,𝒯1\\mathcal\{T\}\_\{1\}subtopics are assigned toAAand𝒯2\\mathcal\{T\}\_\{2\}toBB; in the post\-2011 period, the assignment reverses\. We pass one binary time\-period covariate \(pre\-2011=1\{=\}1\) to covariate\-aware methods, and evaluate recovery of the five𝒯1\\mathcal\{T\}\_\{1\}subtopic differences\. By construction, the difference reverses sign across periods, violating NQI assumption\. We expect interaction\-lasso to be the appropriate method, while demeaning\-based methods are expected to fail\.

Table 1:Stratum Imbalance:\|ℋ\|=3\|\\mathcal\{H\}\|\{=\}3\.†\\dagger/†⁣†\\dagger\\dagger:p<0\.05p\{<\}0\.05/0\.010\.01vs\. lasso, exact paired sign\-flip test\.

### 4\.2Evaluation Metrics

We evaluate how well the methods recover the targeted ground\-truth differences on the held\-out set\. For each scenario, we use anllmto annotate generated hypotheses on held\-out texts and match them to reference hypotheses using the Hungarian algorithm on reference\-vs\.\-generated annotation correlations\. See Appendix[B](https://arxiv.org/html/2606.03029#A2)for detailed process\. We report two metrics computed over matched pairs\.Surface similarity: for each matched reference/inferred hypothesis pair, we promptgpt\-4\.1to assess whether the two hypotheses are the*same*,*related*, or*distinct*, with corresponding scores of 1\.0, 0\.5, and 0\.0 \(prompt in Table[10](https://arxiv.org/html/2606.03029#A4.T10), examples in Table[9](https://arxiv.org/html/2606.03029#A3.T9)\)\. To improve stability, we sample five outputs at temperature0\.70\.7and average them, then report the mean score across\|ℋ\|\|\\mathcal\{H\}\|pairs\.F1 similarity: for each matched pair, we compute the F1\-score between the reference and inferred hypothesistext\-levelannotations on the held\-out set\.

### 4\.3Baselines

Since no existing method handles covariates, we compare against SAE\-based methods\(Movvaet al\.,[2025](https://arxiv.org/html/2606.03029#bib.bib32)\)as the primary baselines\. For fair comparison, we usegpt\-4\.1for proposing hypotheses if not specified otherwise, and OpenAI’s text\-embedding\-3\-small for embeddings\.

sae\-based\.\(1\)lasso: use Eq\.[1](https://arxiv.org/html/2606.03029#S3.E1)to select the top\|ℋ\|\|\\mathcal\{H\}\|features\. \(2\)separation score: select features by the mean ofyyamong the highest\-activating texts vs\. zero\-activating texts\.111https://github\.com/rmovva/HypotheSAEs

llm\-direct\.We include direct prompting baselines to test whether covariate\-aware discovery can be obtained simply by showing sampled texts to anllm, without a corpus\-level feature\-selection step\.llm\-globalis prompted with sampled corpus texts, whilellm\-covariateis also shown covariate\-profile labels for each text\. Because these baselines only observe the prompt sample, we use a generous sample budget in the main tables\. Appendix[C\.3](https://arxiv.org/html/2606.03029#A3.SS3)gives the sensitivity analysis of the sample size\.

Reference baselines\.To calibrate the evaluation, we include:Oracleas an upper reference that gives thellm2020examples sampled from the exact positive and negative sides of each ground\-truth contrast\.Randomsamples2020training texts into arbitrary positive and negative halves before the same interpretation step, measuring how much recovery can come from generic topical cues alone\. We use2020examples to match the SAE interpretation stage\. More implementation details for these baseline methods are provided in Appendix[C](https://arxiv.org/html/2606.03029#A3)\.

Proposed methods\.We compareinteraction\-lasso\(§[3\.1](https://arxiv.org/html/2606.03029#S3.SS1)\),demeaned\-reweighted\-lasso\(§[3\.2](https://arxiv.org/html/2606.03029#S3.SS2)\), anddemeaned\-lassoas ablation with the baselines\.

Statistical testing\.Each proposed method is compared tolassoon seed\-level surface similarity and F1, with1010seeds per setting\. We test whether the mean paired difference differs from zero with an exact two\-sided sign\-flip test\.

Table 2:Sign reversal:\|ℋ\|=5\|\\mathcal\{H\}\|\{=\}5\.†⁣†\\dagger\\dagger:p<0\.01p\{<\}0\.01vs\. lasso, exact paired sign\-flip test\.
### 4\.4Results and Discussion

##### Demeaned\-reweighted\-lasso steers discovery toward specified covariates and recovers suppressed within\-stratum signals\.

Table[1](https://arxiv.org/html/2606.03029#S4.T1)evaluates conditional discovery as the target social\-policy strata become increasingly underrepresented\. Across all imbalance levels, demeaned\-reweighted\-lasso remains near the oracle in surface similarity and significantly improves F1 overlasso, the strongest globalsaebaseline\. Even atr=0\.50r\{=\}0\.50, it reaches\.700\.700surface similarity, near the oracle’s\.750\.750and abovelasso’s\.557\.557\(p=0\.016p\{=\}0\.016\), showing that covariates steer feature selection toward the target𝒯s​o​c\\mathcal\{T\}\_\{soc\}differences\. The gap widens under stronger imbalance:lassofalls to\.383\.383–\.393\.393surface similarity, while demeaned\-reweighted\-lasso remains at\.730\.730\. The demeaned\-lasso ablation improves overlassounder imbalance but remains below the reweighted variant, indicating that residualizing within strata helps but reweighting further helps rare target strata from being suppressed\.

The covariate structure determines the appropriate selection model\.Table[2](https://arxiv.org/html/2606.03029#S4.T2)evaluates sign reversal, where the time\-period covariate reverses the differences across strata and makes the aggregate signal misleading\. As §[3](https://arxiv.org/html/2606.03029#S3)notes, residualization is appropriate when the covariate induces fixed stratum\-level shifts, but not when the difference itself changes sign\. As expected, neither demeaned variant significantly improves overlasso\.

Interaction\-lasso accounts for sign reversal\.Interaction\-lasso is the only covariate\-aware method that significantly improves overlassoin Table[2](https://arxiv.org/html/2606.03029#S4.T2), increasing surface similarity from\.484\.484to\.600\.600\(p=0\.008p\{=\}0\.008\) and F1 from\.362\.362to\.576\.576\(p=0\.002p\{=\}0\.002\)\. This suggests modeling feature\-by\-time interactions allows a feature to be selected when it is discriminative within a period even if its global effect cancels\.

Interaction\-lasso faces computational and sparsity limitations\.Despite this advantage in the sign\-reversal setting, interaction\-lasso does not improve over global SAE baselines in Table[1](https://arxiv.org/html/2606.03029#S4.T1), despite being the more general model\. This empirically validates our discussion in §[3\.1](https://arxiv.org/html/2606.03029#S3.SS1)that interaction\-lasso faces feature\-space expansion and sparsity limitations\. These limitations may also help explain why interaction\-lasso is best in Table[2](https://arxiv.org/html/2606.03029#S4.T2)but still falls short of the oracle\.

Statistical modeling of covariates, not prompt\-level exposure, drives gains\.Both tables show thatllm\-direct methods fall behindsae\-based methods\. The sensitivity analysis in Appendix[C\.3](https://arxiv.org/html/2606.03029#A3.SS3)further shows that llm\-covariate does not consistently outperform llm\-global across sample sizes or with a stronger reasoning model \(gpt\-5\.4\)\.

## 5Validation on Real\-World Datasets

The synthetic experiments above test recovery under known conditional structure\. For real\-world corpora, where no ground\-truth hypotheses exist, we evaluate whether conditional generation surfaces hypotheses that domain experts find useful for comparing outcome groups within researcher\-specified covariate strata\. Because our goal is to generate hypotheses discriminative within strata of𝐂\\mathbf\{C\}while retaining globally discriminative findings, we focus on hypotheses unique to the covariate\-aware method relative to the global baseline\.

### 5\.1Real\-World Datasets

We include two real\-world datasets and instantiate𝐂\\mathbf\{C\}with one researcher\-specified binary covariate for each, chosen from domain knowledge: \(1\)congress\(Gentzkow and Shapiro,[2010](https://arxiv.org/html/2606.03029#bib.bib15)\): U\.S\. speeches from the 109th Congress \(2005\-07\), a longstanding application in computational social science\(Grimmeret al\.,[2021](https://arxiv.org/html/2606.03029#bib.bib14)\)\. The outcome variableyyis party affiliation \(Republican vs\. Democrat\)\. The covariate indicates whether a speech contains substantive public\-policy discussion rather than procedural talk \(e\.g\., scheduling and unanimous\-consent requests\)\. Congressional speeches mix these forms of activities, while procedural talk does not reflect the substantive policy disagreements that political scientists typically seek to measure\(Gentzkowet al\.,[2019](https://arxiv.org/html/2606.03029#bib.bib21)\)\. Conditioning on this covariate allows us to guide hypothesis generation to reflect policy discussion beyond procedural activity\. \(2\)ncte\(Demszky and Hill,[2023](https://arxiv.org/html/2606.03029#bib.bib13)\): the largest publicly available dataset of math classroom transcripts with rich expert annotations of instructional qualityHillet al\.\([2008](https://arxiv.org/html/2606.03029#bib.bib9)\)and classroom environmentPiantaet al\.\([2012](https://arxiv.org/html/2606.03029#bib.bib12)\), collected by the National Center for Teacher Effectiveness \(ncte\) between 2010\-2013\. The outcome variableyyis low vs\. high qualityremed, a measure of how well teachers remediate student math errors and difficulties in class\. This teaching practice is central to math instruction as it provides students with opportunities for conceptual learningBray \([2011](https://arxiv.org/html/2606.03029#bib.bib11)\)\. We use behavioral management \(e\.g\., organizing activities and redirecting students\) as the covariate because management and instruction are intertwined: teachers often need to minimize time spent on behavioral management to focus on content instructionEmmer and Stough \([2003](https://arxiv.org/html/2606.03029#bib.bib10)\)\. Conditioning on this covariate allows us to examine whether the model surfaces remediation\-oriented teaching practices beyond behavioral management\.

### 5\.2Study Setup

We expect the two covariates to steer hypothesis discovery toward within\-stratum differences, analogous to the stratum\-imbalance setting in §[4\.1\.1](https://arxiv.org/html/2606.03029#S4.SS1.SSS1)\. Prior work suggests that sign reversal is rare in practice\(Higginset al\.,[2024](https://arxiv.org/html/2606.03029#bib.bib39)\), so we compare the globallassobaseline with demeaned\-reweighted\-lasso as the covariate\-aware method\. Our main metric is the*helpfulness*of each hypothesis, on a 1–5 scale, for understanding the group difference while accounting for the specified covariate\. Each hypothesis is accompanied by its prevalence in the two outcome groups\. After rating helpfulness, annotators reviewed the same hypotheses with global and covariate\-stratified statistics and rated*conditional interpretive value*on a 1–5 scale: whether the breakdown by the specified covariate added interpretive value beyond the global finding\. We set\|ℋ\|=10\|\\mathcal\{H\}\|\{=\}10to limit the cognitive burden on annotators\.

We recruit research scholars for each domain\. Specifically, we invited two computational social science scholars familiar with U\.S\. politics to annotatecongress\. And we invited two educational researchers who have usednctein prior research to annotate thenctedataset\. More details on annotation instruction, expert recruitment, and covariate operationalization are in Appendix[D](https://arxiv.org/html/2606.03029#A4)\.

### 5\.3Results and Discussion

To focus on complementary discoveries, we manually match semantically similar hypotheses across the two method outputs\. Matched hypotheses capture patterns recovered by both methods and are therefore less diagnostic of the added value of covariate\-aware selection\. We report the full lists and matching decisions for both datasets in Appendix[D\.1](https://arxiv.org/html/2606.03029#A4.SS1)\. Differences among matched items may also reflect LLM interpretation noise, such as wording specificity\. Table[3](https://arxiv.org/html/2606.03029#S5.T3)lists the uniquenctehypotheses by method as qualitative examples; aggregate expert ratings appear separately in Table[4](https://arxiv.org/html/2606.03029#S5.T4)\.

Covariate\-aware selection surfaces useful within\-stratum hypotheses\.Table[3](https://arxiv.org/html/2606.03029#S5.T3)illustrates this shift inncte: unique covariate\-aware hypotheses move from classroom\-management context toward remediation or other instructional activities\. The globallassobaseline surfaced two unique hypotheses about classroom management, the covariate\-defined context we aim to account for\. By contrast, the covariate\-aware method surfaced more instruction\-oriented hypotheses about units\-of\-weight discussion, individualized follow\-up, and peer explanation\. The small\-group hypothesis is especially illustrative: rather than only identifying group organization, it emphasizes teachers prompting students to help and explain concepts to one another\. The same pattern appears more modestly incongress: among the unique hypotheses, the global baseline retains a procedural hypothesis, while the covariate\-aware method surfaces policy\-related hypotheses about economic performance and border security\. Together, these examples suggest that covariate\-aware selection can surface unique hypotheses aligned with the researcher\-specified strata\. The averaged ratings in Table[4](https://arxiv.org/html/2606.03029#S5.T4)further confirm that these hypotheses are more useful than their counterparts \(3\.103\.10vs\.2\.502\.50\)\.

Table 3:NCTE hypotheses unique to each method after semantic matching\. Hypotheses are shortened for readability\. Full wording appears in Appendix[D\.1](https://arxiv.org/html/2606.03029#A4.SS1)\.Table 4:Expert ratings for hypotheses unique to each method after semantic matching\. Helpfulness and conditional interpretive value \(Cond\.\) are 1–5 Likert means\.##### The value of the covariate depends on the dataset\.

Conditional interpretive value measures whether the𝐂\\mathbf\{C\}\-stratified statistics change how annotators interpret a generated hypothesis\. Inncte, behavioral management helps distinguish classroom organization from instructional activities, and unique covariate\-aware hypotheses received higher conditional\-value ratings than uniquelassohypotheses \(3\.50 vs\. 2\.33\)\. Incongress, the policy/procedure covariate was less informative as a displayed breakdown: procedural talk remained strong in both method outputs, and the two unique covariate\-aware hypotheses were already clearly policy\-related from their text\. As one annotator noted, the policy/procedure breakdown often confirmed what was already expected rather than changing the interpretation\. Accordingly, unique covariate\-aware hypotheses incongresswere rated as more helpful \(3\.25 vs\. 2\.75\) but lower in conditional interpretive value \(1\.50 vs\. 2\.50\)\.

## 6Related Work

##### LLM\-based hypothesis discovery\.

Complementing the task overview in §[2\.2](https://arxiv.org/html/2606.03029#S2.SS2), we situatellm\-based hypothesis generation among classical text\-analysis tools and applications\. Classical text analysis methods such as n\-gram frequency comparisons\(Gentzkowet al\.,[2019](https://arxiv.org/html/2606.03029#bib.bib21)\)and topic models, including LDA\(Bleiet al\.,[2003](https://arxiv.org/html/2606.03029#bib.bib20)\)and structural topic models\(Robertset al\.,[2014](https://arxiv.org/html/2606.03029#bib.bib25)\), remain popular tools for relating text to target variables\(Grimmeret al\.,[2022](https://arxiv.org/html/2606.03029#bib.bib26)\), but typically produce word lists that require further interpretation\. Recentllm\-based methods instead generate natural\-language descriptions of corpus differences from sampled examples\(Zhonget al\.,[2022](https://arxiv.org/html/2606.03029#bib.bib30),[2023](https://arxiv.org/html/2606.03029#bib.bib31)\), iterative error analysis\(Zhouet al\.,[2024](https://arxiv.org/html/2606.03029#bib.bib24)\), literature\-grounded candidates\(Liuet al\.,[2025](https://arxiv.org/html/2606.03029#bib.bib23)\), or statistical modeling\(Zhonget al\.,[2024](https://arxiv.org/html/2606.03029#bib.bib33); Movvaet al\.,[2025](https://arxiv.org/html/2606.03029#bib.bib32); Agarwalet al\.,[2026](https://arxiv.org/html/2606.03029#bib.bib1)\)\. Related work has applied this idea to settings such as headline click rates\(Batista and Ross,[2024](https://arxiv.org/html/2606.03029#bib.bib19)\), social media engagement\(Xuet al\.,[2026](https://arxiv.org/html/2606.03029#bib.bib5)\), and causal measurement from text\(Modarressiet al\.,[2025](https://arxiv.org/html/2606.03029#bib.bib18)\)\. OpenAI recently released a text measurement toolkit that also supports hypothesis discovery\(Asirvathamet al\.,[2026](https://arxiv.org/html/2606.03029#bib.bib40)\), demonstrating its potential for real\-world applications\. Our work differs by incorporating domain knowledge through statistical models of covariate structure, steering discovery toward researcher\-specified differences of interest\.

##### Statistical methods for covariate adjustment\.

Our methods draw on standard statistical and econometric tools for making comparisons conditional on covariates\. Fixed effects and within\-group residualization are classical ways to remove nuisance variation\(Angrist and Pischke,[2009](https://arxiv.org/html/2606.03029#bib.bib36)\), with the Frisch–Waugh–Lovell theorem providing the corresponding partialling\-out justification\(Lovell,[1963](https://arxiv.org/html/2606.03029#bib.bib34)\)\. Simpson’s paradox\(Simpson,[1951](https://arxiv.org/html/2606.03029#bib.bib29)\)shows why such adjustment matters: marginal associations can reverse relative to associations within each stratum\(Blyth,[1972](https://arxiv.org/html/2606.03029#bib.bib37)\)\. When conditional effects change sign across strata, the problem becomes one of qualitative interaction\(Gail and Simon,[1985](https://arxiv.org/html/2606.03029#bib.bib38)\)\. These results provide the statistical basis for using residualization, reweighting, and interaction terms as principled feature\-selection mechanisms for conditional hypothesis generation\.

## 7Conclusion

We introduced conditional hypothesis generation for text analysis, a framework for steering natural\-language hypothesis discovery toward differences that hold within researcher\-specified covariate strata\. This setting makes explicit two statistical challenges:*stratum imbalance*, where signals from underrepresented strata are suppressed, and*sign reversal*, where aggregation can cancel or misrepresent within\-stratum differences\. By incorporating covariates into thesaefeature selection, our methods let researchers encode domain knowledge about which strata should shape discovery\.

We proposed two complementary covariate\-aware methods\. Interaction\-lasso models feature–covariate interactions and is suited to sign reversal, while demeaned\-reweighted\-lasso removes stratum\-level variation and reweights underrepresented strata to recover within\-stratum signals under a consistent\-direction assumption\. Synthetic experiments show that these modeling choices matter: demeaned\-reweighted\-lasso recovers hypotheses suppressed by stratum imbalance, whereas interaction\-lasso is needed when the direction of the difference reverses\. Expert validation on two real\-world datasets,congressandncte, further suggests that covariate\-aware method can surface more useful hypotheses for researchers’ stated comparisons than those unique to a global baseline\.

## Limitations

##### Dependence on researcher\-specified covariates\.

Our framework intentionally makes covariate specification a design choice: researchers decide which contexts should shape hypothesis discovery based on domain knowledge and the comparison they want to make\. This choice gives researchers control over the target of discovery, but it also means the methods are only as useful as the covariates supplied\. They can reduce the influence of known covariate structure, but they cannot discover which covariates should matter, correct for omitted covariates, or make the resulting hypotheses causal claims\. Poorly chosen or noisy covariates may steer discovery toward unhelpful strata, and overly fine\-grained covariates may create sparse cells where within\-stratum comparisons are unreliable\.

##### Categorical strata\.

Both proposed methods are designed around observed covariate strata\. In this paper we focus on binary or categorical covariates, such as policy/procedure status, behavior\-management level, topic indicators, and time period\. Continuous covariates would require discretization or a different residualization strategy, The quality of the conditional hypotheses therefore depends on the validity of the covariate operationalization, including any automated annotations used to define covariates\.

##### Method\-specific assumptions\.

The two methods address different statistical regimes\. Demeaned\-reweighted\-lasso assumes no qualitative interaction: the relevant group difference should have a consistent direction across covariate strata\. When the direction reverses, as in our sign\-reversal experiment, residualization can remove nuisance variation but cannot recover the stratum\-specific effects of interest\. Interaction\-lasso is better suited to such cases, but it expands the feature space and can be unstable when bothsaeactivations and covariate strata are sparse\. In practice, researchers should inspect stratum\-level statistics when possible and choose the selection model based on whether sign reversal is plausible\.

##### Dependence on thesaeinterpretation pipeline\.

Our implementation builds on thesae\-based hypothesis\-generation pipeline ofMovvaet al\.\([2025](https://arxiv.org/html/2606.03029#bib.bib32)\)\. It therefore inherits the pipeline’s requirements and failure modesPenget al\.\([2025](https://arxiv.org/html/2606.03029#bib.bib2)\): access to the full corpus for feature extraction, a trainedsaewhose features are sufficiently interpretable for the domain, hyperparameter choices that affect the granularity of learned features, and anllminterpretation step that can phrase selected features imperfectly\. Our contribution modifies feature selection, not the underlying representation learning or natural\-language verbalization stages\.

##### Evaluation of discovery quality\.

The synthetic experiments provide controlled ground truth for stratum imbalance and sign reversal, but their reference hypotheses are constructed from bill\-summary topics and may not cover the full range of linguistic phenomena or covariate choices encountered in applied text analysis\. The real\-world validation complements these experiments with expert judgments in two substantively different domains,congressandncte, while broader generalization will require additional datasets, covariates, and expert panels\. Such validation is difficult because scientific discovery has no single objective metric: what counts as a useful hypothesis depends on domain expertise, the research question, and the implicit norms of the relevant research community rather than explicit deductive criteria alone\(Polanyiet al\.,[2000](https://arxiv.org/html/2606.03029#bib.bib6)\)\. The ratings should therefore be interpreted as evidence that covariate\-aware selection can surface useful hypotheses, not as a complete benchmark of performance across domains\. Future evaluations should test more covariates, multi\-valued and continuous settings, larger expert panels, and downstream research workflows where generated hypotheses are refined and validated by domain specialists\.

## References

- E\. S\. Adler and J\. Wilkerson \(2018\)Congressional bills project: 1995\-2018\.NSF\.Cited by:[Appendix A](https://arxiv.org/html/2606.03029#A1.p1.17)\.
- D\. Agarwal, B\. P\. Majumder, R\. Adamson, M\. Chakravorty, S\. R\. Gavireddy, A\. Parashar, H\. Surana, B\. Dalvi Mishra, A\. McCallum, A\. Sabharwal,et al\.\(2026\)Autodiscovery: open\-ended scientific discovery via bayesian surprise\.Advances in Neural Information Processing Systems38,pp\. 25181–25219\.Cited by:[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- J\. D\. Angrist and J\. Pischke \(2009\)Mostly harmless econometrics: an empiricist’s companion\.Princeton university press\.Cited by:[§3\.1](https://arxiv.org/html/2606.03029#S3.SS1.p1.7),[§3\.2](https://arxiv.org/html/2606.03029#S3.SS2.p1.4),[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px2.p1.1)\.
- H\. Asirvatham, E\. Mokski, and A\. Shleifer \(2026\)GPT as a measurement tool\.Working PaperTechnical Report34834,National Bureau of Economic Research\.External Links:[Document](https://dx.doi.org/10.3386/w34834)Cited by:[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- R\. M\. Batista and J\. Ross \(2024\)Words that work: using language to generate hypotheses\.Available at SSRN 4926398\.Cited by:[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- J\. Bien, J\. Taylor, and R\. Tibshirani \(2013\)A lasso for hierarchical interactions\.Annals of statistics41\(3\),pp\. 1111\.Cited by:[§3\.1](https://arxiv.org/html/2606.03029#S3.SS1.p2.4)\.
- D\. M\. Blei, A\. Y\. Ng, and M\. I\. Jordan \(2003\)Latent dirichlet allocation\.Journal of machine Learning research3\(Jan\),pp\. 993–1022\.Cited by:[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- C\. R\. Blyth \(1972\)On simpson’s paradox and the sure\-thing principle\.Journal of the American Statistical Association67\(338\),pp\. 364–366\.Cited by:[§3](https://arxiv.org/html/2606.03029#S3.p1.1),[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px2.p1.1)\.
- W\. S\. Bray \(2011\)A collective case study of the influence of teachers’ beliefs and knowledge on error\-handling practices during class discussion of mathematics\.Journal for Research in Mathematics education42\(1\),pp\. 2–38\.Cited by:[§5\.1](https://arxiv.org/html/2606.03029#S5.SS1.p1.3)\.
- D\. Card \(2019\)Accelerating text\-as\-data research in computational social science\.Ph\.D\. Thesis,University of Washington\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p1.1)\.
- R\. K\. Crump, V\. J\. Hotz, G\. W\. Imbens, and O\. A\. Mitnik \(2009\)Dealing with limited overlap in estimation of average treatment effects\.Biometrika96\(1\),pp\. 187–199\.Cited by:[§3\.1](https://arxiv.org/html/2606.03029#S3.SS1.p2.4)\.
- D\. Demszky and H\. Hill \(2023\)The ncte transcripts: a dataset of elementary math classroom transcripts\.InProceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2023\),pp\. 528–538\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p8.1),[§5\.1](https://arxiv.org/html/2606.03029#S5.SS1.p1.3)\.
- E\. T\. Emmer and L\. M\. Stough \(2003\)Classroom management: a critical part of educational psychology, with implications for teacher education\.InEducational psychology,pp\. 103–112\.Cited by:[§5\.1](https://arxiv.org/html/2606.03029#S5.SS1.p1.3)\.
- M\. Gail and R\. Simon \(1985\)Testing for qualitative interactions between treatment effects and patient subsets\.Biometrics,pp\. 361–372\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p5.1),[§3\.2](https://arxiv.org/html/2606.03029#S3.SS2.p4.1),[§3](https://arxiv.org/html/2606.03029#S3.p1.1),[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px2.p1.1)\.
- M\. Gentzkow, J\. M\. Shapiro, and M\. Taddy \(2019\)Measuring group differences in high\-dimensional choices: method and application to congressional speech\.Econometrica87\(4\),pp\. 1307–1340\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p3.1),[§5\.1](https://arxiv.org/html/2606.03029#S5.SS1.p1.3),[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- M\. Gentzkow and J\. M\. Shapiro \(2010\)What drives media slant? evidence from us daily newspapers\.Econometrica78\(1\),pp\. 35–71\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p8.1),[§5\.1](https://arxiv.org/html/2606.03029#S5.SS1.p1.3)\.
- J\. Grimmer, M\. E\. Roberts, and B\. M\. Stewart \(2021\)Machine learning for social science: an agnostic approach\.Annual Review of Political Science24\(1\),pp\. 395–419\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p8.1),[§5\.1](https://arxiv.org/html/2606.03029#S5.SS1.p1.3)\.
- J\. Grimmer, M\. E\. Roberts, and B\. M\. Stewart \(2022\)Text as data: a new framework for machine learning and the social sciences\.Princeton University Press\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p1.1),[§1](https://arxiv.org/html/2606.03029#S1.p3.1),[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- J\. Grimmer and B\. M\. Stewart \(2013\)Text as data: the promise and pitfalls of automatic content analysis methods for political texts\.Political analysis21\(3\),pp\. 267–297\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p1.1),[§1](https://arxiv.org/html/2606.03029#S1.p3.1)\.
- J\. P\. T\. Higgins, J\. Thomas, J\. Chandler, M\. Cumpston, T\. Li, M\. J\. Page, and V\. A\. Welch \(2024\)Cochrane handbook for systematic reviews of interventions\.Version 6\.5 edition,Cochrane\.External Links:[Link](https://training.cochrane.org/handbook/current)Cited by:[§5\.2](https://arxiv.org/html/2606.03029#S5.SS2.p1.1)\.
- H\. C\. Hill, M\. L\. Blunk, C\. Y\. Charalambous, J\. M\. Lewis, G\. C\. Phelps, L\. Sleep, and D\. L\. Ball \(2008\)Mathematical knowledge for teaching and the mathematical quality of instruction: an exploratory study\.Cognition and instruction26\(4\),pp\. 430–511\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p8.1),[§5\.1](https://arxiv.org/html/2606.03029#S5.SS1.p1.3)\.
- A\. Hoyle, P\. Goel, R\. Sarkar, and P\. Resnik \(2022\)Are neural topic models broken?\.InFindings of the Association for Computational Linguistics: EMNLP 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 5321–5344\.External Links:[Link](https://aclanthology.org/2022.findings-emnlp.390/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-emnlp.390)Cited by:[Appendix A](https://arxiv.org/html/2606.03029#A1.p1.17),[§4\.1](https://arxiv.org/html/2606.03029#S4.SS1.p1.1)\.
- R\. Huben, H\. Cunningham, L\. Smith, A\. Ewart, and L\. Sharkey \(2024\)Sparse autoencoders find highly interpretable features in language models\.pp\. 7827–7845\.Cited by:[§2\.3](https://arxiv.org/html/2606.03029#S2.SS3.p1.3)\.
- T\. J\. Kane and D\. O\. Staiger \(2012\)Gathering feedback for teaching: combining high\-quality observations with student surveys and achievement gains\. research paper\. met project\.\.Bill & Melinda Gates Foundation\.Cited by:[§D\.2](https://arxiv.org/html/2606.03029#A4.SS2.p1.3)\.
- H\. Liu, Y\. Zhou, M\. Li, C\. Yuan, and C\. Tan \(2025\)Literature meets data: a synergistic approach to hypothesis generation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 245–281\.External Links:[Link](https://aclanthology.org/2025.acl-long.12/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.12),ISBN 979\-8\-89176\-251\-0Cited by:[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- M\. C\. Lovell \(1963\)Seasonal adjustment of economic time series and multiple regression analysis\.Journal of the American Statistical Association58\(304\),pp\. 993–1010\.Cited by:[§3\.2](https://arxiv.org/html/2606.03029#S3.SS2.p1.4),[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px2.p1.1)\.
- I\. Modarressi, J\. Spiess, and A\. Venugopal \(2025\)Causal inference on outcomes learned from text\.arXiv preprint arXiv:2503\.00725\.Cited by:[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- R\. Movva, K\. Peng, N\. Garg, J\. Kleinberg, and E\. Pierson \(2025\)Sparse autoencoders for hypothesis generation\.pp\. 44997–45023\.Cited by:[Appendix B](https://arxiv.org/html/2606.03029#A2.p1.1),[Appendix B](https://arxiv.org/html/2606.03029#A2.p2.1),[§C\.1](https://arxiv.org/html/2606.03029#A3.SS1.p1.3),[§C\.4](https://arxiv.org/html/2606.03029#A3.SS4.p1.9),[§C\.5](https://arxiv.org/html/2606.03029#A3.SS5.p1.23),[§D\.2](https://arxiv.org/html/2606.03029#A4.SS2.p1.3),[Table 10](https://arxiv.org/html/2606.03029#A4.T10),[§1](https://arxiv.org/html/2606.03029#S1.p2.1),[§1](https://arxiv.org/html/2606.03029#S1.p6.1),[§2\.2](https://arxiv.org/html/2606.03029#S2.SS2.p2.3),[§2\.3](https://arxiv.org/html/2606.03029#S2.SS3.p1.3),[§4\.3](https://arxiv.org/html/2606.03029#S4.SS3.p1.1),[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1),[Dependence on thesaeinterpretation pipeline\.](https://arxiv.org/html/2606.03029#Sx1.SS0.SSS0.Px4.p1.1)\.
- K\. Peng, R\. Movva, J\. Kleinberg, E\. Pierson, and N\. Garg \(2025\)Use sparse autoencoders to discover unknown concepts, not to act on known concepts\.arXiv preprint arXiv:2506\.23845\.Cited by:[Dependence on thesaeinterpretation pipeline\.](https://arxiv.org/html/2606.03029#Sx1.SS0.SSS0.Px4.p1.1)\.
- R\. C\. Pianta, B\. K\. Hamre, and S\. Mintz \(2012\)Classroom assessment scoring system: class: upper elementary manual\.Teachstone\.Cited by:[§D\.2](https://arxiv.org/html/2606.03029#A4.SS2.p1.3),[§1](https://arxiv.org/html/2606.03029#S1.p8.1),[§5\.1](https://arxiv.org/html/2606.03029#S5.SS1.p1.3)\.
- M\. Polanyi, J\. Ziman, and S\. Fuller \(2000\)The republic of science: its political and economic theory minerva, i \(1\)\(1962\), 54\-73\.Minerva38\(1\),pp\. 1–32\.Cited by:[Evaluation of discovery quality\.](https://arxiv.org/html/2606.03029#Sx1.SS0.SSS0.Px5.p1.1)\.
- M\. E\. Roberts, B\. M\. Stewart, D\. Tingley, C\. Lucas, J\. Leder\-Luis, S\. K\. Gadarian, B\. Albertson, and D\. G\. Rand \(2014\)Structural topic models for open\-ended survey responses\.American journal of political science58\(4\),pp\. 1064–1082\.Cited by:[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- E\. H\. Simpson \(1951\)The interpretation of interaction in contingency tables\.Journal of the Royal Statistical Society: Series B \(Methodological\)13\(2\),pp\. 238–241\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p5.1),[§3](https://arxiv.org/html/2606.03029#S3.p1.1),[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px2.p1.1)\.
- M\. Taddy \(2013\)Multinomial inverse regression for text analysis\.Journal of the American Statistical Association108\(503\),pp\. 755–770\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p3.1)\.
- P\. Xu, J\. Liu, N\. Jones, J\. Cohen, and W\. Ai \(2024\)The promises and pitfalls of using language models to measure instruction quality in education\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 4375–4389\.Cited by:[§D\.2](https://arxiv.org/html/2606.03029#A4.SS2.p1.3)\.
- P\. Xu, L\. Raschid, and V\. Frias\-Martinez \(2026\)Does geo\-co\-location matter? a case study of public health conversations during covid\-19\.InProceedings of the International AAAI Conference on Web and Social Media,Vol\.20,pp\. 2518–2537\.Cited by:[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- R\. Zhong, C\. Snell, D\. Klein, and J\. Steinhardt \(2022\)Describing differences between text distributions with natural language\.InProceedings of the 39th International Conference on Machine LearningInternational Conference on Machine LearningInternational Conference on Learning Representations,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvari, G\. Niu, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research, Vol\.1622024,pp\. 27099–27116\.External Links:[Link](https://proceedings.mlr.press/v162/zhong22a.html)Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p2.1),[§2\.2](https://arxiv.org/html/2606.03029#S2.SS2.p1.1),[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- R\. Zhong, H\. Wang, D\. Klein, and J\. Steinhardt \(2024\)Explaining datasets in words: statistical models with natural language parameters\.Advances in Neural Information Processing Systems37,pp\. 79350–79380\.Cited by:[Appendix B](https://arxiv.org/html/2606.03029#A2.p1.1),[Table 10](https://arxiv.org/html/2606.03029#A4.T10),[§1](https://arxiv.org/html/2606.03029#S1.p2.1),[§2\.2](https://arxiv.org/html/2606.03029#S2.SS2.p2.3),[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- R\. Zhong, P\. Zhang, S\. Li, J\. Ahn, D\. Klein, and J\. Steinhardt \(2023\)Goal driven discovery of distributional differences via language descriptions\.Advances in Neural Information Processing Systems36,pp\. 40204–40237\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p2.1),[§2\.2](https://arxiv.org/html/2606.03029#S2.SS2.p1.1),[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.
- Y\. Zhou, H\. Liu, T\. Srivastava, H\. Mei, and C\. Tan \(2024\)Hypothesis generation with large language models\.InProceedings of the 1st Workshop on NLP for Science \(NLP4Science\),pp\. 117–139\.Cited by:[§1](https://arxiv.org/html/2606.03029#S1.p2.1),[§2\.2](https://arxiv.org/html/2606.03029#S2.SS2.p1.1),[§6](https://arxiv.org/html/2606.03029#S6.SS0.SSS0.Px1.p1.1)\.

## Appendix ASynthetic Dataset Details

Topics are drawn from the BILLS dataset\(Adler and Wilkerson,[2018](https://arxiv.org/html/2606.03029#bib.bib16); Hoyleet al\.,[2022](https://arxiv.org/html/2606.03029#bib.bib17)\), which provides 21 high\-level topics and 114 low\-level subtopics\. For Scenario 1: conditional discovery in Section[4\.1\.1](https://arxiv.org/html/2606.03029#S4.SS1.SSS1), we define two thematic pools:𝒯g​o​v\\mathcal\{T\}\_\{gov\}\(Government & Economy\) and𝒯s​o​c\\mathcal\{T\}\_\{soc\}\(Social Policy\)\. A topic is eligible if it has at least two subtopics each with≥320\\geq 320samples, ensuring sufficient data to support the imbalanced sampling under the largest imbalance ratio tested\. The eligible topics retained for each theme are listed in Table[5](https://arxiv.org/html/2606.03029#A1.T5)\. Each Scenario 1 corpus containsN=2,400N\{=\}2\{,\}400bill summaries, balanced acrossAAandBB\(NA=NB=1,200N\_\{A\}\{=\}N\_\{B\}\{=\}1\{,\}200\)\. For each selected topic,400400examples are allocated across the two corpora\. For example, whenr=0\.20r\{=\}0\.20, corpusAAcontains3×400×0\.20=2403\{\\times\}400\{\\times\}0\.20\{=\}240examples from𝒯s​o​c\\mathcal\{T\}\_\{soc\}and3×400×0\.80=9603\{\\times\}400\{\\times\}0\.80\{=\}960examples from𝒯g​o​v\\mathcal\{T\}\_\{gov\}, with corpusBBreceiving the mirror allocation\. Each Scenario 2 corpus containsN=3,000N\{=\}3\{,\}000bill summaries \(NA=NB=1,500N\_\{A\}\{=\}N\_\{B\}\{=\}1\{,\}500\), with equal\-sized pre\-2011 and post\-2011 strata so that the within\-period signals cancel at the aggregate level\.

Table 5:Eligible topic pools for conditional discovery\. In each seed run, three topics are randomly sampled from each pool\.
## Appendix BEvaluation Metrics

We follow previous work\(Zhonget al\.,[2024](https://arxiv.org/html/2606.03029#bib.bib33); Movvaet al\.,[2025](https://arxiv.org/html/2606.03029#bib.bib32)\)in evaluating synthetic experiments by matching generated hypotheses to the known ground\-truth differences\. For each scenario, we repeat the experiment for 10 random seeds and report the mean and standard deviation of each metric\. We split each synthetic dataset into training, validation, and test sets in a60:20:2060\{:\}20\{:\}20ratio\. The validation set is used forsaehyperparameter selection, as described in Appendix[C\.5](https://arxiv.org/html/2606.03029#A3.SS5)\.

On the test set, reference annotations are determined directly by the synthetic construction\. For each generated hypothesis, we useQwen3\-30B\-A3B\-Instruct\-2507to produce binary annotations on the same test texts\. Hypothesis annotation is an expensive operation\.Movvaet al\.\([2025](https://arxiv.org/html/2606.03029#bib.bib32)\)usesGPT\-4o\-mini, and we use this open\-sourced model because we can host it locally to reduce costs, and it matchesGPT\-4o\-miniin annotation performances\. We internally evalutated the two models on a sample 20 hypotheses and 30 bill summaries per hypothesis, resulting in 600 text\-concept pairs\. The agreement is95\.3%95\.3\\%, so we useQwen3\-30B\-A3B\-Instruct\-2507for hypothesis annotation throughout the experiments\. We hosted the model on two RTXA6000 GPUs with vLLM for fast inference\.

Now we have one reference annotation vector \(obtained from the dataset\) and one generated annotation vector for each hypothesis\. We compute the correlation between every reference/generated pair, forming a\|ℋ\|×\|ℋ\|\|\\mathcal\{H\}\|\\times\|\\mathcal\{H\}\|correlation matrix, and use the Hungarian algorithm to find the maximum\-correlation one\-to\-one matching\. Thellmused for judging surface similarity isgpt\-4\.1\-2025\-04\-14\.

lassohypothesisΔ\\Deltademeaned\-reweighed\-lasso hypothesisΔ\\DeltaMatched hypothesesthe teacher repeatedly prompts students to explain their reasoning or clarify their answers at multiple steps in multi\-step fraction or mixed number problems, often after an initial incorrect or incomplete response\+24\.7the teacher prompts students to explain or justify their reasoning after a partially correct or incorrect answer, and facilitates peer discussion to resolve confusion\+21\.7teacher prompts students to explicitly explain or justify their procedural steps or reasoning during problem\-solving, guiding them through multi\-step solutions by asking follow\-up questions when confusion or errors occur\+21\.9teacher repeatedly prompts students to show all intermediate steps in their math work \(e\.g\. requiring students to write down each step, not just the final answer, asking them to explain or justify each step, correcting students who do steps in their head or skip showing borrowing/regrouping, explicitly tying process visibility to grading or real\-life situations\)\+16\.4the teacher prompts students to explicitly explain or justify their mathematical thinking while solving arithmetic problems, with multiple follow\-up questions driving students to clarify errors or reasoning \(rather than simply confirming answers or having students display work\)\+23\.0teacher asks students to explain or justify their mathematical reasoning, often following up on errors or confusions, rather than just stating answers or procedures\+21\.0the teacher asks students to explicitly explain or justify their reasoning or chosen strategy for solving a problem\+19\.1the teacher prompts students to explain or compare their solution strategies to each other\+11\.2teacher prompts students to explicitly identify and correct procedural mistakes in place value or decimal operations\+17\.2the teacher frequently prompts students to explain their reasoning, describe their solution process, or justify place value choices during multi\-digit arithmetic problems\+15\.3the teacher assigns or reminds students about their math homework specifically during lesson closure or class transitions\+0\.3teacher assigns or discusses specific math homework or practice problems for students to complete outside of class\+10\.2teacher gives procedural classroom management instructions related to cleaning up materials, lining up, or transitioning between activities\-7\.1teacher directing students to physically move between locations or groups for collaborative or group work\-5\.2Unique hypothesesthe teacher directs students to perform classroom management or logistical tasks, such as handing out papers, putting away materials, collecting items, or resetting groups\-9\.7teacher prompts students to reflect in writing \(e\.g\. journals or notebooks\) on what they learned, their confusions, or their problem\-solving process at the end of the lesson\-2\.6frequent use of whole\-class management language to enforce procedural compliance and orderly transitions, including directives for materials \(e\.g\. whiteboards, slates\), timing cues \(e\.g\. minute warnings, counting down\), and group routines \(e\.g\. ‘boards up’, ‘push your chair in’\), delivered alongside academic activities\-6\.2teacher organizes students to work in small groups or pairs and prompts them to help and explain concepts to one another\+6\.6teacher provides individualized follow\-up or checks in with specific students about their understanding or needs during or after whole\-class instruction\+20\.6contains extended classroom discussions about units of weight \(grams, kilograms, pounds, ounces, tons\), including teacher\-student exchanges involving conversions, comparisons, or real\-life analogies related to weight measurement\-0\.6Table 6:NCTE full hypothesis list used in the expert evaluation, with matched and unique sections in one table\. Color coding:green shadesindicate↑\\uparrowmid/high REMED andred shadesindicate↑\\uparrowlow REMED; light/lighter shades alternate by row for readability\.Δ=H/M−Low\\Delta=\\text\{H/M\}\-\\text\{Low\}\.lassohypothesisΔ\\Deltademeaned\-reweighted\-lasso hypothesisΔ\\DeltaMatched hypothesesrequests unanimous consent for procedural actions in the Senate, such as granting floor privileges, scheduling committee meetings, agreeing to resolutions, making technical amendments, or permitting specific actions during Senate proceedings\+6\.9requests unanimous consent for procedural actions within the Senate or House, such as granting floor privileges, modifying amendments, scheduling votes, printing materials in the record, or controlling debate time\+7\.4requests unanimous consent to authorize a committee or subcommittee meeting or hearing in the Senate, specifying the committee/subcommittee, date, and topic or title of the meeting\+2\.9announces or authorizes official Senate committee hearings or meetings, specifying committee names, dates, times, and locations\+3\.3gives or discusses the schedule and timing of upcoming Senate votes, bill considerations, or legislative actions \(including mentioning specific days, times, or order of business\)\+7\.5discusses the scheduling and process of votes and amendments on the Senate floor, including references to specific times, sessions, and expectations for voting activity\+4\.6criticizes government or presidential leadership for mismanagement or policy failures, particularly in areas such as war, defense, budgets, or major federal programs\-20\.0criticizes government or institutional incompetence or failure to act, particularly in relation to policies, disasters, or administration\-25\.1discusses or criticizes tax benefits, tax cuts, or loopholes that disproportionately benefit the wealthiest Americans or top income percentiles \(e\.g\. top 1%, billionaires, richest families\), often contrasting with the needs of the middle or lower class\-3\.3criticizes tax cuts or fiscal policies that disproportionately benefit wealthy individuals or families at the expense of the poor, the middle class, or public programs\-8\.5discusses the United States national debt, including specific figures or references to the growing debt, debt ceiling, and its impact on citizens and the country\-2\.8discusses the increase of the national debt or debt ceiling of the United States, with reference to specific amounts, time periods, or presidential administrations\-2\.9criticizes proposed federal budget cuts, especially emphasizing negative impacts on socially vulnerable groups or essential services \(such as veterans, students, low\-income families, healthcare, or education\)\-9\.6discusses government spending, budget deficits, and the need for fiscal responsibility or limiting federal expenditures\-4\.3criticizes or discusses actions, policies, or leadership of the Republican Party or House Republicans\-12\.1criticizes House or congressional Republicans for their policies, actions, or leadership\-11\.7Unique hypothesesdiscusses progress, specific events, or achievements related to democratization and government\-building in Iraq, particularly referencing Iraqi elections, voting, establishment of new governmental bodies, and the training or accomplishments of Iraqi security forces\+0\.1criticizes perceived procedural unfairness or lack of accountability within Congress or government institutions\-16\.8discusses United States border security, including measures such as border fences, patrols, or enforcement against illegal crossings\+1\.6presents recent U\.S\. economic statistics or indicators \(such as GDP growth, job creation, unemployment rate, or manufacturing output\) to argue that the U\.S\. economy is strong or improving, often attributing this to Republican or presidential policies\+0\.7Table 7:Congress full hypothesis list used in the expert evaluation, with matched and unique sections in one table\. Color coding:red shadesindicate↑\\uparrowRepublican andblue shadesindicate↑\\uparrowDemocrat; light/lighter shades alternate by row for readability\.Δ=Rep\.−Dem\.\\Delta=\\text\{Rep\.\}\-\\text\{Dem\.\}
## Appendix CMethod Implementation Details

### C\.1SAE\-Based Baselines

We compare against two globalsaefeature\-selection baselines\.SAE\-lassouses Eq\.[1](https://arxiv.org/html/2606.03029#S3.E1)to select the top\|ℋ\|\|\\mathcal\{H\}\|features, using the implementation provided byMovvaet al\.\([2025](https://arxiv.org/html/2606.03029#bib.bib32)\)\.222https://github\.com/rmovva/HypotheSAEsSAE\-separation scoreranks each feature by the mean ofyyamong the highest\-activating texts versus zero\-activating texts, then selects the top\|ℋ\|\|\\mathcal\{H\}\|features\. Both baselines use the samesaeactivation matrix as the proposed methods\.

### C\.2Prompts

SAE Interpretation PromptYou are a machine learning researcher who has trained a neural network on a text dataset\. You are trying to understand what text features cause a specific neuron in the neural network to fire\.You are given two sets of SAMPLES: POSITIVE SAMPLES that strongly activate the neuron, and NEGATIVE SAMPLES from the same distribution that do not activate the neuron\. Your goal is to identify a feature that is present in the positive samples but absent in the negative samples\.\{goal\}POSITIVE SAMPLES:\{positive\_texts\} NEGATIVE SAMPLES:\{negative\_texts\} Rules about the feature you identify:The feature should be objective, focusing on concrete attributes rather than abstract concepts\. The feature should be present in the positive samples and absent in the negative samples\. Do not output a generic feature which also appears in negative samples\. The feature should be as specific as possible, while still applying to all of the positive samples\.Do not output anything besides the feature\. Please suggest exactly one feature, starting with\-and surrounded by quotes\.

Table 8:Prompt template used for thesaeinterpretation step, with dataset\-specific substitutions for the\{goal\}field\.Table 9:Example matched ground\-truth and generated hypotheses from the synthetic bills experiment, with averagedllmsurface\-similarity judgment scores\.
### C\.3LLM\-Direct Baselines

The LLM\-direct baselines test whether covariate\-aware discovery can be obtained simply by showing sampled texts to anllm\.llm\-globalgenerates\|ℋ\|\|\\mathcal\{H\}\|hypotheses from sampled corpus texts without covariate information\.llm\-covariateuses the same direct prompting setup but exposes covariate\-profile labels\. For fair comparison, we usegpt\-4\.1\-2025\-04\-14for hypothesis proposal in the main tables andQwen3\-30B\-A3B\-Instruct\-2507for hypothesis annotation\.

The LLM\-direct prompts differ from thesaeinterpretation prompt in Table[8](https://arxiv.org/html/2606.03029#A3.T8)in two ways\. First, their positive and negative examples are sampled corpus texts, rather than examples chosen by activation of a selectedsaefeature\. Second, they ask thellmto propose up to\|ℋ\|\|\\mathcal\{H\}\|corpus\-level differences directly, rather than to name exactly one feature explaining one selected neuron\. Thellm\-covariateprompt makes one additional change relative tollm\-global: each text is prefixed with its covariate value\(s\), and the model is instructed to look for patterns that are strong either globally or within a covariate stratum\.

We sweptnprompt∈\{25,50,100\}n\_\{\\text\{prompt\}\}\\in\\\{25,50,100\\\}sampled texts per corpus and proposal modelsgpt\-4\.1\-2025\-04\-14andgpt\-5\.4\-2026\-03\-05\. Figures[2](https://arxiv.org/html/2606.03029#A3.F2)–[3](https://arxiv.org/html/2606.03029#A3.F3)visualize the seed\-level distributions across the full prompt\-budget sweep\. The sweep shows that increasing the prompt budget does not consistently improve either LLM\-direct baseline; covariate labels and the stronger proposal model also do not produce consistent gains\. We therefore usenprompt=100n\_\{\\text\{prompt\}\}\{=\}100withgpt\-4\.1as a generous fixed\-budget setting rather than tuning the LLM\-direct baseline per condition\.

![Refer to caption](https://arxiv.org/html/2606.03029v1/x1.png)Figure 2:Prompt\-budget sensitivity for LLM\-direct baselines usinggpt\-4\.1\. Each panel varies the number of sampled texts per corpus used in the direct prompt\. Lines show seed means; shaded bands show±1\\pm 1standard deviation across seed runs\. The trends are non\-monotonic: larger prompt budgets and covariate labels do not consistently improve surface similarity or F1\.![Refer to caption](https://arxiv.org/html/2606.03029v1/x2.png)Figure 3:Prompt\-budget sensitivity for LLM\-direct baselines usinggpt\-5\.4\. Each panel varies the number of sampled texts per corpus used in the direct prompt\. Lines show seed means; shaded bands show±1\\pm 1standard deviation across seed runs\. The qualitative pattern remains non\-monotonic, and covariate\-labeled prompting does not consistently dominate global prompting\.
### C\.4Reference Baselines

We include two reference baselines to calibrate the synthetic evaluation\.Oracleis an upper reference: for each targeted ground\-truth difference,2020texts are randomly sampled from the exact positive and negative sides of the contrast\. In Scenario 1, these are the pairedAAandBBsubtopics for the target social\-policy topic\. In Scenario 2, these are one𝒯1\\mathcal\{T\}\_\{1\}subtopic versus all𝒯2\\mathcal\{T\}\_\{2\}subtopics combined\. These texts and their corpus labels \(AAorBB\) are directly provided to the LLM for interpretation using the prompt in Table[8](https://arxiv.org/html/2606.03029#A3.T8)\.Randommeasures how much recovery can be obtained from general samples alone:2020training texts are randomly sampled into positive and negative halves, then passed to the same prompt\. This is not a pure lower bound, since sampled texts still contain topical cues that can receive partial credit\. We set the sample size to2020to be consistent with the SAE interpretation stage\(Movvaet al\.,[2025](https://arxiv.org/html/2606.03029#bib.bib32)\)\.

### C\.5SAE Hyperparameter Tuning

We tunesaehyperparameters on the validation split and use the selectedsaeto extract feature activations for all SAE\-based and proposed methods\. Following the released HypotheSAEs implementation333https://github\.com/rmovva/HypotheSAEs, we treat the number of learned featuresMMand the per\-example sparsityKKas the mainsaehyperparameters:MMcontrols how many concepts the model can represent, whileKKcontrols how many concepts are active for each text\. The implementation recommends vanilla Top\-KKchoices of approximately\(M,K\)=\(64,4\)\(M,K\)=\(64,4\)for∼\\sim1K examples,\(256,8\)\(256,8\)for∼\\sim10K examples, and\(1024,8\)\(1024,8\)for∼\\sim100K examples, with largerMMyielding finer\-grained concepts andKKtypically in the range11–3232\. For the synthetic bill corpora \(N=2400N=2400\) and thenctecorpus \(∼\\sim10K examples\), we therefore sweep a small grid around the rule\-of\-thumb setting:M∈\{128,256,512\}M\\in\\\{128,256,512\\\}andK∈\{4,8,16\}K\\in\\\{4,8,16\\\}\. Forcongress, we instead match the larger range used for the Congressional speech experiments inMovvaet al\.\([2025](https://arxiv.org/html/2606.03029#bib.bib32)\), sweepingM∈\{1024,2048,4096\}M\\in\\\{1024,2048,4096\\\}andK∈\{8,16,32\}K\\in\\\{8,16,32\\\}\. For each candidate pair, we train a vanilla Top\-KKsaeon the training embeddings with validation embeddings for early stopping, select predictive features, and generate candidate interpretations\. On real\-world datasets, the selected configuration is the one with the best validation predictive score \(AUROC for binary classification, orR2R^\{2\}for regression\)\. On synthetic datasets, where reference hypotheses are available, the selected configuration is the one with the highest validation surface similarity to the ground\-truth hypotheses\.

## Appendix DReal\-World Validation Details

### D\.1Matching Decisions and Full Hypothesis Lists

Tables[7](https://arxiv.org/html/2606.03029#A2.T7)and[6](https://arxiv.org/html/2606.03029#A2.T6)report the complete matched and unique lists used in the expert analysis\. Colored cells indicate direction by global prevalence difference: Congress uses red for↑\\uparrowRepublican and blue for↑\\uparrowDemocrat; NCTE uses green for↑\\uparrowmid/high REMED and red for↑\\uparrowlow REMED\. All prevalence values are from the user\-study annotation artifacts;Δ\\Deltais the difference between the two group prevalences shown in each table\.

Surface Similarity PromptAre text\_a and text\_b similar in meaning? Respond with yes, related, or no\.Here are a few examples\.Example 1:text\_a: has a topic of protecting the environmenttext\_b: has a topic of environmental protection and sustainabilityoutput: yesExample 2:text\_a: has a language of Germantext\_b: has a language of Deutschoutput: yesExample 3:text\_a: has a topic of the relation between political figurestext\_b: has a topic of international diplomacyoutput: relatedExample 4:text\_a: has a topic of the sportstext\_b: has a topic of sports team recruiting new membersoutput: relatedExample 5:text\_a: has a named language of Koreantext\_b: uses archaic and poetic dictionoutput: noExample 6:text\_a: has a named language of Koreantext\_b: has a named language of Japaneseoutput: noExample 7:text\_a: describes an important 20th century historical eventtext\_b: describes a 20th century European politicianoutput: noTarget:text\_a: \{text\_a\}text\_b: \{text\_b\}output:Table 10:Prompt used for judging the surface similarity between ground\-truth and inferred hypotheses\. Adapted fromZhonget al\.\([2024](https://arxiv.org/html/2606.03029#bib.bib33)\); Movvaet al\.\([2025](https://arxiv.org/html/2606.03029#bib.bib32)\)\.
### D\.2Covariate operationalization

Forcongress, the covariate is a binaryllmannotation of whether a speech segment “contains substantive discussion of public policy, rather than only congressional procedure\.” We use the samecongresssplit asMovvaet al\.\([2025](https://arxiv.org/html/2606.03029#bib.bib32)\): approximately 114K training, 16K validation, and 12K held\-out speeches\. Forncte, the covariate is derived from the CLASS Behavior Management score \(CLBM\)Piantaet al\.\([2012](https://arxiv.org/html/2606.03029#bib.bib12)\): segments withCLBM<6<6are marked as low\-quality behavior management, while segments withCLBM≥6\\geq 6are marked as high\-quality behavior management\. We expect to see less management language in the high\-quallity segments\. This threshold marks the low and mid ranges on the CLASS scale and captures roughly 30% of segments\. This imbalanced distribution also allows us to test the stratum imbalance scenario in a real\-world dataset, which is important forncteas high\-quality teaching samples are rare in generalKane and Staiger \([2012](https://arxiv.org/html/2606.03029#bib.bib3)\); Xuet al\.\([2024](https://arxiv.org/html/2606.03029#bib.bib4)\)\. The dataset size has around 10K samples in total with 5K rated as mid/high qualityremedand 5K rated as low qualityremed\. We split the dataset into train/val/testsets in a60:20:2060\{:\}20\{:\}20ratio\. The scores were provided by trained annotators and come withnctedataset\. These covariates were pre\-specified before expert annotation\.

### D\.3Annotation instructions

Annotators reviewed method\-blinded hypothesis sets fromlassoand demeaned\-reweighted\-lasso, shown as Set A and Set B\. They were instructed to use their domain judgment; there were no right or wrong answers\. First, they rated each hypothesis for helpfulness in understanding the outcome\-group difference on a 1–5 scale \(1: not helpful; 3: somewhat helpful; 5: very helpful\)\. They then reviewed the same hypotheses with global and covariate\-stratified statistics and rated whether the covariate breakdown added interpretive value beyond the global finding, also on a 1–5 scale \(1: no value; 3: some value; 5: high value\)\. Annotators could optionally provide free\-text notes explaining their ratings\. For each dataset, we use manually curated one\-to\-one matched pairs and mark all remaining hypotheses as method\-unique\.

### D\.4Recruitment and IRB\.

Participation was voluntary\. We invited research scholars with relevant domain expertise: computational social science researchers familiar with U\.S\. politics forcongress, and education researchers familiar withnctefor the classroom\-transcript study\. Onencteannotator also had years of elementary teaching experience\. The study was reviewed by an institutional review board and determined exempt under 45 CFR 46\.104\(d\)\(2\)\(ii\)\.