Benchmarking Language Models for Statistical Problem Formulation

# Benchmarking Language Models for Statistical Problem Formulation
Source: [https://arxiv.org/html/2609.01982](https://arxiv.org/html/2609.01982)
Chen Wang111Equal contribution\.Affiliation:Department of Statistics and Data Science, Tsinghua UniversityEmail:[wangchen23@mails\.tsinghua\.edu\.cn](mailto:)Junzhe Zhao111Equal contribution\.Affiliation:Department of Statistics and Data Science, Tsinghua UniversityEmail:[zhaojz26@mails\.tsinghua\.edu\.cn](mailto:)Xin Cong222Corresponding author\.Affiliation:Department of Statistics and Data Science, Tsinghua UniversityEmail:[congxin1995@tsinghua\.edu\.cn](mailto:)Wanlu DengAffiliation:Department of Statistics and Data Science, Tsinghua UniversityEmail:[wanludeng@tsinghua\.edu\.cn](mailto:)Ke DengAffiliation:Department of Statistics and Data Science, Tsinghua UniversityEmail:[kdeng@tsinghua\.edu\.cn](mailto:)

###### Abstract

Large language models \(LLMs\) are increasingly used as assistants for statistical and data science work, yet existing evaluations largely assume the analysis target is already specified\. In practice, users arrive with informal goals and heterogeneous data, leaving the model to decide what statistical task is implied and which data are relevant\. We first formalize this upstream step as Statistical Problem Formulation and decompose it into two subtasks: \(1\) Statistical Problem Classification and \(2\) Variable Identification & Role Assignment\. We then introduceStatFormBench, a benchmark built from five cross\-domain statistics textbooks and a data science case library, covering diverse problem types, data representations, and scenario styles\. It contains 1,013 samples spanning 20 coarse\-grained and 85 fine\-grained statistical problem categories\. Across 14 open\- and closed\-source LLMs, the best zero\-shot models reach only 72\.0 fine\-grained classification accuracy and 63\.2 variable set overlap\. No model performs consistently best across the two subtasks, while enhanced prompting strategies yield only limited or inconsistent gains\. We release the benchmark data on Hugging Face at[https://huggingface\.co/datasets/THU\-CongLab/StatFormBench](https://huggingface.co/datasets/THU-CongLab/StatFormBench)and the evaluation code on GitHub at[https://github\.com/THU\-CongLab/StatFormBench](https://github.com/THU-CongLab/StatFormBench)\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2609.01982v1/fig1_v3.png)Figure 1:Overview of statistical problem formulation\. Given a scenario with heterogeneous data, the model must classify the statistical problem type and extract the relevant variables with their roles\.Large language models \(LLMs\) are increasingly expected to serve as general\-purpose assistants for data science and statistical analysis work, helping users inspect datasets, choose analyses, and answer questions grounded in data\([Huang et al\., 2024](https://arxiv.org/html/2609.01982#bib.bib5);[Zhang et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib6);[Chen et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib7)\)\. Much of this work assumes that the analysis target has already been made explicit, for example by specifying a prediction task, a hypothesis test, or a modeling objective\([Lu et al\., 2026](https://arxiv.org/html/2609.01982#bib.bib2);[Liu et al\., 2024](https://arxiv.org/html/2609.01982#bib.bib3);[Huang et al\., 2024](https://arxiv.org/html/2609.01982#bib.bib5);[Zhang et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib6)\)\. In practice, however, users often begin with informal goals rather than formal analysis specifications\. As illustrated in Figure[1](https://arxiv.org/html/2609.01982#S1.F1), even a simple practical request may require the model to distinguish the intended hypothesis test from a superficially plausible prediction task\. It also needs to determine which data objects are required for that test, rather than treating all available fields as relevant\. These implicit choices determine whether the subsequent analysis is statistically valid\. We refer to this upstream specification step as statistical problem formulation\.

We defineStatistical Problem Formulationas the task of mapping an informal, natural\-language analysis request to \(i\) the type of statistical problem it implies and \(ii\) the relevant data objects together with their analytical roles\. These two components correspond to two closely connected decisions\. The first,Statistical Problem Classification, determines what type of statistical task a scenario calls for, such as comparing group means, examining associations between variables, or predicting future outcomes\. The second,Variable Identification and Role Assignment, determines which data objects are relevant to the task and what analytical role each object plays, such as an independent variable or a dependent variable\. Once these decisions have been made, the subsequent choice of statistical methods becomes more constrained\. By contrast, formulation itself often depends on interpreting the user’s intent, the available data, and the surrounding domain context\. Whether LLMs can reliably perform this step remains largely unexplored\.

Recent benchmarks have examined LLMs in statistics and data science, covering statistical knowledge and reasoning\([Lu et al\., 2026](https://arxiv.org/html/2609.01982#bib.bib2);[Liu et al\., 2024](https://arxiv.org/html/2609.01982#bib.bib3);[Su et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib4)\)as well as data science execution\([Huang et al\., 2024](https://arxiv.org/html/2609.01982#bib.bib5);[Zhang et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib6);[Chen et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib7)\)\. These studies provide useful evidence about whether LLMs can reason with statistical concepts or carry out specified analyses, but they generally assume that the analysis target is already given or strongly implied\. As a result, they do not directly test whether a model can first formulate the statistical task from an informal real\-world request\.

Evaluating this capability is challenging because formulation differs from standard question answering or task execution in several ways\. First, the input is often not written in statistical language: users may describe practical goals with ambiguity, domain\-specific wording, and distracting contextual details\. Second, the relevant data may be represented in heterogeneous forms, including raw tabular variables, summary statistics such as means and standard deviations, contingency tables, and other structured or semi\-structured objects\. Third, the target label space is broad and weakly delimited\. Deciding whether a scenario calls for comparison, association analysis, prediction, reliability assessment, or another statistical task requires jointly interpreting the user’s objective, the data structure, and the domain context\. The same problem\-level understanding is also needed to separate relevant variables from incidental information and to assign their analytical roles\.

To this end, we introduceStatFormBench, a benchmark for statistical problem formulation in realistic, unstructured settings\.StatFormBenchoperationalizes formulation through two tasks aligned with the decisions above\.Statistical Problem Classificationasks a model to identify the statistical problem type expressed by a scenario, andVariable Identification & Role Assignmentasks it to identify the data objects relevant to the task and assign their analytical roles\. The benchmark is constructed from statistics textbook exercises across applied domains and a data science case library reflecting diverse business scenarios, allowing us to cover both controlled pedagogical problems and more open\-ended real\-world analysis requests\. Each sample is built through a unified pipeline that extracts raw triples of question, answer, and data, splits multi\-part problems into individual instances, filters for quality, and annotates problem types and variable roles\. The problem descriptions are then rewritten into realistic scenario\-style language\. We evaluate 14 open\-source and proprietary LLMs\. The best zero\-shot fine\-grained classification accuracy is 72\.0, while the highest variable set overlap is 63\.2, with the two results achieved by different models\. Fine\-grained analysis further shows that performance varies substantially across problem categories and data sources, while standard prompting strategies yield limited and inconsistent gains\.

Our contributions are as follows: \(1\) we are the first to study statistical problem formulation as an evaluation problem for LLMs, decomposing it into statistical problem classification and variable identification & role assignment; \(2\) we construct theStatFormBenchbenchmark covering diverse statistical problem types, data representations, and scenario styles, with samples derived from both textbooks and real\-world data science cases; \(3\) we evaluate 14 open\-source and proprietary LLMs, showing that no model performs consistently best across formulation tasks and that current models still struggle to map informal requests to precise statistical formulations regardless of the prompting strategy used\.

## 2Related Work

### 2\.1LLMs for Statistics and Data Science

Recent benchmarks evaluate LLMs for statistics and data science mainly along two lines\. Both are closely related to our setting, but these studies generally assume that the analytical goal is already specified or implied\.

#### Statistical knowledge and reasoning\.

The first line studies whether LLMs can reason with statistical concepts once the analytical goal is explicit or strongly implied\. StatQA\([Zhu et al\., 2024](https://arxiv.org/html/2609.01982#bib.bib1)\)evaluates method applicability and variable selection for pre\-formulated questions over structured tables, StatEval\([Lu et al\., 2026](https://arxiv.org/html/2609.01982#bib.bib2)\)extends this direction to foundational exercises and research\-level proof tasks, and QRData\([Liu et al\., 2024](https://arxiv.org/html/2609.01982#bib.bib3)\)and ClimateViz\([Su et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib4)\)examine statistical or causal reasoning over data sheets and scientific charts\. These benchmarks assess statistical competence in settings where the target analytical goal is already defined, whereas we focus on inferring the statistical problem itself and identifying the data objects required for solving it\.

#### Data science execution\.

The second line evaluates LLMs as executors of specified analytical tasks\. DA\-Code\([Huang et al\., 2024](https://arxiv.org/html/2609.01982#bib.bib5)\)and DataSciBench\([Zhang et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib6)\)focus on code generation for data preprocessing, analysis, and modeling, while ScienceAgentBench\([Chen et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib7)\), LMR\-Bench\([Yan et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib8)\), DSBench\([Jing et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib11)\), and BLADE\([Gu et al\., 2024](https://arxiv.org/html/2609.01982#bib.bib12)\)extend evaluation to data\-driven science, ML research reproduction, and realistic data science workflows\. These works increasingly consider intermediate decisions, but they still largely assume that research questions or analytical objectives have been given\.

### 2\.2LLMs for Problem Formulation

#### Optimization modeling\.

Another line of work evaluates whether LLMs can map natural\-language descriptions to formal problem representations\. Optimization modeling is the most developed example\. NL4Opt\([Ramamonjison et al\., 2022](https://arxiv.org/html/2609.01982#bib.bib14)\)and subsequent benchmarks such as OptiMUS\([AhmadiTeshnizi et al\., 2024](https://arxiv.org/html/2609.01982#bib.bib15)\), Chain\-of\-Experts\([Xiao et al\., 2024](https://arxiv.org/html/2609.01982#bib.bib19)\), ORLM\([Huang et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib16)\), LLMOPT\([Jiang et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib17)\), and OptiBench\([Yang et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib18)\)treat formulation as an explicit target rather than a hidden precursor to solving\.

#### Mathematical and domain\-specific formulation\.

Related benchmarks extend this formulation perspective to broader domains\. ModelingAgent\([Qian et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib9)\)and MM\-Agent\([Liu et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib20)\)study mathematical modeling from competition problems, MedCalc\-Bench\([Khandekar et al\., 2024](https://arxiv.org/html/2609.01982#bib.bib21)\)and MedRaC\([Wang et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib10)\)evaluate clinical calculation through formula selection, entity extraction, and computation, and DiscoveryBench\([Majumder et al\., 2025](https://arxiv.org/html/2609.01982#bib.bib13)\)evaluates data\-driven hypothesis generation in scientific discovery through context, variables, and relationships\. Together, these studies show the value of evaluating intermediate abstractions rather than only final answers\.

Statistical problem formulation shares this mapping challenge, but is less anchored by a fixed schema\. Unlike optimization formulation, which largely amounts to instantiating a fixed template of elements such as objectives and constraints, statistical formulation must infer the analytical question from domain\-specific descriptions and select from a large, heterogeneous, and weakly delimited space of problem types\. It must also identify diverse data objects, ranging from raw variables to grouped summaries and contingency tables\. To our knowledge, no existing benchmark systematically evaluates this upstream formulation ability\.

![Refer to caption](https://arxiv.org/html/2609.01982v1/fig2_v3.png)Figure 2:Overview of ourStatFormBenchbenchmark construction pipeline\.

## 3Task Formulation

We decompose statistical problem formulation into two tasks defined over an input triple\(B,Q,D\)\(B,Q,D\), whereBBdescribes the practical background,QQstates the user request, andDDrefers to data\.DDvaries in both form and content \(e\.g\., tables, textual descriptions, separate files, etc\.\)\. In content, it ranges from raw observations to summary statistics and contingency tables\.

### 3\.1Task 1: Statistical Problem Classification

Given\(B,Q,D\)\(B,Q,D\), the task is to predict

c=\(cCG,cFG\),cCG∈𝒞CG,cFG∈𝒞FG,c=\(c\_\{\\mathrm\{CG\}\},c\_\{\\mathrm\{FG\}\}\),\\quad c\_\{\\mathrm\{CG\}\}\\in\\mathcal\{C\}\_\{\\mathrm\{CG\}\},\\;c\_\{\\mathrm\{FG\}\}\\in\\mathcal\{C\}\_\{\\mathrm\{FG\}\},\(1\)whereccdenotes the statistical problem type\. The label space is a two\-level hierarchy\.𝒞CG\\mathcal\{C\}\_\{\\mathrm\{CG\}\}captures coarse analytical intent, such as descriptive statistics, classification and prediction, and hypothesis testing and interval estimation, while𝒞FG\\mathcal\{C\}\_\{\\mathrm\{FG\}\}subdivides each first\-level category into specific problem types\. We classify the problem type rather than the solving method because the same statistical problem can admit multiple methods, and committing to one method would blur the distinction between problem identification and method selection\.

### 3\.2Task 2: Variable Identification and Role Assignment

Given the same input, the task is to produce a setV=\{v1,…,vk\}V=\\\{v\_\{1\},\\ldots,v\_\{k\}\\\}of variables required for solvingQQ, where each variable object is represented as

vi=\(idi,desci,valuei,rolei\)\.v\_\{i\}=\(\\mathrm\{id\}\_\{i\},\\;\\mathrm\{desc\}\_\{i\},\\;\\mathrm\{value\}\_\{i\},\\;\\mathrm\{role\}\_\{i\}\)\.\(2\)Hereidi\\mathrm\{id\}\_\{i\}is a short identifier, typically reusing a name fromDD\.desci\\mathrm\{desc\}\_\{i\}describes its domain meaning, andvaluei\\mathrm\{value\}\_\{i\}stores the corresponding values fromDD\. The role field specifies howviv\_\{i\}participates in solvingQQand takes one of four values:predictor,response,both, orN/A\. The rolespredictorandresponseare assigned when the problem implies a directional relationship\. The rolebothis used when variables are related without a required direction, as in correlation analysis, or when a variable plays both directional roles\. The roleN/Amarks variables that are needed but have no directional role, such as identifiers, filter variables, and inputs to unsupervised analyses\. Notably, roles are question\-conditional, since the same data may play different roles under different user requests\.

Figure 3:Dataset statistics\.Left: Distribution of samples across coarse\-grained categories\.Right: Distribution of the number of variables per sample\.

## 4Benchmark Construction

### 4\.1Data Sources

We buildStatFormBenchfrom two complementary sources\. The first source consists of worked examples and exercises from five statistics textbooks spanning diverse applied domains, including public health\([Gerstman, 2007](https://arxiv.org/html/2609.01982#bib.bib22)\), business and economics\([McClave et al\., 2013](https://arxiv.org/html/2609.01982#bib.bib23)\), reliability analysis\([Pham, 2022](https://arxiv.org/html/2609.01982#bib.bib24)\), environmental science\([Cook and Wheater, 2000](https://arxiv.org/html/2609.01982#bib.bib25)\), and clinical research\([Altman, 1991](https://arxiv.org/html/2609.01982#bib.bib26)\)\. These samples provide broad coverage of canonical statistical problem types and usually include compact data in text or inline tables\. The second source is a case\-based data science library from GouXiongHui\([GouXiongHui Org\.,](https://arxiv.org/html/2609.01982#bib.bib27)\), where each case centers on a substantive analytical question, accompanied by an analysis report, structured datasets, and reference code\. These samples provide more open\-ended analytical contexts with larger data files in diverse formats\.

### 4\.2Construction Pipeline

To enable uniform evaluation, both sources are converted into samples of the form\(B,Q,D,c,V\)\(B,Q,D,c,V\), where\(B,Q,D\)\(B,Q,D\)is the evaluation input,ccis the problem\-type label, andVVis the reference variable set\. Figure[2](https://arxiv.org/html/2609.01982#S2.F2)summarizes the construction pipeline\. We describe the main stages below with further details in Appendix[A](https://arxiv.org/html/2609.01982#A1)\.

#### Sample extraction and splitting\.

Raw materials are first parsed into entries\(Qo,Ao,D\)\(Q^\{o\},A^\{o\},D\), where the superscriptoodenotes the original form\. Specifically,QoQ^\{o\}is the original problem statement,AoA^\{o\}is the original reference answer, andDDcontains the associated data\. Entries with multiple sub\-questions are then split into individual samples\(B~,Q~,A~,D\)\(\\tilde\{B\},\\tilde\{Q\},\\tilde\{A\},D\)by extracting the shared backgroundB~\\tilde\{B\}fromQoQ^\{o\}and pairing each sub\-questionQ~\\tilde\{Q\}with its answerA~\\tilde\{A\}\.

#### Quality filtering\.

We remove samples that are too narrow to test formulation, such as value lookup and pure concept recall, and discard cases with incomplete information, such as questions that depend on the answer to a previous problem\.

#### Initial label construction\.

Each retained sample is annotated with a categoryc=\(cCG,cFG\)c=\(c\_\{\\mathrm\{CG\}\},c\_\{\\mathrm\{FG\}\}\)and a variable setV=\{v1,…,vk\}V=\\\{v\_\{1\},\\ldots,v\_\{k\}\\\}\. Domain experts trained in statistics at the undergraduate and doctoral level first construct an initial taxonomy from standard statistical practice, and the taxonomy is refined during annotation when new problem types arise\. Specifically, the annotator LLM proposes candidate categories, and a domain expert decides whether to adopt or merge them\. The final taxonomy contains\|𝒞CG\|=20\|\\mathcal\{C\}\_\{\\mathrm\{CG\}\}\|=20first\-level and\|𝒞FG\|=85\|\\mathcal\{C\}\_\{\\mathrm\{FG\}\}\|=85second\-level categories \(Appendix[B](https://arxiv.org/html/2609.01982#A2)\)\. Variable labels are grounded in the solving process ofA~\\tilde\{A\}, so variables mentioned in the scenario but not needed for the solution are excluded\. For the textbook subset, Claude Opus 4\.6[Anthropic \(2026a\)](https://arxiv.org/html/2609.01982#bib.bib31)produces the initial annotations of bothccandVV\. For the case\-library subset, separate GPT\-4o[OpenAI et al\. \(2024\)](https://arxiv.org/html/2609.01982#bib.bib28)calls produce the initial annotations ofccandVV\. The model\-generated labels serve only as initial candidates, and every final label is subsequently verified by human reviewers as described in Section[4\.3](https://arxiv.org/html/2609.01982#S4.SS3)\.

#### Scenario rewriting\.

As both sources are pedagogical in origin, the original\(B~,Q~\)\(\\tilde\{B\},\\tilde\{Q\}\)often use instructional or explicitly statistical language and rarely include information beyond what the problem requires\. We therefore use GPT\-5 to rewrite each problem from the perspective of a domain practitioner without statistical training\. The rewrite preserves the analytical intent, numerical values, and named entities, while removing direct cues about the target statistical problem category, introducing minor contextual details not directly relevant to the solution, and rephrasing the request in more natural scenario\-style language\. The rewritten results\(B,Q\)\(B,Q\), derived from\(B~,Q~\)\(\\tilde\{B\},\\tilde\{Q\}\), combine withDD,cc, andVVfrom earlier stages to form the final samples\.

#### Source\-specific adaptations\.

The textbook subset follows the pipeline above directly, with PDF extraction assisted by MinerU111[https://mineru\.net/](https://mineru.net/)and manual proofreading\. The case\-library subset differs only in extraction\. We retain cases with both a complete analysis report and a structured dataset, induce\(B~,Q~,A~\)\(\\tilde\{B\},\\tilde\{Q\},\\tilde\{A\}\)from the report through a multi\-call LLM chain, and summarize raw data files into a uniform data description used asDD\. To avoid directly releasing the source data, we further replace the row\-level values inDDwith simulated values generated by GPT\-5\. Further details are provided in Appendix[A\.2](https://arxiv.org/html/2609.01982#A1.SS2)\.

### 4\.3Human Verification

To ensure the reliability of the benchmark, all 1,013 samples inStatFormBenchare manually reviewed\. The review covers the problem categoryccand the variable setVV, with both verified against the reference answers in the source materials\.

For the 629 textbook samples, we conduct two rounds of review by different PhD\-level experts in statistics\. The first reviewer checks and corrects each initial annotation against the exercise and its reference answer\. A second reviewer then re\-examines the corrected annotation\. Any disagreements between the two rounds are resolved by a senior reviewer, who determines the final labels\. For the 384 case\-library samples from GouXiongHui, the initial annotations are easier to verify because each sample is derived from a complete analysis report that documents the underlying analytical process\. A PhD\-level expert reviews every annotation against the report and associated structured data\.

### 4\.4Dataset Statistics

The final benchmark contains 1,013 samples, with 629 from textbooks and 384 from the case library\. As shown in Figure[3](https://arxiv.org/html/2609.01982#S3.F3), the dataset covers a broad range of problem categories, with substantial mass in data visualization, mathematical calculation, and hypothesis testing and interval estimation\. The number of variables per sample is concentrated at small values but has a long tail, reflecting both compact textbook exercises and more complex case\-library scenarios\. A more detailed breakdown of the category distribution is provided in Appendix[D](https://arxiv.org/html/2609.01982#A4)\.

## 5Evaluation Metrics

For thejj\-th sample, letc\(j\)=\(cCG\(j\),cFG\(j\)\)c^\{\(j\)\}=\(c\_\{\\mathrm\{CG\}\}^\{\(j\)\},c\_\{\\mathrm\{FG\}\}^\{\(j\)\}\)andc^\(j\)=\(c^CG\(j\),c^FG\(j\)\)\\hat\{c\}^\{\(j\)\}=\(\\hat\{c\}\_\{\\mathrm\{CG\}\}^\{\(j\)\},\\hat\{c\}\_\{\\mathrm\{FG\}\}^\{\(j\)\}\)denote the reference and predicted problem categories, and letV\(j\)V^\{\(j\)\}andV^\(j\)\\hat\{V\}^\{\(j\)\}denote the reference and predicted variable sets\.

### 5\.1Metrics for Statistical Problem Classification

We report coarse\-grained and fine\-grained classification accuracy,

ACCCG\\displaystyle\\mathrm\{ACC\}\_\{\\mathrm\{CG\}\}=1N∑j=1N𝟏\[c^CG\(j\)=cCG\(j\)\],\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\\mathbf\{1\}\[\\hat\{c\}\_\{\\mathrm\{CG\}\}^\{\(j\)\}=c\_\{\\mathrm\{CG\}\}^\{\(j\)\}\],\(3\)ACCFG\\displaystyle\\mathrm\{ACC\}\_\{\\mathrm\{FG\}\}=1N∑j=1N𝟏\[c^FG\(j\)=cFG\(j\)\]\.\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\\mathbf\{1\}\[\\hat\{c\}\_\{\\mathrm\{FG\}\}^\{\(j\)\}=c\_\{\\mathrm\{FG\}\}^\{\(j\)\}\]\.\(4\)Since a correct fine\-grained prediction implies a correct coarse\-grained one, the gap between the two reflects how often models identify the broad analytical intent but miss the specific problem type\.

### 5\.2Metrics for Variable Identification and Role Assignment

#### Variable matching\.

Because variable identifiers are model\-generated, we first match each predicted variablev^∈V^\(j\)\\hat\{v\}\\in\\hat\{V\}^\{\(j\)\}to a reference variablev∈V\(j\)v\\in V^\{\(j\)\}when theirvaluefields are exactly equal\. LetM\(j\)M^\{\(j\)\}be the resulting set of matched pairs\. Unmatched predictions and unmatched reference variables are treated as false positives and false negatives\. Further implementation details and results from a human validation study of this matching rule are provided in Appendix[E\.1](https://arxiv.org/html/2609.01982#A5.SS1)\.

#### Variable identification metrics\.

To evaluate variable identification, we report three set\-level metrics\. The Jaccard coefficient of variable sets \(JCV\) measures overall overlap,

JCV=1N​∑j=1N\|M\(j\)\|\|V^\(j\)\|\+\|V\(j\)\|−\|M\(j\)\|\.\\mathrm\{JCV\}=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\\frac\{\|M^\{\(j\)\}\|\}\{\|\\hat\{V\}^\{\(j\)\}\|\+\|V^\{\(j\)\}\|\-\|M^\{\(j\)\}\|\}\.\(5\)To further distinguish over\-prediction from missed variables, we also report variable precision \(PV\) and variable recall \(RV\),

PV\\displaystyle\\mathrm\{PV\}=1N​∑j=1N\|M\(j\)\|\|V^\(j\)\|,\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\\frac\{\|M^\{\(j\)\}\|\}\{\|\\hat\{V\}^\{\(j\)\}\|\},\(6\)RV\\displaystyle\\mathrm\{RV\}=1N​∑j=1N\|M\(j\)\|\|V\(j\)\|\.\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\\frac\{\|M^\{\(j\)\}\|\}\{\|V^\{\(j\)\}\|\}\.\(7\)

#### Variable role identification \(VRI\)\.

Letrole⁡\(v\)\\mathrm\{role\}\(v\)denote the role field ofvv\. We measure the fraction of reference variables that are both recovered and assigned the correct role using VRI,

VRI=1N​∑j=1N∑\(v^,v\)∈M\(j\)𝟏\[role\(v^\)=role\(v\)\]\|V\(j\)\|\.\\mathrm\{VRI\}=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\\frac\{\\sum\_\{\(\\hat\{v\},v\)\\in M^\{\(j\)\}\}\\mathbf\{1\}\[\\mathrm\{role\}\(\\hat\{v\}\)=\\mathrm\{role\}\(v\)\]\}\{\|V^\{\(j\)\}\|\}\.\(8\)This metric jointly penalizes missed variables and incorrect role labels\.

Table 1:Main results onStatFormBench\.Bolddenotes the best result, andunderlinedenotes the second best\.

## 6Experiments

### 6\.1Setup

We evaluate 14 LLMs from six model families on the two tasks defined in Section[3](https://arxiv.org/html/2609.01982#S3)\. The closed\-source group includes Claude Opus 4\.6[Anthropic \(2026a\)](https://arxiv.org/html/2609.01982#bib.bib31), Claude Sonnet 4\.6[Anthropic \(2026b\)](https://arxiv.org/html/2609.01982#bib.bib32), GPT\-5\.5[OpenAI \(2026b\)](https://arxiv.org/html/2609.01982#bib.bib29), GPT\-5\.4[OpenAI \(2026a\)](https://arxiv.org/html/2609.01982#bib.bib30), Gemini 3\.1 Pro[Google \(2026\)](https://arxiv.org/html/2609.01982#bib.bib34), and Gemini 3 Flash[Google \(2025\)](https://arxiv.org/html/2609.01982#bib.bib33)\. The open\-source group includes Kimi\-K2\.5[Team et al\. \(2026\)](https://arxiv.org/html/2609.01982#bib.bib37), DeepSeek\-V4\-Pro, DeepSeek\-V4\-Flash[DeepSeek\-AI \(2026\)](https://arxiv.org/html/2609.01982#bib.bib35), and five Qwen3\.5 variants \(397B\-A17B, 9B, 4B, 2B, and 0\.8B\)[Qwen Team \(2026\)](https://arxiv.org/html/2609.01982#bib.bib36)that cover a broad range of model scales\. All models are evaluated on the full benchmark under zero\-shot prompting, and additional prompting strategies are examined in Section[6\.3](https://arxiv.org/html/2609.01982#S6.SS3)\.

Figure 4:Classification accuracy by category\. Top: coarse\-grained categories with more than 10 samples\. Bottom: fine\-grained categories with the highest and lowest average accuracy\. Category abbreviations are defined in Table[5](https://arxiv.org/html/2609.01982#A2.T5)\.
### 6\.2Main Results

Table[1](https://arxiv.org/html/2609.01982#S5.T1)presents the main results across 14 LLMs and reveals persistent challenges in both formulation tasks\. For statistical problem classification, Gemini 3\.1 Pro achieves the highest fine\-grained accuracy at 72\.0, followed by Claude Opus 4\.6 at 69\.5 and Gemini 3 Flash at 68\.4\. The strongest open\-source models, Kimi\-K2\.5, DeepSeek\-V4\-Flash, and Qwen3\.5\-397B\-A17B, perform similarly on this task and trail the best closed\-source model by about 7 points\. Across all models, coarse\-grained accuracy is consistently higher than fine\-grained accuracy, indicating that models can often capture the broad analytical intent but still struggle to distinguish closely related subtypes\.

Model rankings differ for variable identification and role assignment\. Claude Opus 4\.6 achieves the highest JCV at 63\.2 and also leads on RV and VRI, while Gemini 3 Flash obtains the highest PV\. Meanwhile, performance of models can also diverge substantially between the two tasks\. Both GPT\-5\.4 and GPT\-5\.5 exceed 65 onACCFG\\mathrm\{ACC\}\_\{\\mathrm\{FG\}\}but remain near 52 on JCV\. Even within the same model family, the relative ordering varies across metrics\. Gemini 3 Flash surpasses Gemini 3\.1 Pro on all four variable\-related metrics, while DeepSeek\-V4\-Flash outperforms DeepSeek\-V4\-Pro on RV and VRI but trails it on JCV and PV\. Taken together, these results show that no model consistently leads across both formulation tasks, leaving statistical problem formulation beyond the reliable reach of current LLMs\.

### 6\.3Fine\-Grained Analysis

#### Per\-category performance\.

As shown in Figure[4](https://arxiv.org/html/2609.01982#S6.F4), performance varies substantially across problem categories\. At the coarse level, categories with explicit analytical goals, such as mathematical calculation and hypothesis testing, are easier, whereas categories that require distinguishing closely related analytical intents, such as model comparison and model interpretation, are more difficult\. This contrast becomes more pronounced at the fine\-grained level\. Models perform well on categories with clear operational cues but struggle when category boundaries depend on subtle differences in the user request or data representation\.

Table 2:Model performance across data sources\.
#### Data source impact\.

We evaluate representative models separately on the two data sources and observe a clear asymmetry, as shown in Table[2](https://arxiv.org/html/2609.01982#S6.T2)\. On textbook samples,ACCFG\\mathrm\{ACC\}\_\{\\mathrm\{FG\}\}ranges from 74\.1 to 84\.1, while JCV ranges from 33\.6 to 51\.0\. On case\-library samples, the pattern reverses, with JCV between 82\.4 and 83\.6 butACCFG\\mathrm\{ACC\}\_\{\\mathrm\{FG\}\}between 34\.6 and 54\.7\. This contrast reflects the characteristics of the two subsets\. Textbook problems align better with standard statistical categories because they are typically designed around clearly defined problem types, making classification easier, but their compact data presentations demand careful variable extraction\. Case\-library problems instead provide structured datasets whose columns correspond directly to variables, making extraction easier, while their proximity to real\-world analytical scenarios makes the problem types harder to classify\.

Table 3:Impact of prompting strategies\. Zero\-shot reports absolute scores, and other rows reportpositiveornegativechanges from zero\-shot\.Figure 5:Few\-shot performance across prompting settings and metrics\.
#### Prompting with category definitions\.

Prompting strategies are known to affect LLM performance, and we examine their impact on formulation\. As shown in Table[3](https://arxiv.org/html/2609.01982#S6.T3), providing category definitions improves fine\-grained classification for all three models by 1\.9 to 3\.8 points, indicating that explicit taxonomy information helps models distinguish closely related labels\. Its effects on JCV and VRI are smaller and mixed, with most changes within one point\. This suggests that explicit definitions can clarify category distinctions, but benefit to variable identification and role assignment is inconsistent and may be offset by the additional context\.

Table 4:Representative errors, with irrelevant fields omitted\.Red textmarks incorrect parts of model outputs\.
#### Few\-shot prompting\.

Figure[5](https://arxiv.org/html/2609.01982#S6.F5)shows the effect of few\-shot examples across six models\. ForACCFG\\mathrm\{ACC\}\_\{\\mathrm\{FG\}\}, models with stronger zero\-shot performance improve slightly as the number of shots increases, whereas lower\-performing models generally drop at one shot before partially recovering\. For VRI, Gemini 3\.1 Pro and GPT\-5\.5 improve, while the remaining models stay nearly flat or decline\. JCV improves modestly for most models, with DeepSeek\-V4\-Pro as the main exception\. This pattern suggests that examples may help models align their variable outputs with the expected representation but do not resolve the conceptual distinctions required for category classification and role assignment\. Overall, few\-shot examples do not substantially improve statistical problem formulation\.

![Refer to caption](https://arxiv.org/html/2609.01982v1/confusion_matrix_combined.png)Figure 6:Confusion matrices for hierarchical classification, focusing on frequently confused categories \(we only present thekey partsof matrices\)\. Values are normalized by true labels\.

### 6\.4Error Analysis

The confusion matrices in Figure[6](https://arxiv.org/html/2609.01982#S6.F6)show that models often confuse categories that share surface cues but differ in analytical intent\. For example, model comparison is frequently predicted as model interpretation\. At the fine\-grained level, many errors remain within the correct first\-level category, suggesting that models may recover the broad analytical intent yet fail to distinguish closely related subtypes\.

#### Representative cases\.

Table[4](https://arxiv.org/html/2609.01982#S6.T4)presents two cases that illustrate how these confusions arise in individual samples\. In the first case, GPT\-5\.4 correctly identifies the problem type but extracts a redundant derived quantity, whereas the reference label records only the data objects required for the calculation\. This over\-specification is consistent with the tendency toward redundant variable extraction observed in the main results\. In the second case, DeepSeek\-V4\-Pro correctly identifies the relevant variable but treats it as continuous rather than discrete, which leads to an incorrect fine\-grained visualization category\. These cases confirm that statistical formulation demands joint reasoning across data representation and analytical intent, where a single misjudgment can propagate through the entire formulation chain\.

## 7Conclusion

We study statistical problem formulation as an evaluation target for LLMs and decompose it into statistical problem classification and variable identification with role assignment\. To support this evaluation, we constructStatFormBench, a benchmark of 1,013 samples drawn from five cross\-domain statistics textbooks and a data science case library\. The benchmark covers 20 coarse\-grained and 85 fine\-grained problem categories, and presents problems in scenario\-style language that reflects how practitioners describe analytical needs\.

Experiments on 14 LLMs show that no model performs consistently best across the two formulation tasks\. The highest zero\-shot fine\-grained classification accuracy is 72\.0, while the highest variable set overlap is 63\.2, with the two results achieved by different models\. Fine\-grained analyses further show that performance of LLMs varies substantially across problem categories, while prompting strategies yield only limited and inconsistent gains\. Taken together, these findings indicate that statistical problem formulation remains beyond the reliable reach of current models\. Bridging this gap may require tighter integration of domain knowledge, statistical reasoning, and data\-aware inference\. We hopeStatFormBenchhelps clarify where current models fall short and supports the development of more capable statistical formulation assistants\.

## Limitations

Our benchmark has several limitations\. First, the samples are derived from solved textbook exercises and a case library rather than real\-world statistical consultations\. This reflects a trade\-off between realism and data accessibility, since real consultations often contain sensitive information and are subject to confidentiality constraints\. We use constrained scenario rewriting to bring these curated sources closer to real consultations, but the resulting samples may still not fully capture the ambiguity and underspecification of real consulting\. Second, the category distribution is imbalanced, with a substantial portion of samples concentrated in a few common types, such as data visualization and hypothesis testing\. As a result, performance estimates for low\-frequency categories may be less reliable\. Third, the textbook subset is originally in English, whereas the case\-library subset is originally in Chinese and translated for evaluation\. This bilingual origin may introduce subtle cross\-lingual effects that are not explicitly considered in the current evaluation\.

## Ethical Statements

This work does not involve human subjects, and no personally identifiable information appears in the dataset\. The textbook exercises are drawn from published educational materials\. The case\-library data were pre\-anonymized by the data provider before we received them, with all personal names, organizational identifiers, and other potentially identifying attributes removed or replaced with placeholders\. We manually verified a random subset and confirmed the absence of residual identifiable content\.

## Acknowledgements

This work is supported by Shenzhen Science and Technology Program \(Grant No\. AI2026019\)\. We thank GouXiongHui for providing the case\-library data used to construct the benchmark\. We also thank Shuang Chen, Jiepeng Lai, Zikai Lin, Zichong Wang, and Ruitong Zhang for their careful review and verification of the textbook annotations\.

## References

- A\. AhmadiTeshnizi, W\. Gao, and M\. UdellOptiMUS: scalable optimization modeling with \(MI\)LP solvers and large language models\.InInternational Conference on Machine Learning,Vol\.235,pp\. 577–596\.Cited by:[§2\.2](https://arxiv.org/html/2609.01982#S2.SS2.SSS0.Px1.p1.1)\.
- Altman \(1991\)D\. G\. AltmanPractical statistics for medical research\.Chapman & Hall\.Cited by:[§4\.1](https://arxiv.org/html/2609.01982#S4.SS1.p1.1)\.
- Anthropic \(2026a\)AnthropicIntroducing claude opus 4\.6\.Note:[https://www\.anthropic\.com/news/claude\-opus\-4\-6](https://www.anthropic.com/news/claude-opus-4-6)Accessed: 2026\-05\-26Cited by:[§4\.2](https://arxiv.org/html/2609.01982#S4.SS2.SSS0.Px3.p1.1),[§6\.1](https://arxiv.org/html/2609.01982#S6.SS1.p1.1)\.
- Anthropic \(2026b\)AnthropicIntroducing claude sonnet 4\.6\.Note:[https://www\.anthropic\.com/news/claude\-sonnet\-4\-6](https://www.anthropic.com/news/claude-sonnet-4-6)Accessed: 2026\-05\-26Cited by:[§6\.1](https://arxiv.org/html/2609.01982#S6.SS1.p1.1)\.
- Chenet al\.\(2025\)Z\. Chen, S\. Chen, Y\. Ning, Q\. Zhang, B\. Wang, B\. Yu, Y\. Li, Z\. Liao, C\. Wei, Z\. Lu, V\. Dey, M\. Xue, F\. N\. Baker, B\. Burns, D\. Adu\-Ampratwum, X\. Huang, X\. Ning, S\. Gao, Y\. Su, and H\. SunScienceAgentBench: toward rigorous assessment of language agents for data\-driven scientific discovery\.External Links:2410\.05080,[Link](https://arxiv.org/abs/2410.05080)Cited by:[§1](https://arxiv.org/html/2609.01982#S1.p1.1),[§1](https://arxiv.org/html/2609.01982#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.01982#S2.SS1.SSS0.Px2.p1.1)\.
- Cook and Wheater \(2000\)P\. A\. Cook and C\. P\. WheaterUsing statistics to understand the environment\.Routledge\.Cited by:[§4\.1](https://arxiv.org/html/2609.01982#S4.SS1.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-v4: towards highly efficient million\-token context intelligence\.Note:[https://huggingface\.co/deepseek\-ai/DeepSeek\-V4\-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro)Cited by:[§6\.1](https://arxiv.org/html/2609.01982#S6.SS1.p1.1)\.
- Gerstman \(2007\)B\. B\. GerstmanBasic biostatistics: statistics for public health practice\.1st edition,Jones & Bartlett Learning\.Cited by:[§4\.1](https://arxiv.org/html/2609.01982#S4.SS1.p1.1)\.
- Google \(2025\)GoogleGemini 3 flash: frontier intelligence built for speed\.Note:[https://blog\.google/products\-and\-platforms/products/gemini/gemini\-3\-flash/](https://blog.google/products-and-platforms/products/gemini/gemini-3-flash/)Accessed: 2026\-05\-26Cited by:[§6\.1](https://arxiv.org/html/2609.01982#S6.SS1.p1.1)\.
- Google \(2026\)GoogleGemini 3\.1 pro: best for complex tasks and bringing creative concepts to life\.Note:[https://deepmind\.google/models/gemini/pro/](https://deepmind.google/models/gemini/pro/)Accessed: 2026\-05\-26Cited by:[§6\.1](https://arxiv.org/html/2609.01982#S6.SS1.p1.1)\.
- \[11\]GouXiongHui Org\.GouXiongHui: Statistics Second Classroom\.Note:[https://www\.xiong99\.com\.cn/](https://www.xiong99.com.cn/)Cited by:[§4\.1](https://arxiv.org/html/2609.01982#S4.SS1.p1.1)\.
- Guet al\.\(2024\)K\. Gu, R\. Shang, R\. Jiang, K\. Kuang, R\. Lin, D\. Lyu, Y\. Mao, Y\. Pan, T\. Wu, J\. Yu, Y\. Zhang, T\. M\. Zhang, L\. Zhu, M\. A\. Merrill, J\. Heer, and T\. AlthoffBLADE: benchmarking language model agents for data\-driven science\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 13936–13971\.External Links:[Document](https://dx.doi.org/10.18653/V1/2024.FINDINGS-EMNLP.815)Cited by:[§2\.1](https://arxiv.org/html/2609.01982#S2.SS1.SSS0.Px2.p1.1)\.
- Huanget al\.\(2025\)C\. Huang, Z\. Tang, S\. Hu, R\. Jiang, X\. Zheng, D\. Ge, B\. Wang, and Z\. WangORLM: a customizable framework in training large models for automated optimization modeling\.Operations Research73\(6\),pp\. 2986–3009\.External Links:[Document](https://dx.doi.org/10.1287/opre.2024.1233)Cited by:[§2\.2](https://arxiv.org/html/2609.01982#S2.SS2.SSS0.Px1.p1.1)\.
- Huanget al\.\(2024\)Y\. Huang, J\. Luo, Y\. Yu, Y\. Zhang, F\. Lei, Y\. Wei, S\. He, L\. Huang, X\. Liu, J\. Zhao, and K\. LiuDA\-code: agent data science code generation benchmark for large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 13487–13521\.External Links:[Document](https://dx.doi.org/10.18653/V1/2025.EMNLP-MAIN.1196)Cited by:[§1](https://arxiv.org/html/2609.01982#S1.p1.1),[§1](https://arxiv.org/html/2609.01982#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.01982#S2.SS1.SSS0.Px2.p1.1)\.
- Jianget al\.\(2025\)C\. Jiang, X\. Shu, H\. Qian, X\. Lu, J\. Zhou, A\. Zhou, and Y\. YuLLMOPT: learning to define and solve general optimization problems from scratch\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 101580–101606\.Cited by:[§2\.2](https://arxiv.org/html/2609.01982#S2.SS2.SSS0.Px1.p1.1)\.
- Jinget al\.\(2025\)L\. Jing, Z\. Huang, X\. Wang, W\. Yao, W\. Yu, K\. Ma, H\. Zhang, X\. Du, and D\. YuDSBench: how far are data science agents from becoming data science experts?\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 32597–32649\.Cited by:[§2\.1](https://arxiv.org/html/2609.01982#S2.SS1.SSS0.Px2.p1.1)\.
- Khandekaret al\.\(2024\)N\. Khandekar, Q\. Jin, G\. Xiong, S\. Dunn, S\. S\. Applebaum, Z\. Anwar, M\. Sarfo\-Gyamfi, C\. W\. Safranek, A\. A\. Anwar, A\. Zhang, A\. Gilson, M\. B\. Singer, A\. Dave, A\. Taylor, A\. Zhang, Q\. Chen, and Z\. LuMedCalc\-Bench: evaluating large language models for medical calculations\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 84730–84745\.External Links:[Document](https://dx.doi.org/10.52202/079017-2690)Cited by:[§2\.2](https://arxiv.org/html/2609.01982#S2.SS2.SSS0.Px2.p1.1)\.
- Liuet al\.\(2025\)F\. Liu, Z\. Yang, C\. Liu, T\. Song, X\. Gao, and H\. LiuMM\-agent: llm as agents for real\-world mathematical modeling problem\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 20881–20934\.External Links:[Document](https://dx.doi.org/10.52202/085713-0704)Cited by:[§2\.2](https://arxiv.org/html/2609.01982#S2.SS2.SSS0.Px2.p1.1)\.
- Liuet al\.\(2024\)X\. Liu, Z\. Wu, X\. Wu, P\. Lu, K\. Chang, and Y\. FengAre LLMs capable of data\-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 9215–9235\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.548)Cited by:[§1](https://arxiv.org/html/2609.01982#S1.p1.1),[§1](https://arxiv.org/html/2609.01982#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.01982#S2.SS1.SSS0.Px1.p1.1)\.
- Luet al\.\(2026\)Y\. Lu, R\. Yang, Y\. Zhang, S\. Yu, Z\. Wang, J\. Xiang, W\. E, C\. Zhu, and F\. ZhouStatEval: a comprehensive benchmark for large language models in statistics\.External Links:2510\.09517,[Link](https://arxiv.org/abs/2510.09517)Cited by:[§1](https://arxiv.org/html/2609.01982#S1.p1.1),[§1](https://arxiv.org/html/2609.01982#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.01982#S2.SS1.SSS0.Px1.p1.1)\.
- Majumderet al\.\(2025\)B\. P\. Majumder, H\. Surana, D\. Agarwal, B\. Dalvi Mishra, A\. Meena, A\. Prakhar, T\. Vora, T\. Khot, A\. Sabharwal, and P\. ClarkDiscoveryBench: towards data\-driven discovery with large language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 4556–4579\.Cited by:[§2\.2](https://arxiv.org/html/2609.01982#S2.SS2.SSS0.Px2.p1.1)\.
- McClaveet al\.\(2013\)J\. T\. McClave, P\. G\. Benson, and T\. T\. SincichStatistics for business and economics\.12th edition,Pearson\.Cited by:[§4\.1](https://arxiv.org/html/2609.01982#S4.SS1.p1.1)\.
- OpenAIet al\.\(2024\)OpenAI, :, A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford, A\. Mądry, A\. Baker\-Whitcomb, A\. Beutel, A\. Borzunov, A\. Carney, A\. Chow, A\. Kirillov, A\. Nichol, A\. Paino, A\. Renzin, A\. T\. Passos, A\. Kirillov, A\. Christakis, A\. Conneau, A\. Kamali, A\. Jabri, A\. Moyer, A\. Tam, A\. Crookes, A\. Tootoochian, A\. Tootoonchian, A\. Kumar, A\. Vallone, A\. Karpathy, A\. Braunstein, A\. Cann, A\. Codispoti, A\. Galu, A\. Kondrich, A\. Tulloch, A\. Mishchenko, A\. Baek, A\. Jiang, A\. Pelisse, A\. Woodford, A\. Gosalia, A\. Dhar, A\. Pantuliano, A\. Nayak, A\. Oliver, B\. Zoph, B\. Ghorbani, B\. Leimberger, B\. Rossen, B\. Sokolowsky, B\. Wang, B\. Zweig, B\. Hoover, B\. Samic, B\. McGrew, B\. Spero, B\. Giertler, B\. Cheng, B\. Lightcap, B\. Walkin, B\. Quinn, B\. Guarraci, B\. Hsu, B\. Kellogg, B\. Eastman, C\. Lugaresi, C\. Wainwright, C\. Bassin, C\. Hudson, C\. Chu, C\. Nelson, C\. Li, C\. J\. Shern, C\. Conger, C\. Barette, C\. Voss, C\. Ding, C\. Lu, C\. Zhang, C\. Beaumont, C\. Hallacy, C\. Koch, C\. Gibson, C\. Kim, C\. Choi, C\. McLeavey, C\. Hesse, C\. Fischer, C\. Winter, C\. Czarnecki, C\. Jarvis, C\. Wei, C\. Koumouzelis, D\. Sherburn, D\. Kappler, D\. Levin, D\. Levy, D\. Carr, D\. Farhi, D\. Mely, D\. Robinson, D\. Sasaki, D\. Jin, D\. Valladares, D\. Tsipras, D\. Li, D\. P\. Nguyen, D\. Findlay, E\. Oiwoh, E\. Wong, E\. Asdar, E\. Proehl, E\. Yang, E\. Antonow, E\. Kramer, E\. Peterson, E\. Sigler, E\. Wallace, E\. Brevdo, E\. Mays, F\. Khorasani, F\. P\. Such, F\. Raso, F\. Zhang, F\. von Lohmann, F\. Sulit, G\. Goh, G\. Oden, G\. Salmon, G\. Starace, G\. Brockman, H\. Salman, H\. Bao, H\. Hu, H\. Wong, H\. Wang, H\. Schmidt, H\. Whitney, H\. Jun, H\. Kirchner, H\. P\. de Oliveira Pinto, H\. Ren, H\. Chang, H\. W\. Chung, I\. Kivlichan, I\. O’Connell, I\. O’Connell, I\. Osband, I\. Silber, I\. Sohl, I\. Okuyucu, I\. Lan, I\. Kostrikov, I\. Sutskever, I\. Kanitscheider, I\. Gulrajani, J\. Coxon, J\. Menick, J\. Pachocki, J\. Aung, J\. Betker, J\. Crooks, J\. Lennon, J\. Kiros, J\. Leike, J\. Park, J\. Kwon, J\. Phang, J\. Teplitz, J\. Wei, J\. Wolfe, J\. Chen, J\. Harris, J\. Varavva, J\. G\. Lee, J\. Shieh, J\. Lin, J\. Yu, J\. Weng, J\. Tang, J\. Yu, J\. Jang, J\. Q\. Candela, J\. Beutler, J\. Landers, J\. Parish, J\. Heidecke, J\. Schulman, J\. Lachman, J\. McKay, J\. Uesato, J\. Ward, J\. W\. Kim, J\. Huizinga, J\. Sitkin, J\. Kraaijeveld, J\. Gross, J\. Kaplan, J\. Snyder, J\. Achiam, J\. Jiao, J\. Lee, J\. Zhuang, J\. Harriman, K\. Fricke, K\. Hayashi, K\. Singhal, K\. Shi, K\. Karthik, K\. Wood, K\. Rimbach, K\. Hsu, K\. Nguyen, K\. Gu\-Lemberg, K\. Button, K\. Liu, K\. Howe, K\. Muthukumar, K\. Luther, L\. Ahmad, L\. Kai, L\. Itow, L\. Workman, L\. Pathak, L\. Chen, L\. Jing, L\. Guy, L\. Fedus, L\. Zhou, L\. Mamitsuka, L\. Weng, L\. McCallum, L\. Held, L\. Ouyang, L\. Feuvrier, L\. Zhang, L\. Kondraciuk, L\. Kaiser, L\. Hewitt, L\. Metz, L\. Doshi, M\. Aflak, M\. Simens, M\. Boyd, M\. Thompson, M\. Dukhan, M\. Chen, M\. Gray, M\. Hudnall, M\. Zhang, M\. Aljubeh, M\. Litwin, M\. Zeng, M\. Johnson, M\. Shetty, M\. Gupta, M\. Shah, M\. Yatbaz, M\. J\. Yang, M\. Zhong, M\. Glaese, M\. Chen, M\. Janner, M\. Lampe, M\. Petrov, M\. Wu, M\. Wang, M\. Fradin, M\. Pokrass, M\. Castro, M\. O\. T\. de Castro, M\. Pavlov, M\. Brundage, M\. Wang, M\. Khan, M\. Murati, M\. Bavarian, M\. Lin, M\. Yesildal, N\. Soto, N\. Gimelshein, N\. Cone, N\. Staudacher, N\. Summers, N\. LaFontaine, N\. Chowdhury, N\. Ryder, N\. Stathas, N\. Turley, N\. Tezak, N\. Felix, N\. Kudige, N\. Keskar, N\. Deutsch, N\. Bundick, N\. Puckett, O\. Nachum, O\. Okelola, O\. Boiko, O\. Murk, O\. Jaffe, O\. Watkins, O\. Godement, O\. Campbell\-Moore, P\. Chao, P\. McMillan, P\. Belov, P\. Su, P\. Bak, P\. Bakkum, P\. Deng, P\. Dolan, P\. Hoeschele, P\. Welinder, P\. Tillet, P\. Pronin, P\. Tillet, P\. Dhariwal, Q\. Yuan, R\. Dias, R\. Lim, R\. Arora, R\. Troll, R\. Lin, R\. G\. Lopes, R\. Puri, R\. Miyara, R\. Leike, R\. Gaubert, R\. Zamani, R\. Wang, R\. Donnelly, R\. Honsby, R\. Smith, R\. Sahai, R\. Ramchandani, R\. Huet, R\. Carmichael, R\. Zellers, R\. Chen, R\. Chen, R\. Nigmatullin, R\. Cheu, S\. Jain, S\. Altman, S\. Schoenholz, S\. Toizer, S\. Miserendino, S\. Agarwal, S\. Culver, S\. Ethersmith, S\. Gray, S\. Grove, S\. Metzger, S\. Hermani, S\. Jain, S\. Zhao, S\. Wu, S\. Jomoto, S\. Wu, Shuaiqi, Xia, S\. Phene, S\. Papay, S\. Narayanan, S\. Coffey, S\. Lee, S\. Hall, S\. Balaji, T\. Broda, T\. Stramer, T\. Xu, T\. Gogineni, T\. Christianson, T\. Sanders, T\. Patwardhan, T\. Cunninghman, T\. Degry, T\. Dimson, T\. Raoux, T\. Shadwell, T\. Zheng, T\. Underwood, T\. Markov, T\. Sherbakov, T\. Rubin, T\. Stasi, T\. Kaftan, T\. Heywood, T\. Peterson, T\. Walters, T\. Eloundou, V\. Qi, V\. Moeller, V\. Monaco, V\. Kuo, V\. Fomenko, W\. Chang, W\. Zheng, W\. Zhou, W\. Manassra, W\. Sheu, W\. Zaremba, Y\. Patil, Y\. Qian, Y\. Kim, Y\. Cheng, Y\. Zhang, Y\. He, Y\. Zhang, Y\. Jin, Y\. Dai, and Y\. MalkovGPT\-4o system card\.External Links:2410\.21276,[Link](https://arxiv.org/abs/2410.21276)Cited by:[§4\.2](https://arxiv.org/html/2609.01982#S4.SS2.SSS0.Px3.p1.1)\.
- OpenAI \(2026a\)OpenAIIntroducing gpt‑5\.4: designed for professional work\.Note:[https://openai\.com/index/introducing\-gpt\-5\-4/](https://openai.com/index/introducing-gpt-5-4/)Accessed: 2026\-05\-26Cited by:[§6\.1](https://arxiv.org/html/2609.01982#S6.SS1.p1.1)\.
- OpenAI \(2026b\)OpenAIIntroducing gpt‑5\.5: a new class of intelligence for real work\.Note:[https://openai\.com/index/introducing\-gpt\-5\-5/](https://openai.com/index/introducing-gpt-5-5/)Accessed: 2026\-05\-26Cited by:[§6\.1](https://arxiv.org/html/2609.01982#S6.SS1.p1.1)\.
- Pham \(2022\)H\. PhamStatistical reliability engineering: methods, models and applications\.Springer Series in Reliability Engineering,Springer\.Cited by:[§4\.1](https://arxiv.org/html/2609.01982#S4.SS1.p1.1)\.
- Qianet al\.\(2025\)C\. Qian, H\. Du, H\. Wang, X\. Chen, Y\. Zhang, A\. Sil, C\. Zhai, K\. McKeown, and H\. JiModelingAgent: bridging llms and mathematical modeling for real\-world challenges\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 1599–1633\.External Links:[Document](https://dx.doi.org/10.18653/V1/2025.FINDINGS-EMNLP.85)Cited by:[§2\.2](https://arxiv.org/html/2609.01982#S2.SS2.SSS0.Px2.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.Note:https://qwen\.ai/blog?id=qwen3\.5Cited by:[§6\.1](https://arxiv.org/html/2609.01982#S6.SS1.p1.1)\.
- Ramamonjisonet al\.\(2022\)R\. Ramamonjison, T\. T\. L\. Yu, R\. Li, H\. Li, G\. Carenini, B\. Ghaddar, S\. He, M\. Mostajabdaveh, A\. Banitalebi\-Dehkordi, Z\. Zhou, and Y\. ZhangNL4Opt competition: formulating optimization problems based on their natural language descriptions\.InNeurIPS 2022 Competition Track,Proceedings of Machine Learning Research, Vol\.220,pp\. 189–203\.Cited by:[§2\.2](https://arxiv.org/html/2609.01982#S2.SS2.SSS0.Px1.p1.1)\.
- Suet al\.\(2025\)R\. Su, J\. Si, Z\. Guo, and J\. B\. PierrehumbertClimateViz: A benchmark for statistical reasoning and fact verification on scientific charts\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 23436–23458\.External Links:[Document](https://dx.doi.org/10.18653/V1/2024.FINDINGS-EMNLP.815)Cited by:[§1](https://arxiv.org/html/2609.01982#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.01982#S2.SS1.SSS0.Px1.p1.1)\.
- Teamet al\.\(2026\)K\. Team, T\. Bai, Y\. Bai, Y\. Bao, S\. H\. Cai, Y\. Cao, Z\. Chai, Y\. Charles, H\. S\. Che, C\. Chen, G\. Chen, H\. Chen, J\. Chen, J\. Chen, J\. Chen, K\. Chen, L\. Chen, R\. Chen, X\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Z\. Chen, Z\. Chen, D\. Cheng, Y\. Cheng, M\. Chu, J\. Cui, J\. Deng, M\. Diao, H\. Ding, M\. Dong, M\. Dong, Y\. Dong, Y\. Dong, A\. Du, C\. Du, D\. Du, L\. Du, Y\. Du, Y\. Fan, S\. Fang, Q\. Feng, Y\. Feng, G\. Fu, K\. Fu, H\. Gao, T\. Gao, Y\. Ge, S\. Geng, C\. Gong, X\. Gong, Z\. Gongque, Q\. Gu, X\. Gu, Y\. Gu, L\. Guan, S\. Guan, Y\. Guo, X\. Hao, D\. He, T\. He, W\. He, W\. He, Y\. He, Y\. He, C\. Hong, H\. Hu, J\. Hu, Y\. Hu, Z\. Hu, K\. Huang, R\. Huang, W\. Huang, Z\. Huang, C\. Jia, T\. Jiang, Z\. Jiang, X\. Jin, Y\. Jing, G\. Lai, A\. Li, C\. Li, C\. Li, F\. Li, G\. Li, G\. Li, H\. Li, H\. Li, J\. Li, J\. Li, J\. Li, L\. Li, M\. Li, W\. Li, W\. Li, X\. Li, X\. Li, Y\. Li, Y\. Li, Y\. Li, Y\. Li, Z\. Li, Z\. Li, Z\. Li, W\. Liao, J\. Lin, X\. Lin, Y\. Lin, Z\. Lin, Z\. Lin, C\. Liu, C\. Liu, H\. Liu, L\. Liu, S\. Liu, S\. Liu, S\. Liu, T\. Liu, T\. Liu, W\. Liu, X\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Z\. Liu, Z\. Liu, E\. Lu, H\. Lu, Z\. Lu, G\. Luo, J\. Luo, T\. Luo, Y\. Luo, L\. Ma, S\. Mao, Y\. Mei, X\. Men, F\. Meng, Z\. Meng, Y\. Miao, M\. Ni, K\. Ouyang, S\. Pan, B\. Pang, Y\. Qian, R\. Qin, Z\. Qin, J\. Qiu, B\. Qu, Z\. Shang, Y\. Shao, T\. Shen, Z\. Shen, J\. Shi, L\. Shi, S\. Shi, F\. Song, P\. Song, T\. Song, X\. Song, H\. Su, J\. Su, Z\. Su, L\. Sui, J\. Sun, J\. Sun, T\. Sun, F\. Sung, Y\. Tai, C\. Tang, H\. Tang, X\. Tang, Z\. Tang, J\. Tao, S\. Teng, C\. Tian, P\. Tian, B\. Wang, C\. Wang, C\. Wang, C\. Wang, D\. Wang, D\. Wang, D\. Wang, F\. Wang, H\. Wang, H\. Wang, H\. Wang, H\. Wang, H\. Wang, H\. Wang, J\. Wang, J\. Wang, J\. Wang, K\. Wang, L\. Wang, Q\. Wang, S\. Wang, S\. Wang, S\. Wang, W\. Wang, X\. Wang, X\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, C\. Wei, M\. Wei, C\. Wen, Z\. Wen, C\. Wu, H\. Wu, J\. Wu, R\. Wu, W\. Wu, Y\. Wu, Y\. Wu, Y\. Wu, Z\. Wu, C\. Xiao, J\. Xie, X\. Xie, Y\. Xie, B\. Xing, B\. Xu, J\. Xu, J\. Xu, J\. Xu, L\. H\. Xu, L\. Xu, S\. Xu, W\. Xu, X\. Xu, X\. Xu, Y\. Xu, Y\. Xu, Y\. Xu, Z\. Xu, Z\. Xu, J\. Yan, Y\. Yan, G\. Yang, H\. Yang, J\. Yang, K\. Yang, N\. Yang, X\. Yang, X\. Yang, X\. Yang, Y\. Yang, Y\. Yang, Y\. Yang, Z\. Yang, Z\. Yang, Z\. Yang, H\. Yao, D\. Ye, H\. Ye, W\. Ye, Z\. Ye, P\. Yebo, B\. Yin, C\. Yu, L\. Yu, T\. Yu, T\. Yu, E\. Yuan, M\. Yuan, X\. Yuan, Y\. Yue, W\. Zeng, D\. Zha, H\. Zhan, D\. Zhang, H\. Zhang, J\. Zhang, P\. Zhang, Q\. Zhang, R\. Zhang, X\. Zhang, X\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Z\. Zhang, C\. Zhao, F\. Zhao, J\. Zhao, S\. Zhao, X\. Zhao, X\. Zhao, Y\. Zhao, Z\. Zhao, H\. Zheng, R\. Zheng, S\. Zheng, T\. Zheng, J\. Zhong, L\. Zhong, W\. Zhong, M\. Zhou, R\. Zhou, X\. Zhou, Z\. Zhou, J\. Zhu, L\. Zhu, X\. Zhu, Y\. Zhu, Z\. Zhu, J\. Zhuang, W\. Zhuang, Y\. Zou, and X\. ZuKimi k2\.5: visual agentic intelligence\.External Links:2602\.02276,[Link](https://arxiv.org/abs/2602.02276)Cited by:[§6\.1](https://arxiv.org/html/2609.01982#S6.SS1.p1.1)\.
- Wanget al\.\(2025\)B\. Wang, I\. Xia, Y\. Zhang, J\. Wang, F\. Ouyang, S\. Han, A\. Cohan, H\. Yu, and Z\. YaoFrom scores to steps: diagnosing and improving LLM performance in evidence\-based medical calculations\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 10809–10833\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.548)Cited by:[§2\.2](https://arxiv.org/html/2609.01982#S2.SS2.SSS0.Px2.p1.1)\.
- Xiaoet al\.\(2024\)Z\. Xiao, D\. Zhang, Y\. Wu, L\. Xu, Y\. Wang, X\. Han, X\. Fu, T\. Zhong, J\. Zeng, M\. Song, and G\. ChenChain\-of\-experts: when llms meet complex operations research problems\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 48519–48537\.Cited by:[§2\.2](https://arxiv.org/html/2609.01982#S2.SS2.SSS0.Px1.p1.1)\.
- Yanet al\.\(2025\)S\. Yan, R\. Li, Z\. Luo, Z\. Wang, D\. Li, L\. Jing, K\. He, P\. Wu, J\. Ni, G\. Michalopoulos, Y\. Zhang, Z\. Zhang, M\. Zhang, Z\. Chen, and X\. DuLMR\-BENCH: evaluating LLM agent’s ability on reproducing language modeling research\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 6164–6186\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.314)Cited by:[§2\.1](https://arxiv.org/html/2609.01982#S2.SS1.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025\)Z\. Yang, Y\. Wang, Y\. Huang, Z\. Guo, S\. Shi, X\. Han, L\. Feng, L\. Song, X\. Liang, and J\. TangOptiBench meets resocratic: measure and improve llms for optimization modeling\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 24726–24759\.Cited by:[§2\.2](https://arxiv.org/html/2609.01982#S2.SS2.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025\)D\. Zhang, S\. Zhoubian, M\. Cai, F\. Li, L\. Yang, W\. Wang, T\. Dong, Z\. Hu, J\. Tang, and Y\. YueDataSciBench: an llm agent benchmark for data science\.External Links:2502\.13897,[Link](https://arxiv.org/abs/2502.13897)Cited by:[§1](https://arxiv.org/html/2609.01982#S1.p1.1),[§1](https://arxiv.org/html/2609.01982#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.01982#S2.SS1.SSS0.Px2.p1.1)\.
- Zhuet al\.\(2024\)Y\. Zhu, S\. Du, B\. Li, Y\. Luo, and N\. TangAre large language models good statisticians?\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 62697–62731\.External Links:[Document](https://dx.doi.org/10.52202/079017-2005)Cited by:[§2\.1](https://arxiv.org/html/2609.01982#S2.SS1.SSS0.Px1.p1.1)\.

## Appendix AData Processing and Annotation Details

We provide implementation details for the pipeline stages described in Section[4\.2](https://arxiv.org/html/2609.01982#S4.SS2)\. Prompt templates for each LLM\-powered step are collected in Appendix[C](https://arxiv.org/html/2609.01982#A3)\.

### A\.1Textbook Processing

#### Sample extraction\.

We use MinerU as the initial PDF parser, which emits structured outputs for body text, equations, tables, and figures\. Annotators then manually correct the parser output for each problem, including fixing OCR errors in equations, replacing rasterized math with LaTeX, and transcribing tables that the parser misses\. For each problem they produce a JSON entry containing the question text, reference answer, data \(tabular, symbolic, or figure\), and metadata \(problem index, chapter/section identifiers, page range\)\. The structured triple\(Qo,Ao,D\)\(Q^\{o\},A^\{o\},D\)is directly extracted from this entry\. Worked examples and end\-of\-section exercises share the same protocol, except that for the latter the reference answer typically lives in a separate solutions appendix\. All entries undergo a final proofreading pass before entering the next stage\.

#### Sub\-question splitting\.

Once extraction is complete, an LLM \(gpt\-4o\) identifies the shared scenario in each problem and decomposes it into a backgroundB~\\tilde\{B\}together with individual\(Q~,A~\)\(\\tilde\{Q\},\\tilde\{A\}\)pairs\. A deterministic flattening step then turns each pair into an independent sample that inheritsB~\\tilde\{B\}andDD, ensuring that every downstream sample carries exactly one analytical question\.

#### Quality filtering\.

Each split sample is then evaluated by an LLM \(gpt\-5\.4\) against a set of exclusion criteria, producing a binary retain\-or\-discard label\. Discarded samples include problems too basic to test statistical reasoning \(such as looking up a single datum or pure concept recall without computation, though graphing and numerical calculation problems are retained\), problems with incomplete information \(e\.g\., those depending on an earlier problem’s result\), and problems whose reference answer is empty\.

#### Initial label construction\.

For retained samples, the annotator LLM \(claude\-opus\-4\-6\) produces initial labels in two stages\. In the category labeling stage, the model selects a category inL1\\L2format from the current taxonomy, or proposes a new category when none fits\. In the subsequent variable extraction stage, it produces variable labels in the schema\{id,value,class,role,description\}\\\{\\textit\{id\},\\textit\{value\},\\textit\{class\},\\textit\{role\},\\textit\{description\}\\\}, grounding each variable in the solving steps ofA~\\tilde\{A\}\. These model\-generated labels serve only as initial candidates and are subsequently verified by human reviewers as described in Section[4\.3](https://arxiv.org/html/2609.01982#S4.SS3)\.

#### Scenario rewriting\.

Finally,gpt\-5rewrites\(B~,Q~\)\(\\tilde\{B\},\\tilde\{Q\}\)into\(B,Q\)\(B,Q\), rephrasing statistical terminology into intuitive language, switching to a first\-person collective voice, and preserving all numerical values\. A self\-evaluation prompt then checks whether the rewrite drifts from the original intent and triggers a one\-shot correction when it does\.

### A\.2Case\-Library Adaptations

Compared with the textbook subset, the case\-library subset differs only in the sample extraction stage, as the raw materials arrive in a fundamentally different form\.

#### Project selection and data simulation\.

The source collection of cases from GouXiongHui contains 142 data\-analysis projects with reports, code, and data in heterogeneous formats\. We use only projects that include a complete analysis report and at least one data file that can be converted into a structured tabular representation\. Image, audio, and other unstructured media files are then excluded, while the remaining tabular, serialized, and text data are converted intopandas\.DataFrameobjects\. Each data frame is then summarized into a data dictionary containing the table shape, column names, data types, per\-column descriptive statistics, missing\-value counts, and a five\-row preview\. To protect the data provider’s copyright, we further replace the original five\-row preview with simulated rows generated by GPT\-5 under constraints that preserve each field’s data type, value range, and distributional characteristics\.

#### Question induction and annotation\.

For each retained project, the HTML analysis report and data dictionary are provided togpt\-4o\-mini, which produces a case overview and a set of independent analysis questions\. Each question is paired with a report\-grounded answer and the variables needed for analysis\. A secondgpt\-4o\-minicall checks whether the questions are of moderate difficulty, mutually independent, and supported by the report\. Outputs that fail these criteria are revised using the model’s feedback for up to five rounds\. After a question passes this check, separategpt\-4ocalls produce its initial category and variable annotations\. This process directly produces independent\(B~,Q~,A~\)\(\\tilde\{B\},\\tilde\{Q\},\\tilde\{A\}\)tuples, making sub\-question splitting unnecessary for the case\-library subset\.

## Appendix BStatistical Problem Taxonomy

The taxonomy contains2020first\-level and8585second\-level categories, designed by jointly considering canonical statistical problem types from standard practice and problem types that surface in our collected samples\. Table[5](https://arxiv.org/html/2609.01982#A2.T5)lists all categories with brief descriptions derived from the operational definitions used in annotation\. We do not claim that the taxonomy exhausts every statistical problem type, and we expect it to grow as additional data sources are incorporated\.

Table 5:Complete statistical problem taxonomy \(2020first\-level,8585second\-level categories\)\.†L2 categories under Hypothesis Testing also cover the corresponding interval\-estimation tasks\.
## Appendix CPrompt Templates

We present the prompt templates used in each LLM\-powered pipeline step \(Section[4\.2](https://arxiv.org/html/2609.01982#S4.SS2)\), listed in pipeline order\.

Quality Filtering Prompt \(gpt\-5\.4\)You are a professional statistics instructor\. Judge whether the following exercise is suitable for testing statistical proficiency\.Input:\{background, data, question, answer\}Judge the exercise asunsuitable\(judge=0\) if any of the following conditions applies:•Overly basic:the exercise requires only a single data lookup or pure concept recall without computation\. Retain graphing and numerical calculation problems\.•Incomplete information:the description is ambiguous, or the question depends on a preceding problem’s result that is not provided\.•Empty answer:theanswerfield is blank\.Note: tables referenced inquestionmay appear as LaTeX code indata\. This does not count as missing information\.Output:\{"judge": 1/0, "explanation": "\.\.\."\}Category Annotation Prompt \(claude\-opus\-4\-6\)You are a strict statistical problem category annotator\. Given a sample’sbackground,data, andquestion, select the most appropriate category label from the statistical problem taxonomy below\.Requirements:1\.If a fitting L2 category exists, write it strictly as “L1\\L2” insuggested\_category\.2\.If no L2 category fits, fillproposed\_new\_categoryinstead and leavesuggested\_categoryempty\.3\.reasonmust explain the choice or why a new category is needed\.Statistical Problem Taxonomy\(20 L1 and 85 L2 categories, abridged\):
•Data Preprocessing: \[Missing Value Handling, Outlier Handling, Standardization, Binning, Transformation\]
•Mathematical Calculation: \[Probability Space, Event Probability & Independence, Expectation, Min\. Sample Size for Target Probability, Distribution Derivation, Other\]
•Hypothesis Testing & Interval Estimation: \[Independence Test, Mean Test, Variance Test, ANOVA, Distribution Test, Proportion Test, Likelihood Ratio Test, Sequential Test, Randomness, Regression Parameter Test, Test Properties\]
… \(The remaining 17 L1 categories are listed in Appendix[B](https://arxiv.org/html/2609.01982#A2)\)
Each L2 category is accompanied by a one\-sentence definition in the actual prompt\.Input:\{background, data, question\}
Output:\{"suggested\_category": "\.\.\.", "proposed\_new\_category": "\.\.\.", "reason": "\.\.\."\}Variable Extraction Prompt \(claude\-opus\-4\-6\)You are a strict statistical annotator\. Extract the variable objects that theansweractually usesin its solving process\.Key requirements\(8 rules\):
1\. Variables are data objects required to answer the research question, including raw columns, sample sizes, means, SDs, and contingency tables\.Excludemodel parameters and test statistics \(zz,pp,tt,FF\)\.
2\. Ground extraction in the answer’s actual reasoning\. Omit variables the answer never uses, and include variables that appear symbolically inquestionoranswereven whendatais empty\.
3\. For grouped data, create one variable object per group\.
4\. For per\-variable summary statistics, store aggregates in thevaluefield as JSON\.
5\. Variable objects include raw columns, grouped statistics, contingency tables, sample sizes, and success counts\. When the answer constructs a derived variable from multiple originals, retain all originals and omit the derived one\.
6\. For “Mathematical Calculation” problems, probabilities, sample sizes, and other numerical objects used in the answer are also treated as variable objects\.
7\. If the answer’s solving process is missing or overly brief, infer the minimal variable set needed to reach the final answer\. If no variables are needed, output empty JSON\.
8\. Output must be valid JSON with variableidvalues as top\-level keys\. Do not include extra text\.Fields per variable:
id: A variable identifier\.
value: Store a table column as an array, aggregated measures as JSON, and any other value in its original recorded form\. When multiple numerical values are present, preserve their order in the source problem\.
class: One ofnumerical,categorical, orothers\.
role: One ofpredictor,response,both, orN/A\.
description: A brief description of the variable\.
Roles:predictordenotes a predictor variable,responsedenotes a response variable, andbothdenotes an undirected association or a dual role\.N/Aindicates that no directional role is needed, as with identifiers, filters, unsupervised features, and single\-variable distributions\. Special\-case rules for time\-trend problems and variables serving dual construction roles are included in the full prompt\.Input:\{background, data, question, answer\}
Output:\{"VAR": \{"id", "value", "class", "role", "description"\}, …\}Scenario Rewriting Prompt \(gpt\-5\)You are a statistician\. Rewrite thebackgroundandquestionof the given statistical problem to create a more realistic formulation challenge from a domain practitioner’s perspective\.Requirements:•Use business language from the relevant domain\. Do not use statistical terminology or explicitly state the target statistical problem category\.•Switch to a first\-person collective voice \(“we”\)\.•Preserve all numerical values, named entities, and table/figure labels\.•The rewritten problem must still map to the same formalized statistical model\.•The wording may be appropriately vague, and distracting context may be added\.The full statistical problem taxonomy and variable specification used in the two prompts above are also provided so the model can verify that the mapping is preserved\. They are omitted here for brevity\. Two worked examples demonstrating the rewriting style are included in the actual prompt\.Input:\{background, data, question\}\+ target statistical model
Output:\{"background": "\.\.\.", "question": "\.\.\."\}Rewrite Self\-Evaluation Prompt \(gpt\-5\)You are an expert evaluator of problem reformulation\. Given the original problem, its formal statistical model, the derivation reasoning, and the rewritten problem, check whether the rewrite meets all of the following standards:1\.Numerical values in the original are preserved\. No unauthorized new data is introduced\.2\.The analytical direction of the original problem is maintained\. No new problem is created\.3\.Table and figure labels are unchanged\.4\.The rewritten question does not explicitly state the target statistical problem category\.5\.The rewritten problem still matches the given formal statistical model\.Input:original problem, statistical model, derivation reasoning, rewritten problem
Output:\{"pass": true/false, "feedback": "\.\.\."\}Statistical Problem Formulation PromptYou are a professional statistician\. Abstract the statistical formulation of the following problem\. The input format is:\{"background": "Relevant description of the problem context \(which may also contain some data\)","data": "The data used, presented as a table in LaTeX code","question": "The core problem"\}The statistical formulation includes the following two parts:1\. Statistical problem category\{statistical problem taxonomy\}2\. Relevant variables and their rolesKey requirements:1\.Variables refer specifically to data objects collected for the research objective \(e\.g\., sample size, mean, standard deviation\), excluding model parameters, z\-scores, p\-values, t\-values, or F\-values\.2\.Variable objects can be raw data columns, grouped statistics, contingency tables, sample sizes, success counts, or time variables\.3\.For "Mathematical Calculation" problems, treat numerical objects \(probabilities, sample sizes\) used in the answer as variables\.4\.If there is no explicit data table or ifdatais empty, variables must still be extracted when symbolic variables or statistics appear\.5\.When the data have a grouping structure, split them into multiple variables by group\.6\.If aggregated measures are provided for multiple variables, create a separate variable object for each, writing measures into thevaluefield as JSON\.7\.If the information is insufficient, output an empty JSON object:\{\}\.8\.Output must be valid JSON with variable IDs as top\-level keys\. Do not include extra text\.Field specifications:•Top\-level key: Variable name \(extracted or assigned\)\.•id: Must match top\-level key\.•value: An array for table columns, JSON for aggregated measures, or the original form \(e\.g\., a TeX table\)\. When multiple numerical values are present, preserve their order in the source problem\.•class: One ofnumerical,categorical,others\.•role: One ofpredictor,response,both, orN/A\. Usebothfor an undirected association or a dual role\. UseN/Awhen no directional role is needed, as with identifiers, filters, unsupervised methods, and distribution descriptions\.•description: Brief description\.Specialrolecases:•For "Trends Over Time" visualization, assignpredictorto the time variable andresponseto the changing variable\.•If a variable serves as a predictor and is also used to construct the response, assignpredictor\.•If a variable serves as a response and is also used to construct the predictor, assignresponse\.Note: Shallow correlation analyses typically use "Data Visualization\." Deeper analyses of direction, magnitude, or significance use modeling and interpretation\. Time series forecasting uses "Continuous Value Prediction\." "Time Series Analysis" focuses on interpretation\.Output format: JSON\. Example:\{"category": "Relationship Between Continuous Variables","variables": \{"height": \{"id": "height","value": \{"μ\\mu": 138, "σ\\sigma": 7\},"class": "numerical","role": "N/A","description": "Heights of 10\-year\-old boys modeled as Normal\(μ\\mu=138 cm,σ\\sigma=7 cm\)\."\},"threshold": \{"id": "threshold","value": 150,"class": "numerical","role": "N/A","description": "Cutoff value \(150 cm\) used in the z\-score calculation z=\(150\-138\)/7\."\}\}\}
## Appendix DAdditional Dataset Statistics

Figure[7](https://arxiv.org/html/2609.01982#A4.F7)complements the dataset overview in Figure[3](https://arxiv.org/html/2609.01982#S3.F3)by showing the fine\-grained composition of the most frequent coarse\-grained categories\.

Figure 7:Category distribution at both taxonomy levels\. The inner ring shows coarse\-grained categories, with those containing no more than 20 samples grouped as Others\. The outer ring shows the fine\-grained composition of each displayed coarse\-grained category\.
## Appendix ESupplementary Results

### E\.1Exact\-Value Matching Implementation and Validation

#### Matching procedure\.

As specified in the variable extraction prompt above, thevaluefield records the complete contents of a variable under a standardized schema\. We use this field as the matching key so that synonymous variable names or descriptions do not affect the score\. The predicted and referencevaluefields are serialized, and a pair is matched only when the resulting strings are identical\. No additional normalization or semantic matching is applied, which keeps the evaluation rule deterministic and reproducible\. Each predicted and reference variable can appear in at most one matched pair, with unmatched predictions and reference variables counted as false positives and false negatives, respectively\. Consequently, per\-sample JCV equals one only when every predicted and reference variable can be paired through exact value equality\.

#### Validation study\.

To quantify how often exact\-value matching rejects semantically correct correspondences, we conduct a targeted validation study on a challenging subset\. We define a matching\-induced error as an output in which exact\-value matching rejects at least one semantically correct variable correspondence\. Such an error occurs when a correctly identified variable is treated as unmatched solely because the prediction uses a different variable decomposition or an equivalent value representation\. For example, a model may represent a sample size and a mean as separate variables while the reference stores both within a single variable entry, or represent a percentage as50rather than50%\. The challenging subset is defined using three representative closed\-source models: Claude Opus 4\.6, GPT\-5\.5, and Gemini 3\.1 Pro\. It contains 387 of the 629 textbook samples on which all three models have per\-sample JCV below one\. We randomly sample 60 cases from this subset, and a PhD\-level expert compares each model’s predicted variable set with the reference label to determine whether any semantically correct correspondence is rejected\.

Table 6:Matching\-induced error rates on the validation subset\.
#### Results\.

As shown in Table[6](https://arxiv.org/html/2609.01982#A5.T6), only about 10% of the sampled outputs for each model contain a matching\-induced error, even in this challenging subset\. The results suggest that although exact\-value matching can underestimate variable\-identification performance to some extent, it does not account for the substantially lower JCV scores in the main experiments\.

### E\.2Prompting Strategies

Figure[8](https://arxiv.org/html/2609.01982#A5.F8)supplements the few\-shot analysis in Section[6\.3](https://arxiv.org/html/2609.01982#S6.SS3)by reporting performance on all six evaluation metrics under the same experimental setting\. The complete comparison leads to the same conclusion that adding examples to the prompt does not yield consistent gains across the two evaluated tasks\.

Figure 8:Performance across prompting strategies on all six evaluation metrics\.
### E\.3Full Comparisons by Data Source

Table[7](https://arxiv.org/html/2609.01982#A5.T7)extends the source\-based analysis to ten models and all six metrics under the same evaluation setting\. Consistent with the pattern observed in Section[6\.3](https://arxiv.org/html/2609.01982#S6.SS3), classification accuracy is higher on textbook samples, whereas variable\-related scores are higher on case\-library samples\.

Table 7:Model performance across six metrics for each data source\.
### E\.4All Results of Classification Accuracy by Category

Figure[9](https://arxiv.org/html/2609.01982#A5.F9)complements the category\-level analysis with results for all categories containing more than 10 samples across ten models under the same evaluation setting\. In line with the findings in Section[6\.3](https://arxiv.org/html/2609.01982#S6.SS3), performance is higher when the analytical objective is explicit and lower when classification depends on subtle distinctions between related analytical intents\.

Figure 9:Classification accuracy for all categories with more than 10 samples\. Top: coarse\-grained categories\. Middle and bottom: fine\-grained categories\.