Error as a Lens: Probing LLM Reasoning through Synthetic Misconception Generation

# Probing LLM Reasoning through Synthetic Misconception Generation
Source: [https://arxiv.org/html/2605.29007](https://arxiv.org/html/2605.29007)
###### Abstract

Personalized tutoring, teacher training, and education research need access to*targeted*synthetic misconceptions, but privacy and IRB constraints make labelled corpora of real student errors scarce\. LLMs could in principle generate synthetic errors at scale, but producing an arbitrary wrong answer is easy for a modern LLM while producing one that matches a*specified*cognitive failure mode is much harder\. We present a framework that generates errors targeted to a five\-class taxonomy adapted from the revised Bloom’s taxonomy, evaluated on questions from the TheoremQA dataset\. A Generation Agent \(GA\) drafts a candidate erroneous solution conditioned on a target class, and an Examination Agent \(EA\) judges whether the draft is incorrect and class\-consistent\. The framework yields a reusable recipe for building class\-stratified synthetic error datasets where authentic student corpora are unavailable\. As a secondary diagnostic, targeted error generation is substantially harder than free\-form incorrect\-answer generation, and answer\-grounding contributes more than expanded examples or external textbook content\.

Error as a Lens: Probing LLM Reasoning through Synthetic Misconception Generation

Xinming YangCUNY Graduate CenterJun LiCUNY Queens College & CUNY Graduate Center

## 1Introduction

Educational technology depends on access to*targeted*misconceptions\. Personalized tutoring systems, whose effectiveness relies on calibrating practice to the specific weaknesses of a learner\(VanLehn,[2011](https://arxiv.org/html/2605.29007#bib.bib22)\), need worked\-out examples of the mistakes a student is likely to make so that practice can be directed at weak areas rather than already\-mastered material\. Teacher\-preparation programs use catalogs of common misconceptions to train instructors to recognize them in student work, and empirical evidence indicates that teachers’ ability to identify likely student misconceptions is a stronger predictor of classroom learning gains than knowledge of the correct answer alone\(Sadleret al\.,[2013](https://arxiv.org/html/2605.29007#bib.bib30)\)\. Education researchers studying how misconceptions form, persist, and are repaired need controlled collections of errors stratified by cognitive type\(Smithet al\.,[1993](https://arxiv.org/html/2605.29007#bib.bib28); Andersonet al\.,[2001](https://arxiv.org/html/2605.29007#bib.bib4)\)\. Exam\-item designers likewise need distractor options that correspond to plausible student reasoning rather than arbitrary wrong answers, a property without which standard psychometric models systematically misestimate student understanding\(Sadler,[1998](https://arxiv.org/html/2605.29007#bib.bib29)\)\.

The bottleneck shared by all of these uses is data\. Authentic student errors at the scale needed to train or evaluate downstream tools are extraordinarily difficult to obtain\. Privacy regulations, IRB constraints, and the ethical sensitivity of storing identifiable assessment data combine to make large labeled corpora of student mistakes scarce in nearly every subject: learning\-analytics work has repeatedly identified consent, privacy, de\-identification, data governance, and disclosure constraints as central barriers to using student records at scale\(Slade and Prinsloo,[2013](https://arxiv.org/html/2605.29007#bib.bib23); Pardo and Siemens,[2014](https://arxiv.org/html/2605.29007#bib.bib24); U\.S\. Department of Education, Student Privacy Policy Office,[2026](https://arxiv.org/html/2605.29007#bib.bib25)\)\. More recent surveys of open learning\-analytics datasets likewise find that publicly released educational datasets remain the exception rather than the norm and are unevenly distributed across subject areas\(Švábenskýet al\.,[2026](https://arxiv.org/html/2605.29007#bib.bib26)\)\. More importantly for this paper, even available corpora rarely annotate the cognitive mechanism behind a wrong answer,i\.e\., whether it reflects a lapse of attention, a missing concept, an inappropriate solution procedure, or a structural misframing\(Andersonet al\.,[2001](https://arxiv.org/html/2605.29007#bib.bib4); Brown and Burton,[1978](https://arxiv.org/html/2605.29007#bib.bib5); VanLehn,[1990](https://arxiv.org/html/2605.29007#bib.bib13)\)\.

*Synthetic student errors*therefore offer a scalable path for many of these applications, analogous to prior uses of synthetic educational data as a privacy\-preserving substitute for restricted student records\(Vieet al\.,[2022](https://arxiv.org/html/2605.29007#bib.bib27)\)\. A practical synthetic\-error generator must, at minimum, produce incorrect answers that are \(i\) genuinely wrong, \(ii\) consistent with a*specified*cognitive failure mode rather than arbitrarily incorrect, and \(iii\) stable across subjects so the same recipe can be reused beyond the one curriculum it was prototyped on\. The central empirical question is whether modern LLMs can satisfy all three constraints: authentic student errors are not random, but reflect identifiable cognitive deficits whose surface forms are subtle\.

This paper presents a framework that meets those three constraints and uses it as a measurement instrument\. We adopt a five\-class error taxonomy adapted from the revised Bloom’s taxonomy\(Andersonet al\.,[2001](https://arxiv.org/html/2605.29007#bib.bib4)\)and build a two\-agent system in which a Generation Agent \(GA\) drafts a candidate erroneous solution conditioned on a target class and an Examination Agent \(EA\) judges whether the draft is incorrect and class\-consistent\. The implementation study varies three design choices: whether theGAsees the correct solution, whether failed attempts are fed back into the next generation, and whether class judgment is done by a prompted model or by a classifier fine\-tuned on post\-hoc human corrections\.

The framework yields a reusable recipe for building targeted\-error datasets in subjects where authentic\-student corpora are unavailable\. Along the way, we make a secondary observation: targeted error generation is substantially harder than uncontrolled wrong\-answer generation, and within a single backend several pipeline variants reach similar rates\. The design axis whose effect is largest and most consistent is theEAfeedback loop on top of answer\-grounded generation, especially on weaker backends; expanded few\-shot exemplars and external textbook excerpts do not yield a clear improvement on top of that\.

#### Contributions\.

This paper makes one primary methodological contribution and one secondary empirical observation:

1. 1\.A scalable framework for taxonomy\-targeted synthetic\-error generation\.We introduce a two\-agent pipeline \(GAas drafter,EAas judge\) that keeps the generator pre\-trained and moves human supervision, when used, to the judging side\. We release the framework, prompted\-EAand fine\-tuned\-EAvariants, and a 1,800\-record\(question, target class, response\)replication artefact\.
2. 2\.Secondary observation: targeted error generation is a controllability problem\.Eight configurations show the targeted errors are substantially harder than free\-form ones; that answer\-grounding theGAaccounts for most of the per\-backend quality gap; and that on a strong backend several pipelines cluster within a narrow band\. We position the targeted\-error rate as a complement to correctness\-based reasoning benchmarks\.

Taken together, the contributions advance a view of NLP progress in which evaluation methodology matters as much as benchmark accuracy: the targeted\-error generation rate exposes behavior that correctness\-only benchmarks miss, while the framework supplies a practical instrument for using that measurement to build downstream educational applications\.

## 2Related Work

#### LLM reasoning evaluation\.

Chain\-of\-thought prompting\(Weiet al\.,[2022](https://arxiv.org/html/2605.29007#bib.bib1)\), process\-reward modeling\(Lightmanet al\.,[2024](https://arxiv.org/html/2605.29007#bib.bib3); Uesatoet al\.,[2022](https://arxiv.org/html/2605.29007#bib.bib2)\), and self\-consistency decoding\(Wanget al\.,[2023b](https://arxiv.org/html/2605.29007#bib.bib12)\)have established that intermediate reasoning quality is partially decoupled from final answer accuracy: divergent chains can reach the same answer, and stepwise quality and final correctness can be scored separately\. Probing approaches \(faithfulness tests\(Turpinet al\.,[2023](https://arxiv.org/html/2605.29007#bib.bib10)\)and perturbation studies\(Mirzadehet al\.,[2025](https://arxiv.org/html/2605.29007#bib.bib11)\)\) ask whether models truly compute the intermediate steps they verbalize\. Our framing extends this line: rather than perturbing the input, we ask the model to perturb the*solution itself*in a controlled, taxonomy\-aligned way\.

#### Misconception detection and educational error analysis\.

Misconception detection has a long history in education and intelligent tutoring\(Brown and Burton,[1978](https://arxiv.org/html/2605.29007#bib.bib5); VanLehn,[1990](https://arxiv.org/html/2605.29007#bib.bib13)\)and more recently in NLP applications to math word problems\(Sunet al\.,[2025](https://arxiv.org/html/2605.29007#bib.bib6); Oteroet al\.,[2024](https://arxiv.org/html/2605.29007#bib.bib7)\), programming exercises\(Oliet al\.,[2024](https://arxiv.org/html/2605.29007#bib.bib8)\), physics problem solutions\(Kortemeyer,[2023](https://arxiv.org/html/2605.29007#bib.bib9)\), and broader scientific questions\(Wanget al\.,[2023a](https://arxiv.org/html/2605.29007#bib.bib14)\)\. Most existing work classifies pre\-existing student responses or learns to predict the next\-likely error; relatively little work studies whether LLMs can themselves*generate*categorically coherent student errors at scale\. Our framework is complementary:Sunet al\.\([2025](https://arxiv.org/html/2605.29007#bib.bib6)\),Oteroet al\.\([2024](https://arxiv.org/html/2605.29007#bib.bib7)\), andOliet al\.\([2024](https://arxiv.org/html/2605.29007#bib.bib8)\)take authentic student responses as input and classify or diagnose the underlying error\. None of these works produce new\(question, target class, response\)triples on demand for a specified failure mode; they assume the student response already exists\. We instead use a pretrainedGAas a substitute student under controlled grounding, relieving the corpus bottleneck\.

#### LLMs as judges and multi\-agent evaluation\.

LLM\-as\-judge methods\(Zhenget al\.,[2023](https://arxiv.org/html/2605.29007#bib.bib15)\)and self\-refinement loops\(Madaanet al\.,[2023](https://arxiv.org/html/2605.29007#bib.bib16)\)have demonstrated that auxiliary models can productively evaluate and revise primary\-model output\. OurGA/EApair is a lightweight, two\-agent instance of this idea specialized to error\-type judging, deliberately avoiding the complexity of large multi\-agent frameworks\(Wuet al\.,[2024](https://arxiv.org/html/2605.29007#bib.bib17); Honget al\.,[2024](https://arxiv.org/html/2605.29007#bib.bib18)\): two agents are the minimum needed to decouple drafting from judging, and we did not find additional agents necessary for the diagnostic question this paper addresses\.

#### Synthetic data and error datasets\.

Synthetic data construction has become standard for training mathematical reasoners\(Yuet al\.,[2024](https://arxiv.org/html/2605.29007#bib.bib19); Luoet al\.,[2023](https://arxiv.org/html/2605.29007#bib.bib20)\)and code models\(Luoet al\.,[2024](https://arxiv.org/html/2605.29007#bib.bib21)\)\. Educational error datasets, in contrast, are scarce: widely used benchmarks such as GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2605.29007#bib.bib34)\), TheoremQA\(Chenet al\.,[2023](https://arxiv.org/html/2605.29007#bib.bib32)\), and APPS\(Hendryckset al\.,[2021](https://arxiv.org/html/2605.29007#bib.bib33)\)provide only question–answer pairs with no annotation of how a wrong answer would fail, and process\-reward datasets such as PRM800K\(Lightmanet al\.,[2024](https://arxiv.org/html/2605.29007#bib.bib3)\)label reasoning steps as correct or incorrect without identifying the underlying cognitive error type\. We contribute a recipe for constructing datasets with controllable error\-class labels\.

## 3Error Taxonomy

We adopt the revised Bloom’s taxonomy of cognitive processes\(Andersonet al\.,[2001](https://arxiv.org/html/2605.29007#bib.bib4)\),i\.e\.,remember,understand,apply,analyze,evaluate, andcreate, and use a derived classification of errors that maps each cognitive process to a characteristic failure mode\(adapted from Andersonet al\.,[2001](https://arxiv.org/html/2605.29007#bib.bib4)\)\. The full seven\-class taxonomy includes two classes \(lack of innovation,quality gap\) that describe sub\-optimal but still essentially correct answers; we drop these because they do not correspond to a student arriving at an incorrect final answer through a coherent misconception\. The retained five classes are summarized in Table[1](https://arxiv.org/html/2605.29007#S3.T1)\.

Table 1:Five\-class student error taxonomy used throughout\. Classes 0 \(lack of innovation\) and 6 \(quality gap\) from the source taxonomy are excluded because they describe*suboptimal correct*rather than*incorrect*responses\.This taxonomy gives us a controlled vocabulary for the rest of the paper: an*error generation*task is now a 3\-tuple\(question, target class, generated response\), and an evaluator can ask whether the response is incorrect and whether it falls in the target class\.

## 4TheGA/EAFramework

Recall from §[1](https://arxiv.org/html/2605.29007#S1)that a practical synthetic\-error generator must produce responses that are \(i\) genuinely wrong, \(ii\) consistent with the specified target class, and \(iii\) stable across subjects\. A single prompt\-and\-generate step cannot reliably enforce \(i\) and \(ii\) together: the model may inadvertently solve the problem correctly, or it may produce a wrong answer whose failure mode does not match the requested class\. To make this concrete, we ran a small pilot in which GPT\-5 was asked to produce a target\-class wrong answer for each of 20 questions drawn from TheoremQA\(Chenet al\.,[2023](https://arxiv.org/html/2605.29007#bib.bib32)\), which is a theorem\-driven QA dataset of domain\-expert\-curated science problems covering math, physics, EE/CS, and finance, paired with each of 5 target classes \(yieldingN=100N=100\(question, target\-class\) cells\), with the gold answer and class definitions supplied as grounding\. On author inspection of the resulting drafts, GPT\-5 fails to match the requested class on18%18\\%of all 100 cells, and on50%50\\%of the 20 cells whose target is structural blindness\. Adding an LLM judge can reduce the mismatch rate to8%8\\%overall and to20%20\\%on structural\-blindness cells\. This is a3939–60%60\\%relative reduction in mismatch rate attributable to the judging step alone\.

We therefore decouple drafting from judging into two pre\-trained agents: a Generation Agent \(GA\) optimized for productive error construction, and an Examination Agent \(EA\) optimized for class\-conditional filtering\. The framework takes a questionqqand a target error classc∈\{1,…,5\}c\\in\\\{1,\\ldots,5\\\}as input and returns a responserrthat theEAhas accepted as an instance of classcc\. Figure[1](https://arxiv.org/html/2605.29007#S4.F1)sketches this loop\. TheEArejects drafts that drift into neighboring classes and routes them back to theGAfor a retry, up to a chosen limit \(five attempts in our experiments\)\. This architecture is subject\-agnostic by design: theGAandEAconsume only the question text and the class definitions, with no subject\-specific module, so the same configuration and prompts transfer across subjects unchanged\. Constraint \(iii\) is therefore satisfied at the design level; the realised cross\-subject*rate*is not invariant in practice and is reported in Appendix[G](https://arxiv.org/html/2605.29007#A7)\. The subsections below describe each agent and the evaluation metric\.

### 4\.1Generation Agent

The Generation Agent \(GA\) receives a questionqqtogether with the five class definitions, and is asked to produce a responserrthat exemplifies a specified target classc∈\{1,…,5\}c\\in\\\{1,\\ldots,5\\\}\. We deliberately use a*single*multi\-classGArather than five class\-specialist agents: in preliminary experiments, class\-specialist agents produced outputs whose realized error class often drifted into a sibling class \(especially3↔43\{\\leftrightarrow\}4\), because each specialist lacked the cross\-class context needed to distinguish its target from adjacent failure modes\.

Question\+\+target error classGeneration Agent \(GA\)Candidate erroneous responseExamination Agent \(EA\)Accepted outputYesNoFigure 1:TheGA/EAloop\. The Generation Agent drafts a candidate incorrect response for a target error class; the Examination Agent judges whether the response is incorrect and matches the requested class\. On rejection, the draft is regenerated, up to a cap of five attempts\.
### 4\.2Examination Agent

The Examination Agent \(EA\) receives the\(question, target class, generated response\)3\-tuple and outputs a binary judgment along two axes: whether the response is actually incorrect, and whether it matches the target class\. Crucially, theEAdoes*not*see theGA’s in\-context examples; it judges using the class definitions alone\. This asymmetry guards against theEAsimply matching surface patterns of theGA’s prompt\.

For lighter\-weight and more stable scoring at scale, we additionally fine\-tune a BERT\-base\-uncased\(Devlinet al\.,[2019](https://arxiv.org/html/2605.29007#bib.bib31)\)classifier as an alternativeEAbackend, trained on1,6001\{,\}600author\-annotated\(3\-tuple, judgment\)examples\. TheGAitself is never fine\-tuned in any pipeline; this keeps the framework a measurement instrument whose numbers reflect the underlying LLM’s off\-the\-shelf capacity rather than our training data, and lets any pre\-trained LLM serve as a drop\-in substitute\.

Table 2:The eight pipeline configurations\. P0–P7 vary information available toGAand the role ofEA\. P8 replaces the promptedEAwith a fine\-tuned classifier\.
### 4\.3Eight Pipelines

We instantiate theGA/EAframework in eight configurations \(Table[2](https://arxiv.org/html/2605.29007#S4.T2)\)\. The baseline \(P0\) givesGAonly the question and target error type\. P1–P7 vary along three axes\. First, some variants expose the correct answer toGA, testing whether knowing the right solution helps it construct a*targeted*wrong one rather than an arbitrary mistake\. Second, some variants add richer conditioning, such as expanded few\-shot examples or textbook excerpts, testing whether additional scaffolding helpsGAshape how the error is expressed\. Third, some variants letEAfeed rejection signals back toGA, turning generation into an iterative search over candidate errors rather than a single\-pass draft\. P8 replaces the promptedEAwith the fine\-tuned BERT\-base\-uncased classifier described above, making it the only pipeline that incorporates human\-supervised training, and only on theEAside\.

## 5Experiments

### 5\.1Evaluation Metric

By author verification, each generation is assigned one of three outcome labels:*incorrect\-and\-wrong\-class*,*incorrect\-and\-right\-class*, or*correct*\. The desired outcome is*incorrect\-and\-right\-class*: the answer is wrong, but wrong in the requested error class\. We call the fraction of generations in this category thetargeted\-error rate, and use it as the headline metric\. The three\-way breakdown is used for failure\-mode analysis \(§[6\.1](https://arxiv.org/html/2605.29007#S6.SS1), §[6\.2](https://arxiv.org/html/2605.29007#S6.SS2)\)\.

#### Refusal\-as\-wrong\-class\.

For pedagogical use, a targeted error must be a*concrete*wrong answer that exhibits the requested cognitive failure mode, not an epistemic refusal\. Responses that admit not knowing \(e\.g\., “I don’t recall”, “I’m not sure”\) are therefore treated as*incorrect\-and\-wrong\-class*\. The detection rule and full pattern list are in Appendix[B](https://arxiv.org/html/2605.29007#A2); the per\-recordis\_refusalflag is released with the dataset \(Appendix[J](https://arxiv.org/html/2605.29007#A10)\)\.

Process\- and cost\-side metrics \(retry count, token consumption, $\-cost, latency, verifier agreement\) are reported alongside the appendices that use them\. We compare these auxiliary metrics in Appendix[H](https://arxiv.org/html/2605.29007#A8)\(retry counts\), Appendix[I](https://arxiv.org/html/2605.29007#A9)\(verifier agreement\), and Appendix[K](https://arxiv.org/html/2605.29007#A11)\(token consumption, $\-cost, and latency\)\.

### 5\.2Main Phase

The main evaluation draws up to 200 questions from TheoremQA\(Chenet al\.,[2023](https://arxiv.org/html/2605.29007#bib.bib32)\), a theorem\-driven QA dataset of domain\-expert\-curated science problems covering combinatorics, numerical analysis, graph algorithms, real and complex analysis, physics, finance, optics, and computer networks\. Different tiers use different subsets: a 20\-question Tier\-1 \(*Verified Benchmark*\) sweep \(§[6\.1](https://arxiv.org/html/2605.29007#S6.SS1)–[6\.3](https://arxiv.org/html/2605.29007#S6.SS3)\) and a 200\-question Tier\-2 \(*Scale Run*\) pool \(§[6\.4](https://arxiv.org/html/2605.29007#S6.SS4)\)\.

### 5\.3Models

We evaluateGAandEAbacked by three OpenAI backbones: a mixed OpenAI o3 / GPT\-4o configuration \(GA= OpenAI o3,EA= GPT\-4o\), GPT\-5, and GPT\-5\-mini\. We use the OpenAI API’s system\-prompt slot for the class definitions across all configurations\. Section[6](https://arxiv.org/html/2605.29007#S6)reports per\-model aggregates; full per\-cell breakdowns are in the appendix\.

### 5\.4Prompting Protocol

For each\(question, target class\)pair,GAreceives the class definitions, in\-context examples of the form⟨\\langlequestion, erroneous answer, explanation⟩\\rangle, and \(depending on the pipeline\) the correct answer and textbook excerpts\.EAreceives only the class definitions and the 3\-tuple to judge\. Few\-shot examples are manually curated by the authors and held fixed across models for comparability\.

## 6Results

Table 3:Tier\-1 results: average targeted\-error rate per pipeline by backend on the first\-20 TheoremQA sweep, human\-verified, unlimited GA/EA retries\. Per\-class breakdown in Appendix[F](https://arxiv.org/html/2605.29007#A6), Table[8](https://arxiv.org/html/2605.29007#A6.T8)\. “mini” = GPT\-5\-mini\.### 6\.1Targeted Error Generation Is Hard

Tier\-1 \(*Verified Benchmark*\) is the paper’s core correctness benchmark: a 20\-question sweep across all 9 pipelines and all 3 backends, with unlimitedGA/EAretries and final human verification of the accepted output\. All targeted\-error\-rate claims in the paper are grounded in this tier\.

Producing a controllable wrong answer is not automatic even when the question and target class are explicitly supplied: the naive baseline \(P0\) leaves2323–3939percentage points below the ceiling on every backend \(Table[3](https://arxiv.org/html/2605.29007#S6.T3)\)\. The shortfall is concentrated on class E5 \(structural blindness:0\.500\.50–0\.550\.55on the stronger configurations,0\.350\.35on GPT\-5\-mini\); E1 and E2 sit close to the ceiling on the stronger backends, but on GPT\-5\-mini both E1 \(0\.600\.60\) and E2 \(0\.300\.30\) are well below \(full per\-class breakdown in Appendix[F](https://arxiv.org/html/2605.29007#A6)\)\. The asymmetry reflects what each class asks of the model: E1 and E2 can be produced by perturbing the correct solution in a single place \(a slip or a substituted formula\), whereas E5 requires the model to*misframe*the problem globally while still producing surface\-coherent reasoning,i\.e\., a confident answer that hides a structural confusion\.

rate0\.250\.50\.751\.0P1P2P3P4P5P6P7P8E1E2E3E4E5Figure 2:Tier\-1 per\-pipeline targeted\-error rate by error class on the first 20 TheoremQA questions, GPT\-5 backend\. Each bar is a per\-class proportion overn=20n\{=\}20cells\. E5 \(violet\) is the hardest target across all configurations \(0\.350\.35–0\.700\.70\)\. P8 does not close the E5 gap on GPT\-5; on o3\+\+GPT\-4o, P8 has the highest point estimate \(Table[3](https://arxiv.org/html/2605.29007#S6.T3)\)\.
### 6\.2Pipeline Comparison on the Verified First\-20

The pipeline\-axis effects in Table[3](https://arxiv.org/html/2605.29007#S6.T3)\(per\-class breakdown on GPT\-5 in Figure[2](https://arxiv.org/html/2605.29007#S6.F2)\) share a common shape: controlled wrong\-answer generation imposes opposing demands on theGA: enough information to fit the requested class, but not so much that the model is dragged toward the correct answer\.

Answer\-grounding \(P1, P2\) helps, with per\-class side effects\. On GPT\-5 and GPT\-5\-mini, supplying the correct answer to theGAimproves the headline rate \(P0→\\toP2==\+4\+4/\+5\+5\), and adding a promptedEAfilter on top further improves it \(P2→\\toP1==\+6\+6/\+5\+5\)\. On o3\+\+GPT\-4o both steps invert \(P0→\\toP2==−3\-3, P2→\\toP1==−12\-12\) because the o3\+\+GPT\-4oGAcannot cleanly suppress the supplied answer and folds pieces of the correct trajectory into its draft, producing partial\-correct reasoning that theEAreads as a slip \(E1\) rather than a missing definition \(E2\); the same failure causes theEAloop to re\-generate near\-duplicates and become a no\-op\. The per\-class signature is concrete: o3\+\+GPT\-4o E2 drops from0\.800\.80\(P0\) to0\.250\.25\(P2\), whereas GPT\-5 P1 E2 stays at0\.950\.95\. As a consequence, E2 is the bottleneck class on o3\+\+GPT\-4o for every prompted answer\-grounded variant \(P1–P7\), while E5 caps the stronger backends\.

TheEAfeedback loop \(P3\) helps, the most on weak backends\. P3 augments P1’sEAloop with the rejected draft and theEA’s justification, giving theGAa negative exemplar to diverge from — diversification temperature\-1 sampling alone cannot reliably produce\. The effect is largest on the weakest reasoner: P1→\\toP3 adds\+21\+21pp on o3\+\+GPT\-4o, concentrated on the harder strategy\-level classes \(E3, E4, E5: each\+25\+25–3030pp; Appendix[F](https://arxiv.org/html/2605.29007#A6)\)\. On GPT\-5 the same step is only\+1\+1\(0\.87→0\.880\.87\\\!\\to\\\!0\.88\), since a strong reasoner already diversifies adequately, at the cost of a modestly longer retry tail on weak backends \(P3 averages3\.093\.09GAattempts on o3\+\+GPT\-4o vs\.2\.452\.45for P1; Appendix[H](https://arxiv.org/html/2605.29007#A8)\)\.

Richer in\-context scaffolding \(P4–P7\) rarely helps\. On GPT\-5, P4–P7 sit at0\.750\.75–0\.860\.86while P1/P3 sit at0\.870\.87/0\.880\.88\. The textbook variants show a clean cross\-backend asymmetry: on o3\+\+GPT\-4o, P7 reaches0\.760\.76\(the strongest prompted configuration on that backend\) because textbook excerpts supply declarative knowledge a weakGAotherwise lacks; on GPT\-5 the same context dilutes the class signal without adding information the model already has, and P7 drops to0\.770\.77\. Expanded examples \(P6\) behave similarly\. The additional scaffolding in P4–P7 therefore does not improve on the lean P1/P3 pair on GPT\-5, leaving P1, P3, and P8 as the configurations worth characterising at scale\.

Fine\-tuned vs\. promptedEA\(P8 vs\. P1\): a deployment trade\-off\. P8 replaces P1’s promptedEAwith a fine\-tuned BERT classifier, keeping every other component identical\. The pure EA\-implementation effect \(P1 vs\. P8\) is large on the weakest backend \(\+25\+25pp on o3\+\+GPT\-4o,0\.54→0\.790\.54\\\!\\to\\\!0\.79\) and small elsewhere \(−4\-4on GPT\-5,\+3\+3on GPT\-5\-mini\)\. The twoEAs also differ structurally: the promptedEA’s acceptance threshold is tunable per call via its system prompt, whereas P8’s is fixed in the classifier weights\. P8 is cheaper at inference \(single forward pass, zero API tokens; §[6\.4](https://arxiv.org/html/2605.29007#S6.SS4)\) but less adaptable; framing P8 vs\. a promptedEAas an API\-cost vs\. on\-device trade\-off is at least as informative as framing it as a quality contest\.

### 6\.3Backend Comparison

The between\-backend gap is large and asymmetric: GPT\-5 averages0\.820\.82over P1–P7, while GPT\-5\-mini \(0\.700\.70\) and o3\+\+GPT\-4o \(0\.680\.68\) sit1212and1414points lower, dwarfing the within\-backend differences across pipelines\. The two abilities surfaced in §[6\.2](https://arxiv.org/html/2605.29007#S6.SS2)explain most of this gap: \(i\) suppressing the supplied answer cleanly enough to produce a class\-consistent wrong trajectory, and \(ii\) diversifying drafts when theEArejects them\. GPT\-5 exhibits both \(i\) clean suppression \(P1 E2=0\.95=0\.95\) and \(ii\) already\-adequate diversification \(P1 / P3=0\.87=0\.87/0\.880\.88\); GPT\-5\-mini is intermediate on \(i\) \(P1 E2=0\.35=0\.35\) and strong on \(ii\) \(P1→\\toP3==\+0\+0\); o3\+\+GPT\-4o is weak on \(i\) \(P2 E2=0\.25=0\.25\) but \(ii\) recovers once explicit feedback compensates \(P1→\\toP3==\+21\+21\)\. The two abilities are largely orthogonal to raw problem\-solving accuracy on TheoremQA: a backend that solves problems well can still fail at controlled error generation if it cannot suppress a supplied answer or diversify under feedback, which is why between\-backend gaps on targeted\-error rate are larger than the gaps on raw correctness\.

### 6\.4Retry Behaviour and Cost

Retry counts surface a hidden cost dimension: two configurations can reach the same targeted\-error rate while one spends22–3×3\\timestheGAcompute\. On Tier\-1, the backend\-retry gap is almost entirely E5: P1 averages2\.452\.45attempts on o3\+\+GPT\-4o vs\.1\.281\.28on GPT\-5 \(Appendix[H](https://arxiv.org/html/2605.29007#A8), Table[10](https://arxiv.org/html/2605.29007#A8.T10)\), with the per\-class breakdown showing the gap concentrated on E5\. Weak backends are not uniformly slower; they are unreliable specifically on cells where theGAre\-generates near\-clones until one happens to land in the target class, so the retry budget is spent on diversification rather than judging\. P8 reshapes*which*class drives the long tail: the prompted pipelines spend retries on E5 \(Table[4](https://arxiv.org/html/2605.29007#S6.T4): P1 E5=0\.90=0\.90\), but P8’s largest retry pocket is on E3 \(0\.900\.90\), not E5\. Its training data over\-weighted E5, so it accepts structural\-blindness drafts quickly but is unusually strict on misconception drafts\. The residual “hard class” shifts with theEAimplementation because the BERT calibration is set offline rather than by a per\-call adaptable prompt\.

Table 4:Tier\-1 per\-pipeline mean retries per cell by error class on the first 20 TheoremQA questions, GPT\-5 backend\. Each cell is the average over 20 \(question, target class\) instances of the number ofEA\-induced retries \(retries==GAattempts−\-1, floor0\)\. A value of0\.000\.00means every instance was accepted on the first draft; a value of0\.900\.90means an average of0\.900\.90retries per instance, with boldface marking notable retry peaks within the table\. P2 is omitted because it has noEAloop\. Source: Appendix[H](https://arxiv.org/html/2605.29007#A8), Table[10](https://arxiv.org/html/2605.29007#A8.T10)\.Tier\-1’s unlimited retries are impractical for deployment; the 5\-attempt cap for Tier\-2 is set by the Tier\-1 retry distribution, in which only2\.5%2\.5\\%of cells exceed 5 retries \(≤1%\\leq 1\\%on GPT\-5; Appendix[H](https://arxiv.org/html/2605.29007#A8)\)\. Tier\-2 \(*Scale Run*\) is a larger 5\-attempt\-cap run on GPT\-5 covering P1, P3, and P8, extended to the full 200\-question pool\. The prompted variants P2 and P4–P7 are not extended because each is dominated by P1/P3 on the GPT\-5 deployment backend \(Table[3](https://arxiv.org/html/2605.29007#S6.T3),0\.770\.77–0\.860\.86vs\.0\.870\.87/0\.890\.89\); P8 is selected on architectural grounds \(local BERT classifier vs\. API round\-trip\) to test whether its near\-single\-pass behaviour persists outside the curated first\-20\. This section reports only*label\-free*quantities \(with cost details in Appendix[K](https://arxiv.org/html/2605.29007#A11)\)\.

Two patterns hold at scale\. First, the prompted\-EAloops and the BERTEAdiffer in*shape*, not just in absolute acceptance \(Table[5](https://arxiv.org/html/2605.29007#S6.T5)\): P1/P3 climb gradually and only saturate at the cap, while P8 is near\-saturated atk=1k\{=\}1\. A deployer shortening the retry budget therefore pays very differently: clipping P1/P3 atk=3k\{=\}3leaves∼\\sim15–25 points of acceptance on the table, while clipping P8 atk=1k\{=\}1loses only∼\\sim5\. The contrast between P1 and P3 also persists across the whole budget range rather than appearing only at the cap: P3 stays above P1 at everykk, which indicates that feeding back the rejected draft andEArationale improves not only eventual acceptance but also*speed to acceptance*\. This matters operationally because many real deployments do not run to exhaustion\. If the effective budget is two or three attempts rather than five, the quality ranking seen under unlimited or near\-unlimited retries can change by much more than the final capped numbers alone suggest\.

Second, the cost gap widens at scale rather than washing out \(Table[6](https://arxiv.org/html/2605.29007#S6.T6)\)\. P8’s cost is*bounded*by its single\-pass cost: the BERT classifier adds no API tokens, so its worst case scales with theGAalone, while the prompted\-EAworst case compoundsGAandEAretry distributions and reaches p95 above 63k tokens / 5 min per cell\. The median rows already show a large separation, but the tail behaviour is the more consequential result: P1/P3 can be acceptable on average while still generating a non\-trivial fraction of very expensive cases, whereas P8 remains tightly concentrated — a higher and less predictable compute burden that is harder to budget for in production\.

Table 5:Tier\-2 loop\-acceptance vs\. retry budget on GPT\-5 \(q20–q199, 5\-attempt cap\)\. P8 is near\-saturated in one shot; P3 dominates P1 at everykk\.Table 6:Tier\-2 per\-cell cost shape on GPT\-5 \(q20–q199, 5\-cap; full Tier\-1\+\+q20\+ inventory in Appendix[K](https://arxiv.org/html/2605.29007#A11)\)\. The prompted\-EAloops \(P1, P3\) carry heavy tails\. P8’s row counts only GA\-side GPT\-5 tokens: itsEAis a local forward pass through the fine\-tuned BERT classifier and contributes no API tokens \($0EA\-side cost\), which is why both median and p95 collapse\.

## 7Analysis and Discussion

#### Hard\-class patterns\.

Beyond E5, Figure[2](https://arxiv.org/html/2605.29007#S6.F2)suggests two secondary patterns: E4 \(wrong choice\) and E5 \(structural blindness\) share a strategy\-level signature, so the promptedEAsometimes accepts an E4 response when an E5 was requested; and E1 \(mental typo\) and E5 sit at opposite ends of the locality axis, so theGAoccasionally produces an E1\-style local slip when asked for E5\. A quantitative confusion matrix requires annotating the realised class of every rejected draft and is flagged in §[Limitations](https://arxiv.org/html/2605.29007#Sx1)\.

#### Answer\-grounding vs\. knowledge\.

Of the design axes we vary, only two have a large and consistent effect: answer\-grounding and theEAfeedback loop\. This suggests targeted\-error generation is more a controllability problem under answer\-grounding than a knowledge\-acquisition problem under expanded examples or textbook context\. We stop short of the stronger “trajectory\-control” claim because no pipeline conditions theGAon the gold reasoning trace \(§[Limitations](https://arxiv.org/html/2605.29007#Sx1)\)\. The safe take\-away is the weaker one: targeted\-error rate adds information invisible to correctness\-only benchmarks\.

#### Deployment implications\.

The strongest GPT\-5 pipelines reach∼\\sim0\.83–0\.88, viable for downstream educational use provided anEAis in the loop and E5 outputs receive additional human review\. “Ask the LLM for a wrong answer” does*not*produce class\-controlled errors: class\-controlled generation requires either answer\-grounding or fine\-tuned post\-hoc judging\. Whether the resulting errors match what real students produce is a separate validity question this evaluation does not answer \(§[Limitations](https://arxiv.org/html/2605.29007#Sx1)\)\.

## 8Conclusion

We presented a two\-agent framework that drafts a targeted student\-like error and filters for class consistency, giving a reusable recipe for class\-stratified synthetic error datasets where labelled student errors are unavailable\. Targeted error generation is substantially harder than uncontrolled wrong\-answer generation; E5 \(structural blindness\) is the hardest class on GPT\-5, and answer\-grounding flips the o3\+\+GPT\-4o bottleneck from E5 to E2\. The released Tier\-1 pool and fine\-tunedEAserve as a replication artefact; real\-student validation is the load\-bearing next step\.

## Limitations

Scale and human labelling\.The Tier\-1 evaluation uses 20 questions per \(pipeline, backend\) cell\. The binding constraint is not API cost but the manual validation step: every targeted\-error cell is ultimately verified by an author against the class definitions \(§[3](https://arxiv.org/html/2605.29007#S3)\), and that labelling cost scales as\|pipelines\|×\|questions\|×5\|\{\\rm pipelines\}\|\\times\|\{\\rm questions\}\|\\times 5classes×\\times\|backends\|\|\{\\rm backends\}\|\. The full Tier\-1 pool contains2,5302\{,\}530author\-verified⟨\\langlequestion, target class, response⟩\\ranglecells across nine pipelines and three backends\. Tier\-2 \(§[6\.4](https://arxiv.org/html/2605.29007#S6.SS4)\) extends the question count, but we found that automatic verifier substitution is not yet reliable enough for us to use its labels as a stand\-in for correctness\. Scaling correctness rates further therefore requires either expert annotators or an automaticEAthat meets a human\-level bar; the latter is what P8 is testing\.

Single taxonomy\.Our five\-class taxonomy is one of several plausible coordinate systems for student errors\. Results may shift under alternative taxonomies \(e\.g\., subject\-specific misconception catalogs\)\.

Closed\-source models\.GPT\-4o, OpenAI o3, GPT\-5, and GPT\-5\-mini are opaque; we cannot disentangle whether observed behavior reflects pretraining content, RLHF, or system\-prompt scaffolding\.

No real\-student baseline \(validity gap\)\.Throughout the paper the term*targeted student error*refers to a response that fits the requested class definition; we do not measure whether the realised surface form matches what authentic students actually produce on the same item\. This is the load\-bearing external\-validity assumption of the framework and we did not test it in this submission\. The closest existing student\-misconception corpora\(Sunet al\.,[2025](https://arxiv.org/html/2605.29007#bib.bib6); Oteroet al\.,[2024](https://arxiv.org/html/2605.29007#bib.bib7); Oliet al\.,[2024](https://arxiv.org/html/2605.29007#bib.bib8)\)each cover different subjects and label schemes from ours, so matching against them requires a label re\-projection step we have not done\. Until that comparison exists, the framework should be read as producing*taxonomy\-targeted synthetic errors*, not as empirically validated student\-like errors\.

No reasoning\-trace ablation\.The paper uses “answer\-grounding” to describe the strongest pipelines \(P1, P3, P4, P7, P8 all show the gold final answer to theGA\)\. No pipeline conditions on the gold*reasoning trace*\(the step\-by\-step derivation that produces the answer\)\. A new natural pipeline would supply the trace and let us cleanly distinguish answer\-only grounding from trajectory grounding; we did not run this and the “trajectory” framing in the discussion is therefore weaker than “answer\-grounding\.”

Per\-cell sampling noise\.Tier\-1 is one stochastic draw per\(question, target class\)cell at temperature 1: each cell was drawn once fromGArather than averaged over multiple completions\. The reported targeted\-error rates \(Tables[3](https://arxiv.org/html/2605.29007#S6.T3),[8](https://arxiv.org/html/2605.29007#A6.T8), and[10](https://arxiv.org/html/2605.29007#A8.T10)\) aggregate over 20 questions×\\times5 target classes per pipeline, so cross\-question variance is implicit in the means and the relative ordering across pipelines is stable in the per\-class breakdowns; but the absolute rates should be read as point estimates that repeated multi\-run sampling \(5–10 draws per cell, the natural next step within the same evaluation budget\) would smooth\.

Single\-annotator labelling\.Labels on the 1,600\-cell training set and the Tier\-1 verification pass were produced by the authors in a single pass; we did not collect independent labels from external experts, so inter\-annotator agreement is not quantified\. External expert validation on a stratified subset is left to future work\.

## Ethical considerations

This work studies the generation of taxonomy\-targeted synthetic incorrect answers for educational use\. The intended use is pedagogical: exposing students to a controlled distribution of plausible misconceptions to support practice and self\-assessment\. The generated errors are not intended for deployment without educator review\. Risks include propagating errors that are not representative of authentic student misconceptions, especially in subjects where our model coverage is thin\. We mitigate this by \(i\) tying generation to an explicit taxonomy, \(ii\) judging outputs with anEA, and \(iii\) publishing the fine\-tunedEAfor downstream verification\.

## Acknowledgements

We used AI\-based coding and writing assistants for software scaffolding, LaTeX formatting, and prose editing\. All experimental design, analysis, and claims are the authors’ own; the commercial LLMs evaluated in this paper are objects of study rather than writing aids\.

## References

- L\. W\. Anderson, D\. R\. Krathwohl, P\. W\. Airasian, K\. A\. Cruikshank, R\. E\. Mayer, P\. R\. Pintrich, J\. Raths, and M\. C\. Wittrock \(2001\)A taxonomy for learning, teaching, and assessing: a revision of bloom’s taxonomy of educational objectives\.Longman,New York\.External Links:ISBN 978\-0801319037Cited by:[§1](https://arxiv.org/html/2605.29007#S1.p1.1),[§1](https://arxiv.org/html/2605.29007#S1.p2.1),[§1](https://arxiv.org/html/2605.29007#S1.p4.1),[§3](https://arxiv.org/html/2605.29007#S3.p1.1)\.
- J\. S\. Brown and R\. R\. Burton \(1978\)Diagnostic models for procedural bugs in basic mathematical skills\.Cognitive Science2\(2\),pp\. 155–192\.Cited by:[§1](https://arxiv.org/html/2605.29007#S1.p2.1),[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px2.p1.1)\.
- W\. Chen, M\. Yin, M\. Ku, P\. Lu, Y\. Wan, X\. Ma, J\. Xu, X\. Wang, and T\. Xia \(2023\)TheoremQA: a theorem\-driven question answering dataset\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 7889–7901\.External Links:2305\.12524Cited by:[Appendix J](https://arxiv.org/html/2605.29007#A10.SS0.SSS0.Px2.p1.1),[Appendix D](https://arxiv.org/html/2605.29007#A4.p1.1),[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2605.29007#S4.p1.7),[§5\.2](https://arxiv.org/html/2605.29007#S5.SS2.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px4.p1.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\),pp\. 4171–4186\.External Links:1810\.04805Cited by:[§4\.2](https://arxiv.org/html/2605.29007#S4.SS2.p2.1)\.
- D\. Hendrycks, S\. Basart, S\. Kadavath, M\. Mazeika, A\. Arora, E\. Guo, C\. Burns, S\. Puranik, H\. He, D\. Song, and J\. Steinhardt \(2021\)Measuring coding challenge competence with APPS\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,External Links:2105\.09938Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, C\. Zhang, J\. Wang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin,et al\.\(2024\)MetaGPT: meta programming for a multi\-agent collaborative framework\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px3.p1.1)\.
- G\. Kortemeyer \(2023\)Toward AI grading of student problem solutions in introductory physics: a feasibility study\.Physical Review Physics Education Research19\(2\),pp\. 020163\.External Links:[Document](https://dx.doi.org/10.1103/PhysRevPhysEducRes.19.020163)Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px4.p1.1)\.
- H\. Luo, Q\. Sun, C\. Xu, P\. Zhao, J\. Lou, C\. Tao, X\. Geng, Q\. Lin, S\. Chen, and D\. Zhang \(2023\)WizardMath: empowering mathematical reasoning for large language models via reinforced evol\-instruct\.arXiv preprint arXiv:2308\.09583\.Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px4.p1.1)\.
- Z\. Luo, C\. Xu, P\. Zhao, Q\. Sun, X\. Geng, W\. Hu, C\. Tao, J\. Ma, Q\. Lin, and D\. Jiang \(2024\)WizardCoder: empowering code large language models with evol\-instruct\.InInternational Conference on Learning Representations \(ICLR\),External Links:2306\.08568Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px4.p1.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. Clark \(2023\)Self\-Refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px3.p1.1)\.
- I\. Mirzadeh, K\. Alizadeh, H\. Shahrokhi, O\. Tuzel, S\. Bengio, and M\. Farajtabar \(2025\)GSM\-Symbolic: understanding the limitations of mathematical reasoning in large language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:2410\.05229Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Oli, R\. Banjade, A\. M\. Olney, and V\. Rus \(2024\)Can LLMs identify gaps and misconceptions in students’ code explanations?\.arXiv preprint arXiv:2501\.10365\.Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px2.p1.1),[Limitations](https://arxiv.org/html/2605.29007#Sx1.p4.1)\.
- N\. Otero, S\. Druga, and A\. Lan \(2024\)A benchmark for math misconceptions: bridging gaps in middle school algebra with AI\-supported instruction\.arXiv preprint arXiv:2412\.03765\.Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px2.p1.1),[Limitations](https://arxiv.org/html/2605.29007#Sx1.p4.1)\.
- A\. Pardo and G\. Siemens \(2014\)Ethical and privacy principles for learning analytics\.British Journal of Educational Technology45\(3\),pp\. 438–450\.External Links:[Document](https://dx.doi.org/10.1111/bjet.12152)Cited by:[§1](https://arxiv.org/html/2605.29007#S1.p2.1)\.
- P\. M\. Sadler, G\. Sonnert, H\. P\. Coyle, N\. Cook\-Smith, and J\. L\. Miller \(2013\)The influence of teachers’ knowledge on student learning in middle school physical science classrooms\.American Educational Research Journal50\(5\),pp\. 1020–1049\.External Links:[Document](https://dx.doi.org/10.3102/0002831213477680)Cited by:[§1](https://arxiv.org/html/2605.29007#S1.p1.1)\.
- P\. M\. Sadler \(1998\)Psychometric models of student conceptions in science: reconciling qualitative studies and distractor\-driven assessment instruments\.Journal of Research in Science Teaching35\(3\),pp\. 265–296\.Cited by:[§1](https://arxiv.org/html/2605.29007#S1.p1.1)\.
- S\. Slade and P\. Prinsloo \(2013\)Learning analytics: ethical issues and dilemmas\.American Behavioral Scientist57\(10\),pp\. 1510–1529\.External Links:[Document](https://dx.doi.org/10.1177/0002764213479366)Cited by:[§1](https://arxiv.org/html/2605.29007#S1.p2.1)\.
- J\. P\. Smith, A\. A\. diSessa, and J\. Roschelle \(1993\)Misconceptions reconceived: a constructivist analysis of knowledge in transition\.Journal of the Learning Sciences3\(2\),pp\. 115–163\.External Links:[Document](https://dx.doi.org/10.1207/s15327809jls0302%5F1)Cited by:[§1](https://arxiv.org/html/2605.29007#S1.p1.1)\.
- Y\. Sun, Z\. Yin, X\. Huang, X\. Qiu, and H\. Zhao \(2025\)Error classification of large language models on math word problems: a dynamically adaptive framework\.InFindings of the Association for Computational Linguistics: EMNLP 2025,External Links:2501\.15581Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px2.p1.1),[Limitations](https://arxiv.org/html/2605.29007#Sx1.p4.1)\.
- V\. Švábenský, B\. Flanagan, E\. D\. López Zapata, and A\. Shimada \(2026\)Open datasets in learning analytics: trends, challenges, and best PRACTICE\.ACM Transactions on Knowledge Discovery from Data\.External Links:[Document](https://dx.doi.org/10.1145/3798096),2602\.17314Cited by:[§1](https://arxiv.org/html/2605.29007#S1.p2.1)\.
- M\. Turpin, J\. Michael, E\. Perez, and S\. R\. Bowman \(2023\)Language models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px1.p1.1)\.
- U\.S\. Department of Education, Student Privacy Policy Office \(2026\)Privacy and data sharing\.Note:[https://studentprivacy\.ed\.gov/privacy\-and\-data\-sharing](https://studentprivacy.ed.gov/privacy-and-data-sharing)Accessed 2026\-05\-17Cited by:[§1](https://arxiv.org/html/2605.29007#S1.p2.1)\.
- J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins \(2022\)Solving math word problems with process\- and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px1.p1.1)\.
- K\. VanLehn \(1990\)Mind bugs: the origins of procedural misconceptions\.MIT Press\.Cited by:[§1](https://arxiv.org/html/2605.29007#S1.p2.1),[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px2.p1.1)\.
- K\. VanLehn \(2011\)The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems\.Educational Psychologist46\(4\),pp\. 197–221\.External Links:[Document](https://dx.doi.org/10.1080/00461520.2011.611369)Cited by:[§1](https://arxiv.org/html/2605.29007#S1.p1.1)\.
- J\. Vie, T\. Rigaux, and S\. Minn \(2022\)Privacy\-preserving synthetic educational data generation\.InEducating for a New Future: Making Sense of Technology\-Enhanced Learning Adoption \(EC\-TEL 2022\),Lecture Notes in Computer Science, Vol\.13450,pp\. 393–406\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-16290-9%5F29),2207\.03202Cited by:[§1](https://arxiv.org/html/2605.29007#S1.p3.1)\.
- X\. Wang, Z\. Hu, P\. Lu, Y\. Zhu, J\. Zhang, S\. Subramaniam, A\. R\. Loomba, S\. Zhang, Y\. Sun, and W\. Wang \(2023a\)SciBench: evaluating college\-level scientific problem\-solving abilities of large language models\.arXiv preprint arXiv:2307\.10635\.Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023b\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px1.p1.1)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. Wang \(2024\)AutoGen: enabling next\-gen LLM applications via multi\-agent conversation\.InConference on Language Modeling \(COLM\),External Links:2308\.08155Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px3.p1.1)\.
- L\. Yu, W\. Jiang, H\. Shi, J\. Yu, Z\. Liu, Y\. Zhang, J\. T\. Kwok, Z\. Li, A\. Weller, and W\. Liu \(2024\)MetaMath: bootstrap your own mathematical questions for large language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px4.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Cited by:[§2](https://arxiv.org/html/2605.29007#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AWorked Examples of the Evaluation Metric

#### Example A: correct output despite a class\-1 request\.

> Q\.Fifteen more than a quarter of a number is 27\. What is the number? Requested class\.1 \(mental typo\)\. Generated response\.A quarter of the number isx/4x/4\. So,x/4\+15=27x/4\+15=27\. Subtract1515:x/4=12x/4=12\. Multiply both sides by 4:x=48x=48\.*Therefore, the number is 48\.* Label\.correct – the model answered the problem correctly instead of producing a class\-1 error\.

#### Example B: incorrect\-and\-right\-class – desired output\.

> Q\.Same as above\.Requested class\.1 \(mental typo\)\. Generated response\.\(1/4\)​n\+15=27\(1/4\)\\,n\+15=27\. Subtract 15:\(1/4\)​n=12\(1/4\)\\,n=12\. Multiplying both sides by 3 to solve fornn:n=12×3=36n=12\\times 3=36\. So the number is 36\. Label\.\(0,1\)\(0,1\)– incorrect answer caused by multiplying by 3 rather than 4, exactly the mental\-typo failure mode\.

#### Example C: epistemic refusal – wrong\-class under the refusal rule\.

A response can read as class\-consistent in tone yet fail to produce a substantive student\-like error because it admits not knowing\. By the rule in §[5\.1](https://arxiv.org/html/2605.29007#S5.SS1), such refusals are labelled*incorrect\-and\-wrong\-class*\.

> Q\.IsW=\{\[x,y\]∈ℝ2:x≥0​and​y≥0\}W=\\\{\[x,y\]\\in\\mathbb\{R\}^\{2\}:x\\geq 0\\text\{ and \}y\\geq 0\\\}a subspace ofℝ2\\mathbb\{R\}^\{2\}?Gold answer\.False\.Requested class\.2 \(knowledge gap\)\. Generated response\.First, I know that to show a set is a subspace ofℝ2\\mathbb\{R\}^\{2\}we have to check certain properties, but I’m not fully sure which ones are required\. I vaguely recall that it has something to do with being “closed under addition” and also something with “scalar multiplication,” yet I’m not certain what “closed” really means in this context\. Since I’m unsure of the exact formal definition of a subspace, I can’t confidently decide if the setWWqualifies\. Label\.*incorrect\-and\-wrong\-class*\. The response expresses ignorance rather than committing to a concrete wrong trajectory, so it is not a usable knowledge\-gap exemplar even though its tone superficially matches class 2\. The automaticis\_refusalflag fires on “I’m not fully sure”, “I vaguely recall”, and “I can’t confidently decide”\.

## Appendix BRefusal Heuristic

The refusal\-as\-wrong\-class rule \(§[5\.1](https://arxiv.org/html/2605.29007#S5.SS1)\) is implemented as a case\-insensitive regex over the generation’s final output text\. A response that matches any of the patterns below is flagged as an epistemic refusal and treated as*incorrect\-and\-wrong\-class*\.

> I don’t \(actually\) know I can’t recall I’m not \(fully/entirely\) sure I’m uncertain I don’t remember I vaguely recall lacking that \(crucial\) fact/detail/knowledge I don’t have the precise/exact I’ll \(make a\) cautious guess I have no idea I’m not familiar with I’m not entirely certain I don’t fully know/understand/recall cannot confidently/fully decide/answer/recall not fully/completely sure I’m \(still\) \(a bit\) unclear/confused I’m unable to I fail to recall don’t \(exactly\) know how

The same flag appears in the released JSONL \(is\_refusal, see Appendix[J](https://arxiv.org/html/2605.29007#A10)\)\. The heuristic is intentionally permissive: any explicit epistemic\-refusal phrasing triggers the flag, even when the response also commits to a concrete final answer\. This conservative choice errs on the side of excluding borderline cases from the targeted\-error rate; downstream users who want a stricter criterion can filter onis\_refusal == False AND human\_examination == 1, and users who want the looser original criterion can ignore the flag\.

#### Scope\.

Across the Tier\-1 sweep the rule applies to8888cells, all on class E2 \(knowledge gap\), where the model expresses ignorance rather than producing a concrete wrong\-formula trajectory\. The unfiltered counts can be recovered by ignoring the flag in downstream analysis\.

## Appendix CPrompt Templates

#### Zero\-shot vs\. few\-shot\.

The Generation Agent receives zero\-shot prompts of the form below for simple cases, and few\-shot prompts containing Q\-A\-E 3\-tuples \(Question, erroneous Answer, Explanation of why this exemplifies the target class\) for harder items\.

> Zero\-shot example for class 1: \*\*Mental Typo\*\*: This error happens when a student is sloppy\. Generate an answer of error type 1 to the following question\.

> Few\-shot example for class 1: \*\*Mental Typo\*\*: This error happens when a student is sloppy\. Q1: Twice Angie’s age, plus 4, is 20\. How old is Angie? A1: Twice Angie’s age is20−4=1620\-4=16\. Angie is16/2=716/2=7\. E1: The student mistakenly calculates 7 instead of 8\. Q2: A roll of 25 m wire weighs 5 kg\. How much does a 75 m roll weigh? A2: The 75 m roll is three times bigger\.\.\. so the 75 m roll weighs15​kg×3=45​kg15\\,\\mathrm\{kg\}\\times 3=45\\,\\mathrm\{kg\}\. E2: The weight in the last step should be 5 kg, not 15 kg\. Generate an answer of error type 1 to the following question\.

A representative few\-shot exemplar for each remaining class is shown below\. The full sets used in our experiments \(six exemplars per class for theGA; class definitions only for theEA\) are distributed with the released code\.

#### Class 2 \(Knowledge gap\)\.

> \*\*Knowledge Gap\*\*: The student lacks a definition or formula\. Q: What’s the remainder when 7 is divided by 17? A: The remainder of 7/17 is 7\. E: The student is unable to distinguish ‘‘divide’’ and ‘‘be divided by’’\. The expression A/B can be read ‘‘A divided by B’’ or ‘‘B divides A’’, but not ‘‘A divides B’’ or ‘‘B divided by A’’\.

#### Class 3 \(Misconception\)\.

> \*\*Misconception\*\*: A faulty mental model fitted to prior experience\. Q: A scuba diver, looking up at the surface, sees total internal reflection at25∘25^\{\\circ\}from vertical\. What is the index of refraction of the water? A: I incorrectly assume cosine relates the critical\-angle indices; settingcos⁡\(25∘\)=nair/nwater\\cos\(25^\{\\circ\}\)=n\_\{\\rm air\}/n\_\{\\rm water\}and solving givesnwater≈1\.10n\_\{\\rm water\}\\approx 1\.10\. E: The student conflated the critical\-angle relation \(which uses sine, not cosine\), a stable conceptual error rather than a local slip\.

#### Class 4 \(Wrong choice\)\.

> \*\*Wrong Choice\*\*: Selection of the wrong solution procedure\. Q: A 1000 kg car with worn shocks sinks 2\.8 cm under a 980 N person\. Model as a spring; find the SHM frequency\. A: Treating the system as a simple pendulum withL=0\.028L=0\.028m,f=\(1/2​π\)​g/L≈3\.0f=\(1/2\\pi\)\\sqrt\{g/L\}\\approx 3\.0Hz\. E: The student applied the pendulum formula to a mass\-on\-spring problem, a category error in solution procedure rather than arithmetic\.

#### Class 5 \(Structural blindness\)\.

> \*\*Structural Blindness\*\*: Failure to model component interaction\. Q: TCP\-Reno transfer time computation \(file size 350 KB,RTT=20\\mathrm\{RTT\}=20ms, link 10 Mbps, slow\-start threshold 32\)\. A: Adding RTT delay and transmission delay sequentially:5×20\+288=3885\\times 20\+288=388ms; I then mistakenly add an extra RTT per segment, getting 655 ms\. E: The student treated per\-segment RTT contributions as non\-overlapping when they actually overlap with concurrent transmission, a structural misframing rather than a calculation slip\.

#### Examination Agent system prompt \(paraphrased\)\.

TheEAis provided with: \(i\) the five class definitions verbatim from Table[1](https://arxiv.org/html/2605.29007#S3.T1); \(ii\) the question; \(iii\) theGA’s candidate response; \(iv\) the target class\. It is asked to output two binary judgments: \(a\) whether the response’s final answer is incorrect; \(b\) whether the reasoning matches the target class definition\. No few\-shot exemplars are shown to theEA\.

## Appendix DTheorem\-Set Questions Used in the First\-20q Sweep

The 20\-question Tier\-1 subset from TheoremQA\(Chenet al\.,[2023](https://arxiv.org/html/2605.29007#bib.bib32)\)is used throughout Tables[3](https://arxiv.org/html/2605.29007#S6.T3),[8](https://arxiv.org/html/2605.29007#A6.T8), and[10](https://arxiv.org/html/2605.29007#A8.T10)\. The exact first\-20 list is shown in Table[7](https://arxiv.org/html/2605.29007#A4.T7); each question is paired with each of the five target error classes, yielding 100 \(question, class\) cells per pipeline\.

Table 7:The first 20 TheoremQA questions used in the per\-pipeline sweep\. Long problem statements are summarized here; the exact verbatim prompts and gold answers are released in the released Tier\-1 question\-and\-answer subset\.
## Appendix EAnnotation Protocol for the 1,600\-ExampleEATraining Set

#### Training examples\.

The 1,600 \(3\-tuple, judgment\) examples used to fine\-tune the BERT\-base\-uncasedEAare human\-annotated, sampled to balance the five target error classes \(approximately 320 examples per class\)\.

#### Label space\.

Each example carries a single binary “human examination” label \(released in the CSVs\):11iff an author verified that the response is both genuinely incorrect*and*an instance of the requested target class, and0otherwise \(a correct answer, or a wrong answer in the wrong class\)\. The classifier input is the 3\-tuple"Question: \{q\} Answer: \{a\} Error Class: \{c\}", with the human\-examination bit as the classification target\. Annotators referred to the five class definitions and the few\-shot exemplars shown in Appendix[C](https://arxiv.org/html/2605.29007#A3)\.

#### Annotator pool\.

Labels on the 1,600\-example training set were produced by the authors during the iterative pilot phase rather than by recruited external annotators\. No IRB approval was required because no external human\-subjects data were collected\.

#### Label resolution\.

Author labels were assigned by a single annotator pass with subsequent self\-review; we did not run a separate adjudication protocol on the training set because labels were not double\-annotated\.

## Appendix FPer\-Pipeline Per\-Class Targeted\-Error Rate

This section gives the per\-class breakdown for the same Tier\-1 sweep summarised by averages in main\-paper Table[3](https://arxiv.org/html/2605.29007#S6.T3)\. Because Tier\-1 uses unlimitedGA/EAretries, every cell is eventually accepted by theEAand labelled by an author; the targeted\-error rate per \(pipeline, backend, class\) is therefore both the end\-to\-end framework accuracy and the human\-agreement rate of the final accepted output\.

Two patterns visible in Table[8](https://arxiv.org/html/2605.29007#A6.T8): \(i\) backend strength dominates – GPT\-5 averages0\.820\.82over P1–P7 vs\.0\.700\.70for GPT\-5\-mini and0\.680\.68for o3\+\+GPT\-4o; \(ii\) class E5 \(structural blindness\) is the bottleneck across all pipelines and all backends, and the fine\-tunedEA\(P8\) does not close it – on GPT\-5, P8 E5=0\.60=0\.60vs\. P3 E5=0\.70=0\.70\. P8 is strongest only on the weakest backend \(o3\+\+GPT\-4o, where it reaches0\.790\.79vs\. P3’s0\.750\.75\)\.

Table 8:Tier\-1 per\-class targeted\-error rate on the first\-20 TheoremQA sweep, broken out by paper\-label pipeline and backend\. Human\-verified, unlimitedGA/EAretries\. Per\-class peaks across backends are in bold\.
## Appendix GPer\-Subject Tier\-1 Breakdown

§[4](https://arxiv.org/html/2605.29007#S4)describes theGA/EAframework as subject\-agnostic in design, because no module is specialised by subject\. To check whether the realised Tier\-1 targeted\-error rate is also subject\-stable in practice, we re\-bucket the same per\-cell labels that feed Table[3](https://arxiv.org/html/2605.29007#S6.T3)by the subject of the underlying TheoremQA question\. The 20 questions in the Tier\-1 sweep fall into four buckets: combinatorics and graphs \(Q1, Q4, Q5, Q6, Q14, Q16\), analysis \(Q2, Q3, Q7, Q9, Q13, Q17, Q18\), linear algebra \(Q10, Q12, Q15\), and physics and probability \(Q8, Q11, Q19, Q20\)\. The per\-subject means are reported in Table[9](https://arxiv.org/html/2605.29007#A7.T9); the “All” column reproduces Table[3](https://arxiv.org/html/2605.29007#S6.T3)as a sanity check on the bucketing\.

Two patterns are visible\. First, the cross\-subject*ordering*is stable: combinatorics/graphs is the easiest subject for nearly every \(pipeline, backend\) row \(0\.500\.50–1\.001\.00\); analysis is the hardest \(0\.460\.46–0\.830\.83\); linear algebra and physics/probability sit between\. Second, the cross\-subject*spread*within a single \(pipeline, backend\) row is wider than the within\-backend pipeline spread that §[6\.2](https://arxiv.org/html/2605.29007#S6.SS2)debates: on GPT\-5 P3, for example, the per\-subject range is0\.800\.80–1\.001\.00\(Δ=0\.20\\Delta\\\!=\\\!0\.20\), whereas the GPT\-5 P1→\\toP3 headline gap is0\.010\.01\. The cross\-subject claim in the introduction is therefore better read as “the within\-row*ordering*of subjects is stable across pipelines and backends” than as “the absolute rate is invariant to subject\.”

Table 9:Per\-subject Tier\-1 targeted\-error rate \(mean over all 5 error classes\), broken out by paper\-label pipeline and backend on the first\-20 TheoremQA sweep\. Cell counts per subject are: combinatorics/graphn=30n\{=\}30\(6 questions×\\times5 classes\); analysisn=35n\{=\}35\(7×\\times5\); linear algebran=15n\{=\}15\(3×\\times5\); physics/probabilityn=20n\{=\}20\(4×\\times5\); and “All”n=100n\{=\}100matches Table[3](https://arxiv.org/html/2605.29007#S6.T3)\. Within\-row variation across subjects is the relevant signal for the cross\-subject\-stability claim in §[4](https://arxiv.org/html/2605.29007#S4)\.
## Appendix HRetry Counts

Table[10](https://arxiv.org/html/2605.29007#A8.T10)reports the mean number ofGAattempts needed before a candidate is accepted by theEA\. Thus a value of 1\.00 means the first generation was accepted, and the number of actual retries is the table value minus one\. P2 has noEAloop, so it is included only as a reference row with value 1\.00\.

The table explains the cost behavior behind the backend comparison in Table[3](https://arxiv.org/html/2605.29007#S6.T3)\. The o3\+\+GPT\-4o stack often needs multiple attempts, especially for class 5; GPT\-5\-mini usually reduces this to one or two attempts; GPT\-5 reduces most cells to a near\-single\-pass regime\. The exception is class 5, where even GPT\-5 still needs modest extra search in P1/P3/P6/P5\. This supports the paper’s interpretation that structural blindness is not merely a weak\-model artifact but a persistent hard target\.

Table 10:Mean number ofGAattempts per accepted generation \(*attempts*, not retries; actual retries equal attempts minus one\)\. Pipeline 2 has no examination loop and trivially equals 1\. Class 5 \(structural blindness\) is the most retry\-intensive class across pipelines and backends, except for P8 where the BERT classifier is near\-single\-pass on all backends, with its largest retry pocket on class 3 \(misconception\) for GPT\-5 and GPT\-5\-mini and on class 5 for o3\+\+GPT\-4o\.#### How often does the loop need more than 5 retries?

Tier\-1 uses unlimited retries, but Tier\-2 \(§[6\.4](https://arxiv.org/html/2605.29007#S6.SS4)\) deploys a 5\-attempt cap\. Table[11](https://arxiv.org/html/2605.29007#A8.T11)counts, for each \(pipeline, backend, error class\), how many of the 20\(question, target class\)cells required more than 5 retries to be accepted\. Of 2,100 Tier\-1 cells with anEAloop \(7 pipelines×\\times3 backends×\\times100 cells\), 53 \(2\.5%\) crossed the 5\-retry mark; 48 of those \(91%\) are on the o3\+\+GPT\-4o backend, concentrated on E5 \(structural blindness\) and secondarily E2 \(knowledge gap\)\. On GPT\-5 and GPT\-5\-mini the cap rarely binds; in particular P3 on GPT\-5 has*zero*cells exceeding 5 retries, supporting the deployment choice of a 5\-attempt cap for the Tier\-2 scale runs\.

Table 11:Tier\-1 redo\>\>5 counts per \(pipeline, backend, error class\) on the first\-20 TheoremQA sweep\. Per\-class cells are counts out ofn=20n\{=\}20; the “All” column is the per\-\(pipeline, backend\) total out ofn=100n\{=\}100with per\-row percentage\. Pipelines P0 and P2 are omitted because they have noEAloop\. The tail concentrates on the o3\+\+GPT\-4o backend \(91% of all redo\>\>5 cells\) and on classes E5 and E2; on GPT\-5 and GPT\-5\-mini the 5\-attempt cap rarely binds\.

## Appendix IThree\-Agent Cascade Ablation

To test whether adding more judges to theGA/EApipeline would help, we run a small ablation in which a third agent \(an independent freshEAcall using the same prompt and the same model, GPT\-5\) re\-judges each of the 100 accepted P3 outputs\. The question is whether a third judge would reject cells the first judge accepted\. If so, end\-to\-end acceptance in a 3\-agent cascade would drop proportionally; if not, the third judge contributes no new information\.

Table 12:Three\-agent cascade ablation on P3 GPT\-5\. Fresh independent re\-judging agrees on96%96\\%of EA1\-accepted cells\.The per\-judge retention rate is0\.960\.96\. With P3’s baseline 2\-agent acceptance at∼0\.89\\sim 0\.89\(Table[3](https://arxiv.org/html/2605.29007#S6.T3)\), a 3\-agent cascadeGA→EA1→EA2\\textsc\{GA\}\\to\\textsc\{EA\}\_\{1\}\\to\\textsc\{EA\}\_\{2\}would have expected acceptance0\.89×0\.96≈0\.850\.89\\times 0\.96\\approx 0\.85, a44\-percentage\-point drop\. Each additional cascaded judge would compound this loss multiplicatively unless it contributed information orthogonal to the existing judges\. The 4 disagreement cells span all five error classes and stem from genuine ambiguity in class boundaries \(e\.g\., distinguishing a misconception from a knowledge gap\) rather than systematic failure modes that a specialist agent could address\. We therefore use the two\-agent setup throughout\.

## Appendix JReleased Dataset Schema and Counts

The supplementary release contains 1,800 records over the TheoremQA main phase \(20 questions×\\times5 error classes×\\times9 pipelines×\\times2 backends\), packaged as JSONL\. The full per\-record schema and datasheet are included with the release\.

#### Per\-record fields\.

Each JSONL line contains the source question and gold answer \(verbatim from TheoremQA\), the target error class \(1–5\) and its human\-readable name, the pipeline and backend identifier, the generating GA’s final accepted response, theEA’s class judgment, the number of retries the response required to be accepted, a binaryhuman\_examinationlabel produced by the authors during the original training\-set construction, and anis\_refusalboolean flagging cells whose final response is an epistemic refusal \(§[5\.1](https://arxiv.org/html/2605.29007#S5.SS1)\)\. The paper’s targeted\-error rate treats a cell as success iffhuman\_examination = 1 AND not is\_refusal; cells withhuman\_examination = 0are EA false\-positives and cells withis\_refusal = Trueare refusal responses, each useful as negative training data for stricter discriminators\.

#### Availability, license, and attribution\.

The main\-phase records and the fine\-tuned BERT\-base\-uncasedEAweights from pipeline P8 will be released upon publication under CC BY\-SA 4\.0; the source questions are drawn from TheoremQA\(Chenet al\.,[2023](https://arxiv.org/html/2605.29007#bib.bib32)\)\(MIT licensed\), acknowledged in the upstream attribution file\.

#### Privacy and content\.

Because the framework is designed precisely to avoid authentic student data, the release contains no personally identifying information: every record is a synthetically generated solution to a publicly released TheoremQA science question, and no real student data was collected or stored\. Each released record carries thehuman\_examinationverification label described above, and that author review surfaced no offensive content, consistent with the technical math and science domain of the source questions\.

## Appendix KCompute and Reproducibility

#### Generation backends\.

AllGAand prompted\-EAinferences use OpenAI’s commercial API: an OpenAI o3\+\+GPT\-4o mixed configuration withGA= OpenAI o3 andEA= GPT\-4o, GPT\-5, and GPT\-5\-mini\. Generations were collected in an offline batch regime, decoupling pipeline depth from end\-to\-end latency at downstream deployment time\. Per\-pipeline cost was a binding constraint and influenced the decision to cap retries at five attempts and to restrict the per\-pipeline sweep to the first 20 questions of the TheoremQA subset\.

#### Fine\-tunedEA\.

The BERT\-base\-uncased classifier in P8 is the standardbert\-base\-uncasedmodel \(12 layers, 768\-dimensional hidden states, 12 attention heads; vocabulary size 30,522\) configured for sequence classification with two output labels\. Fine\-tuning was carried out on a single Google Colab GPU using the HuggingFacetransformersTrainer\(version 4\.56\.1\) with the following hyperparameters: 3 epochs, per\-device train batch size 16, per\-device eval batch size 64, 500 warmup steps, weight decay0\.010\.01, AdamW optimizer with the HuggingFace default learning rate \(5×10−55\\times 10^\{\-5\}\) and linear schedule, and the HuggingFaceTrainer’s default seed\. The 1,600\-example training set was split 60/20/20 into train/validation/test \(960/320/320\) usingsklearn’strain\_test\_split\(random\_state=42\)\. Wall\-clock fine\-tuning time was on the order of minutes given the small training\-set size\.

#### API usage\.

Per\-pipeline API spend was bounded by the offline\-batch regime and the five\-attempt retry cap described in §[4](https://arxiv.org/html/2605.29007#S4)\. The released JSON logs store per\-call token counts for the o3\+\+GPT\-4o, GPT\-5, and GPT\-5\-mini runs\. The o3\+\+GPT\-4o backend was only run on the Tier\-1 first\-20 sweep, whereas GPT\-5 and GPT\-5\-mini also have q20\+ 5\-cap extensions\. Table[13](https://arxiv.org/html/2605.29007#A11.T13)reports average per\-cell token consumption, $\-cost, and end\-to\-end latency by backend and pipeline\. Token counts include input, output, and \(billed\-as\-output\) reasoning tokens across every retry, with costs computed at OpenAI’s public pricing as of 2026\-05 \(o3: input $2\.002\.00/11M, output $8\.008\.00/11M; GPT\-4o: input $2\.502\.50/11M, output $1010/11M; GPT\-5: input $1\.251\.25/11M, output $1010/11M; GPT\-5\-mini: input $0\.250\.25/11M, output $22/11M; reasoning tokens billed at the output rate\)\. P8’s EA call is the fine\-tuned BERT classifier and is treated as free; only the GA call contributes to its API cost\.

#### Sampling parameters\.

All LLM calls use the API provider’s default temperature unless otherwise noted, with top\-ppleft at default\. The offline batch regime caches generations so that downstream measurement is reproducible without re\-querying the APIs\.

#### Run counts and variance\.

Each\(question, target class\)cell in Tables[3](https://arxiv.org/html/2605.29007#S6.T3),[8](https://arxiv.org/html/2605.29007#A6.T8), and[10](https://arxiv.org/html/2605.29007#A8.T10)is a single stochastic draw at temperature 1; we did not collect multiple completions per cell to average over per\-cell variance\. Cross\-question variance is built into the means \(20 questions×\\times5 classes per pipeline\), but the absolute targeted\-error rates should be interpreted as estimates rather than calibrated point values, as noted in the Limitations section\.

#### Random seeds\.

The train/validation/test split usesrandom\_state=42\. The BERT\-EAfine\-tune itself relies on the HuggingFaceTrainer’s default seed; we did not setseedexplicitly in theTrainingArguments, so replications should expect minor run\-to\-run variation in absolute targeted\-error rates of order a few percentage points, consistent with the single\-sample\-per\-cell disclosure in the Limitations section\.

Table 13:Average per\-cell token consumption, $\-cost, and end\-to\-end latency by backend and pipeline, aggregated over all available logged cells\. “tok\./cell” sums GA and EA input, output, and billed\-as\-output reasoning tokens across every retry\. “$/cell” uses OpenAI public pricing as of 2026\-05 \(o3 input $2\.00/1M, output $8\.00/1M; GPT\-4o input $2\.50/1M, output $10/1M; GPT\-5 input $1\.25/1M, output $10/1M; GPT\-5\-mini input $0\.25/1M, output $2/1M\)\. P8’sEAis a local BERT classifier and contributes zero API cost\.