Grammar Concept Annotation at Scale: Deployed Fine-Tuned Small Language Models Outperform Prompted Frontier Models

# Grammar Concept Annotation at Scale: Deployed Fine-Tuned Small Language Models Outperform Prompted Frontier Models
Source: [https://arxiv.org/html/2610.10827](https://arxiv.org/html/2610.10827)
###### Abstract

Corrective feedback is among the best\-evidenced drivers of second\-language acquisition, yet corrections delivered during lessons rarely accumulate into an actionable view of grammar mastery\. Prompted frontier models can provide such a view from learner–tutor lesson transcripts, but they are costly at scale\. We close this gap by fine\-tuning Qwen3\.5 small language models \(SLMs\) on filtered and rebalanced teacher\-generated supervision, then deploying an efficient 0\.8B model in an end\-to\-end grammar mastery tracker for all English learners on our platform\. Internalizing the annotation contract into adapter weights enables pairing the 0\.8B model with a compact matched prompt rather than verbose instructions\. On two human\-curated benchmarks, both the deployed 0\.8B model and a 4B reference comparator outperform prompted GPT\-5\.4 and GPT\-5\.6 Sol in precision and recall under nested matching criteria of increasing strictness: concept, evidence span, and correctness\. The deployed 0\.8B SLM reduces serving cost by approximately 16×\\times\. A feature\-level online experiment shows significant gains in learner engagement \(\+15\.8%\+15\.8\\%\) and key business metrics, including scheduled hours \(\+2\.1%\+2\.1\\%\) and GMV from new lessons \(\+13\.2%\+13\.2\\%\)\.

## 1Introduction

Corrective feedback is one of the best\-evidenced drivers of second\-language acquisition\([Li, 2010](https://arxiv.org/html/2610.10827#bib.bib16)\)\. In live tutoring, however, corrections are often given in the moment rather than retained as structured evidence of progress\. This makes it difficult for learners and tutors to track which grammatical concepts have been mastered, which have merely been avoided, and where the largest learning gaps remain\. An explicit, longitudinal record bridges this gap: learners see which concepts to practice, while tutors see what to revisit in future lessons\.

We present a grammar mastery tracking system deployed to all English learners on our platform\. It processes speech\-to\-text transcripts of lesson audio111Lesson transcripts are analyzed exclusively from sessions for which we have a lawful basis to do so \(see Ethical Considerations\)\.and aggregates correct and incorrect concept evidence into overall and per\-concept grammar mastery scores \(Figure[1](https://arxiv.org/html/2610.10827#S1.F1)\)\. Evidence is weighted by concept frequency, CEFR level\([Council of Europe, 2020](https://arxiv.org/html/2610.10827#bib.bib32)\), and a forgetting curve that discounts older evidence, consistent with trainable memory models for language learning\([Settles and Meeder, 2016](https://arxiv.org/html/2610.10827#bib.bib27)\)\. Every piece of evidence is traceable to a highlighted span in the learner’s own sentences; incorrect uses also receive a short explanation \(Figure[2](https://arxiv.org/html/2610.10827#S2.F2)c\)\.

At the core of the system is the*grammar concept annotation*task that identifies which curricular concepts a learner attempts, extracts the corresponding evidence spans, and judges each use as correct or incorrect\. For incorrect uses, it also generates learner\-facing explanations\. The model returns these predictions in a single structured pass per learner turn, capturing both correct mastery and actionable errors\.

Our initial product pilot prompted GPT\-5\.4 for grammar concept annotation in learner utterances extracted from lesson transcripts\. While it achieved the desired annotation quality, prompting GPT\-5\.4 at production scale would have far exceeded the feature’s serving budget \(Section[7](https://arxiv.org/html/2610.10827#S7)\)\. Each request repeated the complete concept inventory, tagging rules, examples, and span conventions\. Therefore, the pilot was initially rolled out to a small fraction of learners\. The same pattern is widely reported for enterprise AI: many products remain in the pilot phase, and only about 5% reach production\([Challapally et al\., 2025](https://arxiv.org/html/2610.10827#bib.bib33)\)\.

To make full rollout viable and improve quality, we fine\-tuned Qwen3\.5 SLMs on filtered and rebalanced teacher\-generated supervision, internalizing the task contract through demonstrations rather than verbose instructions\. We deployed an efficient 0\.8B SLM with a compact matched prompt, using a 4B model as a reference comparator to confirm that aggressive prompt compaction and parameter reduction incur no quality penalty\. On human\-curated benchmarks, both models surpass prompted GPT\-5\.4 and GPT\-5\.6 Sol\. The contributions of this work are as follows:

45%Grammarmastery score

ConceptUsedMasteryRelative clauses2k10%Modal probability32025%Some / any / no65025%\+5Past perfect24040%\+5Present perfect cont\.1k70%…

Figure 1:Product view of the aggregated grammar mastery score \(left\) and per\-concept breakdown \(right; teal values indicate recent mastery gains\)\.1. 1\.A scalable end\-to\-end grammar mastery tracking system, deployed and evaluated in a feature\-level online experiment showing significant uplift in learner engagement, tutoring hours, and gross merchandise value \(GMV\) from new lessons\.
2. 2\.A formulation of learner grammar feedback as*grammar concept annotation*\(Section[2](https://arxiv.org/html/2610.10827#S2)\), with two human\-curated benchmarks and nested criteria for concept, exact evidence span, and correctness\. On both benchmarks, our deployed 0\.8B model and 4B reference comparator outperform prompted GPT\-5\.4 and GPT\-5\.6 Sol at every matching level in precision and recall\.
3. 3\.A data quality pipeline combining evidence\-grounded review, deterministic validation, filtering, and synthetic enrichment\. Relative to matched 4B and 0\.8B models trained on raw teacher annotations, the full pipeline raises recall by 36\.9–40\.7 percentage points when concept, exact span, and correctness must all match, with further gains on rare concepts and incorrect uses\.
4. 4\.A production serving architecture pairing the task\-specialized 0\.8B SLM with prompt compaction, native vLLM, and autoscaling, reducing serving cost by approximately 16×\\timesrelative to GPT\-5\.4 based on measured production traffic and load tests\.

## 2Grammar Concept Annotation Task

The production pipeline turns speech\-to\-text transcripts of lesson audio into structured evidence for concept\-level mastery tracking\. It extracts learner utterances and relevant context, then an upstream LLM produces a corrected sentence that represents the learner’s intended meaning\. The annotation model receives the original utterance and this correction, rather than the full transcript\.

For an input pair\(ui,ci\)\(u\_\{i\},c\_\{i\}\), whereuiu\_\{i\}is the learner utterance andcic\_\{i\}its correction, the system produces a set of annotations

A⁡\(ui,ci\)=\{\(kj,sj,yj,ej\)\}j=1mi,mi≥0,A\(u\_\{i\},c\_\{i\}\)=\\\{\(k\_\{j\},s\_\{j\},y\_\{j\},e\_\{j\}\)\\\}\_\{j=1\}^\{m\_\{i\}\},\\qquad m\_\{i\}\\geq 0,wherekj∈𝒦k\_\{j\}\\in\\mathcal\{K\}is one of the\|𝒦\|=34\|\\mathcal\{K\}\|=34grammar concepts,222The 34 concepts correspond to an initial version of the system; the architecture extends and scales to a much larger number of concepts, with the next iteration containing more than 300 fine\-grained concepts\.sjs\_\{j\}is an exact contiguous substring ofuiu\_\{i\},yj∈\{correct,incorrect\}y\_\{j\}\\in\\\{\\textsc\{correct\},\\textsc\{incorrect\}\\\}judges the use ofkjk\_\{j\}insjs\_\{j\}, andeje\_\{j\}is a learner\-facing explanation that is non\-empty if and only ifyj=incorrecty\_\{j\}=\\textsc\{incorrect\}\. The model emits this set in a single pass: one utterance may yield several concepts or none\. Becausesjs\_\{j\}must be an exact contiguous substring ofuiu\_\{i\}, the same span can be highlighted in the learner’s sentence and compared with gold spans by string identity at evaluation\.

Figure[2](https://arxiv.org/html/2610.10827#S2.F2)shows an example input, model output, and learner\-facing rendering\. Appendix Table[11](https://arxiv.org/html/2610.10827#A5.T11)states the same contract field by field\. The runtime representation is one pipe\-delimited annotation per line\. The format was chosen during the product pilot to balance quality and cost\.

\(a\) Input Original: I have finish almost all\. Corrected: I have finished almost all\. \(b\) Model output 1\|PRESENT\_PERFECT\|N\|have finish\|Use ‘‘finished’’\. 1\|QUANTIFIERS\_BASIC\|Y\|almost all\| \(c\) Learner\-facing rendering Present perfect Ihave finishalmost all\. →\\rightarrowI havefinishedalmost all\. Use the past participle “finished” after “have”\.

Figure 2:Illustrative synthetic example\. \(a\) Learner utterance and upstream correction\. \(b\) Model output: sentence index, concept, correctness \(Y/N\), exact source span, and explanation for incorrect use only\. \(c\) Learner\-facing rendering of one incorrect annotation; correct annotations contribute to mastery without appearing as feedback\.The nested structure of the tuple defines the evaluation hierarchy in Section[5](https://arxiv.org/html/2610.10827#S5): matching onkjk\_\{j\}, on\(kj,sj\)\(k\_\{j\},s\_\{j\}\), and on\(kj,sj,yj\)\(k\_\{j\},s\_\{j\},y\_\{j\}\)yields progressively stricter criteria, whileeje\_\{j\}is assessed separately because explanations are free text\.

## 3Data and Quality Pipeline

### 3\.1Training Data Collection, Cleaning, and Enrichment

Directly fine\-tuning on raw production logs is hindered by teacher annotation noise, span boundary drift, and severe class imbalance favoring frequent, error\-free structures\. We address these challenges through a three\-stage curation pipeline: \(i\) evidence\-grounded review and programmatic validation of production annotations; \(ii\) structured synthetic generation to rebalance rare concepts and pedagogical errors; and \(iii\) corpus consolidation, deduplicating across sources \(1\.4% overlap\) and strictly removing any utterance present in either golden benchmark\. The resulting training corpus contains approximately 187k utterance groups \(learner utterance, correction, and annotations\), consolidated from 100k reviewed production records and 90k synthetic records prior to deduplication\.

#### Production data and cleaning\.

Each production group pairs a learner utterance and upstream correction with span\-level annotations from the product pilot\. We sample 50k groups uniformly to preserve production traffic distributions, and 50k via heuristic sampling prioritizing utterances suspected of incomplete or excessive annotations \(anomalous counts relative to length, or low span coverage\)\. To clean these records, we adopt a hybrid neuro\-symbolic strategy: deterministic linguistic anchors extracted via spaCy\([Honnibal et al\., 2020](https://arxiv.org/html/2610.10827#bib.bib29)\)are converted into a plain\-text inventory of grammatical elements \(articles, pronouns, prepositions, modals, verb forms, and tense patterns\), alongside hand\-crafted candidate rules\. In the prompt to GPT\-5\.4, these cues are combined with word\-level edit differences from the upstream correction \(pinpointing where the learner diverged from target grammar\) and the pilot annotations\. Rather than filtering alone, GPT\-5\.4 performs a full review and re\-annotation pass \(retaining, correcting, removing, or adding tuples\)\. A deterministic post\-validator then enforces strict contract invariants: rejecting non\-substring spans, pruning invalid labels, and enforcing explanation consistency \(Appendix[B](https://arxiv.org/html/2610.10827#A2)\)\.

#### Synthetic enrichment\.

Unconstrained synthetic generation often hallucinates labels or perpetuates natural class skews\. We instead decouple syntactic scaffolding from lexical realization using structural*backbones*—sequences of concept labels and correctness flags harvested from quality\-filtered production utterances\. To address long\-tail sparsity, we stratify backbone sampling across three 30k subsets: uniform sampling, error\-weighted sampling \(favoring templates with natural mistakes\), and smoothed inverse concept\-frequency sampling \(boosting rare curricular targets\)\. Unlike post\-hoc token corruption, which produces disfluent or unnatural errors, we inject pedagogical errors directly into the generation plan: withp=0\.05p=0\.05, a correct slot is assigned an error type \(within\-concept misuse, omission, or confusable structure substitution\)\. Prompted with the backbone and matching production exemplars, GPT\-5\.4 realizes the plan into learner\-style sentences, exact spans, and targeted explanations\. Deterministic post\-validation then enforces span substring constraints and label alignment \(Appendix[B](https://arxiv.org/html/2610.10827#A2)\)\.

### 3\.2Golden Evaluation Benchmarks

Evaluating an educational tracking system requires measuring both overall operational accuracy and robustness on critical failure modes\. We curate two independent human\-adjudicated benchmarks over the 34 concepts \(Table[1](https://arxiv.org/html/2610.10827#S3.T1)\)\.*Golden*\(689 utterances\) mirrors the natural production distribution, providing an unskewed evaluation of daily traffic\. In contrast,*Golden2*\(2,977 utterances, constructed via model\-assisted candidate retrieval followed by double\-annotator curation and adjudication\) is an error\-enriched diagnostic benchmark \(95\.6% error\-bearing utterances vs\. 50\.1% in Golden\)\. Golden2 stress\-tests the system against the product’s costliest failure mode—failing to detect a genuine learner error—while providing ample support to evaluate explanation fidelity\. In both benchmarks, two annotators worked independently and a third adjudicated disagreements\.

Table 1:Composition of the two golden evaluation benchmarks under the common 34\-concept evaluation scope\. Annotation counts are computed after scope filtering\.Learner ASR transcripts, annotations, and internal prompts are proprietary and privacy\-sensitive, so we report aggregate statistics and methodological details but do not release records\. Appendix[E\.4](https://arxiv.org/html/2610.10827#A5.SS4)reports an additional teacher\-synthetic distribution\-shift probe\.

## 4Fine\-Tuning Method

We adapted Qwen3\.5\([Qwen Team, 2026](https://arxiv.org/html/2610.10827#bib.bib8)\)and Qwen3\([Yang et al\., 2025](https://arxiv.org/html/2610.10827#bib.bib30)\)base models with completion\-only LoRA\([Hu et al\., 2022](https://arxiv.org/html/2610.10827#bib.bib9)\)\. At training and serving time, packing five utterance–correction pairs into one conversation \(Section[7](https://arxiv.org/html/2610.10827#S7)\) balanced throughput and memory; scaling further yielded diminishing returns while increasing activation memory\. We used a 90/10 train–validation split and discarded examples exceeding the maximum sequence length rather than truncating targets\. Checkpoints were selected by minimum teacher\-forced validation loss evaluated four times per epoch with early\-stopping patience of three evaluations \(Appendix[A](https://arxiv.org/html/2610.10827#A1)\)\.

#### Prompt internalization and model sizing\.

Our primary deployment target is an efficient, sub\-1B parameter model \(Qwen3\.5\-0\.8B\) capable of serving high\-concurrency traffic on single\-GPU commodity instances \(g6\.2xlarge\)\. In the production pilot, GPT\-5\.4 required a 3,810\-token prompt detailing all 34 concepts, edge cases, and examples; at scale, repeatedly ingesting this verbose prompt induces severe prefill latency and dominates KV\-cache memory in vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.10827#bib.bib7)\), restricting concurrency\. However, task fine\-tuning enables*prompt internalization*\([Zou et al\., 2024](https://arxiv.org/html/2610.10827#bib.bib6);[Shin et al\., 2025](https://arxiv.org/html/2610.10827#bib.bib25)\): the model absorbs concept definitions and span conventions directly into adapter weights from demonstrations\. We therefore designed a compact prompt \(375 Qwen3\.5 tokens, a 90\.2% reduction; Appendix[E\.2](https://arxiv.org/html/2610.10827#A5.SS2)\) specifying only concept names, the schema contract, and brief rules, shrinking the maximum sequence length from 7,000 to 2,500 tokens \(used for both 0\.8B and 0\.6B; Appendix[A](https://arxiv.org/html/2610.10827#A1)\)\. To provide sufficient capacity for these rules, the 0\.8B model used a higher adapter rank \(r=128r=128across all linear layers vs\.r=16r=16for 4B; Appendix Table[5](https://arxiv.org/html/2610.10827#A1.T5)\)\. The larger Qwen3\.5\-4B model retained the detailed prompt strictly as a reference comparator to confirm that prompt compaction and parameter reduction incur no quality penalty\. Differences between sizes thus reflect distinct operational recipes rather than an isolated parameter ablation\.

## 5Evaluation Methodology

Table 2:Micro precision and recall \(%\) on Golden and Golden2\. GPT\-5\.4 is the production prompted model; GPT\-5\.6 Sol is a quality\-only comparator\. Bold marks the best value in each row\. Base denotes untuned Qwen3\.5 models; fine\-tuned denotes selected full\-pipeline checkpoints \(raw\-supervision controls in Table[3](https://arxiv.org/html/2610.10827#S6.T3)\)\. The two fine\-tuned models use independently tuned recipes; the 0\.8B checkpoint, marked†, combines a compact prompt with separately tuned hyperparameters, so differences between sizes are not a controlled comparison\. Bootstrap 95% confidence intervals appear in Appendix Table[7](https://arxiv.org/html/2610.10827#A3.T7); key\-macro results in Appendix Table[8](https://arxiv.org/html/2610.10827#A3.T8)\. All fine\-tuned Qwen–GPT\-5\.4 differences are significant under paired randomization tests \(p<0\.0001p<0\.0001\)\.#### Annotation evaluation\.

Each system predicts a setA^​\(ui,ci\)\\hat\{A\}\(u\_\{i\},c\_\{i\}\)over the 34\-concept inventory, scored against goldA⁡\(ui,ci\)A\(u\_\{i\},c\_\{i\}\)from Section[2](https://arxiv.org/html/2610.10827#S2)\. We benchmark against two reference classes: frontier models establish operational parity \(production GPT\-5\.4\) and a strong quality comparator \(GPT\-5\.6 Sol\), while base\-model comparisons isolate task\-adaptation gains against untuned counterparts\. Unparseable outputs are recorded as format\-adherence failures \(parse errors\) and treated as empty predictions, measuring syntactic reliability under autoregressive generation\.

We score the nested criteria of Section[2](https://arxiv.org/html/2610.10827#S2)at L1 \(kjk\_\{j\}\), L2 \(\(kj,sj\)\(k\_\{j\},s\_\{j\}\)\), and L3 \(\(kj,sj,yj\)\(k\_\{j\},s\_\{j\},y\_\{j\}\)\), reporting micro precision and recall across utterances \(key\-macro averages in Appendix[C\.2](https://arxiv.org/html/2610.10827#A3.SS2)\)\. Test\-set uncertainty is estimated via percentile 95% confidence intervals from 20,000 paired nonparametric cluster bootstrap resamples of packed five\-utterance inference batches, preserving output dependencies\. Differences from GPT\-5\.4 are assessed via two\-sided paired randomization tests with 20,000 permutations\([Koehn, 2004](https://arxiv.org/html/2610.10827#bib.bib11);[Dror et al\., 2018](https://arxiv.org/html/2610.10827#bib.bib23)\)\(Appendix[C\.4](https://arxiv.org/html/2610.10827#A3.SS4)\)\.

#### Explanation evaluation\.

Because surface n\-gram metrics correlate poorly with pedagogical feedback quality, we adopt an LLM\-as\-a\-judge framework\([Zheng et al\., 2023](https://arxiv.org/html/2610.10827#bib.bib12);[Kobayashi et al\., 2024](https://arxiv.org/html/2610.10827#bib.bib24)\): GPT\-5\.5 evaluates explanation acceptability given the utterance, span, reference correction, and concept definition \(rubric in Appendix[E\.3](https://arxiv.org/html/2610.10827#A5.SS3)\)\. To counter potential model leniency, a human reviewer audits the judgments, and an independent second reviewer reconciles disagreements\.

#### Pipeline ablation\.

To isolate pipeline contributions, we trained raw\-supervision controls using unreviewed GPT\-5\.4 annotations with identical prompts, splits, hyperparameters, and matched optimizer\-update budgets\. Because model sizes use independently tuned recipes, each control is compared only against its same\-size counterpart\. To evaluate how well the pipeline addresses specific distribution skews, we report equal\-weight L3 recall across all 34 concepts \(Mean\-34; curricular balance\), micro L3 recall restricted to gold incorrect uses \(Incorrect; error sensitivity\), and equal\-weight L3 recall on*Rare\-9*\(the nine least frequent concepts in raw training data, validating synthetic tail\-rebalancing; Appendix[C\.3](https://arxiv.org/html/2610.10827#A3.SS3)\)\.

## 6Results

#### Annotation accuracy and specialization\.

Table[2](https://arxiv.org/html/2610.10827#S5.T2)reports precision and recall on both benchmarks\. Both fine\-tuned models outperform GPT\-5\.6 Sol across all criteria, with the largest uplift under strict L3 matching \(concept, exact span, and correctness\)\.

#### Base\-model controls and format adherence\.

Untuned 4B and 0\.8B models fail to parse on 26\.9–32\.8% and 75\.3–92\.0% of examples, respectively, demonstrating a wide adaptation gap\. An earlier\-generation Qwen3\-0\.6B checkpoint trained with the 0\.8B recipe produced unparseable output on 57\.4% of Golden2 versus 0\.03% for 0\.8B despite smooth validation convergence \(generational differences prevent attributing this gap solely to parameter count; Appendix[A](https://arxiv.org/html/2610.10827#A1)\)\.

Table 3:Pipeline ablation \(%\) on Golden2 and Golden\. Bold marks the best value in each column within each benchmark\. Mean\-34 is equal\-weight recall over all concepts; Rare\-9 is equal\-weight recall over the nine concepts fixed from the raw training split; Incorrect restricts recall to gold incorrect\-use annotations\. Full\-pipeline rows are the selected checkpoints in Table[2](https://arxiv.org/html/2610.10827#S5.T2)\.
#### Data\-pipeline ablation\.

Matched within\-size controls isolate the combined supervision pipeline \(Table[3](https://arxiv.org/html/2610.10827#S6.T3)\)\. On Golden2 it raises L3 precision by 28\.37 \(4B\) and 29\.93 \(0\.8B\) points and L3 recall by 36\.91 \(95% CI \[35\.93, 37\.87\]\) and 39\.95 \(\[39\.01, 40\.89\]\) points; on Golden, L3 recall rises by 37\.33 \(\[35\.15, 39\.48\]\) and 40\.66 \(\[38\.44, 42\.81\]\) points\. Gains extend across all subsets, notably Rare\-9 \(up 45\.31 and 45\.81 points on Golden2\)\. On Golden2, raw\-supervision recall drops by 23\.64–23\.78 points from L1 to exact\-span L2, compared to 6\.28–6\.31 points for the full pipeline, underscoring an acute exact\-span degradation under raw supervision\.

#### Explanation quality\.

On Golden2, GPT\-5\.5 judged 99\.06% of GPT\-5\.4 explanations acceptable versus 94\.91% \(4B\) and 93\.88% \(0\.8B\)\. However, following human audit and reconciliation, all three systems achieved similar acceptable rates \(≈\\approx95%\) with no significant differences\. This disparity indicates that the uncalibrated LLM judge favored its frontier sibling, underscoring the necessity of human calibration\.

#### Online product impact\.

We deployed the complete system with Qwen3\.5\-0\.8B \(Section[7](https://arxiv.org/html/2610.10827#S7)\) in a 35\-day feature\-level A/B experiment randomized at the learner level with 50% exposure\. At the fixed pre\-scaling snapshot \(Table[4](https://arxiv.org/html/2610.10827#S6.T4)\), learner engagement with progress and all three business metrics improved significantly\. These gains apply to the end\-to\-end product feature rather than the SLM in isolation\.

Table 4:Relative effects in the online experiment at the fixed pre\-scaling evaluation snapshot \(all corresponding 95% CIs exclude zero; bold marksp<0\.05p<0\.05\)\.

## 7Deployment and Operational Considerations

Our production serving architecture prioritizes high concurrent throughput and cost\-efficient horizontal scaling\. We deploy Qwen3\.5\-0\.8B with vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.10827#bib.bib7)\)on autoscaled, single\-GPUg6\.2xlargeSageMaker instances that allow fine\-grained capacity scaling\. Before inference, the application packs prepared utterance–correction pairs into five\-pair requests to an internal inference service\. Requests are sequential within each transcript chunk, and vLLM batches concurrent requests arriving at each replica\. The service applies the compact prompt \(Appendix[E\.2](https://arxiv.org/html/2610.10827#A5.SS2)\), parses the structured output, and returns annotations aligned to input indices\. Validation or service failures route work to GPT\-5\.4 \(topology in Appendix Figure[3](https://arxiv.org/html/2610.10827#A4.F3); autoscaling and load testing in Appendix[D\.1](https://arxiv.org/html/2610.10827#A4.SS1)\)\.

The warmed load test exercised this packed\-request path: 8g6\.2xlargeinstances sustained 4,993\.5 pairs per minute, exceeding the observed post\-rollout peak\. During a 31\-day production APM snapshot, p95 latency was 4\.3 seconds and the request error rate was 0\.0015%\. Combining measured GPT\-5\.4 production cost, observed demand, load\-tested SLM capacity, and public SageMaker prices, recurring detection cost per lesson is 6\.2% of the GPT\-5\.4 baseline—an approximately 16×\\timesreduction \(Appendix[D\.2](https://arxiv.org/html/2610.10827#A4.SS2)\)\.

## 8Related Work

#### Task formulation\.

Grammatical error detection and correction localize or repair deviations\([Rei and Yannakoudakis, 2016](https://arxiv.org/html/2610.10827#bib.bib4);[Bryant et al\., 2023](https://arxiv.org/html/2610.10827#bib.bib2)\), label edits, or generate learner feedback\([Bryant et al\., 2017](https://arxiv.org/html/2610.10827#bib.bib3);[Nagata, 2019](https://arxiv.org/html/2610.10827#bib.bib5);[Liang et al\., 2023](https://arxiv.org/html/2610.10827#bib.bib20);[Song et al\., 2024](https://arxiv.org/html/2610.10827#bib.bib28)\)\. Curriculum\-grounded methods often target predefined activities or dialogue contrasts\([Quixal et al\., 2021](https://arxiv.org/html/2610.10827#bib.bib17);[Akef et al\., 2026](https://arxiv.org/html/2610.10827#bib.bib18);[Ribeiro\-Flucht et al\., 2026](https://arxiv.org/html/2610.10827#bib.bib19)\)\. In contrast, we identify concepts from a 34\-concept inventory in unconstrained transcribed speech turns, jointly predicting evidence spans, correctness, and explanations\.

#### Teacher\-generated and synthetic supervision\.

Prior work generates synthetic GEC data via tagged corruption or LLM prompting and adapts distributions via type\-aware augmentation or reweighting\([Stahlberg and Kumar, 2021](https://arxiv.org/html/2610.10827#bib.bib13);[Li and Lan, 2025](https://arxiv.org/html/2610.10827#bib.bib22);[Vainikko et al\., 2025](https://arxiv.org/html/2610.10827#bib.bib21);[Zhang et al\., 2025](https://arxiv.org/html/2610.10827#bib.bib14)\)\. Our pipeline instead filters production annotations and samples concept–correctness backbones to rebalance the curriculum, benchmarked against matched raw\-supervision controls\.

#### SLM specialization and deployment\.

Prior work internalizes prompts\([Zou et al\., 2024](https://arxiv.org/html/2610.10827#bib.bib6);[Shin et al\., 2025](https://arxiv.org/html/2610.10827#bib.bib25)\), distills teacher data into lightweight models\([Wang et al\., 2025](https://arxiv.org/html/2610.10827#bib.bib15)\), or supplies LLM explanations to SLM GEC systems\([Li et al\., 2025](https://arxiv.org/html/2610.10827#bib.bib26)\)\. We combine prompt internalization, a compact schema prompt, and completion\-only LoRA in a deployed sub\-1B SLM, jointly evaluating accuracy, serving cost, and online impact\.

## 9Conclusion

We presented a deployed grammar tracker jointly predicting curricular concepts, exact evidence spans, correctness, and explanations for errors\. Fine\-tuned Qwen3\.5 models outperformed GPT\-5\.4 and GPT\-5\.6 Sol across all matching levels on two human\-curated benchmarks\. Ablations confirmed our data pipeline yields 36\.9–40\.7 points of strict L3 recall over raw teacher supervision\. The deployed 0\.8B model cut serving cost by≈\\approx16×\\times, while an online experiment demonstrated significant engagement and business gains, showing that data quality, prompt internalization, and efficient serving make accurate feedback viable at scale\.

## Limitations

#### Modeling and training controls\.

Our pipeline ablation benchmarks raw supervision against the combined pipeline \(review, deterministic validation, filtering, and synthetic enrichment\), but does not isolate the individual contribution of each component\. Furthermore, we report results from a single training run per configuration and do not estimate across\-seed variance, meaning fine\-grained metric differences should not be over\-interpreted\. Finally, because the 4B and 0\.8B models rely on independently tuned training recipes and prompt lengths, and the 0\.6B comparison belongs to an earlier Qwen generation, differences across model sizes reflect distinct operational configurations rather than an isolated parameter ablation\.

#### Operational and evaluation scope\.

The online A/B experiment evaluates the end\-to\-end product feature rather than the SLM or serving infrastructure in isolation\. Similarly, our serving cost model isolates recurring grammar\-detection compute and excludes upstream transcription, correction, and fallback calls to frontier APIs\. Regarding explanation evaluation, while human audit reconciled acceptability rates, the leniency gap in the uncalibrated LLM judge demonstrates the need for domain\-specific judge calibration; moreover, explanations were evaluated against gold rubrics rather than through direct perceptual studies with teachers or learners\.

#### Data scope, speech, and supervision biases\.

While our deployed checkpoints target an English\-language curriculum of 34 concepts, the underlying pipeline—curated supervision, prompt internalization, and lightweight serving—is framework\-agnostic and designed to scale to a substantially broader curriculum\. Because learner turns are transcribed via automatic speech recognition \(ASR\), transcription errors can cascade into upstream correction and downstream concept tracking; we do not isolate robustness to ASR noise\. Finally, because both reviewed production supervision and synthetic sentences derive from GPT\-5\.4, systematic teacher biases may persist in ways our benchmarks cannot fully reveal\.

## Ethical Considerations

We analyze lesson transcripts only from sessions for which we have a lawful basis to do so\. Learner utterances, annotations, and internal prompts are not released; the illustrative example in Figure[2](https://arxiv.org/html/2610.10827#S2.F2)is synthetic\. The paper reports aggregate evaluation and online\-experiment results only\.

## References

- Akefet al\.\(2026\)S\. Akef, A\. Mendes, P\. Rebuschat, and D\. MeurersLLM\-powered but rule\-grounded: pedagogically relevant grammatical error characterization for learner model construction\.InProceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2026\),San Diego, California, USA,pp\. 851–862\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.bea-1.58),[Link](https://aclanthology.org/2026.bea-1.58/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px1.p1.1)\.
- Amazon Web Services \(2026\)Amazon Web ServicesAWS price list for amazon SageMaker\.Note:Published 4 September 2026; accessed 7 September 2026External Links:[Link](https://aws.amazon.com/sagemaker/pricing/)Cited by:[§D\.2](https://arxiv.org/html/2610.10827#A4.SS2.p2.1)\.
- Bryantet al\.\(2017\)C\. Bryant, M\. Felice, and T\. BriscoeAutomatic annotation and evaluation of error types for grammatical error correction\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vancouver, Canada,pp\. 793–805\.External Links:[Document](https://dx.doi.org/10.18653/v1/P17-1074),[Link](https://aclanthology.org/P17-1074/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px1.p1.1)\.
- Bryantet al\.\(2023\)C\. Bryant, Z\. Yuan, M\. R\. Qorib, H\. Cao, H\. T\. Ng, and T\. BriscoeGrammatical error correction: a survey of the state of the art\.Computational Linguistics49\(3\),pp\. 643–701\.External Links:[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00478),[Link](https://aclanthology.org/2023.cl-3.4/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px1.p1.1)\.
- Challapallyet al\.\(2025\)A\. Challapally, C\. Pease, R\. Raskar, and P\. ChariThe GenAI divide: state of AI in business 2025\.Research reportMIT Project NANDA\.External Links:[Link](https://www.actuarialpost.co.uk/downloads/cat_1/MIT-ai_report_2025.pdf)Cited by:[§1](https://arxiv.org/html/2610.10827#S1.p4.1)\.
- Chenet al\.\(2016\)T\. Chen, B\. Xu, C\. Zhang, and C\. GuestrinTraining deep nets with sublinear memory cost\.arXiv preprint arXiv:1604\.06174\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1604.06174),[Link](https://arxiv.org/abs/1604.06174)Cited by:[Appendix A](https://arxiv.org/html/2610.10827#A1.p1.1)\.
- Council of Europe \(2020\)Council of EuropeCommon european framework of reference for languages: learning, teaching, assessment—companion volume\.Council of Europe Publishing,Strasbourg\.External Links:ISBN 978\-92\-871\-8621\-8,[Link](https://rm.coe.int/common-european-framework-of-reference-for-languages-learning-teaching/16809ea0d4)Cited by:[§1](https://arxiv.org/html/2610.10827#S1.p2.1)\.
- Droret al\.\(2018\)R\. Dror, G\. Baumer, S\. Shlomov, and R\. ReichartThe hitchhiker’s guide to testing statistical significance in natural language processing\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Melbourne, Australia,pp\. 1383–1392\.External Links:[Document](https://dx.doi.org/10.18653/v1/P18-1128),[Link](https://aclanthology.org/P18-1128/)Cited by:[§5](https://arxiv.org/html/2610.10827#S5.SS0.SSS0.Px1.p2.1)\.
- Honnibalet al\.\(2020\)M\. Honnibal, I\. Montani, S\. Van Landeghem, and A\. BoydspaCy: industrial\-strength natural language processing in python\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.1212303),[Link](https://doi.org/10.5281/zenodo.1212303)Cited by:[§3\.1](https://arxiv.org/html/2610.10827#S3.SS1.SSS0.Px1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§4](https://arxiv.org/html/2610.10827#S4.p1.1)\.
- Kobayashiet al\.\(2024\)M\. Kobayashi, M\. Mita, and M\. KomachiLarge language models are state\-of\-the\-art evaluator for grammatical error correction\.InProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2024\),Mexico City, Mexico,pp\. 68–77\.External Links:[Link](https://aclanthology.org/2024.bea-1.6/)Cited by:[§5](https://arxiv.org/html/2610.10827#S5.SS0.SSS0.Px2.p1.1)\.
- Koehn \(2004\)P\. KoehnStatistical significance tests for machine translation evaluation\.InProceedings of the 2004 Conference on Empirical Methods in Natural Language Processing,Barcelona, Spain,pp\. 388–395\.External Links:[Link](https://aclanthology.org/W04-3250/)Cited by:[§5](https://arxiv.org/html/2610.10827#S5.SS0.SSS0.Px1.p2.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles,pp\. 611–626\.External Links:[Document](https://dx.doi.org/10.1145/3600006.3613165),[Link](https://doi.org/10.1145/3600006.3613165)Cited by:[§4](https://arxiv.org/html/2610.10827#S4.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.10827#S7.p1.1)\.
- Li \(2010\)S\. LiThe effectiveness of corrective feedback in SLA: a meta\-analysis\.Language Learning60\(2\),pp\. 309–365\.External Links:[Document](https://dx.doi.org/10.1111/j.1467-9922.2010.00561.x),[Link](https://doi.org/10.1111/j.1467-9922.2010.00561.x)Cited by:[§1](https://arxiv.org/html/2610.10827#S1.p1.1)\.
- Li and Lan \(2025\)X\. Li and Y\. LanLarge language models are good annotators for type\-aware data augmentation in grammatical error correction\.InProceedings of the 31st International Conference on Computational Linguistics,Abu Dhabi, UAE,pp\. 199–213\.External Links:[Link](https://aclanthology.org/2025.coling-main.14/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2025\)Y\. Li, S\. Qin, J\. Ye, H\. Huang, Y\. Li, S\. Guo, L\. Qin, X\. Hu, W\. Jiang, H\. Zheng, and P\. S\. YuRethinking the roles of large language models in Chinese grammatical error correction\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),Vienna, Austria,pp\. 553–567\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-industry.39),[Link](https://aclanthology.org/2025.acl-industry.39/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px3.p1.1)\.
- Lianget al\.\(2023\)K\. Liang, S\. Davidson, X\. Yuan, S\. Panditharatne, C\. Chen, R\. Shea, D\. Pham, Y\. Tan, E\. Voss, L\. Fryer, and Z\. YuChatBack: investigating methods of providing grammatical error feedback in a GUI\-based language learning chatbot\.InProceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2023\),Toronto, Canada,pp\. 83–99\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.bea-1.7),[Link](https://aclanthology.org/2023.bea-1.7/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px1.p1.1)\.
- Nagata \(2019\)R\. NagataToward a task of feedback comment generation for writing learning\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),Hong Kong, China,pp\. 3206–3215\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1316),[Link](https://aclanthology.org/D19-1316/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px1.p1.1)\.
- Quixalet al\.\(2021\)M\. Quixal, B\. Rudzewitz, E\. Bear, and D\. MeurersAutomatic annotation of curricular language targets to enrich activity models and support both pedagogy and adaptive systems\.InProceedings of the 10th Workshop on NLP for Computer Assisted Language Learning,Online,pp\. 15–27\.External Links:[Link](https://aclanthology.org/2021.nlp4call-1.2/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px1.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4](https://arxiv.org/html/2610.10827#S4.p1.1)\.
- Rajbhandariet al\.\(2020\)S\. Rajbhandari, J\. Rasley, O\. Ruwase, and Y\. HeZeRO: memory optimizations toward training trillion parameter models\.InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis,pp\. 1–16\.External Links:[Document](https://dx.doi.org/10.1109/SC41405.2020.00024),[Link](https://doi.org/10.1109/SC41405.2020.00024)Cited by:[Appendix A](https://arxiv.org/html/2610.10827#A1.p1.1)\.
- Rei and Yannakoudakis \(2016\)M\. Rei and H\. YannakoudakisCompositional sequence labeling models for error detection in learner writing\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Berlin, Germany,pp\. 1181–1191\.External Links:[Document](https://dx.doi.org/10.18653/v1/P16-1112),[Link](https://aclanthology.org/P16-1112/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px1.p1.1)\.
- Ribeiro\-Fluchtet al\.\(2026\)L\. Ribeiro\-Flucht, L\. Huang, and X\. ChenFrom dialogue to learner modeling: identifying candidate signals of productive use in LLM\-based grammar practice\.InProceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2026\),San Diego, California, USA,pp\. 933–940\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.bea-1.62),[Link](https://aclanthology.org/2026.bea-1.62/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px1.p1.1)\.
- Settles and Meeder \(2016\)B\. Settles and B\. MeederA trainable spaced repetition model for language learning\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Berlin, Germany,pp\. 1848–1858\.External Links:[Document](https://dx.doi.org/10.18653/v1/P16-1174),[Link](https://aclanthology.org/P16-1174/)Cited by:[§1](https://arxiv.org/html/2610.10827#S1.p2.1)\.
- Shinet al\.\(2025\)H\. Shin, L\. Ji, Y\. Gong, S\. Kim, E\. Choi, and M\. SeoGenerative prompt internalization\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Albuquerque, New Mexico,pp\. 7338–7363\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.376),[Link](https://aclanthology.org/2025.naacl-long.376/)Cited by:[§4](https://arxiv.org/html/2610.10827#S4.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px3.p1.1)\.
- Songet al\.\(2024\)Y\. Song, K\. Krishna, R\. Bhatt, K\. Gimpel, and M\. IyyerGEE\! grammar error explanation with large language models\.InFindings of the Association for Computational Linguistics: NAACL 2024,Mexico City, Mexico,pp\. 754–781\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.49),[Link](https://aclanthology.org/2024.findings-naacl.49/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px1.p1.1)\.
- Stahlberg and Kumar \(2021\)F\. Stahlberg and S\. KumarSynthetic data generation for grammatical error correction with tagged corruption models\.InProceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications,Online,pp\. 37–47\.External Links:[Link](https://aclanthology.org/2021.bea-1.4/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px2.p1.1)\.
- Vainikkoet al\.\(2025\)M\. Vainikko, T\. Kamarik, K\. Kert, K\. Liin, S\. Maine, K\. Allkivi, A\. Kaivapalu, and M\. FishelParagraph\-level error correction and explanation generation: case study for Estonian\.InProceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2025\),Vienna, Austria,pp\. 953–967\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.bea-1.72),[Link](https://aclanthology.org/2025.bea-1.72/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2025\)C\. Wang, J\. Yan, Y\. Yue, and J\. HuangDistilQwen2\.5: industrial practices of training distilled open lightweight language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),Vienna, Austria,pp\. 32–42\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-industry.4),[Link](https://aclanthology.org/2025.acl-industry.4/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.09388),[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4](https://arxiv.org/html/2610.10827#S4.p1.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, Z\. Xu, S\. Wu, Y\. Zhang, and D\. RamageSynthesizing and adapting error correction data for mobile large language model applications\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),Vienna, Austria,pp\. 1102–1112\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-industry.78),[Link](https://aclanthology.org/2025.acl-industry.78/)Cited by:[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46595–46623\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html)Cited by:[§5](https://arxiv.org/html/2610.10827#S5.SS0.SSS0.Px2.p1.1)\.
- Zouet al\.\(2024\)J\. Zou, M\. Zhou, T\. Li, S\. Han, and D\. ZhangPromptIntern: saving inference costs by internalizing recurrent prompt during large language model fine\-tuning\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 10288–10305\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.602),[Link](https://aclanthology.org/2024.findings-emnlp.602/)Cited by:[§4](https://arxiv.org/html/2610.10827#S4.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2610.10827#S8.SS0.SSS0.Px3.p1.1)\.

## Appendix ATraining Configurations and Convergence

All runs used bfloat16, gradient checkpointing\([Chen et al\., 2016](https://arxiv.org/html/2610.10827#bib.bib31)\), DeepSpeed ZeRO\-2\([Rajbhandari et al\., 2020](https://arxiv.org/html/2610.10827#bib.bib10)\), completion\-only loss, and long\-sample filtering\. Validation was performed four times per epoch with early\-stopping patience of three evaluations\. Table[5](https://arxiv.org/html/2610.10827#A1.T5)records the selected run configurations, and Table[6](https://arxiv.org/html/2610.10827#A1.T6)reports training convergence and structured\-output reliability for the three fine\-tuned models\.

Table 5:Recorded fine\-tuning configurations for Qwen3\.5\-4B, Qwen3\.5\-0\.8B, and Qwen3\-0\.6B\. Effective batch size \(256 for the compact runs\) is the product of per\-device batch size \(1\), gradient accumulation steps \(32\), and 8 distributed GPU workers \(1×32×8=2561\\times 32\\times 8=256\); the 4B run used 1 worker with accumulation 16 \(1×16×1=161\\times 16\\times 1=16\)\. Dashes indicate settings not retained in the available run record\.Table 6:Minimum teacher\-forced loss on each run’s training\-validation split and structured\-output parse errors on Golden2\. Bold highlights format adherence \(≤0\.03%\\leq 0\.03\\%parse errors\) and the lowest validation loss within each distinct recipe \(0\.2201 for detailed 4B; 0\.2245 for compact 0\.8B\)\. The 0\.8B and 0\.6B runs shared prompt and optimization hyperparameters but selected different early\-stopping checkpoints\.†The 4B checkpoint used a different prompt, training target, and schedule; its loss is descriptive rather than a controlled cross\-model comparison\.The matched\-recipe comparison between the 0\.8B and 0\.6B runs reveals a failure mode not visible in the loss curve\. Although the 0\.8B and 0\.6B runs shared prompt and optimization hyperparameters, the 0\.6B model converged smoothly to a minimum validation loss 28\.5% higher, and 57\.4% of its evaluation outputs failed schema validation \(parse errors\), compared with 0\.03% for the 0\.8B model: token\-level optimization did not guarantee reliable autoregressive generation of the required structure\. Because the models also belong to different Qwen generations, this comparison does not isolate parameter count as the sole cause\.

## Appendix BData Pipeline Details

#### Error\-prioritized sampling\.

The heuristic used to select the second 50k production groups flags annotation densitydd\(annotations per non\-noun word\) or annotated\-span coverageccoutside fixed bounds \(d≤0\.75d\\leq 0\.75ord≥1\.25d\\geq 1\.25;c≤0\.75c\\leq 0\.75\)\. Flagged utterances are prioritized by\(\|d−1\|\+max⁡\(0,1−c\)\)×NN\+5\(\|d\-1\|\+\\max\(0,1\-c\)\)\\times\\frac\{N\}\{N\+5\}, whereNNis utterance length in words, dampening extreme ratios on very short utterances\.

#### Deterministic evidence and validation\.

Beyond part\-of\-speech tags and rule\-based construction detection, coherence checks and high\-confidence rules add warnings and candidate concepts to the review checklist\. In addition to the checks listed in Section[3\.1](https://arxiv.org/html/2610.10827#S3.SS1), the validator resolves disfluent repetitions omitted by the correction before checking span containment, and falls back to the pilot annotations if all proposed tuples are rejected\.

#### Review prompt contract\.

Review calls query GPT\-5\.4 at temperature 0 with medium reasoning effort \(one utterance per request\)\. The prompt contract is executed in the standard chat JSON format below, specifying system instructions, user input context, and the expected assistant audit response; proprietary extraction notes are omitted\.

Data Cleaning Review Prompt Contract \(Chat JSON\)\[ \{ "role":"system", "content": "You are a professional linguist reviewing existing grammar annotations\. For each preliminary row: KEEP, FIX, REMOVE, or ADD\.\\n \- Span & Intent: Every text\_span must be an exact substring\. When a corrected line is present, label by intended target grammar, not surface form in isolation\.\\n \- Correctness & Explanation: correct is true iff the concept is used properly\. Non\-empty error\_explanation requires correct=false \(REPETITION is the sole exception: false with empty explanation\)\.\\n \- Output Format: Return a JSON object mapping utterance to \[\[text\_span, concept\_label, correct, error\_explanation\], \.\.\.\]\." \}, \{ "role":"user", "content": "Utterance: She don’t like swimming in the pool\.\\n Error hints: don’t \-\> doesn’t\\n Linguistic evidence: Pronouns: She; Prepositions: in; Articles: the; Verbs: don’t, like, swimming\\n Prior analysis: \[\["don’t", "PRESENT\_SIMPLE", true, ""\], \["swimming", "PRESENT\_CONTINUOUS", true, ""\], \["the pool", "ARTICLES", true, ""\]\]" \}, \{ "role":"assistant", "content": "\{\\n "She don’t like swimming in the pool\.": \[\\n \["She", "PRONOUNS", true, ""\],\\n \["don’t like", "PRESENT\_SIMPLE", false, "Use doesn’t with third\-person She\."\],\\n \["swimming", "VERB\_PATTERNS", true, ""\],\\n \["in the pool", "PREPOSITIONS", true, ""\],\\n \["the pool", "ARTICLES", true, ""\]\\n \]\\n \}" \} \]

#### Synthetic backbone generation\.

Seed backbones are harvested from production records by stripping lexical tokens from annotated utterances, after discarding degenerate sequences \(e\.g\., word repetitions or comma\-heavy enumerations\)\. The 90k synthetic corpus is produced across three 30k strata: \(i\)*uniform sampling*from production backbones; \(ii\)*error\-weighted sampling*, where templates withkknatural error slots are sampled with weight proportional toc×\(2\+k\)0\.75c\\times\(2\+k\)^\{0\.75\}\(ccis template frequency\); and \(iii\)*inverse\-frequency sampling*, where anchor concepts are drawn with probabilityP⁡\(gi\)∝1/\(ni\+1000\)1\.0P\(g\_\{i\}\)\\propto 1/\(n\_\{i\}\+1000\)^\{1\.0\}\(nin\_\{i\}is concept span count\)\.

#### Pedagogical error injection and synthesis\.

Each originally correct slot is mutated with probabilityp=0\.05p=0\.05into one of three error kinds: \(a\)*same\-label misuse*\(e\.g\., wrong verb inflection\); \(b\)*omission*\(dropping an obligatory article or preposition\); or \(c\)*confusion swap*, where the slot is temporarily projected to a confusable substitute label \(e\.g\., present continuous instead of present perfect continuous\) and paired with discourse cues \(e\.g\., duration adverbials\) that make the substitute form erroneous in context\. GPT\-5\.4 \(medium reasoning effort, 5 sentence variations per call\) receives concept definitions for the projected sequence, numberedcorrect/not\_correctslot lines with inline\[Error: \.\.\.\]instructions, and 1–3 retrieved production exemplars in the required JSON shape\. Confusion\-swap slots are shown to the model as the substitute label markedcorrect; post\-processing restores intent labels, coerces incorrect spans to the planned intent, drops sentences with out\-of\-scope labels, and enforces substring validity\. The sanitized chat exchange is structured under the standard message contract below\.

Synthetic Backbone Realization Contract \(Chat JSON\)\[ \{ "role":"system", "content": "You are an expert in SLA and synthetic data generation\. Generate \{num\_sentences\} distinct learner\-style variations from the example structure\.\\n \- Slots: Keep each correct slot fully grammatical; for each not\_correct slot, follow the inline \[Error: \.\.\.\] instruction and keep the named intent label with correct=false\.\\n \- Coverage: Annotate every grammar block in the sentence, including naturally occurring ones not in the sequence \(e\.g\., pronouns, articles\)\.\\n \- Output Format: Return JSON mapping each generated sentence to \{"corrected": \.\.\., "annotations": \[\.\.\.\]\}\." \}, \{ "role":"user", "content": "Definitions: PRESENT\_SIMPLE: main verb in present simple; VERB\_PATTERNS: \-ing complement only\.\\n Sequence & labels:\\n 1\. PRESENT\_SIMPLE: not\_correct \[Error: same\-label misuse\] Subject\-verb agreement \(e\.g\., "She don’t like \.\.\."\)\.\\n 2\. VERB\_PATTERNS: correct\\n Exemplar \(reference\):\\n \{"She doesn’t like swimming in the park\.": \{"corrected": null, "annotations": \[\["She","PRONOUNS",true,""\], \["doesn’t like","PRESENT\_SIMPLE",true,""\], \["swimming","VERB\_PATTERNS",true,""\], \["in the park","PREPOSITIONS",true,""\], \["the park","ARTICLES",true,""\]\] \}\}" \}, \{ "role":"assistant", "content": "\{\\n "He don’t enjoy running at the gym\.": \{\\n "corrected": "He doesn’t enjoy running at the gym\.",\\n "annotations": \[\\n \["He", "PRONOUNS", true, ""\],\\n \["don’t enjoy", "PRESENT\_SIMPLE", false,\\n "Use doesn’t with third\-person He\."\],\\n \["running", "VERB\_PATTERNS", true, ""\],\\n \["at the gym", "PREPOSITIONS", true, ""\],\\n \["the gym", "ARTICLES", true, ""\]\\n \]\\n \}\\n \}" \} \]

## Appendix CExtended Experimental Results and Statistical Methodology

### C\.1Confidence Intervals for Annotation Results

Table[7](https://arxiv.org/html/2610.10827#A3.T7)provides the complete percentile 95% confidence intervals corresponding to the primary evaluation in Table[2](https://arxiv.org/html/2610.10827#S5.T2), derived from 20,000 paired cluster bootstrap resamples\.

Table 7:Percentile 95% confidence intervals for micro precision and recall in Table[2](https://arxiv.org/html/2610.10827#S5.T2), based on 20,000 paired nonparametric cluster bootstrap resamples of packed five\-utterance inference batches\. Surplus generated utterances remain assigned to their originating batch\. Bold marks the best interval in each column within each benchmark\. Base and FT denote untuned and fine\-tuned models, respectively\. All values are percentages; all GPT\-5\.4 and GPT\-5\.6 Sol outputs parsed successfully\.

### C\.2Key\-Macro Annotation Results

Table[8](https://arxiv.org/html/2610.10827#A3.T8)presents key\-macro\-averaged precision and recall, assigning equal weight to each distinct target key at each evaluation level \(target concepts at L1, concept–span pairs at L2, and concept–span–correctness triples at L3\), complementing the class\-level Mean\-34 recall metric in Table[3](https://arxiv.org/html/2610.10827#S6.T3)\. Macro recall and precision are computed as the unweighted mean of per\-key scores across all distinct target and predicted keys, respectively\.

CriterionGPT\-5\.4P / R \(%\)Qwen3\.5\-4B FTP / R \(%\)Qwen3\.5\-0\.8B FTP / R \(%\)*Golden \(selected checkpoints\)*Macro L1: concept74\.08 / 75\.3484\.21 /91\.4385\.60/ 89\.82Macro L2: \+ span52\.44 / 51\.6572\.62 / 72\.9375\.47/75\.70Macro L3: \+ correctness45\.77 / 45\.0864\.48 / 64\.8267\.84/68\.07*Golden2 \(selected checkpoints\)*Macro L1: concept72\.26 / 68\.3184\.41 / 84\.8586\.29/87\.65Macro L2: \+ span47\.64 / 46\.1367\.57 / 67\.1668\.90/68\.68Macro L3: \+ correctness41\.65 / 40\.2659\.91 / 59\.3762\.06/61\.79Table 8:Key\-macro precision and recall \(%\) assigning equal weight to each distinct target matching key at each level, corresponding to the micro results in Table[2](https://arxiv.org/html/2610.10827#S5.T2)\. Bold marks the best value for each metric in each row\.
### C\.3Pre\-Fixed Rare\-Concept Subset

We selected the nine concepts with the fewest annotations in the raw 83,815\-group training split after restricting labels to the 34\-concept inventory\. Selection used no benchmark outcomes; there was no tie at the cutoff\. Table[9](https://arxiv.org/html/2610.10827#A3.T9)reports the selection counts and gold support on the primary Golden2 evaluation benchmark \(Golden serves as an additional robustness check\)\.

Table 9:Pre\-fixed Rare\-9 concepts and annotation support\. Raw\-train counts determined subset membership; Golden2 counts are reported only as evaluation support\.
### C\.4Bootstrap Resampling and Statistical Significance

Bootstrap resampling operates on packed five\-utterance inference batches, so surplus generated utterances remain assigned to the batch that produced them and are scored as false positives\. For paired ablation intervals, metric differences are computed within each aligned cluster resample, taking the 2\.5 and 97\.5 percentiles across 20,000 draws\. For fixed\-set Mean\-34 and Rare\-9 intervals, bootstrap draws without gold support for every included concept were redrawn conditionally \(Golden2: 138 draws for 4B and 153 for 0\.8B; Golden: 769 draws for each size\)\.

For pairwise comparisons against GPT\-5\.4, two\-sided paired randomization tests permute model predictions across systems at the five\-utterance batch cluster level usingB=20,000B=20\{,\}000Monte Carlo permutations\. Empiricalpp\-values are computed asp=\(b\+1\)/\(B\+1\)p=\(b\+1\)/\(B\+1\), wherebbis the number of permuted absolute test statistic differences exceeding or equaling the observed difference; tests yielding zero exceedances \(b=0b=0\) are reported asp<0\.0001p<0\.0001\(p=1/20,001p=1/20\{,\}001\)\.

## Appendix DServing Capacity, Autoscaling, and Cost

Figure[3](https://arxiv.org/html/2610.10827#A4.F3)provides the detailed topology omitted from the page\-limited main paper\. SageMaker distributes invocations across replicas; within each replica, vLLM batches concurrent sequences using chunked prefill, prefix caching, and an FP8 KV cache\.

![Refer to caption](https://arxiv.org/html/2610.10827v1/images/serving_setup.png)Figure 3:Simplified productiong6\.2xlargenative\-vLLM topology\. The application backend and internal inference API are collapsed into the inference\-client block\. SageMaker distributes calls across independently provisioned one\-GPU containers\.#### Workload and measured capacity\.

Before inference, the application partitions prepared utterance–correction pairs into five\-pair requests \(averaging 4\.27 pairs in production\)\. To determine serving capacity, concurrent clients replayed five\-pair requests through a warmed inference service\. Eightg6\.2xlargeinstances sustained 4,993\.5 pairs per minute, which equals4,993\.5/\(8×5×60\)=2\.084\{,\}993\.5/\(8\\times 5\\times 60\)=2\.08five\-pair requests per second \(RPS\) per instance\. Using five pairs per request is conservative relative to the production mean\.

### D\.1Autoscaling Policy and Load Test

The production endpoint uses AWS Application Auto Scaling targeting four concurrent in\-flight requests per instance, with 10\-minute scale\-out and 20\-minute scale\-in cooldowns across a range of one to eight instances\. We validated this policy by deploying the model to a staging endpoint and replaying a two\-hour staged load test ramped from 1 to 6\.4 RPS before declining \(Table[10](https://arxiv.org/html/2610.10827#A4.T10)\)\.

Table 10:Predeployment autoscaling test\. “Instances” gives the actual transition during each traffic phase; the initial zero denotes endpoint startup before the configured one\-instance minimum became active\. Latency is measured per request\.Across the test, the endpoint processed 17,994 requests with an overall error rate of 0\.011% \(two transient errors during the initial scale\-out from two to five instances\)\. At the 6\.4 RPS peak, five instances sustained traffic with zero errors and a 7\.64\-second p99 latency; the resulting1\.281\.28RPS per instance \(61\.5%61\.5\\%of measured saturation capacity\) operated safely within target concurrency thresholds\. As load dropped, the scale\-in cooldown retained capacity before returning to the single\-instance baseline\.

### D\.2Cost Projection

To project monthly serving expenditure under live traffic patterns, we analyzed 26 days of hourly production request volume following full rollout\. For each production hour, required capacity was computed as observed request rate divided by planning capacity \(2\.08×0\.70=1\.462\.08\\times 0\.70=1\.46RPS, derating measured saturation capacity to provide a 30% headroom margin, consistent with the stable peak rate observed in Table[10](https://arxiv.org/html/2610.10827#A4.T10)\), rounded up, and clamped to the deployed 1–8 instance range\.

Average provisioned capacity was multiplied by 730 monthly hours and public on\-demand SageMaker real\-time pricing forg6\.2xlargeinstance\-hours\([Amazon Web Services, 2026](https://arxiv.org/html/2610.10827#bib.bib1)\)\. Compared against measured May 2026 production costs for the low\-reasoning GPT\-5\.4 configuration, the resulting SLM serving cost is 6\.2% of baseline per lesson—an approximately 16×\\timesreduction\.

This projection isolates recurring detection serving: the GPT\-5\.4 baseline is measured in production, while the SLM estimate combines observed demand, load\-tested throughput, and public cloud pricing\. Common upstream filtering and correction, fallback API calls, and one\-time training and deployment work are excluded\. Absolute spend, per\-lesson costs, and eligible\-lesson volumes are withheld\.

## Appendix ETask Specifications and Auxiliary Probes

### E\.1Task Schema and Logical Contract

Table[11](https://arxiv.org/html/2610.10827#A5.T11)states the grammar concept annotation contract of Section[2](https://arxiv.org/html/2610.10827#S2)field by field\.

Table 11:Logical input and output schema for grammar concept annotation\. The first two fields form the input; the remaining fields describe each output annotation\.
### E\.2Prompt Contracts: Verbose Baseline and Compact Inference

We document the operational prompt contracts governing grammar concept annotation in the standard chat JSON format below\. The*verbose baseline prompt*\(3,810 Qwen3\.5 tokens; excerpted in the first contract box\) was executed during the production pilot with GPT\-5\.4 and served as the comparator prompt for Qwen3\.5\-4B\. It specifies comprehensive category definitions, extraction boundaries, and edge\-case rules for all 34 concepts in our curriculum\. In contrast, the*compact inference prompt*\(375 Qwen3\.5 tokens, a 90\.2% prompt reduction; presented in full in the second contract box\) was designed for the deployed Qwen3\.5\-0\.8B SLM\. Because completion\-only LoRA internalizes concept definitions and span boundaries directly into model weights \(Section[4](https://arxiv.org/html/2610.10827#S4)\), runtime serving requires only the list of allowed concept names, the schema contract, and concise formatting constraints, minimizing prefill latency and KV\-cache footprint in vLLM\.

Verbose Baseline Prompt Contract \(Chat JSON\)\[ \{ "role":"system", "content": "You are a deterministic grammar annotation engine\. Given an ORIGINAL sentence \(student version\) and a CORRECTED sentence \(professionally edited\), extract, label, and compare all grammar\-relevant native text spans using the Grammar Categories Table\.\\n\\n \# Instructions\\n 1\. Parse both ORIGINAL and CORRECTED sentences, identifying grammar\-relevant spans\.\\n 2\. Align each span by grammatical role and order, using syntax and structure\.\\n 3\. For each alignment, set correct usage to Yes if identical; if not, set to No\.\\n 4\. Strictly follow Output Format\-\-\-no extra text, explanation, or commentary\.\\n\\n \# Grammar Categories Table \(34 concepts; representative excerpt\)\\n \- PRESENT\_SIMPLE: Main verb in present simple\. Include auxiliary \(do/does/don’t/doesn’t\) in negatives/questions\. Extract copula be \(am/is/are\) and main verb have\. Include contractions \(it’s, I’m\)\.\\n \- ARTICLES: Article \+ head noun\. Extract article spans independently even inside prepositional phrases \(e\.g\., in ’in the morning’, extract ’in the morning’ as PREPOSITIONS and ’the morning’ as ARTICLES\)\.\\n \- PREPOSITIONS: Preposition \+ object\. For phrasal verbs, extract verb/adj \+ preposition\.\\n \[\.\.\. 31 concept specifications and extraction notes omitted \.\.\.\]\\n\\n \# Span Extraction Rules\\n \- Exact contiguous substring; minimal span; no character overlap\.\\n \- Report every detectable grammar feature\-\-\-both correct and incorrect usage\.\\n\\n \# Output Format\\n Output one annotation per line separated by \|: sentence\_index\|grammar\_concept\|correct\|text\_span\|explanation\\n \- correct: Y if correct, N if error\.\\n \- explanation: For N, 6\-20 words describing the fix from CORRECTED; leave empty for Y\. For REPETITION, explanation must be empty\." \}, \{ "role":"user", "content": "Sentence pairs to analyze:\\n 1\|ORIGINAL: She don’t like swimming in the pool\.\\n 1\|CORRECTED: She doesn’t like swimming in the pool\." \}, \{ "role":"assistant", "content": "1\|PRESENT\_SIMPLE\|N\|don’t like\|Use doesn’t with third\-person She\.\\n 1\|VERB\_PATTERNS\|Y\|swimming\|\\n 1\|PREPOSITIONS\|Y\|in the pool\|\\n 1\|ARTICLES\|Y\|the pool\|" \} \]

Compact Inference Prompt Contract \(Chat JSON\)\[ \{ "role":"system", "content": "Grammar annotation task\.\\n Input: numbered ORIGINAL / CORRECTED English sentence pairs\.\\n Output one pipe\-delimited annotation per grammar\-relevant span:\\n sentence\_index\|grammar\_concept\|correct\|text\_span\|explanation\\n\\n Allowed grammar\_concept values \(34 total\):\\n PRESENT\_SIMPLE, PRESENT\_CONTINUOUS, PAST\_SIMPLE, PAST\_CONTINUOUS, PRESENT\_PERFECT, PRESENT\_PERFECT\_CONTINUOUS, PAST\_PERFECT, FUTURE\_SIMPLE, PASSIVE\_VOICE, MODAL\_ABILITY, MODAL\_OBLIGATION, MODAL\_REQUEST\_SUGGEST, MODAL\_PROBABILITY, VERB\_PATTERNS, IMPERATIVES, ZERO\_CONDITIONAL, FIRST\_CONDITIONAL, SECOND\_CONDITIONAL, THERE\_IS\_THERE\_ARE, ARTICLES, PREPOSITIONS, ADVERBS, ADJECTIVES\_BASIC, COMPARATIVES, SUPERLATIVES, QUANTIFIERS\_BASIC, POSSESSIVE\_ADJECTIVES, POSSESSIVE\_S\_WHOSE, DEMONSTRATIVES, SOME\_ANY\_NO\_COMPOUNDS, ONE\_ONES, RELATIVE\_CLAUSES, REPORTED\_SPEECH, CAUSATIVE\_PERMISSIVE\.\\n\\n Rules:\\n \- grammar\_concept: Classify based on the grammar used in CORRECTED, even if ORIGINAL used a different structure\.\\n \- text\_span: Exact text from ORIGINAL\.\\n \- correct: Y if ORIGINAL uses the feature correctly, N if CORRECTED fixes an error in it\.\\n \- explanation: For Y, leave empty \(line must end with a pipe \|\)\. For N, write a fix under 15 words using words from CORRECTED\. Do not use pipes \(\|\) or newlines\.\\n \- Output only annotation lines\. No headers, blank lines, or code fences\." \}, \{ "role":"user", "content": "Sentence pairs to analyze:\\n 1\|ORIGINAL: She don’t like swimming in the pool\.\\n 1\|CORRECTED: She doesn’t like swimming in the pool\." \}, \{ "role":"assistant", "content": "1\|PRESENT\_SIMPLE\|N\|don’t like\|Use doesn’t with third\-person She\.\\n 1\|VERB\_PATTERNS\|Y\|swimming\|\\n 1\|PREPOSITIONS\|Y\|in the pool\|\\n 1\|ARTICLES\|Y\|the pool\|" \} \]

### E\.3Explanation Judgment Rubric

The LLM judge assigns*accept*to correct and actionable feedback and*pass*to useful but pedagogically weak feedback\. It assigns*reject*when feedback targets the wrong locus, states an incorrect rule, proposes an invalid fix, is tautological, or is empty\. Explanation acceptability \(Section[5](https://arxiv.org/html/2610.10827#S5)\) is defined as the proportion of evaluated explanations receiving either*accept*or*pass*, grouping together outputs that provide factually sound pedagogical assistance without misguiding the learner, while isolating actively erroneous or misleading feedback under*reject*\.

In the human validation stage, an independent reviewer evaluated every model\-generated explanation under this rubric, and a second reviewer resolved discrepancies to establish final aligned judgments\. Across all three systems on Golden2, aligned human acceptability rates converged to approximately 95% with no statistically significant differences \(χ2\\chi^\{2\}test of independence,p\>0\.30p\>0\.30\), in contrast to the uncalibrated LLM judge\.

### E\.4Synthetic Distribution\-Shift Probe

We tested the selected Qwen3\.5\-0\.8B model on 200 learner\-like utterances independently authored and annotated by GPT\-5\.5\. Concepts and correctness rates followed Golden2, while topics spanned 15 domains; deterministic checks enforced schema, span, and label constraints\. Because the labels were teacher\-generated, this is a diagnostic probe rather than a third golden benchmark\.*Partial\-span*matching requires at least 50% character overlap with the target span, whereas*exact\-span*requires exact string equality of the text span\.

Table 12:Performance of the same Qwen3\.5\-0\.8B checkpoint under real and synthetic evaluation distributions\. Scores are percentages\. Span axes evaluate localization independently of concept classification; exact\-span performance is diagnostic because span\-boundary conventions were not calibrated across annotators\.As reported in Table[12](https://arxiv.org/html/2610.10827#A5.T12), concept and partial\-span F1 remained stable and all synthetic outputs parsed, but exact\-span F1 fell by 24\.69 points\. The synthetic\-data generator used span\-boundary rules that differed from those used for the human\-annotated benchmarks\. The drop is therefore consistent with an annotation\-protocol mismatch\.