LLM评分器的成功与失败之处：来自两场计算机科学考试的证据

# Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams
Source: [https://arxiv.org/html/2609.29333](https://arxiv.org/html/2609.29333)
## Where LLM Graders Succeed and Break: Evidence from Two Computer\-Science ExamsThanks:Corresponding author\.

Ali Habibullah††thanks:Equal contribution\.Affiliation:, Yazan Alshoibi11footnotemark:1& Mohammad Alshiekh11footnotemark:1KAUST AcademyComputer, Electrical & Mathematical Sciences & Engineering \(CEMSE\)King Abdullah University of Science and Technology \(KAUST\)Thuwal, Saudi ArabiaEmail:[ali\.habibullah@kaust\.edu\.sa](mailto:)Salman KhanVisual Artificial Intelligence LaboratoryOxford Brookes UniversityEmail:[salmankhan@brookes\.ac\.uk](mailto:)Email:[naeemullah\.khan@kaust\.edu\.sa](mailto:)Naeemullah KhanKAUST Academy & CEMSE, KAUSTLady Margaret Hall, University of OxfordEmail:[mohammad\.shiekh@kaust\.edu\.sa](mailto:)

###### Abstract

One long\-form exam in a large course costs hundreds of grader\-hours, and qualified graders are scarce; LLM graders are a tempting alternative\. To show its pitfalls we grade a practical Computer Vision exam \(570570dual\-graded students\) under171171configurations spanning closed and open\-weights models; the best reaches mean absolute error1\.64/351\.64/35, below the2\.61/352\.61/35two human graders achieve against each other\. The catch is the prompt: a short “strict grader” preamble drives1414of1717open\-weights models out of the graded band \(MAE≥8\\text\{MAE\}\\geq 8\), three stopping grading altogether\. The damage traces to the preamble’s two credit\-withholding sentences, not to tone or model scale; one of them, “never give partial credit”, alone makes two of three probed models stop grading\. The closed flagships of three vendors shift calibration under it but stay in the band\. In162162further configurations on a second, independent Machine Learning exam from another course \(1,0381\{,\}038dual\-graded students\), the preamble worsens ten models, moving three out of the band into collapse and one into refusal, yet improves seven whose neutral prompts over\-mark: the vulnerability replicates, but its direction is exam\-specific\. Light LoRA fine\-tuning repairs it: one adapter on the two exams’ pooled∼3,900\\sim 3\{,\}900graded examples brings five small open models to parity or better with a human grader in agreement with the grader pair, and sensitivity to the three harsh personas nearly vanishes \(≤0\.32\\leq 0\.32MAE\)\. We release the anonymised dataset, full ablation grid, and grading, fine\-tuning and analysis pipelines111Code, data and results:[https://github\.com/KAUST\-Academy/where\-llm\-graders\-succeed\-and\-break](https://github.com/KAUST-Academy/where-llm-graders-succeed-and-break)\.\.

*K*eywordsLLM auto\-grading⋅\\cdotprompt brittleness⋅\\cdotinstruction\-following⋅\\cdotpersona effects⋅\\cdotopen\-weights evaluation⋅\\cdotLoRA fine\-tuning⋅\\cdoteducational assessment

## 1Introduction

Large language models \(LLMs\) are being used in computer\-science education\[[1](https://arxiv.org/html/2609.29333#bib.bib14)\], grading included\[[2](https://arxiv.org/html/2609.29333#bib.bib12)\]: hundreds of grader\-hours per cycle and scarce qualified graders make the economic case\. The methodological case is murkier: the usual metrics hide which configuration of model, prompt and persona grades usably, and how it survives an instructor’s small prompt edits\. On a practical Computer Vision \(CV\) exam, dual\-graded by fixed grader pairs, we run171171configurations with bootstrap CIs: closed and open\-weight models \(77B–480480B\), prompt components, personas, few\-shot demonstrations, temperatures\. Six full\-cohort configurations, four Gemini and two GPT\-5\.5, match or beat the exam’s inter\-grader floor of2\.61/352\.61/35\.

The central finding is an asymmetry: for closed models the logical prompt \(reference solution, grading guidelines, rubric breakdown, neutral persona\) is already at a joint optimum \(Section[6](https://arxiv.org/html/2609.29333#S6)\), while open models are one sentence away from failure\. A “strict grader” preamble knocks1414of1717models out of the graded band in all six tested open\-weight families, three stopping grading entirely, the damage is unordered by scale and attributed to its two policy sentences, not its tone; across three closed vendors only Gemini’s cheapest tier leaves the band, and only on the second exam \(Sections[5](https://arxiv.org/html/2609.29333#S5)and[6\.1](https://arxiv.org/html/2609.29333#S6.SS1)\)\. Failure takes three forms MAE alone cannot separate \(zeroed submissions, blanket mark\-downs, one structured field dismantled while the rest grades on\); a conflicting\-instruction account fits only the last, and only as a hypothesis \(Section[5\.3](https://arxiv.org/html/2609.29333#S5.SS3)\)\.

The collapse is not an artifact of one exam: under the same preamble, a162162\-configuration replication on another Machine Learning \(ML\) exam from a different course worsens1010of1717models, three leaving the graded band and one refusing outright, while the best closed and open configurations again land below its floor\. The direction does not carry over: seven models*improve*, the ML exam’s over\-marking cancelling against the persona, calibration masquerading as robustness \(Section[5\.4](https://arxiv.org/html/2609.29333#S5.SS4)\)\.

The collapse trains away\. One LoRA adapter on the two exams’ pooled∼3,900\\sim 3\{,\}900examples, supervised by per\-question grader marks, takes five open models \(44B–3030B; Qwen, Llama, Gemma\) from significantly worse than one human grader to parity or better under a paired third\-grader test on*both*exams; an adapter trained on one exam already improves the unseen other\. The persona sweep costing base models up to2020MAE points moves the pooled adapters by at most0\.320\.32under the three harsh personas, and by at most0\.390\.39under*lenient*bar Qwen3\-Coder\-3030B\-A33B \(up to1\.811\.81; Section[8](https://arxiv.org/html/2609.29333#S8)\)\.

## 2Related Work

#### LLMs as judges and graders\.

LLM judging is standard since[Zheng et al\. \[3\]](https://arxiv.org/html/2609.29333#bib.bib7)showed GPT\-4 matches expert labellers on MT\-Bench and[Liu et al\. \[4\]](https://arxiv.org/html/2609.29333#bib.bib8)formalised prompt\-plus\-chain\-of\-thought scoring\. Programming auto\-grading mostly unit\-tests functional correctness\[[5](https://arxiv.org/html/2609.29333#bib.bib13)\], without rubric\-aligned partial credit, or correlates LLM with human raters on∼100\\sim 100\-submission datasets; the nearest benchmark,[Phung et al\. \[2\]](https://arxiv.org/html/2609.29333#bib.bib12), compares ChatGPT and GPT\-4 with human tutors\. We apply the same scaffolding to*student code grading*on two real, released exams \(n=570n=570and1,0381\{,\}038\) whose ground truth is two independent human graders, not a curated gold reference\. Our few\-shot arm supplies two worked examples per question labelled with D0101’s scores and rationales, not the graders’ marks — in\-context distillation of the best closed grader in the format of[Brown et al\. \[6\]](https://arxiv.org/html/2609.29333#bib.bib15)\(Appendix[D](https://arxiv.org/html/2609.29333#A4)\)\.

#### Prompt brittleness and personas\.

[Sclar et al\. \[7\]](https://arxiv.org/html/2609.29333#bib.bib9)show that semantically meaningless formatting changes shift benchmark performance by tens of percentage points and, with[Mizrahi et al\. \[8\]](https://arxiv.org/html/2609.29333#bib.bib10), urge reporting distributions over prompt variants;[Deshpande et al\. \[9\]](https://arxiv.org/html/2609.29333#bib.bib11)report persona\-dependent toxicity rises up to6×6\\timesin ChatGPT\. Our perturbation instead carries meaning, and we replay it on a second exam, where the vulnerability replicates but its direction and failure mode do not \(Section[5\.4](https://arxiv.org/html/2609.29333#S5.SS4)\)\.

#### Instruction hierarchies and conflicts\.

Section[5\.3](https://arxiv.org/html/2609.29333#S5.SS3)’s failure mode — abandoning one of two incompatible instructions rather than negotiating them — connects to instruction arbitration\. Models treat instructions as equally privileged unless trained on an explicit hierarchy\[[10](https://arxiv.org/html/2609.29333#bib.bib18)\], fail a non\-trivial fraction of simple verifiable instructions\[[11](https://arxiv.org/html/2609.29333#bib.bib19)\], break stated rules even on straightforward tests\[[12](https://arxiv.org/html/2609.29333#bib.bib20)\], degrade sharply when instructions conflict\[[13](https://arxiv.org/html/2609.29333#bib.bib21)\], and rarely flag the conflict\[[14](https://arxiv.org/html/2609.29333#bib.bib22)\]\. Here the conflict arises from an ordinary instructor edit, not an attack, with a clean behavioural signature: the structured score channel surrenders while the prose channel keeps following the rubric\.

## 3Experimental Setup

#### Exams, graders and grid\.

570570students took the CV exam’s four code questions \(Q1transfer learning,Q2CNN\-from\-scratch classification,Q3semantic segmentation,Q4bonus colorization\);Q1–Q3form the3535\-point base scale \(weights12/11/1212/11/12\) used throughout; bonuses graded but excluded\. Two of2020graders in1010fixed pairs \(≈57\\approx 57students each\) grade every submission independently; the grader\-average total is the ground truth \(Section[4](https://arxiv.org/html/2609.29333#S4)\)\. The ML exam — another course,1,0381\{,\}038students, three questions,≈65\\approx 65\-point scale, dual\-graded by4949graders in non\-fixed pairs — carries a162162\-configuration replication \(Section[5\.4](https://arxiv.org/html/2609.29333#S5.SS4), Appendix[H](https://arxiv.org/html/2609.29333#A8)\)\. We evaluate44Gemini models\[[15](https://arxiv.org/html/2609.29333#bib.bib5),[16](https://arxiv.org/html/2609.29333#bib.bib6)\]via the Vertex API, GPT\-5\.5, GPT\-5\.4 and Claude Opus 5 via their batch APIs \(Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1)\), and1717open\-weights variants from six vendors, from77B to480480B, dense and mixture\-of\-experts, served locally with vLLM\[[17](https://arxiv.org/html/2609.29333#bib.bib23)\]on A100 nodes\. The171171CV configurations \(3333closed,138138open\-weights\) span per\-model baselines, prompt\-component removals, thinking, five\-persona sweeps, few\-shot prompting, temperature and mechanism probes \(Section[5\.2](https://arxiv.org/html/2609.29333#S5.SS2)\); Appendix[A](https://arxiv.org/html/2609.29333#A1)documents roster, serving stack, reproducibility, thinking configuration and exclusions; Appendices[I](https://arxiv.org/html/2609.29333#A9)and[K](https://arxiv.org/html/2609.29333#A11)tabulate every run\.

#### The prompt\.

The default prompt supplies the question, its rubric, the reference solution, the student notebook, grading guidelines and a breakdown of each scorable item; the model returns a score, any bonus and a rationale\. Each question is graded in its own call \(three or four per student\) with only its own notebook, serialised cell by cell; prompts run to≈22\\approx 22k characters on the CV exam,≈16\\approx 16k on the ML exam, half to two thirds shared reference material\. Personas are short preambles;*strict*reads “*You are a HARSH teaching assistant\. Award the MINIMUM defensible score for any task that is incomplete, buggy, or deviates from the rubric\. Never give partial credit if the task does not run correctly\.*”, the*neutral*default “*You are a strict but fair teaching assistant\.*”, with*lenient*,*rigorous*and*exacting*analogous\.

#### Metrics\.

The headline metric is MAE of the AI total against the grader\-average total, with95%95\\%percentile bootstrap Confidence Intervals \(CIs\) \(2,0002,000resamples\)\[[18](https://arxiv.org/html/2609.29333#bib.bib16),[19](https://arxiv.org/html/2609.29333#bib.bib17)\], alongside the*mean signed error*\(bias\); the human floor is computed identically,MAE=𝔼⁡\[\|G1−G2\|\]\\text\{MAE\}=\\mathbb\{E\}\[\\,\|\\text\{G\}\_\{1\}\-\\text\{G\}\_\{2\}\|\\,\]over the grader pair, and both are recomputed unchanged on the ML exam’s own scale\. Because a total can hide per\-question errors of opposite sign, Appendix[F](https://arxiv.org/html/2609.29333#A6)decomposes the bias per question for the headline runs and Table[2](https://arxiv.org/html/2609.29333#A1.T2)\(Appendix[A](https://arxiv.org/html/2609.29333#A1)\) gives the item\-level Spearmanρ\\rhoper question alongside the total\. Students whose grading call permanently failed are excluded from that run’s metrics — none in most runs, at most33elsewhere, exceptions in Appendix[A](https://arxiv.org/html/2609.29333#A1)\.

## 4The Human\-Grader Floor

Across all570570dual\-graded submissions of the CV exam the two graders’ totals agree toMAE=2\.61/35\\text\{MAE\}=2\.61/35\(95%95\\%bootstrap CI\[2\.37,2\.85\]\[2\.37,2\.85\]; Pearsonr=0\.868r=0\.868\)\. We treat2\.612\.61as the human\-grader floor: an AI configuration whose MAE sits at or below it makes, in aggregate, no more error against the grader average than the two graders make against each other\. The floor is comparative only, and leans in the AI’s favour because averaging two graders cancels part of their noise\. Beating it yields a parity claim of the kind Section[8\.2](https://arxiv.org/html/2609.29333#S8.SS2)retests with a paired third\-grader test\. The floor pools across pairs whose internal disagreement varies by4\.2×4\.2\\times: per\-pair MAE runs0\.850\.85to3\.553\.55over the1010fixed pairs, so which pair a student draws is a source of variability \(Appendix[G](https://arxiv.org/html/2609.29333#A7)\)\.

## 5The Strict\-Persona Collapse

### 5\.1The collapse, quantified

Table[3](https://arxiv.org/html/2609.29333#A2.T3)\(Appendix[B](https://arxiv.org/html/2609.29333#A2)\) pairs each model’s*strict*and*neutral*runs, identical bar the persona sentence; Figure[2](https://arxiv.org/html/2609.29333#A2.F2)renders all5151strict\-flavoured runs\. MAE alone cannot separate the failures \(zeroing every student scoresMAE≈26\\text\{MAE\}\\approx 26, the distance from the grader mean\), so we add a*behaviour*class: arefusalzeroes at least90%90\\%of students with awarded\-total standard deviation below0\.50\.5, a*near\-refusal*keeps some variation at that zero\-rate, acollapsereaches3\.07×3\.07\\timesits own exam’s inter\-grader floor while still discriminating between students, and anything else is*graded*\. The CV exam’sMAE≥8\\text\{MAE\}\\geq 8fixes that multiple \(3\.07×2\.613\.07\\times 2\.61\) and givesMAE≥15\.7\\text\{MAE\}\\geq 15\.7on the ML exam \(3\.07×5\.133\.07\\times 5\.13\), the same severity on both scales\.

#### Every open\-weight family breaks\.

Prepending a short “strict grader” preamble to the otherwise\-best prompt takes1414of the1717config\-matched pairs out of the graded band, all six families represented, MAE rising×1\.69\\times 1\.69to×7\.11\\times 7\.11\. Three models stop grading outright: Llama\-3\.1\-8B and Mistral\-Small\-24B award0\.00/350\.00/35on average, GLM\-4\-9B0\.560\.56; Qwen2\.5\-Coder\-3232B still marks students \(MAE=20\.29\\text\{MAE\}=20\.29, bias−20\.28\-20\.28, mean awarded5\.75/355\.75/35\)\. No Gemini configuration collapses under any wording here \(Appendix[B](https://arxiv.org/html/2609.29333#A2)\):gemini\-3\.1\-pro\-previewmoves1\.86→2\.751\.86\\to 2\.75under*strict*\(F0303, still at the floor\), Flash\-Lite’s strict\-flavoured runs sit at3\.733\.73–5\.755\.75— an imperfect calibration knob, not a failure mode — and GPT\-5\.5, GPT\-5\.4 and Claude Opus 5 stay in band \(Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1)\)\.

#### Scale does not order the damage\.

Parameter count and the strict/neutral MAE ratio correlate atρ=−0\.21\\rho=\-0\.21\(p=0\.42p=0\.42\) across the1717sized models\. The two largest ratios belong to a2424B model that stops grading and a1414B that zeroes88%88\\%of submissions, while106106B GLM\-4\.5\-Air collapses to21\.0221\.02; GLM*improves*from99B to3232B before worsening at106106B while Gemma*worsens*from1212B to2727B — opposite directions over matched rungs\. Robustness does appear at the top, Qwen3\-235235B\-A2222B and Qwen3\-Coder\-480480B staying in band, but both are Qwen, the only family with rungs above106106B, confounding family with scale exactly where it matters; the third in\-band model, Qwen2\.5\-Coder\-77B, clears the threshold by0\.080\.08from an already\-weak baseline\.

#### The damage is specific to one preset\.

Two of1313models leave the band under*rigorous*and two under*exacting*, against1414of1717under*strict*\(Table[4](https://arxiv.org/html/2609.29333#A2.T4), Appendix[B](https://arxiv.org/html/2609.29333#A2)\); only*strict*carries the two policy sentences, so the gap is about policy content, not vocabulary\.

### 5\.2Attribution: the policy sentences, not the adjective

The*strict*preset is three sentences: a frame \(“You are a HARSH teaching assistant\.”\) and two policy sentences —S1, “Award the MINIMUM defensible score for any task that is incomplete, buggy, or deviates from the rubric\.”, andS2, “Never give partial credit if the task does not run correctly\.” Varying the adjective alone — STRICT, RIGOROUS, FAIR — over verbatim S1 and S2 moves MAE by0\.940\.94to2\.152\.15across five models, in no consistent direction \(Table[5](https://arxiv.org/html/2609.29333#A2.T5), Appendix[B](https://arxiv.org/html/2609.29333#A2)\);FAIRwith the harsh policy leaves Mistral\-Small\-24B at25\.0525\.05, still not grading\.

A2×22\\times 2crosses S1 with S2 on three models, frame fixed: the frame alone moves each less than two points off neutral, both sentences take them to20\.2620\.26–26\.0426\.04, S2 alone to outright refusal on Llama\-3\.1\-8B and Mistral\-Small\-24B\. S1 is not benign — alone it takes Llama\-3\.1\-8B to23\.4523\.45and outdoes S2 on Qwen2\.5\-Coder\-32B \(15\.5515\.55against12\.9212\.92\) — so dominance varies by family: S1 for the3232B, S2 for the other two, on both exams, where S2 again alone stops graders grading \(Appendix[H](https://arxiv.org/html/2609.29333#A8)\)\. S2 contradicts the partial\-credit scale the rubric scaffold mandates — hence the safer wordings, and advice about one instruction, not a tone\.

### 5\.3Mechanism: three failure modes, one MAE range

Three explanations fit the headline numbers: refusal, calibration drift, conflicting instructions \(“be strict” vs\. “award partial credit per rubric”\)\. All three occur, on different models, indistinguishably by MAE: Gemma\-3\-27B reachesMAE=11\.55\\text\{MAE\}=11\.55having zeroed Q11on*zero*of570570students, GLM\-4\-32B a lower9\.669\.66by zeroing273273\. One question separates them — of students scored00on Q11, what fraction received a non\-zero Q22? — and it splits the1919matched strict runs intoblanket zeroing\(five\),selective field collapse\(five\) anduniform severity\(nine\), both thresholds in empty bands \(Table[6](https://arxiv.org/html/2609.29333#A2.T6), Appendix[B](https://arxiv.org/html/2609.29333#A2)\)\.

In the selective group the prompt’s two output channels disagree: on the two hand\-annotated runs,20%20\\%and43%43\\%of zeroed submissions carry a rationale itemising partial credit while the score field reads00\(Appendix[B](https://arxiv.org/html/2609.29333#A2)\) — one channel obeys S2, the other the rubric breakdown\. But this covers neither other group, and its clearest prediction fails: withdrawing the breakdown*worsens*the collapse in five of five families against a clean neutral control \(Appendix[B](https://arxiv.org/html/2609.29333#A2)\)\. An alternative fits that failure — the breakdown may anchor the score field to non\-zero sub\-totals, so withdrawing it lets the credit\-withholding sentence run unopposed — and our grid cannot separate the two; we treat the mechanism as classified rather than explained \(Section[10](https://arxiv.org/html/2609.29333#S10)\)\.

### 5\.4The collapse replicates on the ML exam; its direction does not

We replicate it on the ML exam \(Section[3](https://arxiv.org/html/2609.29333#S3); floor5\.135\.13computed identically\) over162162configurations: neutral and*strict*runs for all1717open\-weights models, five\-persona sweeps, attribution probes, a closed\-model arm \(Appendix[H](https://arxiv.org/html/2609.29333#A8)\)\. Under*strict*four models cross from the graded band to above that exam’s \(MAE≥15\.7\\text\{MAE\}\\geq 15\.7\); Llama\-3\.1\-88B again refuses\. A fixedMAE≥8\\text\{MAE\}\\geq 8, only1\.56×1\.56\\timesthis floor, would instead put ten strict runs out of band and eleven*neutral*baselines with them, against one here \(DeepSeek\-Coder\-V2\-Lite,16\.4116\.41\)\. The collapse is thus no artifact of one rubric, cohort or scale — nor of open weights alone: Flash\-Lite worsens under*strict*\(7\.53→9\.027\.53\\to 9\.02\) and leaves the band under*lenient*\(17\.8617\.86\), whilegemini\-3\.1\-pro\-previewstays below the floor \(3\.40→3\.673\.40\\to 3\.67, IG2323\)\.

Figure 1:The strict persona on the1717models run on both exams, floor\-relative\.*Left, centre:*neutral \(circle\)→\\tostrict MAE; dashed = floor, dotted = each exam’s band \(3\.07×3\.07\\timesfloor:MAE≥8\\text\{MAE\}\\geq 8on the CV exam,≥15\.7\\geq 15\.7on the ML exam\), dashed arrows = refusals \(ceiling MAE\)\.*Right:*the signed strict effect\(MAEstrict−MAEneutral\)/floor\(\\text\{MAE\}\_\{\\text\{strict\}\}\-\\text\{MAE\}\_\{\\text\{neutral\}\}\)/\\text\{floor\}against neutral bias, negative exactly where the persona cancels over\-marking\.The*direction*does not transfer \(Figure[1](https://arxiv.org/html/2609.29333#S5.F1)\): seven of1717improve under the same sentence — calibration, not robustness\. The ML exam’s neutral prompt over\-marks for1515models, and across the1616non\-refusal pairs neutral bias predicts the strict effect atr=−0\.73r=\-0\.73\(permutationp=0\.002p=0\.002\); on the CV exam, whose neutral biases straddle zero \(none above\+1\.68\+1\.68\), zero of1717improve\. The decomposition also splits the two big Qwens’ apparent robustness: the480480B over\-marks by9\.529\.52points at neutral \(MAE=10\.10\\text\{MAE\}=10\.10, twice its floor\) and is pulled back by the sentence; the235235B is the one model above106106B in band under both personas on both exams\. Parity replicates on both sides \(gemini\-3\.1\-pro\-previewat0\.66×0\.66\\timesthe ML exam’s floor, Qwen3\-Coder\-Next at4\.474\.47\), and the failure mode wanders: Qwen2\.5\-Coder\-3232B, selective here, zeroes whole submissions there\.

#### One preamble, two remedies\.

One preamble separates competitive open\-weights graders \(Section[7](https://arxiv.org/html/2609.29333#S7)\) from1414out of band, a gap single\-prompt evaluation cannot see\[[7](https://arxiv.org/html/2609.29333#bib.bib9),[8](https://arxiv.org/html/2609.29333#bib.bib10)\], with two remedies: never forbid partial credit, or fine\-tune lightly on in\-house labels \(Section[8](https://arxiv.org/html/2609.29333#S8)\)\.

## 6Closed\-Model Configurations

On the closed side the baseline recipe sits at the joint optimum: no prompt\-component removal reliably improves on it\.

#### Six full\-cohort configurations at or below the floor\.

Six full\-cohort configurations meet or undercut the human floor of2\.612\.61with CI upper bounds below it \(Table[10](https://arxiv.org/html/2609.29333#A3.T10), Appendix[C](https://arxiv.org/html/2609.29333#A3)\): D01 \(gemini\-3\-flash\-preview, neutral,t=0t=0\) atMAE=1\.64\\text\{MAE\}=1\.64\(bound1\.791\.79\); F02 and D02 \(the3\.13\.1\-pro under*lenient*and*neutral*\); B03 \(Flash\-Lite with thinking\), an order of magnitude cheaper; andgpt\-5\.5under*lenient*and*neutral*\(O032\.252\.25, O012\.432\.43; Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1)\)\. The floor comparison is not like\-for\-like \(Section[4](https://arxiv.org/html/2609.29333#S4)\), so we retest all six with the paired third\-grader statistic of Eq\. \([1](https://arxiv.org/html/2609.29333#S8.E1)\) \(Table[11](https://arxiv.org/html/2609.29333#A3.T11)\): three are better than a human grader — D01d¯=−0\.54\\overline\{d\}=\-0\.54\[−0\.71,−0\.36\]\[\-0\.71,\-0\.36\], F02−0\.37\-0\.37\[−0\.56,−0\.18\]\[\-0\.56,\-0\.18\], D02−0\.37\-0\.37\[−0\.55,−0\.18\]\[\-0\.55,\-0\.18\]— while B03 \(−0\.19\-0\.19\[−0\.39,\+0\.01\]\[\-0\.39,\+0\.01\]\) and bothgpt\-5\.5runs \(−0\.01\-0\.01\[−0\.22,\+0\.21\]\[\-0\.22,\+0\.21\]lenient,\+0\.19\+0\.19\[−0\.02,\+0\.41\]\[\-0\.02,\+0\.41\]neutral\) are at parity\. The Flash\-Lite baseline A01 is significantly worse \(\+1\.03\+1\.03\), as areclaude\-opus\-5\(\+1\.13\+1\.13\) and all1717open\-weights baselines’ closest analogues \(Section[7](https://arxiv.org/html/2609.29333#S7)\)\. Within Gemini the newer Flash beats the older Pro \(D03,4\.104\.10\); D01’s scatter \(Figure[3](https://arxiv.org/html/2609.29333#A3.F3)\) is essentially unbiased, its residual error concentrated in the low\-score band where the graders also disagree more\.

#### On Flash\-Lite, thinking helps and dropping the solution hurts\.

The B\-series varies one prompt component at a time on Flash\-Lite \(Table[12](https://arxiv.org/html/2609.29333#A3.T12)\)\. Dropping the reference solution flips a small under\-grading bias \(−1\.78\-1\.78\) into over\-grading \(\+3\.30\+3\.30\) costing≈0\.8\\approx 0\.8MAE; dropping the guidelines or breakdown moves MAE by at most0\.120\.12; thinking \(B03\) is the one large move, closing about three\-quarters of the gap to D01 on a far cheaper model\. Two combination runs cross a persona with another component: F01 \(Flash\-Lite,*strict*\+\+thinking\) at3\.733\.73recovers most of the*strict*penalty \(5\.755\.75\); F02 \(3\.13\.1Pro\+\+*lenient*\) reaches1\.791\.79against1\.861\.86neutral \(Table[24](https://arxiv.org/html/2609.29333#A9.T24)\)\. The G\-series ablations \(gemini\-3\-flash\-preview, first100100students; Appendix[C](https://arxiv.org/html/2609.29333#A3.SS0.SSS0.Px1)\) beat D01 only as a sample\-size artifact; thet=0t=0variance probes are in Appendix[C](https://arxiv.org/html/2609.29333#A3)\.

### 6\.1The asymmetry holds across vendors

Gemini is one vendor, so we replayed the persona sweep on two more via batch APIs — same prompt bytes, thinking off, temperature00where exposed \(Appendix[C](https://arxiv.org/html/2609.29333#A3), Table[9](https://arxiv.org/html/2609.29333#A3.T9)\): OpenAI’sgpt\-5\.5andgpt\-5\.4under*neutral*,*strict*and*lenient*on both exams, Anthropic’sclaude\-opus\-5under*neutral*on both and*strict*on the CV exam\. None leaves the graded band under*strict*—gpt\-5\.52\.43→4\.432\.43\\to 4\.43\(CV exam\) and3\.54→3\.703\.54\\to 3\.70\(ML exam\),claude\-opus\-53\.54→5\.173\.54\\to 5\.17,gpt\-5\.44\.69→6\.904\.69\\to 6\.90and4\.30→5\.114\.30\\to 5\.11, zero\-total rates at most0\.7%0\.7\\%, no refusal — the calibration shift the Gemini models show \(1\.86→2\.751\.86\\to 2\.75,3\.34→5\.753\.34\\to 5\.75\), against1414of1717open\-weights models leaving it under the identical sentence\. The cheaper tiers are the fragile ones:gpt\-5\.4is worst under*lenient*on the ML exam \(4\.30→8\.304\.30\\to 8\.30, bias\+7\.94\+7\.94\) but stays in band, where Flash\-Lite reaches17\.8617\.86and leaves it; on the CV exam*lenient*nearly halves its error \(4\.69→2\.694\.69\\to 2\.69\) by cancelling a−4\.45\-4\.45under\-marking — calibration masquerading as robustness \(Section[5\.4](https://arxiv.org/html/2609.29333#S5.SS4)\), now on a closed model\. At neutralgpt\-5\.5is below the floor on both exams,claude\-opus\-5on the ML exam only \(4\.374\.37\)\.

## 7Open\-Weights Configurations Under Neutral Prompts

Under neutral prompts nothing stands out: the1717baselines span2\.852\.85to7\.497\.49MAE, no family dominating or collapsing \(Table[14](https://arxiv.org/html/2609.29333#A4.T14), Appendix[D](https://arxiv.org/html/2609.29333#A4)\); none of it predicts which stop grading under a persona sentence\. Scaling is not monotone: Qwen2\.5\-Coder improves from77B to1414B \(5\.68→3\.515\.68\\to 3\.51\) then regresses at3232B \(7\.497\.49, bias−7\.35\-7\.35\); dropping the reference solution recovers3\.123\.12, consistent with over\-anchoring on it, though exam\-locally: the same removal moves it\+0\.16\+0\.16on the ML exam\. Within Qwen3\-Coder the3030B\-A33B MoE beats the8080B Coder\-Next \(4\.504\.50vs\.5\.175\.17\), reversing at480480B \(3\.143\.14\)\.

#### GLM\-4\-32B takes four of the five best open\-weights slots\.

Its best \(rubric breakdown removed, neutral; L\-BD0707\),2\.682\.68, is the grid’s strongest open\-weights grader \(Table[24](https://arxiv.org/html/2609.29333#A9.T24)\); L\-X0404\(*rigorous*\) reaches2\.742\.74, essentially unbiased; the fifth slot, Qwen3\-Coder\-Next\+\+*lenient*\(2\.812\.81\) is a bias\-correction artifact, not capability\. Two gaps to D01 \(1\.641\.64\): just over one point on best achievable,1\.421\.42on best surviving a persona sentence \(Qwen3\-Coder\-480480B under*rigorous*,3\.063\.06; L\-R0404\), since GLM\-4\-32B collapses to9\.669\.66under*strict*\.

#### Model choice beats size\.

The3030B\-A33B beats the8080B, the1414B the3232B, the7272B dense the235235B MoE; the480480B’s edge is persona robustness: at neutral it only ties the7272B\.*Which*model is a property of the exam, not the model: across the1717run on both, neutral MAE is uncorrelated \(ρ=−0\.07\\rho=\-0\.07\) and Qwen2\.5\-Coder\-3232B, weakest here, ties statistically for best there\. Open\-weights graders are practically viable, but only zero\-shot on a prompt guaranteed never to carry a strict\-flavoured persona \(Section[5](https://arxiv.org/html/2609.29333#S5)\), which Section[8](https://arxiv.org/html/2609.29333#S8)removes with in\-house labels\.

## 8Fine\-Tuning Repairs the Strict\-Persona Collapse

So far we have changed only the prompt\. We have better material than prompts: two humans graded every submission in both exams\. In this section we use their grades to fine\-tune the models with LoRA\[[20](https://arxiv.org/html/2609.29333#bib.bib24)\]\. We fine\-tune five small open models \(44B to3030B\)\. Each model gets three adapters: one per exam, and one trained on both exams pooled \(about3,9003\{,\}900examples\)\. Three things happen\. With the pooled adapter, every model grades as well as a human or better, on both exams\. The skill transfers: an adapter trained on one exam also gets better at the other, without ever seeing it\. And the persona problem is gone\. The prompt that adds as much as2020MAE points to a base model barely moves the fine\-tuned one\.

### 8\.1Setup

We fine\-tune five open models across three families and sizes from44B to3030B parameters with LoRA: Qwen2\.5\-Coder\-77B and1414B\[[21](https://arxiv.org/html/2609.29333#bib.bib1)\], Qwen3\-Coder\-3030B\-A33B\[[22](https://arxiv.org/html/2609.29333#bib.bib3)\], Llama\-3\.1\-88B\[[23](https://arxiv.org/html/2609.29333#bib.bib25)\], and Gemma\-4\-E44B\[[24](https://arxiv.org/html/2609.29333#bib.bib32)\]\. Training hyperparameters are in Appendix[E](https://arxiv.org/html/2609.29333#A5)\. Each exam uses a seed\-fixed80/2080/20student split:456/114456/114on the CV exam, giving1,4941\{,\}494training examples, and830/208830/208on the ML exam, giving2,4132\{,\}413\. The ML exam records one grade per question with the bonus folded in, so we split each grade into a score up to the base cap and the remainder as bonus\. We train three adapters per model: one per exam and one pooled\. The pooled training set is the union of the two train splits, so no adapter ever sees a held\-out student\. We evaluate all three, plus the untuned base, on both held\-out sets\. On the held\-out students, the two human graders disagree with each other by2\.832\.83marks on average on the CV exam and5\.195\.19on the ML exam\. These are the floors: a grader that disagrees with the humans by less than this agrees with them better than they agree with each other\. We use two training recipes with identical prompts\. The*marks*recipe averages the two graders’ scores into a single target, one example per student and question, with a \{score, bonus\} output format\. The*bd*\(breakdown\) recipe additionally distils a per\-task breakdown from Gemini 3 Flash Preview \(run D0101\), without its reasoning text\.

### 8\.2One adapter reaches human parity or better on both exams

Every untuned base grades worse than a human on both exams and mis\-calibrates in opposite directions: Llama\-3\.1\-88B reads19\.819\.8on the CV exam where the grader average is25\.325\.3, yet40\.540\.5on the ML exam where it is30\.930\.9\. With a score standard deviation of2\.12\.1, it barely distinguishes the CV exam’s students at all\. The pooled fine\-tune produces better results on both exams: it lands at1\.751\.75–2\.012\.01MAE on the CV exam and3\.293\.29–3\.533\.53on the ML exam \(marks recipe; Table[1](https://arxiv.org/html/2609.29333#S8.T1)\), revives the collapsed score spread on the CV exam \(σ\\sigmafrom2\.1→7\.32\.1\\to 7\.3–7\.57\.5, against8\.08\.0for the grader average\), and replaces the bases’ bias of−5\.9\-5\.9to\+9\.5\+9\.5marks with under one mark of generosity \(\+0\.2\+0\.2to\+0\.9\+0\.9\) on both exams\.

Because MAE against a two\-grader*average*is easier than any single grader faces \(Section[4](https://arxiv.org/html/2609.29333#S4)\), the verdicts come from a paired third\-grader test:

di=12​\(\|AIi−G1,i\|\+\|AIi−G2,i\|\)−\|G1,i−G2,i\|\.d\_\{i\}\\;=\\;\\tfrac\{1\}\{2\}\\big\(\|\\mathrm\{AI\}\_\{i\}\-\\mathrm\{G\}\_\{1,i\}\|\+\|\\mathrm\{AI\}\_\{i\}\-\\mathrm\{G\}\_\{2,i\}\|\\big\)\\;\-\\;\|\\mathrm\{G\}\_\{1,i\}\-\\mathrm\{G\}\_\{2,i\}\|\.\(1\)Pairing removes per\-student exam difficulty; averagingddover the held\-out students with a95%95\\%CI, an interval containing00means the AI is statistically indistinguishable from a human grader, entirely below00that it disagrees with the humans*less than they disagree with each other*, entirely above, worse\. Below zero is possible because\|G1−G2\|\|\\mathrm\{G\}\_\{1\}\-\\mathrm\{G\}\_\{2\}\|carries*two*graders’ idiosyncratic noise while\|AI−Gi\|\|\\mathrm\{AI\}\-\\mathrm\{G\}\_\{i\}\|carries one human’s plus the model’s; the AI beats the floor exactly when its own noise is smaller than a single human’s\. With no ground truth beyond the graders,*better*here means more reliable, not more correct: added to the grading pool, the model would introduce less disagreement than another human\.

All base configurations are significantly worse than a human grader on both exams\. The pooled marks adapter is significantly*better*in99of1010\(model×\\timesexam\) cells: all five models on the ML exam \(d¯\\overline\{d\}from−0\.94\-0\.94to−1\.13\-1\.13, every CI below zero\) and four of five on the CV\. The pooled bd adapter reaches at least parity in all1010cells and is better in two\. No cell on either exam reads worse \(Table[19](https://arxiv.org/html/2609.29333#A5.T19), Appendix[E](https://arxiv.org/html/2609.29333#A5)\)\. The ML exam sets a lower bar because its two graders disagree more \(floor5\.195\.19vs\.2\.832\.83\), but every point estimate there is negative regardless\.

### 8\.3Grading skill transfers across exams

An adapter fine\-tuned on one exam improves grading on the other, which it never saw: mean MAE on the ML exam falls from7\.957\.95\(base\) to5\.365\.36under the CV\-exam adapter, and on the CV exam from5\.545\.54to3\.573\.57under the ML\-exam adapter\. Every model improves in both directions, with one exception discussed in Section[8\.5](https://arxiv.org/html/2609.29333#S8.SS5)\. Transfer stops short of in\-domain performance \(3\.543\.54and1\.901\.90\), but pooling closes the gap for free: the pooled adapter matches or beats each specialist*on the specialist’s own exam*in all eight \(exam×\\timesrecipe\) columns \(Table[1](https://arxiv.org/html/2609.29333#S8.T1)\)\.

Table 1:Held\-out MAE vs\. the grader average by training set \(marks recipe; bd in Table[17](https://arxiv.org/html/2609.29333#A5.T17), Appendix[E](https://arxiv.org/html/2609.29333#A5)\)\. The cross columns are transfer; pooled matches or beats each specialist on its own exam\.‡\\ddagger: the one negative transfer cell \(Section[8\.5](https://arxiv.org/html/2609.29333#S8.SS5)\)\.
### 8\.4Fine\-tuning immunises against the strict\-persona collapse

We re\-run the persona sweep for base and pooled models on both exams \(Tables[16](https://arxiv.org/html/2609.29333#A5.T16)and[18](https://arxiv.org/html/2609.29333#A5.T18), Appendix[E](https://arxiv.org/html/2609.29333#A5)\)\. The bases reproduce the collapse on both exams: under*strict*, Qwen2\.5\-Coder\-1414B goes4\.06→23\.814\.06\\to 23\.81on the CV exam, and Llama\-3\.1\-88B reaches MAE30\.930\.9on the ML with bias−30\.9\-30\.9: it scores nearly every student zero\. The pooled adapters are flat everywhere: at most\+0\.32\+0\.32\(marks\) and\+0\.62\+0\.62\(bd\) across three personas, two exams and five models, and the catastrophic tail the collapse creates is gone: under a neutral prompt the worst base mis\-grades4343of114114CV\-exam students by more than1010marks, while every pooled model is at00–11\. On the CV exam the*strict*preset is the most destructive on every base, as Section[5\.2](https://arxiv.org/html/2609.29333#S5.SS2)predicts: it alone carries the two policy sentences\. The per\-exam adapters had one immunity failure, and pooling fixes it: Llama\-bd under*strict*sat at4\.084\.08when tuned on the CV exam alone and drops to2\.672\.67with pooled training\.

On the ML exam*strict*sometimes improves a base model \(Qwen3\-Coder\-3030B:8\.60→6\.348\.60\\to 6\.34\)\. This is not the persona working\. The bases over\-grade the ML exam, and the induced harshness happens to cancel part of the bias, the accidental calibration of Section[5\.4](https://arxiv.org/html/2609.29333#S5.SS4)again\. So the same wording that destroys a model on one exam helps it on the other\. An effect that flips sign between exams is not a calibration knob\.

### 8\.5What matters, what does not

Size and family remain nearly irrelevant\. The five pooled marks fine\-tunes converge to1\.751\.75–2\.012\.01MAE on the CV exam and3\.293\.29–3\.533\.53on the ML, and starting quality does not predict the tuned result: Llama\-3\.1\-88B starts worst on both exams \(7\.997\.99,12\.3012\.30\) and finishes with everyone else \(2\.012\.01,3\.373\.37\)\. Human labels alone suffice: pooled marks matches or beats pooled bd on every \(model, exam\) cell, decisively on the ML exam \(3\.413\.41vs\.4\.154\.15mean\)\. Nine of the eleven better verdicts in the paired test are marks verdicts, and marks needs no closed\-model teacher\. Which exam the labels come from matters less than having labels at all, with one exception\. Gemma\-4\-E44B is the strongest base on the ML exam \(4\.664\.66\) and the only model that cross\-exam transfer fails to help there \(4\.79‡4\.79^\{\\ddagger\}\), and in\-domain labels move it only to3\.513\.51: fine\-tuning buys the most where the base grades worst, and Gemma had the least room to improve\. The residual errors concentrate on the students the two human graders also disagree on \(correlation\+0\.35\+0\.35to\+0\.48\+0\.48for every pooled model, against no consistent relationship for the bases; Appendix[E](https://arxiv.org/html/2609.29333#A5)\): the tuned models are unsure exactly where the rubric is ambiguous\.

## 9Discussion

#### What the collapse is, and is not\.

Of the three failure modes only selective field collapse looks like a failure to negotiate conflicting instructions, and only as a hypothesis \(Section[5\.3](https://arxiv.org/html/2609.29333#S5.SS3)\); the others are calibration drift and refusal, which never appears on a Qwen model, so a single\-family roster misses that mode\. Spanning all six families unordered by scale, the failure belongs to open\-weights instruction following: no closed model of the three vendors leaves the band on either exam, and robustness above106106B is half illusory once the ML exam decomposes it\. Fine\-tuning shows the failure is zero\-shot, not small\-model\.

#### Why single\-prompt evaluation misses it\.

One well\-behaved prompt leaves the best open\-weights configuration just over one MAE point behind Gemini’s best \(2\.682\.68vs\.1\.641\.64\), one instructor\-style persona preamble nearly fourteen behind \(15\.5815\.58\): single\-prompt evaluation would have told a sunnier story\[[7](https://arxiv.org/html/2609.29333#bib.bib9),[8](https://arxiv.org/html/2609.29333#bib.bib10)\]\.

#### Implications for deployment\.

\(1\)Never instruct a grader to withhold credit: both policy sentences collapse graders on their own \(Section[5\.2](https://arxiv.org/html/2609.29333#S5.SS2)\), unordered by parameter count, and under\-grading harms students who attempted the work in good faith\. Verify the model against the prompt*and the exam*: on the ML exam the same preamble helps seven of1717models purely by cancelling their over\-marking, yet nearly doubles the error of the ML exam’s best\.\(2\)With in\-house dual\-graded examples, fine\-tune rather than prompt\-engineer — and pool the courses\.One adapter on the two exams’ pooled∼3,900\\sim 3\{,\}900examples brings every44B–3030B model to parity or better with a human grader on both exams and removes the collapse on both \(persona drift≤0\.32\\leq 0\.32MAE\), with no closed\-model teacher and no cost to either course\.\(3\)Evaluate several prompt variants, including instructor\-written ones\.\(4\)Report bias alongside MAE: Flash\-Lite’s lenient and strict have comparable MAE with opposite\-signed bias \(\+4\.01\+4\.01vs\.−5\.44\-5\.44\): MAE alone does not show which way the harm runs\.

## 10Limitations

#### Coverage\.

Both exams are from one university: replication spans courses, cohorts, rubrics and scales, not institutions\. Fine\-tuning’s two transfer directions are not size\-matched \(456456vs\.830830training students\), confounding their comparison; Gemma\-4’s ML\-exam baseline, unlike the other four, lacks independent corroboration\. We varied three strict\-flavoured wordings and two preset policy sentences, not every credit\-withholding instruction\. Both models above106106B are Qwen, entangling scale with family where it matters \(Section[5\.4](https://arxiv.org/html/2609.29333#S5.SS4)separates their robustness, not that confound\)\. Thegemini\-3\-flash\-previewablations used the first100100students \(matched\-subset baselines, wider CIs\)\. Closed coverage: four Gemini models; three personas on two OpenAI tiers and one Anthropic model \(Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1); Anthropic*strict*on the CV exam only\), no*rigorous*or*exacting*outside Gemini\. Grading is static: nothing is executed \(Appendix[A](https://arxiv.org/html/2609.29333#A1); no test suites, minutes\-to\-hours GPU cost, no stored outputs\), so “does not run” verdicts are unaudited and their error rate on valid but unconventional code unmeasured\. We know of no public corpus of long\-form notebooks with dual human rubric grades for external benchmarking; automated\-grading corpora are unit\-test based\[[5](https://arxiv.org/html/2609.29333#bib.bib13)\]or score short per\-task programs\[[2](https://arxiv.org/html/2609.29333#bib.bib12)\]\.

#### ML\-exam ground truth\.

Nine of the ML exam’s1,0381\{,\}038rows record0\.00\.0for one grader against real marks from the partner; they are retained in the headline floor but excluded from the maximum\-disagreement figure; excluding them everywhere moves no behaviour class, no table ordering and no held\-out verdict \(Appendix[H](https://arxiv.org/html/2609.29333#A8)\)\. Its pairing is hybrid rather than fixed, so per\-pair statistics are scoped to the1818\-pair backbone covering79%79\\%of the cohort\.

#### Few\-shot demonstrations\.

The K\-series demonstrations carry D0101’s labels, and their five students remain in the evaluated cohort \(5/5705/570\), graded with their own worked answer in the prompt\. The ML exam’s demonstrations likewise carry IG0808’s labels, with three students remaining; its Flash\-Lite arm suffers biased dropout from prompt\-length failures, so only its matched subsample is comparable \(Appendix[H](https://arxiv.org/html/2609.29333#A8)\)\.

#### Claims\.

Section[5\.3](https://arxiv.org/html/2609.29333#S5.SS3)’s three\-way split classifies rather than explains: withdrawing the rubric breakdown contradicts the conflicting\-instruction reading, leaving the causal story open\. The few\-shot uplift is capability, not held\-out generalisation: the demonstrations carry D0101’s own labels\. One seed\-fixed split and one run per configuration make the paired third\-grader test load\-bearing; its normal\-approximation CIs leave three CV\-exam*better\-than\-grader*verdicts marginal: upper bounds−0\.044\-0\.044\(14B marks\),−0\.020\-0\.020\(Gemma marks\) and−0\.001\-0\.001\(7B bd\), all strictly below zero under a20002000\-resample percentile bootstrap, though the weakest would contain zero under att\-interval \(p=0\.052p=0\.052paired,0\.0690\.069Wilcoxon\); the ML exam’s are not \(upper CIs≤−0\.30\\leq\-0\.30\)\. Appendix[J](https://arxiv.org/html/2609.29333#A10)details the remaining caveats\.

## 11Conclusion

For a dual\-graded exam the recipe is short\. The stronger closed models \(Gemini 3 Flash and 3\.1 Pro, GPT\-5\.5\) grade at or below the human floor untuned, from a neutral prompt; an open model gets there only with light fine\-tuning on the exam’s own grades, and only then survives the instructor edits that break it zero\-shot\. Between exams the vulnerability and its repair transfer; the model ranking, the persona’s direction andt=0t=0determinism do not\. So verify a grader on the exam it will grade, under an instructor’s own prompts\.

## Acknowledgments

We thank Prof\. Sultan Albarakati \(KAUST\) for his support, which made this paper possible\. We thank all graders whose grading produced the human ground truth, and the course instructors for permission to use the anonymised exam data\. For computer time, this research used Ibex managed by the Supercomputing Core Laboratory at King Abdullah University of Science & Technology \(KAUST\) in Thuwal, Saudi Arabia\.

## AI Use Statement

We used generative AI tools \(Opus 4\.8, Opus 5\.0 and Fable 5\) as coding and writing assistants throughout this work\. Specifically: for implementing and refactoring the data pipeline, the grading harness, the fine\-tuning scripts, and the analysis code; for drafting and editing prose in this paper; and for exploratory literature search\. We did not use generative AI to generate research ideas, to produce experimental results, or to write any of the analysis numbers reported here — every number in this paper is computed by the released scripts from the released data, and the ledgers those scripts emit are the authority for the manuscript\. All AI\-assisted code was reviewed and executed by the authors, and the statistical results were independently re\-derived before being reported\. We take responsibility for the final content of this work, including all text, claims, and artifacts\.

Note that generative models are also the*object*of study here rather than a tool: the LLM graders evaluated in Sections[6](https://arxiv.org/html/2609.29333#S6)–[8](https://arxiv.org/html/2609.29333#S8)produced the scores we analyse, and those outputs are released in full\.

## Reproducibility Statement

Every number in the paper is regenerated from released artifacts by released code\. The two anonymised exam datasets, the rubrics and reference solutions, one result workbook per configuration, and the analysis, grading, and fine\-tuning pipelines are released at[https://github\.com/KAUST\-Academy/where\-llm\-graders\-succeed\-and\-break](https://github.com/KAUST-Academy/where-llm-graders-succeed-and-break)\. Section[3](https://arxiv.org/html/2609.29333#S3)and Appendix[A](https://arxiv.org/html/2609.29333#A1)specify the prompt, the serving stack, the decoding parameters, and the exclusion rules; Appendix[I](https://arxiv.org/html/2609.29333#A9)tabulates all171171CV\-exam configurations with their bootstrap confidence intervals; Appendix[E](https://arxiv.org/html/2609.29333#A5)gives the fine\-tuning hyperparameters and the seed\-fixed split\. Training uses HuggingFace with PEFT and every evaluated number is served by vLLM, Gemma\-4’s adapter included\. One caveat is stated where it arises: evaluation is not bitwise reproducible because vLLM batching reorders reductions \(∼0\.01\\sim 0\.01MAE\); Gemma\-4’s vLLM serving path was cross\-validated against HuggingFace generation to within∼0\.1\\sim 0\.1MAE \(Appendix[E](https://arxiv.org/html/2609.29333#A5)\)\.

## Ethics Statement

The data are student exam submissions and grader marks from two university courses, released with the course instructors’ permission\. Both datasets are anonymised before release: student identifiers, names, and email addresses are removed from notebook code, markdown, outputs, and file metadata, and graders appear only as opaque identifiers\. No demographic attributes are collected or released, and no student is identifiable from the released artifacts\.

The application this paper studies carries real risk to the people being graded, which is why we report the failure mode rather than only the headline accuracy\. An under\-grading collapse of the kind documented in Section[5](https://arxiv.org/html/2609.29333#S5)harms students who attempted the work in good faith, and it is invisible to single\-prompt evaluation\. We therefore recommend against deploying an LLM grader on the basis of one prompt’s measured accuracy, and we report bias alongside MAE throughout so that the direction of harm is visible\. We regard human oversight of consequential grading decisions as a requirement, not an option, and none of the results here should be read as licensing unsupervised automated assessment\.

## References

- \[1\]J\. Prather, P\. Denny, J\. Leinonen, B\. A\. Becker, I\. Albluwi, M\. Craig, H\. Keuning, N\. Kiesler, T\. Kohn, A\. Luxton\-Reilly, S\. MacNeil, A\. Petersen, R\. Pettit, B\. N\. Reeves, and J\. Savelka\(2023\)The robots are here: navigating the generative AI revolution in computing education\.InProceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education,ITiCSE 2023,pp\. 108–159\.External Links:[Document](https://dx.doi.org/10.1145/3623762.3633499),[Link](https://doi.org/10.1145/3623762.3633499)Cited by:[§1](https://arxiv.org/html/2609.29333#S1.p1.1)\.
- \[2\]T\. Phung, V\. Pădurean, J\. Cambronero, S\. Gulwani, T\. Kohn, R\. Majumdar, A\. Singla, and G\. Soares\(2023\)Generative AI for programming education: benchmarking ChatGPT, GPT\-4, and human tutors\.InProceedings of the 2023 ACM Conference on International Computing Education Research \- Volume 2,ICER 2023,pp\. 41–42\.External Links:[Document](https://dx.doi.org/10.1145/3568812.3603476),[Link](https://doi.org/10.1145/3568812.3603476)Cited by:[§1](https://arxiv.org/html/2609.29333#S1.p1.1),[§10](https://arxiv.org/html/2609.29333#S10.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px1.p1.1)\.
- \[3\]L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. Gonzalez, and I\. Stoica\(2023\)Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 46595–46623\.External Links:[Document](https://dx.doi.org/10.52202/075280-2020),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf)Cited by:[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px1.p1.1)\.
- \[4\]Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu\(2023\)G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 2511–2522\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.153/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153)Cited by:[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px1.p1.1)\.
- \[5\]M\. Messer, N\. C\. C\. Brown, M\. Kölling, and M\. Shi\(2024\)Automated grading and feedback tools for programming education: a systematic review\.ACM Transactions on Computing Education24\(1\),pp\. 1–43\.External Links:[Document](https://dx.doi.org/10.1145/3636515),[Link](https://doi.org/10.1145/3636515)Cited by:[§10](https://arxiv.org/html/2609.29333#S10.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px1.p1.1)\.
- \[7\]M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. Suhr\(2024\)Quantifying language models’ sensitivity to spurious features in prompt design or: how I learned to start worrying about prompt formatting\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 25055–25083\.Cited by:[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px2.p1.1),[§5\.4](https://arxiv.org/html/2609.29333#S5.SS4.SSS0.Px1.p1.1),[§9](https://arxiv.org/html/2609.29333#S9.SS0.SSS0.Px2.p1.1)\.
- \[8\]M\. Mizrahi, G\. Kaplan, D\. Malkin, R\. Dror, D\. Shahaf, and G\. Stanovsky\(2024\)State of what art? A call for multi\-prompt LLM evaluation\.Transactions of the Association for Computational Linguistics12,pp\. 933–949\.External Links:[Link](https://aclanthology.org/2024.tacl-1.52/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00681)Cited by:[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px2.p1.1),[§5\.4](https://arxiv.org/html/2609.29333#S5.SS4.SSS0.Px1.p1.1),[§9](https://arxiv.org/html/2609.29333#S9.SS0.SSS0.Px2.p1.1)\.
- \[9\]A\. Deshpande, V\. Murahari, T\. Rajpurohit, A\. Kalyan, and K\. Narasimhan\(2023\)Toxicity in ChatGPT: analyzing persona\-assigned language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 1236–1270\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.88/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.88)Cited by:[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px2.p1.1)\.
- \[10\]E\. Wallace, K\. Xiao, R\. Leike, L\. Weng, J\. Heidecke, and A\. Beutel\(2024\)The instruction hierarchy: training LLMs to prioritize privileged instructions\.External Links:2404\.13208,[Link](https://arxiv.org/abs/2404.13208)Cited by:[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px3.p1.1)\.
- \[11\]J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou\(2023\)Instruction\-following evaluation for large language models\.External Links:2311\.07911,[Link](https://arxiv.org/abs/2311.07911)Cited by:[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px3.p1.1)\.
- \[12\]N\. Mu, S\. Chen, Z\. Wang, S\. Chen, D\. Karamardian, L\. Aljeraisy, B\. Alomair, D\. Hendrycks, and D\. Wagner\(2023\)Can LLMs follow simple rules?\.External Links:2311\.04235,[Link](https://arxiv.org/abs/2311.04235)Cited by:[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px3.p1.1)\.
- \[13\]Z\. Zhang, S\. Li, Z\. Zhang, X\. Liu, H\. Jiang, X\. Tang, Y\. Gao, Z\. Li, H\. Wang, Z\. Tan, Y\. Li, Q\. Yin, B\. Yin, and M\. Jiang\(2025\)IHEval: evaluating language models on following the instruction hierarchy\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 8374–8398\.External Links:[Link](https://aclanthology.org/2025.naacl-long.425/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.425)Cited by:[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px3.p1.1)\.
- \[14\]X\. He, Q\. Zhang, P\. Chen, G\. Chen, L\. Yu, Y\. Yuan, and S\. Yiu\(2026\)ConInstruct: evaluating large language models on conflict detection and resolution in instructions\.Proceedings of the AAAI Conference on Artificial Intelligence40\(37\),pp\. 30969–30977\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i37.40356),[Link](https://doi.org/10.1609/aaai.v40i37.40356)Cited by:[§2](https://arxiv.org/html/2609.29333#S2.SS0.SSS0.Px3.p1.1)\.
- \[15\]Gemini Teamet al\.\(2023\)Gemini: a family of highly capable multimodal models\.External Links:2312\.11805,[Link](https://arxiv.org/abs/2312.11805)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2609.29333#S3.SS0.SSS0.Px1.p1.1)\.
- \[16\]G\. Comaniciet al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.External Links:2507\.06261,[Link](https://arxiv.org/abs/2507.06261)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2609.29333#S3.SS0.SSS0.Px1.p1.1)\.
- \[17\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica\(2023\)Efficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles,SOSP ’23,pp\. 611–626\.External Links:[Document](https://dx.doi.org/10.1145/3600006.3613165),[Link](https://doi.org/10.1145/3600006.3613165)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2609.29333#S3.SS0.SSS0.Px1.p1.1)\.
- \[18\]B\. Efron\(1979\)Bootstrap Methods: Another Look at the Jackknife\.The Annals of Statistics7\(1\),pp\. 1–26\.External Links:[Document](https://dx.doi.org/10.1214/aos/1176344552),[Link](https://doi.org/10.1214/aos/1176344552)Cited by:[§3](https://arxiv.org/html/2609.29333#S3.SS0.SSS0.Px3.p1.1)\.
- \[19\]B\. Efron and R\.J\. Tibshirani\(1994\)An introduction to the bootstrap\.Chapman and Hall/CRC\.External Links:ISBN 9780429246593,[Document](https://dx.doi.org/10.1201/9780429246593),[Link](https://doi.org/10.1201/9780429246593)Cited by:[§3](https://arxiv.org/html/2609.29333#S3.SS0.SSS0.Px3.p1.1)\.
- \[20\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§8](https://arxiv.org/html/2609.29333#S8.p1.1)\.
- \[21\]B\. Huiet al\.\(2024\)Qwen2\.5\-Coder technical report\.External Links:2409\.12186,[Link](https://arxiv.org/abs/2409.12186)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1),[§8\.1](https://arxiv.org/html/2609.29333#S8.SS1.p1.1)\.
- \[22\]A\. Yanget al\.\(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1),[§8\.1](https://arxiv.org/html/2609.29333#S8.SS1.p1.1)\.
- \[23\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian,et al\.\(2024\)The Llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1),[§8\.1](https://arxiv.org/html/2609.29333#S8.SS1.p1.1)\.
- \[24\]Gemma Teamet al\.\(2026\)Gemma 4 technical report\.External Links:2607\.02770,[Link](https://arxiv.org/abs/2607.02770)Cited by:[§8\.1](https://arxiv.org/html/2609.29333#S8.SS1.p1.1)\.
- \[25\]A\. Yanget al\.\(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1)\.
- \[26\]Qwen Team\(2026\)Qwen3\-Coder\-Next technical report\.Technical reportAlibaba Cloud\.Note:Model card:[https://huggingface\.co/Qwen/Qwen3\-Coder\-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next)External Links:[Link](https://github.com/QwenLM/Qwen3-Coder/blob/main/qwen3_coder_next_tech_report.pdf)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1)\.
- \[27\]Meta\(2024\)Llama 3\.3 model card\.Note:[https://github\.com/meta\-llama/llama\-models/blob/main/models/llama3\_3/MODEL\_CARD\.md](https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/MODEL_CARD.md)70B Instruct released 6 December 2024Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1)\.
- \[28\]Team GLMet al\.\(2024\)ChatGLM: a family of large language models from GLM\-130B to GLM\-4 all tools\.External Links:2406\.12793,[Link](https://arxiv.org/abs/2406.12793)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1)\.
- \[29\]Z\.ai\(2025\)GLM\-4\-9B\-0414\.Note:Hugging Face model card,[https://huggingface\.co/zai\-org/GLM\-4\-9B\-0414](https://huggingface.co/zai-org/GLM-4-9B-0414)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1)\.
- \[30\]Z\.ai\(2025\)GLM\-4\-32B\-0414\.Note:Hugging Face model card,[https://huggingface\.co/zai\-org/GLM\-4\-32B\-0414](https://huggingface.co/zai-org/GLM-4-32B-0414)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1)\.
- \[31\]GLM\-4\.5 Teamet al\.\(2025\)GLM\-4\.5: agentic, reasoning, and coding \(ARC\) foundation models\.External Links:2508\.06471,[Link](https://arxiv.org/abs/2508.06471)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1)\.
- \[32\]Gemma Teamet al\.\(2025\)Gemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1)\.
- \[33\]Mistral AI\(2025\)Mistral\-Small\-24B\-Instruct\-2501\.Note:Hugging Face model card,[https://huggingface\.co/mistralai/Mistral\-Small\-24B\-Instruct\-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1)\.
- \[34\]DeepSeek\-AIet al\.\(2024\)DeepSeek\-Coder\-V2: breaking the barrier of closed\-source models in code intelligence\.External Links:2406\.11931,[Link](https://arxiv.org/abs/2406.11931)Cited by:[Appendix A](https://arxiv.org/html/2609.29333#A1.SS0.SSS0.Px2.p1.1)\.
- \[35\]A\. Pathak, R\. Gandhi, V\. Uttam, A\. Ramamoorthy, P\. Ghosh, A\. R\. Jindal, S\. Verma, A\. Mittal, A\. Ased, C\. Khatri, Y\. Nakka, Devansh, J\. S\. Challa, and D\. Kumar\(2025\)Rubric is all you need: improving LLM\-based code evaluation with question\-specific rubrics\.InProceedings of the 2025 ACM Conference on International Computing Education Research V\.1,ICER ’25,pp\. 181–195\.External Links:[Document](https://dx.doi.org/10.1145/3702652.3744220),[Link](https://doi.org/10.1145/3702652.3744220)Cited by:[Appendix B](https://arxiv.org/html/2609.29333#A2.SS0.SSS0.Px3.p2.1)\.
- \[36\]C\. Chan, W\. Chen, Y\. Su, J\. Yu, W\. Xue, S\. Zhang, J\. Fu, and Z\. Liu\(2024\)ChatEval: towards better LLM\-based evaluators through multi\-agent debate\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 9079–9093\.Cited by:[Appendix B](https://arxiv.org/html/2609.29333#A2.SS0.SSS0.Px3.p3.1)\.

## Appendix AExperimental Details

#### The CV exam in full\.

The CV exam’s570570submissions answer four code questions:Q1transfer learning with EfficientNetV2,Q2a CNN from scratch,Q3semantic segmentation with a pretrained encoder,Q4a colorization\-dataset bonus\. Beyond the3535base points \(weights1212/1111/1212\) sit up to1313bonus points \(Q1, Q2 and an all\-or\-nothing55\-point Q4\), graded but excluded from every reported metric\.

#### Model roster and serving\.

On the closed\-model side we use Google’sgemini\-2\.5\-pro\[[16](https://arxiv.org/html/2609.29333#bib.bib6)\]andgemini\-flash\-lite,gemini\-3\-flash\-preview, andgemini\-3\.1\-pro\-preview, for which no technical report was available at the time of writing, via the Vertex API\[[15](https://arxiv.org/html/2609.29333#bib.bib5)\]\. Two further vendors enter only in the cross\-vendor replication of Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1): OpenAI’sgpt\-5\.5andgpt\-5\.4and Anthropic’sclaude\-opus\-5\(Appendix[C](https://arxiv.org/html/2609.29333#A3), Table[9](https://arxiv.org/html/2609.29333#A3.T9)\)\. On the open\-weights side we use1717variants from six vendors\. From Qwen: Qwen2\.5\-Coder at77B /1414B /3232B\[[21](https://arxiv.org/html/2609.29333#bib.bib1)\]and the Qwen2\.57272B dense model\[[25](https://arxiv.org/html/2609.29333#bib.bib2)\], and from the Qwen3 family the Qwen3\-Coder3030B\-A33B MoE, the235235B\-A2222B MoE, and the480480B Qwen3\-Coder MoE in FP8\[[22](https://arxiv.org/html/2609.29333#bib.bib3)\], and the8080B Qwen3\-Coder\-Next MoE\[[26](https://arxiv.org/html/2609.29333#bib.bib4)\]\. From Meta: Llama\-3\.1\-88B\[[23](https://arxiv.org/html/2609.29333#bib.bib25)\]and Llama\-3\.3\-7070B\[[27](https://arxiv.org/html/2609.29333#bib.bib26)\]\. Z\.ai: GLM\-4\-99B and GLM\-4\-3232B, the04140414releases\[[28](https://arxiv.org/html/2609.29333#bib.bib27),[29](https://arxiv.org/html/2609.29333#bib.bib28),[30](https://arxiv.org/html/2609.29333#bib.bib29)\], and the106106B GLM\-4\.5\-Air MoE\[[31](https://arxiv.org/html/2609.29333#bib.bib30)\]\. Google: Gemma\-3\-1212B and Gemma\-3\-2727B\[[32](https://arxiv.org/html/2609.29333#bib.bib31)\]\. Mistral: Mistral\-Small\-2424B\[[33](https://arxiv.org/html/2609.29333#bib.bib33)\]\. DeepSeek: the1616B DeepSeek\-Coder\-V2\-Lite MoE \(2\.42\.4B active\)\[[34](https://arxiv.org/html/2609.29333#bib.bib34)\]\. Serving is vLLM\[[17](https://arxiv.org/html/2609.29333#bib.bib23)\]on one4×4\\timesA100 node, or8×8\\timesfor Qwen2\.5\-7272B,235235B\-A2222B and the480480B FP8 model\.

Open\-weights generation is*unconstrained*: the grader requests JSON in the prompt and parses the reply, with no grammar or schema enforcement at decode time \(Vertex calls do pass a JSON response schema\)\. A controlled A/B confirms that grammar\-constrained decoding changes no score and costs6\.9×6\.9\\timesin latency\. Every open\-weights grid run uses vLLM0\.19\.1or later, greedy decoding at temperature00— thet=0\.5t=0\.5probes L\-E0101/L\-E0202\(Appendix[C](https://arxiv.org/html/2609.29333#A3.SS0.SSS0.Px2)\) excepted — and one request in flight\.

#### Notebook serialisation, parsing and retries\.

Every call receives one question’s notebook as plain text — a\-\-\- Cellkk\[code\|markdown\] \-\-\-header then the cell’s source — with every output dropped: training logs, printed metrics and rendered plots or masks \(base64 media\) never reach the model\. Nothing is truncated in the grid; the fine\-tuning harness alone head\-and\-tail\-truncates to24,00024\{,\}000characters\. Replies are parsed as JSON; the open\-weights path falls back from strict parsing to a lenient parse and then tojson\_repair, the Vertex path parses strictly against its response schema\. A Vertex call is retried up to three times with a2s×2s\\timesattempt back\-off, except that aMAX\_TOKENSfinish is not retried because it is deterministic att=0t=0; the open\-weights path does not retry a truncated or unparseable reply att=0t=0for the same reason\. A question with no score after the run’s retries and resumes is a permanent failure, and the student is excluded from that run’s metrics as described under*Exclusions*\.

#### Ablation\-grid legend\.

Appendix[I](https://arxiv.org/html/2609.29333#A9)tabulates the grid’s3333closed\-model \(2525Gemini,66OpenAI,22Anthropic\) and138138open\-weights runs; the machine\-readable master comparison ships in the accompanying repository\. Gemini series:A= baseline \(n=1n\{=\}1\);B= prompt\-component ablations on Flash\-Lite \(n=4n\{=\}4: B0101drop solution, B0202drop guidelines, B0303thinking on, B0404drop breakdown\);C= strictness persona on Flash\-Lite \(n=2n\{=\}2\);D= model swap \(n=3n\{=\}3\);E= temperature variance \(n=3n\{=\}3\);F= combos and the 3\.1 Pro strict run \(n=3n\{=\}3\);G,M= prompt ablations and persona ongemini\-3\-flash\-preview\(n=4n\{=\}4,n=2n\{=\}2\);K= few\-shot \(n=1n\{=\}1\);P= alternate\-wording personas on Flash\-Lite \(n=2n\{=\}2\)\. Cross\-vendor closed series \(batch APIs; Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1)\):O= OpenAI,gpt\-5\.5\(O0101–O0303\) andgpt\-5\.4\(O1111–O1313\) under*neutral*,*strict*and*lenient*\(n=6n\{=\}6\);N= Anthropicclaude\-opus\-5under*neutral*and*strict*\(n=2n\{=\}2\)\.

Open\-weights series \(prefixedL\-\) fall in three groups\.*Qwen ablations:*L\-A/L\-AN= per\-model baselines \(n=5n\{=\}5\); on Qwen2\.5\-Coder\-32B,L\-B= prompt\-component ablations \(n=4n\{=\}4\),L\-C= persona \(n=2n\{=\}2\) andL\-P= alternate wordings \(n=2n\{=\}2\);L\-D/L\-DN=\+\+thinking on Qwen3 MoE \(n=2n\{=\}2\);L\-E= temperature variance \(n=2n\{=\}2\);L\-G,L\-H= prompt ablations replayed on the3030B and77B \(n=6n\{=\}6\);L\-K= few\-shot \(n=1n\{=\}1\);L\-M= persona on additional Qwen models \(n=7n\{=\}7\);L\-N,L\-Q,L\-R= full five\-persona sweeps on Qwen2\.5\-7272B, Qwen3\-235235B\-A2222B and Qwen3\-Coder\-480480B FP8 \(n=15n\{=\}15\)\.*Cross\-family sweeps*, each five personas on all570570students:L\-S= DeepSeek\-Coder\-V2\-Lite;L\-U= Llama\-3\.1\-88B;L\-V= Llama\-3\.3\-7070B;L\-W= GLM\-4\-99B;L\-X= GLM\-4\-3232B;L\-J= GLM\-4\.5\-Air;L\-F= Gemma\-3\-1212B;L\-Y= Gemma\-3\-2727B;L\-Z= Mistral\-Small\-2424B \(n=45n\{=\}45\)\.*Mechanism probes:*L\-MP= minimal pairs isolating the persona’s headword from its policy sentences, on five models \(n=25n\{=\}25\);L\-PS= the2×22\\times 2over the two policy sentences, on three models \(n=12n\{=\}12\);L\-BD= rubric\-breakdown removal crossed with persona, on five models \(n=10n\{=\}10\)\.

G\- and M\-series Gemini configurations run on the first100100students,gemini\-3\-flash\-previewthroughput being the binding constraint; comparisons against full\-nnbaselines restrict the baseline to the matching subset\.

#### Gemini thinking configuration\.

The Vertex API exposes an internal reasoning budget,thinking\_budget\. We run2121of the2525Gemini ablations with thinking*disabled*\(thinking\_budget=0=0\) and three \(B0303, F0101, G0303\)*enabled*at the dynamic budget \(−1\-1; the model picks the per\-call budget\)\. The exception is D0303:gemini\-2\.5\-prorejectsthinking\_budget=0=0, so it ran under that model’s default dynamic policy and reads as thinking\-enabled \(Section[10](https://arxiv.org/html/2609.29333#S10)\)\.

#### Exclusions\.

A student with any of Q11–Q33ungraded after retries is excluded from that run’s metrics rather than scored with a partial total: no students in most runs, at most33elsewhere, the exceptions being P0202\(1313\) and the temperature\-variance E\-series, where a student is kept only if all of Q11–Q33were graded in at least one rerun \(2121in E0101,77each in E0202/E0303\); per\-runnnaccompanies the affected tables\. P0202’s1313are genuine permanent failures, one question each \(66on Q11,77on Q22\); E0101’s2121are1414such failures plus the77students who submitted no notebook for at least one of Q11–Q33, and E0202/E0303’s77are those non\-submitters alone — neither run suffered a grading failure\. Only the E\-series rerun rule drops a non\-submitter: every single\-call run keeps them and scores the missing question00, and restoring them that way moves the E\-series MAE by−0\.03\-0\.03\(3\.38→3\.363\.38\\to 3\.36,3\.51→3\.483\.51\\to 3\.48,3\.60→3\.573\.60\\to 3\.57\)\. No log survives for these four runs, so we cannot quote afinish\_reason; where logs do survive \(the ML exam’s Gemini grid and F0303\),705705of709709permanently failed cells areMAX\_TOKENStruncations and the other44JSON parse errors, with429429,503503and dropped connections retryable and no safety block at all\. The artifacts agree: a failure burns149149–316316s \(P0202, one call\) or2,0952\{,\}095–2,5292\{,\}529s \(E0101, five reruns\) against a55–66s normal call; E0101’s1414cells fail5/55/5att=0t=0yet grade5/55/5att=0\.5t=0\.5and0\.70\.7; and the longest prompt,39,41839\{,\}418characters \(≈10,000\\approx 10\{,\}000tokens\) against a million\-token window, leaves only the65,53565\{,\}535\-token output cap to bind\. Unlike the ML exam’s few\-shot Flash\-Lite run \(Appendix[H](https://arxiv.org/html/2609.29333#A8)\), the dropout is neither length\- nor ability\-biased: excluded notebooks are no longer than survivors’ \(P020229,43329\{,\}433vs\.31,62531\{,\}625characters, Welchp=0\.35p=0\.35; E0101’s1414,31,65731\{,\}657vs\.31,77231\{,\}772,p=0\.90p=0\.90\) and their grader\-average totals match \(26\.926\.9vs\.26\.026\.0out of3535,p=0\.69p=0\.69;26\.026\.0vs\.26\.226\.2,p=0\.91p=0\.91\)\. E\-series per\-question scores are the mean over all recorded reruns \(five per cell by design; some Gemini cells logged more\) and the total is the Q11–Q33base sum as elsewhere\.

#### Item\-level agreement\.

Table[2](https://arxiv.org/html/2609.29333#A1.T2)gives per\-question and total\-level Spearmanρ\\rhobetween the AI mark and the grader\-average mark for the headline configurations of both exams\.

Table 2:Item\-level agreement for both exams’ headline configurations: Spearmanρ\\rhobetween the AI mark and the grader\-average mark, per question and at total level, over the same students each run’s MAE uses\. The total is the optimistic number: across the4040CV and3838ML runs measured here \(the rows below plus each exam’s1717matched neutral/strict pairs; Table[3](https://arxiv.org/html/2609.29333#A2.T3), Section[5\.4](https://arxiv.org/html/2609.29333#S5.SS4)\), it exceeds the weakest question’sρ\\rhoin all3737CV and3737ML runs where every correlation is defined — median gap0\.200\.20\(CV\),0\.120\.12\(ML\), and at least0\.150\.15in3030CV and1212ML runs\. The other three CV and one ML runs give every student the same mark, leaving their correlations undefined rather than zero\. The weak item is the same within an exam: Q22on CV \(3030of3737\), Q11on ML \(2727of3737\)\. Pearsonrrdiffers fromρ\\rhoby a median of0\.020\.02on both exams\. Generated byanalysis/checks/per\_question\_rank\_correlation\.py\.

## Appendix BStrict\-Persona Collapse: Additional Tables

This appendix carries the full exhibits behind Section[5](https://arxiv.org/html/2609.29333#S5)\.

Table 3:The*strict*persona against each model’s own*neutral*baseline at an identical prompt configuration\. MAE in points out of3535; ratio is strict÷\\divneutral; behaviour classes as in Section[5\.1](https://arxiv.org/html/2609.29333#S5.SS1); a refusal’s MAE is a ceiling artifact and must not be read as severity\. Every row isn=570n=570\. Generated byanalysis/computer\_vision\_make\_paper\_tables\.py\.RunModelFamilyParamsNeutralStrictRatioBehaviour*Open weights — ordered by total parameters*L\-M05Qwen2\.5\-Coder\-7BQwen7B5\.687\.92×1\.39\\times 1\.39gradedL\-U02Llama\-3\.1\-8BLlama8B7\.3126\.04×3\.56\\times 3\.56refusalL\-W02GLM\-4\-9BGLM9B4\.3125\.48×5\.91\\times 5\.91near\-refusalL\-F02Gemma\-3\-12BGemma12B5\.208\.77×1\.69\\times 1\.69collapseL\-M07Qwen2\.5\-Coder\-14BQwen14B3\.5124\.52×6\.99\\times 6\.99collapseL\-S02DeepSeek\-Coder\-V2\-LiteDeepSeek16B5\.9812\.09×2\.02\\times 2\.02collapseL\-Z02Mistral\-Small\-24BMistral24B3\.6626\.03×7\.11\\times 7\.11refusalL\-Y02Gemma\-3\-27BGemma27B4\.3211\.55×2\.67\\times 2\.67collapseL\-M01Qwen3\-Coder\-30B\-A3BQwen30B4\.5014\.91×3\.31\\times 3\.31collapseL\-X02GLM\-4\-32BGLM32B2\.859\.66×3\.39\\times 3\.39collapseL\-C01Qwen2\.5\-Coder\-32BQwen32B7\.4920\.29×2\.71\\times 2\.71collapseL\-V02Llama\-3\.3\-70BLlama70B4\.5213\.52×2\.99\\times 2\.99collapseL\-N02Qwen2\.5\-72BQwen72B3\.149\.82×3\.13\\times 3\.13collapseL\-M03Qwen3\-Coder\-Next 80BQwen80B5\.1711\.70×2\.26\\times 2\.26collapseL\-J02GLM\-4\.5\-AirGLM106B5\.6621\.02×3\.71\\times 3\.71collapseL\-Q02Qwen3\-235B\-A22BQwen235B4\.097\.32×1\.79\\times 1\.79gradedL\-R02Qwen3\-Coder\-480BQwen480B3\.143\.78×1\.20\\times 1\.20graded*Closed — reference*C01Flash\-LiteGemini—3\.345\.75×1\.72\\times 1\.72gradedF033\.1 Pro PreviewGemini—1\.862\.75×1\.48\\times 1\.48gradedFigure 2:Every strict\-flavoured persona run on the full cohort \(5151runs;5050at the default prompt configuration, plus F0101, which additionally enables thinking\), coloured by family and by wording \(*strict*darker;*rigorous*/*exacting*lighter\)\. The dashed line is the human inter\-grader floor \(MAE=2\.61\\text\{MAE\}=2\.61\)\. Runs classified as refusals are marked: their bar height is the distance to the grader mean and is not a severity measurement \(Section[5\.1](https://arxiv.org/html/2609.29333#S5.SS1)\)\. The two default\-configuration strict runs not shown are M0101\(n=99n=99, below the full\-cohort cutoff\) and F0303\(gemini\-3\.1\-pro\-preview,2\.752\.75; Table[3](https://arxiv.org/html/2609.29333#A2.T3)\)\.Table 4:The three strict\-flavoured presets on the models that received all of them\. Bold marks a run outside the graded band\. These presets arenotminimal pairs: only*strict*carries the two policy sentences \(Section[5\.2](https://arxiv.org/html/2609.29333#S5.SS2)\);*rigorous*and*exacting*ask for rubric precision with neither\. Qwen2\.5\-Coder\-77B, Qwen3\-Coder\-3030B\-A33B and Qwen3\-Coder\-Next received only*strict*in the full\-cohort grid; Section[8\.4](https://arxiv.org/html/2609.29333#S8.SS4)covers the first two\.M0101\(n=99n=99, the only strict run ongemini\-3\-flash\-preview\) has a CI of\[3\.60,4\.88\]\[3\.60,4\.88\]spanning most of the Gemini cluster of Section[5\.1](https://arxiv.org/html/2609.29333#S5.SS1), so we do not rank it within that cluster\.

Table 5:Attribution of the collapse to the policy sentences rather than the headword\.*Left:*S1 and S2 held verbatim, only the adjective varies; the*nharsh*control is the neutral frame with its one adjective swapped to “harsh”\.*Right:*the HARSH frame held fixed, crossing S1 with S2 \(both quoted in Section[5\.2](https://arxiv.org/html/2609.29333#S5.SS2)\)\. All rowsn≥569n\\geq 569, default prompt configuration,t=0t=0\. Spreads are max−\-min of the unrounded MAEs, so they can differ from the difference of the printed cells by0\.010\.01\. Bold marks cells whose damage matches the full preset’s\.Table 6:How each matched*strict*run fails\. Q1=01=0is the count of students awarded zero on Q11; the next column is how many of*those*still received a non\-zero Q22\. Generated byanalysis/computer\_vision\_make\_paper\_tables\.py\.The three near\-total zeroers have awarded\-total standard deviations of0\.000\.00,0\.090\.09and2\.422\.42and AI–grader correlations undefined,−0\.01\-0\.01and0\.190\.19\. GLM\-4\.5\-Air blanket\-zeroes63%63\\%of students and still grades the rest \(r=0\.53r=0\.53\), which is why Table[3](https://arxiv.org/html/2609.29333#A2.T3)classes it a collapse rather than a refusal\. Qwen2\.5\-Coder\-1414B \(L\-M0707\) is the second intermediate case: it zeroes88%88\\%of whole submissions \(mean awarded total1\.521\.52,r=0\.25r=0\.25\), just under the90%90\\%refusal cut\. Q11\-zero rates fall at00–5%5\\%or2424–100%100\\%with nothing between, and among the latter the selective fraction is00–15%15\\%or5151–77%77\\%\.

Table 7:Self\-contradiction between the structured Q11score and the prose breakdown, on the two strict\-persona runs whose rationales we annotated by hand\. A row is self\-contradicting when the structured score is00but the rationale itemises positive partial credit\. Both runs sit in the selective\-field\-collapse group of Table[6](https://arxiv.org/html/2609.29333#A2.T6)and match its Q1=01=0counts; the second column differs because this one requires reading the prose, which we did not do at scale\.#### Representative example\.

The clearest case in our sample is L\-C0101student\#​2\\\#2, where both graders gave Q1=12\.0/121=12\.0/12and the model emitted Q1=01=0\. The reasoning blob begins:

> The student completed most tasks correctly, including defining the transforms, creating DataLoaders, … However, the backbone was not properly frozen, and the bonus task for Test Time Augmentation was not implemented\.

The same blob then itemises a breakdown \(“Complete transforms: 1\.5”, “Create DataLoaders: 0\.5”, …\) summing to10\.010\.0points awarded in prose — the largest contradicted total in Table[7](https://arxiv.org/html/2609.29333#A2.T7)\.

#### A prediction of the conflict account that fails\.

If the collapse is driven by a conflict between S2 and the rubric breakdown, withdrawing the breakdown should remove one side of the conflict and*relieve*the collapse\. Removing the breakdown under*strict*worsens MAE by\+6\.42\+6\.42on Llama\-3\.3\-70B \(13\.52→19\.9413\.52\\to 19\.94\),\+5\.92\+5\.92on GLM\-4\-32B \(9\.66→15\.589\.66\\to 15\.58\),\+3\.93\+3\.93on DeepSeek\-Coder\-V2\-Lite \(12\.09→16\.0212\.09\\to 16\.02\),\+1\.40\+1\.40on Qwen2\.5\-Coder\-32B \(20\.29→21\.6920\.29\\to 21\.69\) and\+0\.15\+0\.15on Gemma\-3\-27B \(11\.55→11\.6911\.55\\to 11\.69\) — five of five in the wrong direction\. The same manipulation under*neutral*is a clean control, with a largest shift of\+0\.85\+0\.85and four of five models within±0\.25\\pm 0\.25, so this is specific to the persona rather than an artifact of a shorter prompt\.

The fairest reading is that this weakens the causal story without settling on an alternative; the structural\-scaffolding reading of Section[5\.3](https://arxiv.org/html/2609.29333#S5.SS3)fits the L\-BD result but is itself untested\. Withdrawing the breakdown leaves the rubric table in the prompt, so the conflict is attenuated rather than eliminated; a fully clean test would strip the rubric too, which we have not run\. We flag this as open rather than resolved \(Section[10](https://arxiv.org/html/2609.29333#S10)\)\.

#### Prompt\-only baselines\.

Before fine\-tuning, we asked whether the*strict*\-persona collapse can be undone by prompting alone\. We tried the two repairs the literature suggests on the four CV\-exam models whose*strict*runs collapse or refuse \(Qwen2\.5\-Coder\-3232B, GLM\-4\-3232B, Mistral\-Small\-2424B, Llama\-3\.1\-88B\), over the first100100students \(the G\-series subset\), Q11–Q33, with the paper’s vLLM stack, and compared each repair against the same model’s own*strict*and neutral runs restricted to the same students \(Table[8](https://arxiv.org/html/2609.29333#A2.T8)\)\.

The first repair is*decomposition*\[[35](https://arxiv.org/html/2609.29333#bib.bib35)\]: instead of grading a whole question in one call, the grader makes one call per rubric task row \(1616/1313/1111rows for Q11/Q22/Q33; penalty and bonus rows handled\) and awards only that row’s marks; the row awards are summed and clamped to the question maxima\. The persona, guidelines, reference solution and submission are exactly those of the paper’s prompt\. The hope is that a small, concrete criterion leaves the model less room to zero a whole question\. It does not\. Qwen2\.5\-Coder\-3232B moves from21\.2421\.24to20\.0020\.00MAE, Mistral\-Small\-2424B from27\.2527\.25to26\.2726\.27, and Llama\-3\.1\-88B refuses on every row exactly as it refused on every question \(27\.2627\.26\)\. Only GLM\-4\-3232B re\-enters the graded band \(8\.90→5\.018\.90\\to 5\.01\), and even then it sits at2\.2×2\.2\\timesits neutral error\. The credit\-withholding sentences bind on each criterion just as they bind on the question as a whole\.

The second repair is*grade\-then\-arbitrate*\[[36](https://arxiv.org/html/2609.29333#bib.bib36)\]: the paper’s*strict*prompt and the paper’s neutral prompt each grade the question, and a third call under the neutral persona receives both JSON grades together with the rubric and must resolve every disagreement on the rubric’s partial\-credit scale\. The arbiter’s grade is the run’s grade, at three calls per question instead of one\. This does bring every model back into the graded band, but only to where its own neutral prompt already was: GLM\-4\-3232B2\.642\.64against2\.242\.24neutral, Mistral3\.133\.13against3\.593\.59, Llama7\.467\.46against7\.737\.73\(on9999students: one arbiter reply, student8787, Q22, was truncated at4,0964\{,\}096tokens\), Qwen9\.299\.29against8\.028\.02\. The arbiter defers to the neutral grade, so arbitration repairs the persona by removing it, at three times the cost, and gains nothing beyond the neutral prompt\.

Neither repair therefore reaches what fine\-tuning reaches\. The neutral level the repairs recover is2\.22\.2–8\.08\.0MAE on these four models; the pooled adapters of Section[8](https://arxiv.org/html/2609.29333#S8)start from that level and end at1\.751\.75–2\.012\.01on the held\-out CV students, below the human floor\. Scripts:prompt\_baselines\_grade\.py,analysis/checks/prompt\_baselines\_vs\_paper\.py\.

Table 8:Prompt\-only repairs under the*strict*persona, CV exam, first100100students,3535\-point base scale; the paper’s strict and neutral runs are restricted to the same students\. MAE with95%95\\%bootstrap CI \(20002000resamples\); zero = share of students awarded a total of00; behaviour classes as in Section[5\.1](https://arxiv.org/html/2609.29333#S5.SS1)\. Human floor on these students:3\.433\.43\.

## Appendix CClosed\-Model Details

Table 9:Closed models from three vendors under*neutral*,*strict*and*lenient*on both exams \(Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1)\): MAE against the grader average with95%95\\%bootstrap CI, and bias\. Gemini rows are grid runs \(D0202/F0303/F0202and A0101/C0101/C0202on the CV exam; IG0808/IG2323/IG1414and IG0101/IG0505/IG0606on the ML exam, with theirnnas in Tables[24](https://arxiv.org/html/2609.29333#A9.T24)and[25](https://arxiv.org/html/2609.29333#A11.T25)\)\. OpenAI and Anthropic rows were graded on the full cohorts through the vendors’ batch APIs with the identical prompt, reasoning off and temperature00\(OpenAI\) or thinking disabled \(Anthropic, whose API exposes no temperature\); every reply parsed\. Bold marks a run outside the graded band; no zero\-total rate exceeds1\.1%1\.1\\%\. Generated byanalysis/closed\_vendor\_table\.py\.ModelPersonaCV exam \(floor2\.612\.61\)ML exam \(floor5\.135\.13\)MAE \[95% CI\]biasMAE \[95% CI\]biasclaude\-opus\-5*neutral*3\.543\.54\[3\.31,3\.77\]\[3\.31,\\,3\.77\]−3\.24\-3\.244\.374\.37\[4\.13,4\.63\]\[4\.13,\\,4\.63\]−2\.64\-2\.64*strict*5\.175\.17\[4\.90,5\.45\]\[4\.90,\\,5\.45\]−5\.02\-5\.02——gemini\-3\.1\-pro\-preview*neutral*1\.861\.86\[1\.70,2\.02\]\[1\.70,\\,2\.02\]−0\.82\-0\.823\.403\.40\[3\.16,3\.65\]\[3\.16,\\,3\.65\]\+1\.18\+1\.18*strict*2\.752\.75\[2\.51,2\.99\]\[2\.51,\\,2\.99\]−2\.09\-2\.093\.673\.67\[3\.43,3\.94\]\[3\.43,\\,3\.94\]−1\.00\-1\.00*lenient*1\.791\.79\[1\.62,1\.95\]\[1\.62,\\,1\.95\]\+0\.82\+0\.824\.484\.48\[4\.22,4\.75\]\[4\.22,\\,4\.75\]\+3\.46\+3\.46gemini\-flash\-lite*neutral*3\.343\.34\[3\.15,3\.53\]\[3\.15,\\,3\.53\]−1\.78\-1\.787\.537\.53\[7\.20,7\.87\]\[7\.20,\\,7\.87\]\+5\.98\+5\.98*strict*5\.755\.75\[5\.46,6\.04\]\[5\.46,\\,6\.04\]−5\.44\-5\.449\.029\.02\[8\.61,9\.43\]\[8\.61,\\,9\.43\]−7\.83\-7\.83*lenient*4\.494\.49\[4\.18,4\.82\]\[4\.18,\\,4\.82\]\+4\.01\+4\.0117\.86\\mathbf\{17\.86\}\[17\.30,18\.42\]\[17\.30,\\,18\.42\]\+17\.80\+17\.80gpt\-5\.4*neutral*4\.694\.69\[4\.46,4\.92\]\[4\.46,\\,4\.92\]−4\.45\-4\.454\.304\.30\[4\.07,4\.55\]\[4\.07,\\,4\.55\]−1\.95\-1\.95*strict*6\.906\.90\[6\.63,7\.15\]\[6\.63,\\,7\.15\]−6\.81\-6\.815\.115\.11\[4\.85,5\.37\]\[4\.85,\\,5\.37\]−3\.61\-3\.61*lenient*2\.692\.69\[2\.51,2\.89\]\[2\.51,\\,2\.89\]\+1\.00\+1\.008\.308\.30\[7\.96,8\.64\]\[7\.96,\\,8\.64\]\+7\.94\+7\.94gpt\-5\.5*neutral*2\.432\.43\[2\.28,2\.59\]\[2\.28,\\,2\.59\]−1\.65\-1\.653\.543\.54\[3\.33,3\.77\]\[3\.33,\\,3\.77\]\+1\.23\+1\.23*strict*4\.434\.43\[4\.19,4\.68\]\[4\.19,\\,4\.68\]−4\.25\-4\.253\.703\.70\[3\.48,3\.93\]\[3\.48,\\,3\.93\]−1\.38\-1\.38*lenient*2\.252\.25\[2\.08,2\.44\]\[2\.08,\\,2\.44\]\+1\.13\+1\.136\.296\.29\[6\.02,6\.57\]\[6\.02,\\,6\.57\]\+5\.83\+5\.83

Table 10:The six full\-cohort configurations, and five first\-100100runs, whose point\-estimate MAE meets or undercuts the human inter\-grader floor, with95%95\\%bootstrap CIs \(20002000resamples\), plus reference rows\. The G\-series and M02 rows cover only the first∼100\\sim 100students, an easier subset \(the unmodified D01 recipe scores1\.231\.23there vs\.1\.641\.64on the full cohort; Appendix[C](https://arxiv.org/html/2609.29333#A3.SS0.SSS0.Px1)\); they are not comparable to the full\-cohort rows, and M02’s CI upper bound crosses the floor\.RunModelnnMAE95%95\\%CIBias*Human inter\-grader floor \(Section[4](https://arxiv.org/html/2609.29333#S4)\)*—G1vs\. G25702\.61\[2\.37,2\.85\]\[2\.37,\\ 2\.85\]—*Full\-cohort configurations*D01gemini\-3\-flash\-preview, neutral5701\.64\[1\.51,1\.79\]\[1\.51,\\ 1\.79\]\+0\.01\+0\.01F02gemini\-3\.1\-pro\-preview \+ lenient5701\.79\[1\.62,1\.95\]\[1\.62,\\ 1\.95\]\+0\.82\+0\.82D02gemini\-3\.1\-pro\-preview, neutral5701\.86\[1\.70,2\.02\]\[1\.70,\\ 2\.02\]−0\.82\-0\.82B03gemini\-flash\-lite \+ thinking5702\.05\[1\.89,2\.21\]\[1\.89,\\ 2\.21\]−0\.15\-0\.15O03gpt\-5\.5 \+ lenient \(batch API\)5702\.25\[2\.08,2\.44\]\[2\.08,\\ 2\.44\]\+1\.13\+1\.13O01gpt\-5\.5, neutral \(batch API\)5702\.43\[2\.28,2\.59\]\[2\.28,\\ 2\.59\]−1\.65\-1\.65*First\-100100\-students subset \(easier subset; see caption\)*D01 \(restricted\)gemini\-3\-flash\-preview, baseline recipe on the same100100students1001\.23\[1\.02,1\.49\]\[1\.02,\\ 1\.49\]−0\.18\-0\.18G043\-flash\-preview, no rubric breakdown1001\.25\[1\.01,1\.51\]\[1\.01,\\ 1\.51\]−0\.07\-0\.07G033\-flash\-preview \+ thinking1001\.39\[1\.18,1\.61\]\[1\.18,\\ 1\.61\]−0\.12\-0\.12G023\-flash\-preview, no guidelines991\.60\[1\.30,1\.93\]\[1\.30,\\ 1\.93\]\+0\.79\+0\.79G013\-flash\-preview, no reference solution981\.63\[1\.32,2\.02\]\[1\.32,\\ 2\.02\]\+0\.71\+0\.71M023\-flash\-preview \+ lenient1002\.18\[1\.75,2\.65\]\[1\.75,\\ 2\.65\]\+1\.90\+1\.90*Reference rows*A01gemini\-flash\-lite \(baseline recipe\)5703\.34\[3\.15,3\.53\]\[3\.15,\\ 3\.53\]−1\.78\-1\.78L\-R04Qwen3\-Coder\-480B \+ rigorous \(best robust open\)5703\.06\[2\.88,3\.26\]\[2\.88,\\ 3\.26\]−0\.23\-0\.23

Table 11:The paired third\-grader test \(Eq\. \([1](https://arxiv.org/html/2609.29333#S8.E1)\)\) applied to the zero\-shot configurations, on the full570570\-student CV cohort: the same statistic and decision rule as Table[19](https://arxiv.org/html/2609.29333#A5.T19), the standard the fine\-tuned models face\.*better*means the whole95%95\\%CI ofd¯\\overline\{d\}sits below zero, “==grader” that it contains zero,*worse*that it lies above\. CIs are the normal approximation used in Table[19](https://arxiv.org/html/2609.29333#A5.T19); a20002000\-resample percentile bootstrap gives the same verdict in every row\. With the floor at2\.612\.61,d¯=12​\(\|AI−G1\|\+\|AI−G2\|\)−2\.61\\overline\{d\}=\\tfrac\{1\}\{2\}\(\|\\mathrm\{AI\}\-\\mathrm\{G\}\_\{1\}\|\+\|\\mathrm\{AI\}\-\\mathrm\{G\}\_\{2\}\|\)\-2\.61exactly\. The two single\-grader distances differ by up to0\.630\.63here, so both are shown\. Generated byanalysis/checks/paired\_test\_zero\_shot\_cv\.py\.RunConfiguration\|AI−Gavg\|\|\\mathrm\{AI\}\{\-\}\\mathrm\{G\}\_\{\\mathrm\{avg\}\}\|\|AI−G1\|\|\\mathrm\{AI\}\{\-\}\\mathrm\{G\}\_\{1\}\|\|AI−G2\|\|\\mathrm\{AI\}\{\-\}\\mathrm\{G\}\_\{2\}\|d¯\\overline\{d\}\[95%95\\%CI\]VerdictD013\-flash\-preview, neutral1\.642\.231\.92−0\.54\-0\.54\[−0\.71,−0\.36\]\[\-0\.71,\-0\.36\]betterF023\.1\-pro\-preview\+\+lenient1\.792\.322\.15−0\.37\-0\.37\[−0\.56,−0\.18\]\[\-0\.56,\-0\.18\]betterD023\.1\-pro\-preview, neutral1\.862\.531\.96−0\.37\-0\.37\[−0\.55,−0\.18\]\[\-0\.55,\-0\.18\]betterB03flash\-lite\+\+thinking2\.052\.552\.29−0\.19\-0\.19\[−0\.39,\+0\.01\]\[\-0\.39,\+0\.01\]==graderO03gpt\-5\.5\+\+lenient \(Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1)\)2\.252\.592\.62−0\.01\-0\.01\[−0\.22,\+0\.21\]\[\-0\.22,\+0\.21\]==graderO01gpt\-5\.5, neutral \(Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1)\)2\.433\.102\.51\+0\.19\+0\.19\[−0\.02,\+0\.41\]\[\-0\.02,\+0\.41\]==grader*Reference rows \(do not clear the floor\)*A01flash\-lite, baseline recipe3\.343\.833\.46\+1\.03\+1\.03\[\+0\.77,\+1\.29\]\[\+0\.77,\+1\.29\]worseN01claude\-opus\-5, neutral \(Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1)\)3\.544\.053\.42\+1\.13\+1\.13\[\+0\.88,\+1\.37\]\[\+0\.88,\+1\.37\]worse

Figure 3:D01 \(gemini\-3\-flash\-preview, neutral,t=0t=0\): AI total vs\. grader\-average total over the fulln=570n=570submissions, withy=xy=xas the perfect\-agreement reference\. Bias is\+0\.01\+0\.01, and residual error concentrates in the under\-1515band, where graders also disagree more\.Table 12:Prompt\-component ablations ongemini\-flash\-lite\. B03 vs\. A01 measures dynamic\-budget thinking against the baseline’s disabled thinking \(thinking\_budget=0=0; Appendix[A](https://arxiv.org/html/2609.29333#A1)\)\.n=570n=570except B01 \(567567\) and B02/B04 \(569569each\), which exclude students with a permanently failed grading call\.Table 13:G\-series ongemini\-3\-flash\-preview, all on the first\-100100subset \(n=100n=100, except G01/G02 at9898/9999after excluding permanently failed grading calls\)\. D01\-restricted is the baseline recipe on the same students\. Bootstrap CI on20002000resamples\.#### The apparent G\-series wins are a sample\-size artifact \(n=100n=100\)\.

The G\-series replays the ablations ongemini\-3\-flash\-preview, whose throughput limited these runs to the first100100students\. Against D01’s full\-cohort1\.641\.64, G04 \(no rubric breakdown,1\.251\.25\) appears to win; restricting D01 to the same students gives1\.231\.23, so the baseline ties G04 within noise and leads the other three \(Table[13](https://arxiv.org/html/2609.29333#A3.T13)\)\. Dropping the reference solution or the guidelines costs about0\.40\.4MAE, with CIs still grazing the baseline’s\.

#### Flash\-Lite att=0t=0is deterministic here; sampling adds variance without gain\.

We measured sampling variance ongemini\-flash\-lite\(E01/E02/E03 att=0\.0/0\.5/0\.7t=0\.0/0\.5/0\.7, five reruns per cell\)\. As everywhere in this paper the statistics cover the three graded questions only \(Q44is bonus\-only\), leaving1,6871,687/1,7011,701/1,7011,701\(student, question\) cells per configuration\. Att=0t=0the model is fully deterministic:100%100\\%of cells have zero standard deviation across reruns\. Att=0\.5t=0\.5the median per\-cell standard deviation is0\.650\.65points and22\.3%22\.3\\%of cells have a standard deviation above one point; att=0\.7t=0\.7those figures rise to0\.760\.76and31\.8%31\.8\\%\. Qwen2\.5\-Coder\-3232B and Qwen3\-Coder\-Next att=0\.5t=0\.5\(∼50\\sim 50\-student subset\) show0\.760\.76and0\.820\.82, comparable to Gemini att=0\.7t=0\.7\. Averaging the reruns does not beat a singlet=0t=0call: mean\-of\-reruns MAE is3\.383\.38att=0t=0,3\.513\.51att=0\.5t=0\.5, and3\.603\.60att=0\.7t=0\.7, versus3\.343\.34for the single\-call A01 baseline \(the0\.040\.04gap att=0t=0comes from E01’s smaller cohort, not from nondeterminism\)\. We therefore recommend a single call att=0t=0as the production default\.

## Appendix DOpen\-Weights Details

This appendix carries the exhibits behind Section[7](https://arxiv.org/html/2609.29333#S7): neutral\-prompt baselines for all1717variants and the few\-shot comparison\. L\-A3232rationales often penalise valid stylistic divergence from the reference solution — different variable names or layer orderings — supporting the over\-anchoring account of the3232B regression informally\.

#### Few\-shot prompting moves the open model substantially\.

Two worked examples per question \(Table[15](https://arxiv.org/html/2609.29333#A4.T15)\) improve Flash\-Lite by0\.310\.31MAE and Qwen3\-Coder\-3030B\-A33B by≈1\.16\\approx 1\.16; on the ML exam it nearly halves the same model’s error \(11\.12→5\.8411\.12\\to 5\.84; Appendix[H](https://arxiv.org/html/2609.29333#A8)\)\. Excluding the five demonstration students changes nothing measurable: on the562562\(Flash\-Lite\) and565565\(Qwen3\-Coder\-3030B\-A33B\) students left, the uplift is\+0\.30\+0\.30\[0\.05,0\.54\]\[0\.05,0\.54\]and\+1\.16\+1\.16\[1\.04,1\.30\]\[1\.04,1\.30\]against\+0\.31\+0\.31and\+1\.16\+1\.16on the full cohort, and the ML exam’s arms match \(\+5\.28\+5\.28against\+5\.28\+5\.28for the3030B,\+3\.56\+3\.56against\+3\.55\+3\.55for Flash\-Lite, less its three demonstration students\)\.

Table 14:Open\-weights baselines on the default prompt \(neutral persona,t=0t=0\), ordered by family then total parameters\. MAE in points out of3535with95%95\\%bootstrap CI; bias is mean signed error \(AI−\-grader\-avg\)\. Six vendors and both dense and mixture\-of\-experts architectures\. All rowsn=570n=570\.RunModelParamsMAE95%95\\%CIBiasL\-S01DeepSeek\-Coder\-V2\-Lite \(MoE\)16B5\.98\[5\.69,6\.29\]\[5\.69,\\ 6\.29\]−3\.44\-3\.44L\-W01GLM\-4\-9B9B4\.31\[4\.02,4\.63\]\[4\.02,\\ 4\.63\]\+1\.68\+1\.68L\-X01GLM\-4\-32B32B2\.85\[2\.66,3\.05\]\[2\.66,\\ 3\.05\]\+0\.50\+0\.50L\-J01GLM\-4\.5\-Air \(MoE\)106B5\.66\[5\.38,5\.95\]\[5\.38,\\ 5\.95\]−5\.37\-5\.37L\-F01Gemma\-3\-12B12B5\.20\[4\.94,5\.48\]\[4\.94,\\ 5\.48\]−2\.75\-2\.75L\-Y01Gemma\-3\-27B27B4\.32\[4\.12,4\.55\]\[4\.12,\\ 4\.55\]−1\.52\-1\.52L\-U01Llama\-3\.1\-8B8B7\.31\[6\.97,7\.68\]\[6\.97,\\ 7\.68\]−5\.15\-5\.15L\-V01Llama\-3\.3\-70B70B4\.52\[4\.31,4\.73\]\[4\.31,\\ 4\.73\]−3\.60\-3\.60L\-Z01Mistral\-Small\-24B24B3\.66\[3\.46,3\.87\]\[3\.46,\\ 3\.87\]−2\.41\-2\.41L\-A07Qwen2\.5\-Coder\-7B7B5\.68\[5\.39,5\.98\]\[5\.39,\\ 5\.98\]−4\.84\-4\.84L\-A14Qwen2\.5\-Coder\-14B14B3\.51\[3\.30,3\.74\]\[3\.30,\\ 3\.74\]−1\.16\-1\.16L\-A30MQwen3\-Coder\-30B\-A3B \(MoE\)30B4\.50\[4\.29,4\.72\]\[4\.29,\\ 4\.72\]−2\.70\-2\.70L\-A32Qwen2\.5\-Coder\-32B32B7\.49\[7\.19,7\.78\]\[7\.19,\\ 7\.78\]−7\.35\-7\.35L\-N01Qwen2\.5\-72B \(dense\)72B3\.14\[2\.95,3\.35\]\[2\.95,\\ 3\.35\]\+0\.19\+0\.19L\-ANxtQwen3\-Coder\-Next 80B \(MoE\)80B5\.17\[4\.92,5\.42\]\[4\.92,\\ 5\.42\]−4\.74\-4\.74L\-Q01Qwen3\-235B\-A22B \(MoE\)235B4\.09\[3\.87,4\.32\]\[3\.87,\\ 4\.32\]−3\.56\-3\.56L\-R01Qwen3\-Coder\-480B FP8 \(MoE\)480B3\.14\[2\.95,3\.35\]\[2\.95,\\ 3\.35\]\+0\.51\+0\.51Table 15:Two worked examples per question, drawn from the highest\-agreement grader pair and labelled with D0101’s scores and rationales \(in\-context distillation, not human exemplars\); the five demonstration students remain in the evaluated cohort \(Appendix[J](https://arxiv.org/html/2609.29333#A10)\), though excluding them moves the uplift by at most0\.010\.01\(analysis/checks/fewshot\_demo\_leakage\.py\)\. Few\-shot moves Gemini Flash\-Lite by≈0\.3\\approx 0\.3MAE and Qwen3\-Coder\-3030B\-A33B by≈1\.2\\approx 1\.2MAE points\.

## Appendix EFine\-Tuning Details

All cells use the pinned held\-out students \(114114CV /208208ML exam\)\. Evaluation follows each exam’s ground truth: the CV exam records score and bonus separately, so agreement is score\-only on both sides; the ML exam folds bonus into the grade, so agreement compares AI score\+\+bonus to the grader total\.

Table 16:Worst MAE drift over the three harsh personas \(*strict*,*rigorous*,*exacting*\): the largest signed difference persona−\-neutral; when all three are negative it is the least negative, which is why Qwen3\-Coder\-3030B\-A33B’s ML entry of−1\.26\-1\.26is its*rigorous*drift while*strict*sits at−2\.26\-2\.26\(marks recipe; full grid in Table[18](https://arxiv.org/html/2609.29333#A5.T18)\)\.†\\dagger: negative drift: the persona accidentally*improved*the over\-grading base \(Section[8\.4](https://arxiv.org/html/2609.29333#S8.SS4)\)\.*lenient*, swept after tuning, is in Table[18](https://arxiv.org/html/2609.29333#A5.T18): the pooled marks adapters drift\+0\.05\+0\.05to\+0\.39\+0\.39except Qwen3\-Coder\-3030B\-A33B \(\+1\.02\+1\.02CV,\+1\.81\+1\.81ML\); bd\+0\.20\+0\.20to\+1\.64\+1\.64\.#### Hyperparameters and evaluation\.

We use LoRA with rank1616,α=32\\alpha=32, dropout0\.050\.05, and all\-linear targets \(attention\-only for the3030B MoE\), in bf16 for two epochs at learning rate2×10−42\\times 10^\{\-4\}with cosine decay and effective batch size1616, held fixed across four data\-parallel A100s; Gemma\-4 trains on a single A100, as its unused parameters per step trip DDP’s reduction check\. All evaluation is served by vLLM, including Gemma\-4’s adapter, which reproduces the HuggingFace\-generation numbers to within0\.10\.1MAE across its four re\-run cells\.

Table 17:Held\-out MAE vs\. the grader average, mean over the five models, both recipes \(the per\-model marks grid is Table[1](https://arxiv.org/html/2609.29333#S8.T1)\)\. The pooled adapter is best or tied in all eight columns\.Table 18:The full persona grid \(MAE, marks recipe; per\-persona bias and the bd grid are in the released workbooks\)\.†\\dagger: the sign\-inconsistency cell of Section[8\.4](https://arxiv.org/html/2609.29333#S8.SS4)—*strict*improves the over\-grading base\.*len\.*is the inflating direction, swept after tuning: the pooled marks adapters drift\+0\.05\+0\.05to\+0\.39\+0\.39except Qwen3\-3030B \(\+1\.02\+1\.02CV,\+1\.81\+1\.81ML, the latter above the5\.195\.19floor\); every pooled lenient drift is positive \(bias\+1\.0\+1\.0to\+4\.4\+4\.4\)\. bd worst drifts: base up to\+18\.5\+18\.5on either exam; pooled≤\+0\.62\\leq\+0\.62under the three harsh wordings, with Llama\-bd\-*strict*at2\.672\.67\(down from4\.084\.08under the CV\-exam\-only adapter\); under*lenient*the bd adapters drift\+0\.20\+0\.20to\+1\.64\+1\.64\(Qwen3\-3030B ML4\.07→5\.714\.07\\to 5\.71, Gemma\-44ML4\.22→5\.514\.22\\to 5\.51\)\.Table 19:Paired third\-grader test \(Eq\. \([1](https://arxiv.org/html/2609.29333#S8.E1)\)\) for the pooled adapters\. Floors:2\.832\.83\(CV\),5\.195\.19\(ML\)\.*better*: the whole95%95\\%CI ofd¯\\overline\{d\}below zero; “==grader”: it contains zero\. Both single\-grader distances are shown:\|AI−G1\|\|\\mathrm\{AI\}\-\\mathrm\{G\}\_\{1\}\|is the larger in all ten ML rows and nine of ten CV rows, by up to0\.220\.22\(CV\) and0\.570\.57\(ML\)\. No paired\-bootstrap difference excludes zero, and one ML student whose G1slot reads0\.00\.0against a real G2mark \(Appendix[H](https://arxiv.org/html/2609.29333#A8)\) alone accounts for0\.210\.21–0\.250\.25of that gap, so we read the asymmetry as the grader\-slot artifact, not a systematic preference\. Three CV upper bounds fall inside±0\.05\\pm 0\.05and are printed to three decimals; all stay below zero under both the normal CI used here and a20002000\-resample percentile bootstrap, though 7B bd, CV is marginal \(Section[10](https://arxiv.org/html/2609.29333#S10)\)\. Recomputed byanalysis/checks/ftpaired\_marginal\_verdicts\.pyandftpaired\_symmetry\_guard\.py\.
#### Q33of the CV exam retains a residual ceiling\.

Fine\-tuning lowers MAE on every graded CV\-exam question, but a residual ceiling remains on Q33, semantic segmentation, the most open\-ended: post\-tuning MAE1\.11\.1–1\.31\.3across models under the marks recipe, against0\.60\.6–1\.01\.0on Q11/Q22\.

#### The catastrophic tail disappears, on both exams\.

Under a neutral prompt the base models mis\-grade up to4343of114114CV\-exam students \(Llama\) and6666of208208ML\-exam students by more than1010marks \(\>16\{\>\}16on the ML exam’s larger scale\); pooled, every model is at00–11\(CV\) and22–44\(ML\) under marks \(00–11and44–66under bd\)\.

#### Errors become human\-like, on both exams\.

We correlate each grader’s per\-student absolute error with the exam’s intrinsic ambiguity,\|G1−G2\|\|\\mathrm\{G\}\_\{1\}\-\\mathrm\{G\}\_\{2\}\|\. On the CV exam the bases show no consistent relationship \(−0\.31\-0\.31to\+0\.37\+0\.37\) and four of five blunder on submissions the two graders*agreed*on; after pooled fine\-tuning the correlation is positive for every model on both exams \(\+0\.35\+0\.35to\+0\.48\+0\.48\), the residual errors concentrating on the genuinely ambiguous submissions\. Part of that correlation is arithmetic: the grader\-average target carries the graders’ own noise, so any low\-bias grader’s residual co\-varies with\|G1−G2\|\|\\mathrm\{G\}\_\{1\}\-\\mathrm\{G\}\_\{2\}\|\(the coupling Appendix[G](https://arxiv.org/html/2609.29333#A7)notes for pairs\) and we do not subtract it; the base\-to\-tuned change establishes the loss of blunders on submissions the graders agreed on, not human\-like judgement\.

## Appendix FPer\-Question Error Decomposition

Table[20](https://arxiv.org/html/2609.29333#A6.T20)decomposes the mean signed error per question for the headline runs of both exams; both sides are sums of per\-question marks, so the decomposition is exact, reproducing each run’s published bias \(Tables[10](https://arxiv.org/html/2609.29333#A3.T10),[24](https://arxiv.org/html/2609.29333#A9.T24)and[25](https://arxiv.org/html/2609.29333#A11.T25)\) to within4×10−154\\times 10^\{\-15\}\.

Across the4444runs decomposed — the ten in the table plus the1717matched neutral/strict open\-weights pairs of Table[3](https://arxiv.org/html/2609.29333#A2.T3)— the largest cancellation is L\-A1414\(Qwen2\.5\-Coder\-1414B, neutral\): per\-question biases of\+0\.89\+0\.89,−1\.30\-1\.30and−0\.75\-0\.75sum to−1\.16\-1\.16, hiding1\.781\.78of2\.942\.94points of directional error \(61%61\\%\); the largest in the table is A0101, at1\.251\.25of3\.033\.03\. Cancellation is confined to the near\-unbiased runs: all1717matched*strict*runs and all four ML\-exam Gemini runs haveC=0C=0, so no persona\-collapse claim rests on a cancelled total\.

Table 20:Per\-question mean signed error for the headline runs of both exams, over exactly the students each run’s published metric uses\.bqb\_\{q\}is the mean ofAIq−G¯q\\mathrm\{AI\}\_\{q\}\-\\bar\{\\mathrm\{G\}\}\_\{q\}on questionqq\(CV exam:3535\-point base scale, Q44bonus excluded; ML exam: score plus bonus,≈65\\approx 65\-point scale\)\.*Total*is the run’s published bias\.C=∑q\|bq\|−\|∑qbq\|C=\\sum\_\{q\}\|b\_\{q\}\|\-\|\\sum\_\{q\}b\_\{q\}\|is the cancellation the total hides, zero exactly when every question is pushed the same way\. At two decimals a row can miss∑qbq=\\sum\_\{q\}b\_\{q\}=\{\}*Total*by0\.010\.01through rounding\. Generated byanalysis/checks/per\_question\_signed\_error\.py\.RunConfigurationnnb1b\_\{1\}b2b\_\{2\}b3b\_\{3\}Total∑q\|bq\|\\sum\_\{q\}\|b\_\{q\}\|CC*CV exam —3535\-point base scale \(Q111212, Q221111, Q331212\)*D013\-flash\-preview, neutral570−0\.15\-0\.15\+0\.08\+0\.08\+0\.08\+0\.08\+0\.01\+0\.010\.300\.29D023\.1\-pro\-preview, neutral570−0\.44\-0\.44−0\.07\-0\.07−0\.31\-0\.31−0\.82\-0\.820\.820\.00F023\.1\-pro\-preview\+\+lenient570−0\.01\-0\.01\+0\.28\+0\.28\+0\.55\+0\.55\+0\.82\+0\.820\.840\.02B03flash\-lite\+\+thinking570−0\.50\-0\.50\+0\.04\+0\.04\+0\.30\+0\.30−0\.15\-0\.150\.850\.69A01flash\-lite, neutral \(baseline\)570−0\.28\-0\.28−2\.12\-2\.12\+0\.63\+0\.63−1\.78\-1\.783\.031\.25C01flash\-lite\+\+strict570−1\.80\-1\.80−3\.08\-3\.08−0\.55\-0\.55−5\.44\-5\.445\.440\.00*ML exam —≈65\\approx 65\-point scale \(Q112626, Q221717, Q332222\)*IG083\.1\-pro\-preview, neutral1038\+0\.43\+0\.43\+0\.20\+0\.20\+0\.55\+0\.55\+1\.18\+1\.181\.180\.00IG073\-flash\-preview, neutral1038\+1\.55\+1\.55\+0\.10\+0\.10\+0\.59\+0\.59\+2\.24\+2\.242\.240\.00IG01flash\-lite, neutral1013\+4\.00\+4\.00\+0\.79\+0\.79\+1\.19\+1\.19\+5\.98\+5\.985\.980\.00IG05flash\-lite\+\+strict1008−1\.97\-1\.97−2\.82\-2\.82−3\.04\-3\.04−7\.83\-7\.837\.830\.00
## Appendix GGround\-Truth Structure

Two checks probe the grader\-average target, with D0101as the AI side and, on the ML exam, its best full\-cohort run \(IG0808,gemini\-3\.1\-pro\-preview\)\.

#### Whether the AI inherits per\-pair human noise is exam\-specific\.

If the AI inherited its target’s noise structure, its per\-pair error should track the human MAE, which spans0\.850\.85to3\.553\.55\. It does not \(r=−0\.301r=\-0\.301,p=0\.40p=0\.40; Table[21](https://arxiv.org/html/2609.29333#A7.T21), Figure[4](https://arxiv.org/html/2609.29333#A7.F4)\)\. Agreement with any one grader’s judgements is a different metric, not measurable here\.

*And it does not generalise\.*On the ML exam’s1818stable pairs the test inverts —r=\+0\.571r=\+0\.571\(permutationp=0\.014p=0\.014\), a difference itself significant \(Fisherz=2\.10z=2\.10,p=0\.036p=0\.036\), and the sign holds for every AI side we tried \(\+0\.57\+0\.57to\+0\.72\+0\.72\)\. Some coupling must be arithmetic, since a noisier pair injects its variance into the target, but the correction is not clean enough to rest on; we report the divergence, not a mechanism; the ML exam’s contiguous\-block grading assignment is one candidate \(Appendix[H](https://arxiv.org/html/2609.29333#A8)\)\.

#### Per\-grader harshness is real, modest, and best measured within pairs\.

The2020graders span a6\.556\.55\-point range in mean awarded total\. A one\-way ANOVA over the1,1401\{,\}140grades puts that atη2=0\.051\\eta^\{2\}=0\.051\(p=5×10−6p=5\\times 10^\{\-6\}\), but those grades are not independent: each submission is marked by both members of one fixed pair, and each pair marks its own batch of5555–6060students, confounding between\-pair harshness with batch ability \(Appendix[H](https://arxiv.org/html/2609.29333#A8)\)\. Theη2\\eta^\{2\}splits orthogonally into0\.0340\.034between pairs and0\.0170\.017within pairs, the only part identified from the*same*papers\. A linear mixed modeltotal∼grader\\text\{total\}\\sim\\text\{grader\}with a random intercept per student \(student variance48\.6948\.69, residual5\.735\.73,ICC=0\.89\\mathrm\{ICC\}=0\.89\) gives the ten within\-pair contrasts standard errors of0\.440\.44–0\.460\.46against1\.361\.36–1\.401\.40for the between\-pair ones, and the likelihood\-ratio tests separate them: collapsing graders to pair means costs little \(χ2​\(9\)=21\.1\\chi^\{2\}\(9\)=21\.1,p=0\.012p=0\.012\), dropping the within\-pair contrasts costs a great deal \(χ2​\(10\)=168\.9\\chi^\{2\}\(10\)=168\.9,p=5×10−31p=5\\times 10^\{\-31\}; blocked\-ANOVA partialη2=0\.26\\eta^\{2\}=0\.26of the paired\-difference variance\)\. Within pairs the mean signed difference runs−2\.75\-2\.75to\+3\.11\+3\.11points, six of ten pairs are non\-zero after Holm correction, and a sign\-flip omnibus over the570570paired differences givesp<10−4p<10^\{\-4\}\. A student’s expected score does depend on which pair graded them, and the AI’s MAE incorporates that noise in its target; but only the within\-pair third of the5%5\\%is measurable as harshness rather than as batch ability\. Thatη2\\eta^\{2\}is design\-dependent and does not transfer; the quantity that does — signed bias between two graders on the*same*papers — agrees across exams at0\.61×0\.61\\timesthe floor \(Appendix[H](https://arxiv.org/html/2609.29333#A8)\)\.

Together the two results say the best closed\-model grader acts as aMAE≈1\.6\\text\{MAE\}\\approx 1\.6stabiliser around a noisy human consensus on the CV exam, sitting below all but the tightest pair’s disagreement bar\.

The largest single\-paper disagreement is24\.124\.1points\. The one\-way tests use all1,1401\{,\}140grades \(570570students×\\timestwo graders\), and the non\-parametric Kruskal–Wallis test agrees with the ANOVA \(H=53\.80H=53\.80,p=3\.5×10−5p=3\.5\\times 10^\{\-5\}\); both share the independence assumption that the paired analysis above replaces\. Neither that analysis nor the one\-way tests turn on the one negative total \(aTA1\\mathrm\{TA\}\_\{1\}total of−1\.8\-1\.8\): removing it givesH=54\.34H=54\.34, a within\-pairη2\\eta^\{2\}share of0\.0170\.017, andχ2​\(10\)=171\.6\\chi^\{2\}\(10\)=171\.6\.

Table 21:Per\-pair human disagreement vs\. D0101’s per\-pair AI MAE, sorted by human MAE\.Figure 4:Per\-pair human disagreement \(x\-axis\) against D0101’s per\-pair MAE \(y\-axis\), one point per grader pair\. The two are uncorrelated \(Pearsonr=−0\.30r=\-0\.30,p=0\.40p=0\.40\): the AI grader sits roughly equidistant from all pairs rather than inheriting pair\-level noise\.

## Appendix HML\-Exam Replication Details

The ML exam of Section[5\.4](https://arxiv.org/html/2609.29333#S5.SS4)is an introductory AI practical from another course at the same institution:1,0381\{,\}038students after excluding2424whose uploads contained no gradable notebook, dual\-graded by4949graders, three questions \(regression, PyTorch, classification\) worth23\+323\{\+\}3,14\+314\{\+\}3and19\+319\{\+\}3marks including bonuses \(5656base points plus up to99bonus points,6565in all\)\. Question maxima come from the marking scheme’s task tables, not its header totals, which contradict them in all three questions; two instructor corrections issued by email mid\-grading are folded in, both affecting Q33\. Per\-question grader marks fold bonus into the score, so the comparable AI total is score plus bonus\. A student who did not submit a question is scored00by graders and models alike\.

The grading instrument is identical to Section[3](https://arxiv.org/html/2609.29333#S3)\(same persona strings, JSON schema, output rules and grading scale, reference solution, rubric breakdown, no chain\-of\-thought, temperature00, one sample per question\), except that this exam has no student\-facing guidelines document, so thegdcomponent is fixed at00throughout\.

#### The grid\.

The replication comprises162162configurations —3030closed\-model \(2323Gemini IG\-,66OpenAI IO\-,11Anthropic IN\-\) and132132open\-weights \(IA\-\) — replaying the CV exam’s programme: neutral and matched*strict*runs for all1717open\-weights models, five\-persona sweeps on1313, the mechanism probes of Section[5\.2](https://arxiv.org/html/2609.29333#S5.SS2), component removals, few\-shot demonstrations on one closed and one open model, temperature\-variance probes, and a closed arm covering all four Gemini models,gpt\-5\.5,gpt\-5\.4andclaude\-opus\-5\. Table[25](https://arxiv.org/html/2609.29333#A11.T25)\(Appendix[K](https://arxiv.org/html/2609.29333#A11)\) lists every run\.

Table 22:ML exam,1,0381\{,\}038dual\-graded students: matched neutral/strict pairs for all1717open\-weights models, sorted by damage ratio\. MAE against the grader average on a≈65\\approx 65\-point scale,95%95\\%bootstrap CI on the strict run, strict MAE as a multiple of the ML exam’s human floor \(5\.135\.13, defined as in Section[4](https://arxiv.org/html/2609.29333#S4)\)\. Behaviour classes are those of Section[5\.1](https://arxiv.org/html/2609.29333#S5.SS1)and describe the strict run, at this exam’s floor\-matched bandMAE≥15\.7\\text\{MAE\}\\geq 15\.7\(3\.07×3\.07\\timesits5\.135\.13floor, the multiple the CV exam’s88fixes\); four strict runs and one neutral baseline cross it\. A fixedMAE≥8\\text\{MAE\}\\geq 8would be only1\.56×1\.56\\timesthis floor and would label ten strict runs and eleven neutral baselines collapse\. A refusal’s MAE is a ceiling artifact, not a severity measurement\. Ratios below11are real improvements, examined below\. Generated byanalysis/introduction\_to\_ai\_make\_paper\_tables\.py\.Figure 5:Every strict\-flavoured persona run on the ML exam’s full cohort \(5050runs:*strict*/*rigorous*/*exacting*at the default prompt configuration, including IG1313, which additionally enables thinking\), coloured by family and by wording \(*strict*darker;*rigorous*/*exacting*lighter\)\. The dashed line is the ML exam’s human inter\-grader floor \(MAE=5\.13\\text\{MAE\}=5\.13\)\. The refusal \(Llama\-3\.1\-8B\+\+*strict*\) is hatched: its bar height is the distance to the grader mean, not a severity measurement\. The ML exam’s counterpart to Figure[2](https://arxiv.org/html/2609.29333#A2.F2)\.
#### When severity helps, it is calibration, not robustness\.

Fifteen of the1717neutral baselines over\-mark, with biases from\+1\.51\+1\.51to\+14\.89\+14\.89\(the exceptions are Qwen2\.5\-Coder\-3232B at−0\.21\-0\.21and GLM\-4\.5\-Air at−1\.34\-1\.34\), and across the1616non\-refusal pairs a model’s neutral bias predicts the strict effect atr=−0\.73r=\-0\.73\(Spearman−0\.73\-0\.73, permutationp=0\.002p=0\.002;r=−0\.50r=\-0\.50with the refusal’s ceiling\-artifact MAE included\), whereas the CV exam’s neutral biases sit at or below zero for all but two models and zero of1717improve\. The seven improvements in Table[22](https://arxiv.org/html/2609.29333#A8.T22)are heavy over\-markers pushed down — Qwen3\-Coder\-3030B\-A33B \(bias\+10\.44\+10\.44\), Qwen2\.5\-7272B \(\+10\.31\+10\.31\), Qwen3\-Coder\-480480B \(\+9\.52\+9\.52\) — two miscalibrations cancelling: the480480B is one such case \(MAE=10\.10\\text\{MAE\}=10\.10at neutral, twice its floor\), whereas Qwen3\-235235B\-A2222B is genuinely indifferent to the sentence on both exams \(here bias\+3\.73\+3\.73, strict effect−0\.22\-0\.22\)\.

#### Mechanism\.

The probe of Section[5\.3](https://arxiv.org/html/2609.29333#S5.SS3)separates the same two shapes on the ML exam \(Table[23](https://arxiv.org/html/2609.29333#A8.T23)\)\. It is read only for runs that left the graded band and zeroed at least5%5\\%of Q11, the volume gate the CV exam’s classifier applies at10%10\\%; two runs fall below it, DeepSeek\-Coder\-V2\-Lite \(66of10381038,0\.6%0\.6\\%\) and Qwen3\-Coder\-Next8080B \(3939of10011001,3\.9%3\.9\\%\)\.

Table 23:ML exam,*strict*runs: blanket versus selective zeroing, all1717models\. Q1=01=0is the count of students awarded zero on Q11out of those the run graded; the next column is how many of*those*still received a non\-zero Q22\. A run is read only if it left the graded band*and*zeroed Q11on at least5%5\\%of the students it graded \(the CV exam applies the same volume gate at10%10\\%,MECH\_ZERO\_RATE; on these runs any cut between4\.8%4\.8\\%and13\.3%13\.3\\%gives the same partition\): below that, the zeros are ordinary non\-attempts and the share is noise on a handful of students\. Generated byanalysis/introduction\_to\_ai\_make\_paper\_tables\.py\.
#### The attribution series replicates\.

The minimal pairs of Section[5\.2](https://arxiv.org/html/2609.29333#S5.SS2)reproduce both findings\. S2 remains the only single component that stops graders grading: alone it takes Llama\-3\.1\-88B to29\.3929\.39\(98%98\\%of students zeroed\) and Mistral\-Small\-2424B to26\.0326\.03\(76%76\\%zeroed\), while the HARSH frame alone moves each of the three models probed by at most2\.42\.4points from neutral, twice in the*improving*direction\. Nor does the adjective carry the damage: with both policy sentences held verbatim, swapping STRICT for RIGOROUS or FAIR moves MAE by0\.550\.55–3\.863\.86across the five models with no consistent direction — a FAIR teaching assistant carrying the harsh policy still more than doubles Qwen2\.5\-Coder\-3232B’s error \(10\.4010\.40against4\.574\.57\) — while the*nharsh*control \(a harsh adjective with no policy\) lands within1\.21\.2points of neutral on all five\. Absolute damage is exam\-dependent — S2 alone leaves Qwen2\.5\-Coder\-3232B in the graded band here \(6\.686\.68, against12\.9212\.92on the CV exam\) — but which sentence dominates a model is not: S1 outweighs S2 for the3232B and S2 for the other two, on both exams\.

#### The closed\-model arm\.

The ML exam also carries all four Gemini models under the same instrument\. The parity result replicates on the closed side — all three full\-cohort neutral configurations sit wholly below the floor’s CI:gemini\-3\.1\-pro\-previewat3\.403\.40\[3\.16,3\.65\]\[3\.16,3\.65\]\(IG0808; Figure[6](https://arxiv.org/html/2609.29333#A8.F6)\),gemini\-2\.5\-proat3\.473\.47, andgemini\-3\-flash\-previewat3\.923\.92, against a floor of5\.135\.13\[4\.80,5\.48\]\[4\.80,5\.48\]; the3\.13\.1\-pro under*lenient*\(4\.484\.48\[4\.22,4\.75\]\[4\.22,4\.75\]\) and under*strict*\(3\.673\.67\[3\.43,3\.94\]\[3\.43,3\.94\], bias−1\.00\-1\.00against\+1\.18\+1\.18at neutral; IG2323\) join them\. The cross\-vendor runs of Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1)agree: no other closed run leaves this exam’s band \(Table[9](https://arxiv.org/html/2609.29333#A3.T9)\)\. What does*not*replicate is Gemini’s persona immunity at the cheaper tier\. Flash\-Lite, at7\.537\.53under neutral \(bias\+5\.98\+5\.98\), moves in*both*directions:*strict*takes it to9\.029\.02\(bias−7\.83\-7\.83\) and*lenient*to17\.8617\.86\(bias\+17\.80\+17\.80\), the one closed run past this exam’s band, which only four open\-weights strict runs cross — while*rigorous*and*exacting*improve it \(5\.525\.52,5\.965\.96\), the same calibration correction the open side shows\. The strongergemini\-3\-flash\-previewfollows the same law on its first\-100100subset: against a matched\-subset baseline of3\.873\.87\(bias\+3\.04\+3\.04\),*strict*lands at2\.872\.87\(bias\+0\.09\+0\.09\) and*lenient*at6\.996\.99\(bias\+6\.60\+6\.60\)\. Component ablations echo the CV exam with larger magnitudes: dropping the reference solution costs Flash\-Lite4\.934\.93points \(7\.53→12\.467\.53\\to 12\.46\) against0\.790\.79there, and thinking helps by2\.162\.16\(7\.53→5\.377\.53\\to 5\.37\) against1\.291\.29\.

Figure 6:IG0808\(gemini\-3\.1\-pro\-preview, neutral,t=0t=0\): AI total vs\. grader\-average total over the ML exam’s fulln=1,038n=1\{,\}038cohort, withy=xy=xas the perfect\-agreement reference\. The best full\-cohort configuration on the ML exam \(0\.66×0\.66\\timesits human floor\), the counterpart to Figure[3](https://arxiv.org/html/2609.29333#A3.F3)\.
#### Few\-shot demonstrations transfer, with one measurement caveat\.

Two worked examples per question \(from the two highest\-agreement grader pairs, labelled by IG0808; the three students remain in the evaluated cohort,3/1,0383/1\{,\}038\) nearly halve the open model’s error: Qwen3\-Coder\-3030B\-A33B goes from11\.1211\.12to5\.845\.84, bias\+10\.44\+10\.44to\+4\.56\+4\.56, on the full cohort \(IA201201\)\. On Flash\-Lite \(IG1818\) they lengthen the prompt enough that334334students’ calls fail permanently, with biased dropout: the dropped students average32\.7832\.78points against the survivors’28\.0228\.02\(Welchp=3×10−7p=3\\times 10^\{\-7\}\)\. On the matched679679\-student subsample the effect holds \(7\.87→4\.327\.87\\to 4\.32MAE, bias\+6\.52→\+0\.37\+6\.52\\to\+0\.37\), but its headline numbers are not comparable to full\-cohort rows and Table[25](https://arxiv.org/html/2609.29333#A11.T25)flags itsnn\.

#### Determinism att=0t=0does not transfer\.

On the CV exam Flash\-Lite reproduced100%100\\%of \(student, question\) cells exactly across fivet=0t=0reruns \(Appendix[C](https://arxiv.org/html/2609.29333#A3.SS0.SSS0.Px2)\)\. Here the same probe \(IG1010\) reproduces only59\.3%59\.3\\%; the median per\-cell standard deviation is still0\.000\.00but a40\.7%40\.7\\%tail varies, and att=0\.5t=0\.5/0\.70\.7the medians rise to0\.980\.98/1\.141\.14\(IG1111/IG1212\) against0\.650\.65/0\.760\.76there\. Two open\-weights probes on4141students att=0\.5t=0\.5\(IA9292, Qwen2\.5\-Coder\-3232B; IA136136, Qwen3\-Coder\-Next\) show medians of0\.650\.65and0\.840\.84\.

#### Component removals do not transfer either\.

The CV exam’s over\-anchoring result \(7\.49→3\.127\.49\\to 3\.12on Qwen2\.5\-Coder\-3232B with the reference solution removed\) does not reproduce: the same removal moves it4\.57→4\.734\.57\\to 4\.73here\. The rubric\-breakdown test of Appendix[B](https://arxiv.org/html/2609.29333#A2)softens: under*strict*, removing the breakdown worsens the collapse on GLM\-4\-3232B \(\+6\.26\+6\.26\) and Llama\-3\.3\-7070B \(\+3\.96\+3\.96\) but is flat on Qwen2\.5\-Coder\-3232B \(\+0\.19\+0\.19\), Gemma\-3\-2727B \(−0\.26\-0\.26\) and DeepSeek\-Coder\-V2\-Lite \(−0\.10\-0\.10\), neutral controls within±0\.52\\pm 0\.52— against five\-of\-five worsening there\.

#### Ground\-truth structure of the ML exam\.

Its floor is5\.13/655\.13/65\(95%95\\%bootstrap CI\[4\.80,5\.48\]\[4\.80,5\.48\]at50,00050,000resamples — the lower bound is not stable to two decimals at2,0002,000; Pearsonr=0\.871r=0\.871\)\. Nine of the1,0381\{,\}038rows record0\.00\.0for exactly one grader while the partner awarded real marks: ungraded slots stored as zeros\. They move the floor by0\.050\.05\(inside the CI, so the headline retains them\) but dominate the extremes, the maximum single\-paper disagreement being41\.041\.0without them and51\.051\.0with\. Excluding them everywhere changes no conclusion: the floor falls to5\.095\.09\[4\.76,5\.42\]\[4\.76,5\.42\], the best full\-cohort configurations move by at most0\.060\.06MAE \(IG08083\.40→3\.353\.40\\to 3\.35, IG09093\.47→3\.423\.47\\to 3\.42, IG07073\.92→3\.863\.92\\to 3\.86, Qwen3\-Coder\-Next neutral4\.47→4\.444\.47\\to 4\.44\), no run of Table[22](https://arxiv.org/html/2609.29333#A8.T22)or Table[25](https://arxiv.org/html/2609.29333#A11.T25)changes behaviour class or damage\-ratio order \(largest move0\.210\.21MAE, the Llama\-3\.1\-88B refusal\), and on the208208held\-out students \(three of the nine; the other six sit in the830830\-student fine\-tuning split\) the pooled adapters grade0\.080\.08–0\.110\.11MAE better, every Table[19](https://arxiv.org/html/2609.29333#A5.T19)verdict intact \(analysis/checks/ml\_unrecorded\_grader\_rows\.py\)\. A further nine rows carry0\.00\.0from both graders and are genuine non\-submissions, retained\.

The pairing is hybrid, unlike the CV exam’s1010fixed pairs:4949assistants work in4848grading slots \(one slot is a pair who marked jointly\) forming5151pairings, of which1818are stable pairs of3838–5151students covering816816of1,0381\{,\}038\(79%79\\%\); the rest are ad hoc combinations around floating graders, and two of the5151on the roster returned no marks\. Every per\-pair figure is scoped to that backbone, across which human MAE runs1\.901\.90to10\.3610\.36— a5\.4×5\.4\\timesspread, against4\.2×4\.2\\timeson the CV exam — with eight of1818pairs above the pooled floor, against five of ten\. Mean within\-pair bias magnitude is3\.143\.14\(0\.61×0\.61\\timesfloor\) against1\.581\.58\(0\.61×0\.61\\timesfloor\): systematic harshness is the same fraction of the noise floor on both exams, whereas a one\-wayη2\\eta^\{2\}over grader identity is not comparable here, because each pair marks a contiguous block of students of widely varying ability\.

The per\-pair test of Appendix[G](https://arxiv.org/html/2609.29333#A7.SS0.SSS0.Px1)inverts here: per\-pair human MAE against IG0808’s per\-pair MAE gives Pearsonr=\+0\.571r=\+0\.571\(permutationp=0\.014p=0\.014,20,00020,000shuffles; Spearman\+0\.447\+0\.447; Figure[7](https://arxiv.org/html/2609.29333#A8.F7)\) against−0\.301\-0\.301on the CV exam, a difference significant at Fisherz=2\.10z=2\.10,p=0\.036p=0\.036, and the sign survives changing the AI side \(Qwen3\-Coder\-Next neutral\+0\.565\+0\.565, Qwen2\.5\-Coder\-3232B\+0\.723\+0\.723\)\. Some of the coupling is arithmetic, a noisier pair injecting its variance into the target\. One candidate for the rest is the assignment itself: ML pairs mark contiguous blocks of students whose ability varies widely, so block difficulty raises both human disagreement and model error within a pair, whereas the CV exam’s fixed pairs each drew a comparable slice\. The AI is still the more uniform grader, its per\-pair MAE spanning3\.6×3\.6\\times\.

Figure 7:ML exam: per\-pair human disagreement \(x\-axis\) against IG0808’s per\-pair MAE \(y\-axis\) over the1818\-pair stable backbone\. Unlike the CV exam \(Figure[4](https://arxiv.org/html/2609.29333#A7.F4)\), the two correlate \(r=\+0\.571r=\+0\.571\): here the AI’s error is largest exactly where the humans disagree most\.
#### What the replication does and does not establish\.

*Strict*worsens ten of1717models and takes four above this exam’s floor\-matched band \(MAE≥15\.7\\text\{MAE\}\\geq 15\.7\), none of them there at neutral\. What transfers is the floor\-relative attainability of parity \(IG0808at0\.66×0\.66\\timesthis exam’s floor; the best open\-weights baseline, Qwen3\-Coder\-Next,4\.474\.47\[4\.24,4\.72\]\[4\.24,4\.72\], also wholly below its CI\), the S2 mechanism, the blanket/selective split and the danger of an untested persona sentence; model choice, persona direction, component effects, failure shape \(Qwen2\.5\-Coder\-3232B inverts from selective field collapse to blanket zeroing\) andt=0t=0determinism are exam\-level properties\.

## Appendix IAll Grader Configurations

Table[24](https://arxiv.org/html/2609.29333#A9.T24)lists every one of the171171grader configurations with its headline metrics, one row per run\. The machine\-readable version, with per\-question breakdowns and the full column set, ships asanalysis/computer\_vision\_master\_comparison\.csvin the accompanying repository\.

Table 24:Every grader configuration in the study, one row per run: the3333closed\-model configurations \(2525Gemini via Vertex,66OpenAI and22Anthropic via their batch APIs; Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1)\) followed by the138138open\-weights \(L\-\) configurations, each sorted by run id\.*Prompt*lists the components present \(S reference solution, G grading guidelines, B rubric breakdown, R thinking enabled, F few\-shot demonstrations\); personas prefixedmpare the mechanism probes of Section[5\.2](https://arxiv.org/html/2609.29333#S5.SS2):mpstrict,mprigorousandmpfairkeep the*strict*preset’s two policy sentences and swap only its headword;mpframeis the HARSH frame alone,mpnoclausethe frame with S1,mps2onlythe frame with S2,mpharshthe frame with both \(the*strict*text, rerun on the current stack\);mpnharshis the neutral frame with “harsh” in place of “strict”\. L\-MP0404and L\-PS0303are one physical run \(qwen2\.5\-coder\-32b,*mpnoclause*\), listed under both the headword series and the policy2×22\\times 2\.nnis the number of students scored against the grader average; MAE and bias are on the3535\-point base scale, the95%95\\%bootstrap CI on MAE; behaviour classes are defined in Section[5\.1](https://arxiv.org/html/2609.29333#S5.SS1)\. A refusal’s MAE is a ceiling artifact, not a severity measurement\. D0303ran with thinking active regardless of its flag \(gemini\-2\.5\-procannot disable it; Appendix[A](https://arxiv.org/html/2609.29333#A1)\)\.claude\-opus\-5exposes no sampling temperature \(thinking disabled; shown as —\)\. The final block lists the88prompt\-only baselines of Appendix[B](https://arxiv.org/html/2609.29333#A2.SS0.SSS0.Px3)\(first100100students; the persona column names the strategy\), which are not configurations of the paper’s grader and are excluded from every count; their CIs use20002000resamples\. Model keys match the released spreadsheets\. Generated byanalysis/computer\_vision\_make\_appendix\_table\.py\.RunModelPersonaPromptttnnMAE95%95\\%CIBiasBehaviour*Closed\-model configurations \(Gemini: A–P series; OpenAI: O; Anthropic: N\)*A01flash\-liteneutralSGB005705703\.343\.34\[3\.15,3\.53\]\[3\.15,\\ 3\.53\]−1\.78\-1\.78gradedB01flash\-liteneutralGB005675674\.134\.13\[3\.83,4\.42\]\[3\.83,\\ 4\.42\]\+3\.30\+3\.30gradedB02flash\-liteneutralSB005695693\.343\.34\[3\.16,3\.52\]\[3\.16,\\ 3\.52\]−1\.42\-1\.42gradedB03flash\-liteneutralSGBR005705702\.052\.05\[1\.89,2\.21\]\[1\.89,\\ 2\.21\]−0\.15\-0\.15gradedB04flash\-liteneutralSG005695693\.223\.22\[3\.03,3\.43\]\[3\.03,\\ 3\.43\]−1\.45\-1\.45gradedC01flash\-litestrictSGB005705705\.755\.75\[5\.46,6\.04\]\[5\.46,\\ 6\.04\]−5\.44\-5\.44gradedC02flash\-litelenientSGB005695694\.494\.49\[4\.18,4\.82\]\[4\.18,\\ 4\.82\]\+4\.01\+4\.01gradedD013\-flash\-previewneutralSGB005705701\.641\.64\[1\.51,1\.79\]\[1\.51,\\ 1\.79\]\+0\.01\+0\.01gradedD023\.1\-pro\-previewneutralSGB005705701\.861\.86\[1\.70,2\.02\]\[1\.70,\\ 2\.02\]−0\.82\-0\.82gradedD032\.5\-proneutralSGB005705704\.104\.10\[3\.83,4\.38\]\[3\.83,\\ 4\.38\]−3\.85\-3\.85gradedE01flash\-liteneutralSGB005495493\.383\.38\[3\.19,3\.59\]\[3\.19,\\ 3\.59\]−1\.85\-1\.85gradedE02flash\-liteneutralSGB0\.50\.55635633\.513\.51\[3\.31,3\.71\]\[3\.31,\\ 3\.71\]−2\.20\-2\.20gradedE03flash\-liteneutralSGB0\.70\.75635633\.603\.60\[3\.40,3\.79\]\[3\.40,\\ 3\.79\]−2\.32\-2\.32gradedF01flash\-litestrictSGBR005705703\.733\.73\[3\.50,3\.98\]\[3\.50,\\ 3\.98\]−3\.28\-3\.28gradedF023\.1\-pro\-previewlenientSGB005705701\.791\.79\[1\.62,1\.95\]\[1\.62,\\ 1\.95\]\+0\.82\+0\.82gradedF033\.1\-pro\-previewstrictSGB005705702\.752\.75\[2\.51,2\.99\]\[2\.51,\\ 2\.99\]−2\.09\-2\.09gradedG013\-flash\-previewneutralGB0098981\.631\.63\[1\.32,2\.02\]\[1\.32,\\ 2\.02\]\+0\.71\+0\.71gradedG023\-flash\-previewneutralSB0099991\.601\.60\[1\.30,1\.93\]\[1\.30,\\ 1\.93\]\+0\.79\+0\.79gradedG033\-flash\-previewneutralSGBR001001001\.391\.39\[1\.18,1\.61\]\[1\.18,\\ 1\.61\]−0\.12\-0\.12gradedG043\-flash\-previewneutralSG001001001\.251\.25\[1\.01,1\.51\]\[1\.01,\\ 1\.51\]−0\.07\-0\.07gradedK01flash\-liteneutralSGBF005675673\.033\.03\[2\.83,3\.24\]\[2\.83,\\ 3\.24\]\+1\.09\+1\.09gradedM013\-flash\-previewstrictSGB0099994\.234\.23\[3\.60,4\.88\]\[3\.60,\\ 4\.88\]−4\.16\-4\.16gradedM023\-flash\-previewlenientSGB001001002\.182\.18\[1\.75,2\.65\]\[1\.75,\\ 2\.65\]\+1\.90\+1\.90gradedN01claude\-opus\-5neutralSGB—5705703\.543\.54\[3\.31,3\.77\]\[3\.31,\\ 3\.77\]−3\.24\-3\.24gradedN02claude\-opus\-5strictSGB—5705705\.175\.17\[4\.90,5\.45\]\[4\.90,\\ 5\.45\]−5\.02\-5\.02gradedO01gpt\-5\.5neutralSGB005705702\.432\.43\[2\.28,2\.59\]\[2\.28,\\ 2\.59\]−1\.65\-1\.65gradedO02gpt\-5\.5strictSGB005705704\.434\.43\[4\.19,4\.68\]\[4\.19,\\ 4\.68\]−4\.25\-4\.25gradedO03gpt\-5\.5lenientSGB005705702\.252\.25\[2\.08,2\.44\]\[2\.08,\\ 2\.44\]\+1\.13\+1\.13gradedO11gpt\-5\.4neutralSGB005705704\.694\.69\[4\.46,4\.92\]\[4\.46,\\ 4\.92\]−4\.45\-4\.45gradedO12gpt\-5\.4strictSGB005705706\.906\.90\[6\.63,7\.15\]\[6\.63,\\ 7\.15\]−6\.81\-6\.81gradedO13gpt\-5\.4lenientSGB005705702\.692\.69\[2\.51,2\.89\]\[2\.51,\\ 2\.89\]\+1\.00\+1\.00gradedP01flash\-literigorousSGB005705703\.923\.92\[3\.70,4\.16\]\[3\.70,\\ 4\.16\]−3\.01\-3\.01gradedP02flash\-liteexactingSGB005575574\.584\.58\[4\.35,4\.85\]\[4\.35,\\ 4\.85\]−4\.05\-4\.05graded*Open\-weights \(L\-\) configurations*L\-A07qwen2\.5\-coder\-7bneutralSGB005705705\.685\.68\[5\.39,5\.98\]\[5\.39,\\ 5\.98\]−4\.84\-4\.84gradedL\-A14qwen2\.5\-coder\-14bneutralSGB005705703\.513\.51\[3\.30,3\.74\]\[3\.30,\\ 3\.74\]−1\.16\-1\.16gradedL\-A30Mqwen3\-coder\-30b\-a3bneutralSGB005705704\.504\.50\[4\.29,4\.72\]\[4\.29,\\ 4\.72\]−2\.70\-2\.70gradedL\-A32qwen2\.5\-coder\-32bneutralSGB005705707\.497\.49\[7\.19,7\.78\]\[7\.19,\\ 7\.78\]−7\.35\-7\.35gradedL\-ANxtqwen3\-coder\-nextneutralSGB005705705\.175\.17\[4\.92,5\.42\]\[4\.92,\\ 5\.42\]−4\.74\-4\.74gradedL\-B01qwen2\.5\-coder\-32bneutralGB005705703\.123\.12\[2\.92,3\.31\]\[2\.92,\\ 3\.31\]−1\.56\-1\.56gradedL\-B02qwen2\.5\-coder\-32bneutralSB005705705\.865\.86\[5\.60,6\.11\]\[5\.60,\\ 6\.11\]−5\.46\-5\.46gradedL\-B03qwen2\.5\-coder\-32bneutralSGBR005705707\.517\.51\[7\.22,7\.80\]\[7\.22,\\ 7\.80\]−7\.36\-7\.36gradedL\-B04qwen2\.5\-coder\-32bneutralSG005705707\.277\.27\[6\.98,7\.55\]\[6\.98,\\ 7\.55\]−7\.11\-7\.11gradedL\-BD01qwen2\.5\-coder\-32bneutralSG005705707\.367\.36\[7\.06,7\.64\]\[7\.06,\\ 7\.64\]−7\.23\-7\.23gradedL\-BD02qwen2\.5\-coder\-32bstrictSG0057057021\.6921\.69\[21\.16,22\.23\]\[21\.16,\\ 22\.23\]−21\.68\-21\.68collapseL\-BD03llama\-3\.3\-70bneutralSG005705704\.694\.69\[4\.46,4\.92\]\[4\.46,\\ 4\.92\]−3\.96\-3\.96gradedL\-BD04llama\-3\.3\-70bstrictSG0057057019\.9419\.94\[19\.43,20\.51\]\[19\.43,\\ 20\.51\]−19\.94\-19\.94collapseL\-BD05deepseek\-coder\-v2\-liteneutralSG005705706\.836\.83\[6\.51,7\.17\]\[6\.51,\\ 7\.17\]−5\.03\-5\.03gradedL\-BD06deepseek\-coder\-v2\-litestrictSG0057057016\.0216\.02\[15\.55,16\.49\]\[15\.55,\\ 16\.49\]−15\.99\-15\.99collapseL\-BD07glm\-4\-32bneutralSG005705702\.682\.68\[2\.49,2\.88\]\[2\.49,\\ 2\.88\]\+0\.70\+0\.70gradedL\-BD08glm\-4\-32bstrictSG0057057015\.5815\.58\[14\.99,16\.20\]\[14\.99,\\ 16\.20\]−15\.46\-15\.46collapseL\-BD09gemma\-3\-27bneutralSG005705704\.324\.32\[4\.11,4\.56\]\[4\.11,\\ 4\.56\]−1\.64\-1\.64gradedL\-BD10gemma\-3\-27bstrictSG0056956911\.6911\.69\[11\.31,12\.11\]\[11\.31,\\ 12\.11\]−11\.64\-11\.64collapseL\-C01qwen2\.5\-coder\-32bstrictSGB0057057020\.2920\.29\[19\.83,20\.74\]\[19\.83,\\ 20\.74\]−20\.28\-20\.28collapseL\-C02qwen2\.5\-coder\-32blenientSGB005705703\.953\.95\[3\.75,4\.17\]\[3\.75,\\ 4\.17\]−2\.15\-2\.15gradedL\-D30Mqwen3\-coder\-30b\-a3bneutralSGBR005705704\.494\.49\[4\.28,4\.71\]\[4\.28,\\ 4\.71\]−2\.70\-2\.70gradedL\-DNxtqwen3\-coder\-nextneutralSGBR005705705\.185\.18\[4\.93,5\.43\]\[4\.93,\\ 5\.43\]−4\.75\-4\.75gradedL\-E01qwen2\.5\-coder\-32bneutralSGB0\.50\.550507\.157\.15\[6\.26,8\.04\]\[6\.26,\\ 8\.04\]−7\.02\-7\.02gradedL\-E02qwen3\-coder\-nextneutralSGB0\.50\.548485\.385\.38\[4\.64,6\.16\]\[4\.64,\\ 6\.16\]−5\.20\-5\.20gradedL\-F01gemma\-3\-12bneutralSGB005705705\.205\.20\[4\.94,5\.48\]\[4\.94,\\ 5\.48\]−2\.75\-2\.75gradedL\-F02gemma\-3\-12bstrictSGB005705708\.778\.77\[8\.42,9\.15\]\[8\.42,\\ 9\.15\]−8\.43\-8\.43collapseL\-F03gemma\-3\-12blenientSGB005705704\.794\.79\[4\.54,5\.06\]\[4\.54,\\ 5\.06\]−0\.82\-0\.82gradedL\-F04gemma\-3\-12brigorousSGB005705705\.455\.45\[5\.19,5\.73\]\[5\.19,\\ 5\.73\]−3\.19\-3\.19gradedL\-F05gemma\-3\-12bexactingSGB005705705\.105\.10\[4\.85,5\.38\]\[4\.85,\\ 5\.38\]−2\.76\-2\.76gradedL\-G01qwen3\-coder\-30b\-a3bneutralGB005705703\.603\.60\[3\.37,3\.86\]\[3\.37,\\ 3\.86\]\+1\.29\+1\.29gradedL\-G02qwen3\-coder\-30b\-a3bneutralSB005705703\.633\.63\[3\.42,3\.85\]\[3\.42,\\ 3\.85\]−0\.18\-0\.18gradedL\-G04qwen3\-coder\-30b\-a3bneutralSG005705704\.804\.80\[4\.59,5\.03\]\[4\.59,\\ 5\.03\]−3\.49\-3\.49gradedL\-H01qwen2\.5\-coder\-7bneutralGB005705704\.294\.29\[4\.00,4\.62\]\[4\.00,\\ 4\.62\]\+2\.16\+2\.16gradedL\-H02qwen2\.5\-coder\-7bneutralSB005705705\.255\.25\[4\.99,5\.53\]\[4\.99,\\ 5\.53\]−4\.02\-4\.02gradedL\-H04qwen2\.5\-coder\-7bneutralSG005705705\.835\.83\[5\.56,6\.14\]\[5\.56,\\ 6\.14\]−4\.92\-4\.92gradedL\-J01glm\-4\.5\-airneutralSGB005705705\.665\.66\[5\.38,5\.95\]\[5\.38,\\ 5\.95\]−5\.37\-5\.37gradedL\-J02glm\-4\.5\-airstrictSGB0057057021\.0221\.02\[20\.49,21\.55\]\[20\.49,\\ 21\.55\]−21\.02\-21\.02collapseL\-J03glm\-4\.5\-airlenientSGB005705704\.684\.68\[4\.39,4\.99\]\[4\.39,\\ 4\.99\]\+4\.51\+4\.51gradedL\-J04glm\-4\.5\-airrigorousSGB005685687\.297\.29\[6\.99,7\.64\]\[6\.99,\\ 7\.64\]−7\.21\-7\.21gradedL\-J05glm\-4\.5\-airexactingSGB0057057011\.2911\.29\[10\.92,11\.65\]\[10\.92,\\ 11\.65\]−11\.27\-11\.27collapseL\-K01qwen3\-coder\-30b\-a3bneutralSGBF005705703\.343\.34\[3\.16,3\.53\]\[3\.16,\\ 3\.53\]−1\.77\-1\.77gradedL\-M01qwen3\-coder\-30b\-a3bstrictSGB0057057014\.9114\.91\[14\.52,15\.30\]\[14\.52,\\ 15\.30\]−14\.91\-14\.91collapseL\-M02qwen3\-coder\-30b\-a3blenientSGB005705704\.844\.84\[4\.50,5\.17\]\[4\.50,\\ 5\.17\]\+2\.66\+2\.66gradedL\-M03qwen3\-coder\-nextstrictSGB0057057011\.7011\.70\[11\.35,12\.04\]\[11\.35,\\ 12\.04\]−11\.70\-11\.70collapseL\-M04qwen3\-coder\-nextlenientSGB005705702\.812\.81\[2\.64,2\.98\]\[2\.64,\\ 2\.98\]−0\.41\-0\.41gradedL\-M05qwen2\.5\-coder\-7bstrictSGB005705707\.927\.92\[7\.54,8\.30\]\[7\.54,\\ 8\.30\]−7\.41\-7\.41gradedL\-M06qwen2\.5\-coder\-7blenientSGB005705703\.883\.88\[3\.65,4\.11\]\[3\.65,\\ 4\.11\]−0\.86\-0\.86gradedL\-M07qwen2\.5\-coder\-14bstrictSGB0057057024\.5224\.52\[23\.94,25\.13\]\[23\.94,\\ 25\.13\]−24\.52\-24\.52collapseL\-MP01qwen2\.5\-coder\-32bmpstrictSGB0057057016\.4116\.41\[16\.01,16\.80\]\[16\.01,\\ 16\.80\]−16\.40\-16\.40collapseL\-MP02qwen2\.5\-coder\-32bmprigorousSGB0057057017\.7417\.74\[17\.29,18\.16\]\[17\.29,\\ 18\.16\]−17\.73\-17\.73collapseL\-MP03qwen2\.5\-coder\-32bmpfairSGB0057057018\.5518\.55\[18\.11,19\.00\]\[18\.11,\\ 19\.00\]−18\.54\-18\.54collapseL\-MP04qwen2\.5\-coder\-32bmpnoclauseSGB0057057015\.5515\.55\[15\.16,15\.95\]\[15\.16,\\ 15\.95\]−15\.55\-15\.55collapseL\-MP05qwen2\.5\-coder\-32bmpnharshSGB005705708\.258\.25\[7\.93,8\.55\]\[7\.93,\\ 8\.55\]−8\.13\-8\.13collapseL\-MP06mistral\-small\-24bmpstrictSGB0057057025\.9925\.99\[25\.42,26\.57\]\[25\.42,\\ 26\.57\]−25\.99\-25\.99near\-refusalL\-MP07mistral\-small\-24bmprigorousSGB0057057025\.8725\.87\[25\.29,26\.45\]\[25\.29,\\ 26\.45\]−25\.87\-25\.87near\-refusalL\-MP08mistral\-small\-24bmpfairSGB0057057025\.0525\.05\[24\.48,25\.64\]\[24\.48,\\ 25\.64\]−25\.05\-25\.05near\-refusalL\-MP09mistral\-small\-24bmpnoclauseSGB0057057010\.4010\.40\[10\.06,10\.73\]\[10\.06,\\ 10\.73\]−10\.34\-10\.34collapseL\-MP10mistral\-small\-24bmpnharshSGB005705704\.234\.23\[4\.02,4\.46\]\[4\.02,\\ 4\.46\]−3\.31\-3\.31gradedL\-MP11llama\-3\.3\-70bmpstrictSGB0057057012\.6712\.67\[12\.28,13\.07\]\[12\.28,\\ 13\.07\]−12\.66\-12\.66collapseL\-MP12llama\-3\.3\-70bmprigorousSGB0057057011\.3711\.37\[11\.00,11\.74\]\[11\.00,\\ 11\.74\]−11\.35\-11\.35collapseL\-MP13llama\-3\.3\-70bmpfairSGB0057057011\.4111\.41\[11\.04,11\.78\]\[11\.04,\\ 11\.78\]−11\.39\-11\.39collapseL\-MP14llama\-3\.3\-70bmpnoclauseSGB005705708\.148\.14\[7\.83,8\.43\]\[7\.83,\\ 8\.43\]−8\.03\-8\.03collapseL\-MP15llama\-3\.3\-70bmpnharshSGB005705704\.604\.60\[4\.38,4\.81\]\[4\.38,\\ 4\.81\]−3\.71\-3\.71gradedL\-MP16glm\-4\-32bmpstrictSGB005705707\.027\.02\[6\.58,7\.49\]\[6\.58,\\ 7\.49\]−6\.61\-6\.61gradedL\-MP17glm\-4\-32bmprigorousSGB005705707\.437\.43\[7\.00,7\.86\]\[7\.00,\\ 7\.86\]−7\.12\-7\.12gradedL\-MP18glm\-4\-32bmpfairSGB005705706\.416\.41\[6\.00,6\.82\]\[6\.00,\\ 6\.82\]−5\.85\-5\.85gradedL\-MP19glm\-4\-32bmpnoclauseSGB005705705\.515\.51\[5\.22,5\.83\]\[5\.22,\\ 5\.83\]−5\.10\-5\.10gradedL\-MP20glm\-4\-32bmpnharshSGB005705702\.882\.88\[2\.71,3\.08\]\[2\.71,\\ 3\.08\]−0\.49\-0\.49gradedL\-MP21gemma\-3\-27bmpstrictSGB0057057010\.4910\.49\[10\.12,10\.85\]\[10\.12,\\ 10\.85\]−10\.42\-10\.42collapseL\-MP22gemma\-3\-27bmprigorousSGB005705709\.239\.23\[8\.87,9\.57\]\[8\.87,\\ 9\.57\]−9\.09\-9\.09collapseL\-MP23gemma\-3\-27bmpfairSGB005705708\.608\.60\[8\.27,8\.93\]\[8\.27,\\ 8\.93\]−8\.42\-8\.42collapseL\-MP24gemma\-3\-27bmpnoclauseSGB005705709\.909\.90\[9\.52,10\.26\]\[9\.52,\\ 10\.26\]−9\.73\-9\.73collapseL\-MP25gemma\-3\-27bmpnharshSGB005705704\.564\.56\[4\.34,4\.79\]\[4\.34,\\ 4\.79\]−2\.36\-2\.36gradedL\-N01qwen2\.5\-72bneutralSGB005705703\.143\.14\[2\.95,3\.35\]\[2\.95,\\ 3\.35\]\+0\.19\+0\.19gradedL\-N02qwen2\.5\-72bstrictSGB005705709\.829\.82\[9\.46,10\.18\]\[9\.46,\\ 10\.18\]−9\.78\-9\.78collapseL\-N03qwen2\.5\-72blenientSGB005705704\.474\.47\[4\.17,4\.81\]\[4\.17,\\ 4\.81\]\+3\.69\+3\.69gradedL\-N04qwen2\.5\-72brigorousSGB005705703\.093\.09\[2\.91,3\.28\]\[2\.91,\\ 3\.28\]−0\.29\-0\.29gradedL\-N05qwen2\.5\-72bexactingSGB005705703\.433\.43\[3\.25,3\.62\]\[3\.25,\\ 3\.62\]−1\.57\-1\.57gradedL\-P01qwen2\.5\-coder\-32brigorousSGB005705708\.708\.70\[8\.38,9\.01\]\[8\.38,\\ 9\.01\]−8\.61\-8\.61collapseL\-P02qwen2\.5\-coder\-32bexactingSGB0057057010\.2310\.23\[9\.91,10\.56\]\[9\.91,\\ 10\.56\]−10\.20\-10\.20collapseL\-PS01qwen2\.5\-coder\-32bmpframeSGB005705709\.379\.37\[9\.04,9\.68\]\[9\.04,\\ 9\.68\]−9\.31\-9\.31collapseL\-PS02qwen2\.5\-coder\-32bmps2onlySGB0057057012\.9212\.92\[12\.56,13\.27\]\[12\.56,\\ 13\.27\]−12\.91\-12\.91collapseL\-PS03qwen2\.5\-coder\-32bmpnoclauseSGB0057057015\.5515\.55\[15\.16,15\.95\]\[15\.16,\\ 15\.95\]−15\.55\-15\.55collapseL\-PS04qwen2\.5\-coder\-32bmpharshSGB0057057020\.2620\.26\[19\.79,20\.73\]\[19\.79,\\ 20\.73\]−20\.26\-20\.26collapseL\-PS05mistral\-small\-24bmpframeSGB005705705\.625\.62\[5\.36,5\.89\]\[5\.36,\\ 5\.89\]−5\.33\-5\.33gradedL\-PS06mistral\-small\-24bmps2onlySGB0057057026\.0426\.04\[25\.45,26\.61\]\[25\.45,\\ 26\.61\]−26\.04\-26\.04refusalL\-PS07mistral\-small\-24bmpnoclauseSGB0057057010\.3310\.33\[9\.99,10\.66\]\[9\.99,\\ 10\.66\]−10\.28\-10\.28collapseL\-PS08mistral\-small\-24bmpharshSGB0057057026\.0326\.03\[25\.44,26\.61\]\[25\.44,\\ 26\.61\]−26\.03\-26\.03refusalL\-PS09llama\-3\.1\-8bmpframeSGB005705708\.468\.46\[8\.10,8\.86\]\[8\.10,\\ 8\.86\]−6\.94\-6\.94collapseL\-PS10llama\-3\.1\-8bmps2onlySGB0057057026\.0426\.04\[25\.45,26\.61\]\[25\.45,\\ 26\.61\]−26\.04\-26\.04refusalL\-PS11llama\-3\.1\-8bmpnoclauseSGB0057057023\.4523\.45\[22\.89,23\.99\]\[22\.89,\\ 23\.99\]−23\.45\-23\.45collapseL\-PS12llama\-3\.1\-8bmpharshSGB0057057026\.0426\.04\[25\.45,26\.61\]\[25\.45,\\ 26\.61\]−26\.04\-26\.04refusalL\-Q01qwen3\-235b\-a22bneutralSGB005705704\.094\.09\[3\.87,4\.32\]\[3\.87,\\ 4\.32\]−3\.56\-3\.56gradedL\-Q02qwen3\-235b\-a22bstrictSGB005705707\.327\.32\[7\.01,7\.61\]\[7\.01,\\ 7\.61\]−7\.24\-7\.24gradedL\-Q03qwen3\-235b\-a22blenientSGB005705706\.016\.01\[5\.65,6\.37\]\[5\.65,\\ 6\.37\]\+5\.94\+5\.94gradedL\-Q04qwen3\-235b\-a22brigorousSGB005705705\.185\.18\[4\.92,5\.45\]\[4\.92,\\ 5\.45\]−4\.90\-4\.90gradedL\-Q05qwen3\-235b\-a22bexactingSGB005705706\.856\.85\[6\.56,7\.13\]\[6\.56,\\ 7\.13\]−6\.75\-6\.75gradedL\-R01qwen3\-coder\-480bneutralSGB005705703\.143\.14\[2\.95,3\.35\]\[2\.95,\\ 3\.35\]\+0\.51\+0\.51gradedL\-R02qwen3\-coder\-480bstrictSGB005705703\.783\.78\[3\.58,3\.98\]\[3\.58,\\ 3\.98\]−2\.52\-2\.52gradedL\-R03qwen3\-coder\-480blenientSGB005705704\.134\.13\[3\.83,4\.44\]\[3\.83,\\ 4\.44\]\+3\.38\+3\.38gradedL\-R04qwen3\-coder\-480brigorousSGB005705703\.063\.06\[2\.88,3\.26\]\[2\.88,\\ 3\.26\]−0\.23\-0\.23gradedL\-R05qwen3\-coder\-480bexactingSGB005705703\.333\.33\[3\.14,3\.52\]\[3\.14,\\ 3\.52\]−1\.56\-1\.56gradedL\-S01deepseek\-coder\-v2\-liteneutralSGB005705705\.985\.98\[5\.69,6\.29\]\[5\.69,\\ 6\.29\]−3\.44\-3\.44gradedL\-S02deepseek\-coder\-v2\-litestrictSGB0057057012\.0912\.09\[11\.66,12\.52\]\[11\.66,\\ 12\.52\]−11\.87\-11\.87collapseL\-S03deepseek\-coder\-v2\-litelenientSGB005705705\.925\.92\[5\.62,6\.22\]\[5\.62,\\ 6\.22\]−3\.31\-3\.31gradedL\-S04deepseek\-coder\-v2\-literigorousSGB005705707\.017\.01\[6\.70,7\.34\]\[6\.70,\\ 7\.34\]−5\.31\-5\.31gradedL\-S05deepseek\-coder\-v2\-liteexactingSGB005705706\.696\.69\[6\.39,7\.01\]\[6\.39,\\ 7\.01\]−4\.83\-4\.83gradedL\-U01llama\-3\.1\-8bneutralSGB005705707\.317\.31\[6\.97,7\.68\]\[6\.97,\\ 7\.68\]−5\.15\-5\.15gradedL\-U02llama\-3\.1\-8bstrictSGB0057057026\.0426\.04\[25\.45,26\.61\]\[25\.45,\\ 26\.61\]−26\.04\-26\.04refusalL\-U03llama\-3\.1\-8blenientSGB005705706\.496\.49\[6\.06,6\.96\]\[6\.06,\\ 6\.96\]\+5\.36\+5\.36gradedL\-U04llama\-3\.1\-8brigorousSGB0057057010\.3110\.31\[9\.88,10\.74\]\[9\.88,\\ 10\.74\]−9\.43\-9\.43collapseL\-U05llama\-3\.1\-8bexactingSGB005705706\.896\.89\[6\.57,7\.23\]\[6\.57,\\ 7\.23\]−4\.50\-4\.50gradedL\-V01llama\-3\.3\-70bneutralSGB005705704\.524\.52\[4\.31,4\.73\]\[4\.31,\\ 4\.73\]−3\.60\-3\.60gradedL\-V02llama\-3\.3\-70bstrictSGB0057057013\.5213\.52\[13\.11,13\.93\]\[13\.11,\\ 13\.93\]−13\.52\-13\.52collapseL\-V03llama\-3\.3\-70blenientSGB005705703\.993\.99\[3\.72,4\.28\]\[3\.72,\\ 4\.28\]\+1\.82\+1\.82gradedL\-V04llama\-3\.3\-70brigorousSGB005705704\.694\.69\[4\.47,4\.91\]\[4\.47,\\ 4\.91\]−3\.89\-3\.89gradedL\-V05llama\-3\.3\-70bexactingSGB005705706\.746\.74\[6\.46,7\.01\]\[6\.46,\\ 7\.01\]−6\.56\-6\.56gradedL\-W01glm\-4\-9bneutralSGB005705704\.314\.31\[4\.02,4\.63\]\[4\.02,\\ 4\.63\]\+1\.68\+1\.68gradedL\-W02glm\-4\-9bstrictSGB0057057025\.4825\.48\[24\.88,26\.04\]\[24\.88,\\ 26\.04\]−25\.48\-25\.48near\-refusalL\-W03glm\-4\-9blenientSGB005705706\.946\.94\[6\.43,7\.46\]\[6\.43,\\ 7\.46\]\+6\.40\+6\.40gradedL\-W04glm\-4\-9brigorousSGB005705704\.534\.53\[4\.21,4\.87\]\[4\.21,\\ 4\.87\]\+1\.70\+1\.70gradedL\-W05glm\-4\-9bexactingSGB005705705\.185\.18\[4\.80,5\.57\]\[4\.80,\\ 5\.57\]\+3\.80\+3\.80gradedL\-X01glm\-4\-32bneutralSGB005705702\.852\.85\[2\.66,3\.05\]\[2\.66,\\ 3\.05\]\+0\.50\+0\.50gradedL\-X02glm\-4\-32bstrictSGB005705709\.669\.66\[9\.15,10\.19\]\[9\.15,\\ 10\.19\]−9\.43\-9\.43collapseL\-X03glm\-4\-32blenientSGB005705705\.335\.33\[4\.99,5\.69\]\[4\.99,\\ 5\.69\]\+5\.00\+5\.00gradedL\-X04glm\-4\-32brigorousSGB005705702\.742\.74\[2\.56,2\.93\]\[2\.56,\\ 2\.93\]\+0\.03\+0\.03gradedL\-X05glm\-4\-32bexactingSGB005705703\.723\.72\[3\.49,3\.96\]\[3\.49,\\ 3\.96\]−2\.34\-2\.34gradedL\-Y01gemma\-3\-27bneutralSGB005705704\.324\.32\[4\.12,4\.55\]\[4\.12,\\ 4\.55\]−1\.52\-1\.52gradedL\-Y02gemma\-3\-27bstrictSGB0057057011\.5511\.55\[11\.16,11\.92\]\[11\.16,\\ 11\.92\]−11\.50\-11\.50collapseL\-Y03gemma\-3\-27blenientSGB005705704\.554\.55\[4\.25,4\.86\]\[4\.25,\\ 4\.86\]\+2\.29\+2\.29gradedL\-Y04gemma\-3\-27brigorousSGB005705704\.884\.88\[4\.65,5\.13\]\[4\.65,\\ 5\.13\]−3\.49\-3\.49gradedL\-Y05gemma\-3\-27bexactingSGB005705705\.475\.47\[5\.21,5\.73\]\[5\.21,\\ 5\.73\]−4\.63\-4\.63gradedL\-Z01mistral\-small\-24bneutralSGB005705703\.663\.66\[3\.46,3\.87\]\[3\.46,\\ 3\.87\]−2\.41\-2\.41gradedL\-Z02mistral\-small\-24bstrictSGB0057057026\.0326\.03\[25\.44,26\.61\]\[25\.44,\\ 26\.61\]−26\.03\-26\.03refusalL\-Z03mistral\-small\-24blenientSGB005705704\.764\.76\[4\.44,5\.12\]\[4\.44,\\ 5\.12\]\+3\.57\+3\.57gradedL\-Z04mistral\-small\-24brigorousSGB005705706\.156\.15\[5\.87,6\.41\]\[5\.87,\\ 6\.41\]−5\.89\-5\.89gradedL\-Z05mistral\-small\-24bexactingSGB005695697\.047\.04\[6\.75,7\.33\]\[6\.75,\\ 7\.33\]−6\.93\-6\.93graded*Prompt\-only baselines \(Appendix[B](https://arxiv.org/html/2609.29333#A2.SS0.SSS0.Px3); not grader configurations\)*PB\-A01qwen2\.5\-coder\-32bstrict \(arbitrated\)SGB001001009\.299\.29\[8\.58,9\.98\]\[8\.58,\\ 9\.98\]−9\.27\-9\.27collapsePB\-A02glm\-4\-32bstrict \(arbitrated\)SGB001001002\.642\.64\[2\.23,3\.10\]\[2\.23,\\ 3\.10\]−1\.53\-1\.53gradedPB\-A03mistral\-small\-24bstrict \(arbitrated\)SGB001001003\.133\.13\[2\.74,3\.53\]\[2\.74,\\ 3\.53\]−2\.31\-2\.31gradedPB\-A04llama\-3\.1\-8bstrict \(arbitrated\)SGB0099997\.467\.46\[6\.60,8\.28\]\[6\.60,\\ 8\.28\]−5\.77\-5\.77gradedPB\-D01qwen2\.5\-coder\-32bstrict \(decomposed\)SGB0010010020\.0020\.00\[19\.10,20\.93\]\[19\.10,\\ 20\.93\]−20\.00\-20\.00collapsePB\-D02glm\-4\-32bstrict \(decomposed\)SGB001001005\.015\.01\[4\.50,5\.51\]\[4\.50,\\ 5\.51\]−4\.95\-4\.95gradedPB\-D03mistral\-small\-24bstrict \(decomposed\)SGB0010010026\.2726\.27\[24\.98,27\.55\]\[24\.98,\\ 27\.55\]−26\.27\-26\.27collapsePB\-D04llama\-3\.1\-8bstrict \(decomposed\)SGB0010010027\.2627\.26\[25\.89,28\.65\]\[25\.89,\\ 28\.65\]−27\.26\-27\.26refusal
## Appendix JFull Limitations Inventory

#### Persona coverage\.

Qwen2\.5\-Coder\-1414B received no persona in the full\-cohort grid and Qwen3\-Coder\-Next only*strict*; the held\-out sweep adds all three harsh wordings and*lenient*on the77B,1414B and3030B \(base and pooled\), leaving the8080B evaluated on only one wording\. On the ML exam the1414B received*neutral*and*strict*only, and Qwen3\-Coder\-Next no*rigorous*or*exacting*\. The headword series covers five models and the policy2×22\\times 2three, with only Qwen2\.5\-Coder\-3232B and Mistral\-Small\-2424B in both, and we did not cross them, so an adjective\-only interaction is untested\. On the closed side only Gemini received all five wordings; Section[10](https://arxiv.org/html/2609.29333#S10)itemises the cross\-vendor coverage, whose decoding differs where the APIs force it \(Anthropic’s current models expose no temperature;gpt\-5\.5is not bit\-reproducible at temperature00\)\.

#### Model and serving confounds\.

gemini\-2\.5\-prorejects a zero thinking budget, so D0303against D0101/D0202compares an older thinking\-on model with newer thinking\-off ones, if anything understating the newer models’ advantage\. The Qwen3\-Coder*Instruct*variants implement no thinking mode, so their thinking\-on configurations are inert replicates\.

#### Fine\-tuning\.

Held\-out MAE CIs are roughly±0\.45\\pm 0\.45, so the paired test carries the verdicts\. The bd arm trains on targets produced by D0101\(CV exam\) and IG0707\(ML exam\), so it measures closed\-model distillation; only the marks recipe trains on human labels alone\. The3030B is not a clean size point \(MoE, attention\-only LoRA\); base rows use the fine\-tuning harness, not identical reruns of Section[5](https://arxiv.org/html/2609.29333#S5)\. vLLM batching reorders reductions \(∼0\.01\\sim 0\.01MAE\); all evaluation, Gemma\-4’s adapter included, is vLLM\-served, with Gemma\-4’s path cross\-validated against HuggingFace generation to within∼0\.1\\sim 0\.1MAE \(Appendix[E](https://arxiv.org/html/2609.29333#A5)\)\.

## Appendix KML\-Exam Run Table

Table[25](https://arxiv.org/html/2609.29333#A11.T25)lists every one of the162162ML\-exam configurations individually —3030closed\-model and132132open\-weights runs — with each run’s own confidence interval, bias, and behaviour class\. Table[22](https://arxiv.org/html/2609.29333#A8.T22)pairs the matched neutral/strict runs for the headline comparison; this one is the flat record, and is the ML exam’s counterpart to Table[24](https://arxiv.org/html/2609.29333#A9.T24)\. The machine\-readable version, with per\-question breakdowns and the full column set, ships asanalysis/introduction\_to\_ai\_master\_comparison\.csvin the accompanying repository\.

Table 25:Every second\-exam grader configuration, one row per run: the3030closed\-model configurations \(2323Gemini IG\-,66OpenAI IO\- and11Anthropic IN\-; Section[6\.1](https://arxiv.org/html/2609.29333#S6.SS1)\) followed by the132132open\-weights \(IA\-\) configurations, each in run\-id order\.*Prompt*lists the components present \(S reference solution, B rubric breakdown, R thinking enabled, F few\-shot demonstrations\); this exam has no student\-facing guidelines document, so the G component of Table[24](https://arxiv.org/html/2609.29333#A9.T24)never appears\. Personas prefixedmpare the mechanism probes of Section[5\.2](https://arxiv.org/html/2609.29333#S5.SS2), replayed on this exam \(keys decoded in Table[24](https://arxiv.org/html/2609.29333#A9.T24)’s caption\)\.nnis the number of students scored against the grader average \(∗\\ast: IG1818’s693693are the survivors of prompt\-length failures, an easier subsample whose MAE is not comparable to full\-cohort rows; Appendix[H](https://arxiv.org/html/2609.29333#A8)\); MAE and bias are on the≈65\\approx 65\-point scale \(score plus bonus, matching how its graders record marks\), the95%95\\%bootstrap CI \(50,00050,000resamples\) on MAE\. Behaviour classes are those of Section[5\.1](https://arxiv.org/html/2609.29333#S5.SS1), at this exam’s floor\-matched collapse band \(MAE≥15\.7\\text\{MAE\}\\geq 15\.7\), and describe a run’s error level rather than persona damage: DeepSeek\-Coder\-V2\-Lite over\-marks badly enough to leave it unprompted \(IA1111, IA1313\)\. A refusal’s MAE is a ceiling artifact, not a severity measurement\. IG1515–IG2020ran on the first100100students; IG1010–IG1212, IA9292and IA136136are repeated\-sampling probes whose scores are the per\-student mean across five reruns \(Appendix[H](https://arxiv.org/html/2609.29333#A8)\)\.claude\-opus\-5exposes no sampling temperature \(thinking disabled; shown as —\)\. Run ids match the released tracker and spreadsheets\. Generated byanalysis/introduction\_to\_ai\_make\_appendix\_table\.py\.RunModelPersonaPromptttnnMAE95%95\\%CIBiasBehaviour*Closed\-model configurations \(Gemini IG\-, OpenAI IO\-, Anthropic IN\-\)*IG01flash\-liteneutralSB00101310137\.537\.53\[7\.20,7\.87\]\[7\.20,\\ 7\.87\]\+5\.98\+5\.98gradedIG02flash\-liteneutralB001032103212\.4612\.46\[12\.03,12\.89\]\[12\.03,\\ 12\.89\]\+12\.16\+12\.16gradedIG03flash\-liteneutralSBR00103710375\.375\.37\[5\.09,5\.66\]\[5\.09,\\ 5\.66\]−4\.04\-4\.04gradedIG04flash\-liteneutralS00101610166\.886\.88\[6\.57,7\.20\]\[6\.57,\\ 7\.20\]\+5\.27\+5\.27gradedIG05flash\-litestrictSB00100810089\.029\.02\[8\.61,9\.43\]\[8\.61,\\ 9\.43\]−7\.83\-7\.83gradedIG06flash\-litelenientSB001020102017\.8617\.86\[17\.30,18\.42\]\[17\.30,\\ 18\.42\]\+17\.80\+17\.80collapseIG073\-flash\-previewneutralSB00103810383\.923\.92\[3\.68,4\.16\]\[3\.68,\\ 4\.16\]\+2\.24\+2\.24gradedIG083\.1\-pro\-previewneutralSB00103810383\.403\.40\[3\.16,3\.65\]\[3\.16,\\ 3\.65\]\+1\.18\+1\.18gradedIG092\.5\-proneutralSB00103510353\.473\.47\[3\.24,3\.72\]\[3\.24,\\ 3\.72\]−0\.83\-0\.83gradedIG10flash\-liteneutralSB009559557\.787\.78\[7\.44,8\.13\]\[7\.44,\\ 8\.13\]\+6\.14\+6\.14gradedIG11flash\-liteneutralSB0\.50\.59639637\.657\.65\[7\.32,8\.00\]\[7\.32,\\ 8\.00\]\+5\.95\+5\.95gradedIG12flash\-liteneutralSB0\.70\.79639637\.637\.63\[7\.29,7\.98\]\[7\.29,\\ 7\.98\]\+5\.85\+5\.85gradedIG13flash\-litestrictSBR00103810387\.767\.76\[7\.43,8\.09\]\[7\.43,\\ 8\.09\]−7\.06\-7\.06gradedIG143\.1\-pro\-previewlenientSB00103310334\.484\.48\[4\.22,4\.75\]\[4\.22,\\ 4\.75\]\+3\.46\+3\.46gradedIG153\-flash\-previewneutralB001001004\.354\.35\[3\.32,5\.64\]\[3\.32,\\ 5\.64\]\+3\.51\+3\.51gradedIG163\-flash\-previewneutralSBR001001003\.763\.76\[2\.78,5\.01\]\[2\.78,\\ 5\.01\]\+2\.78\+2\.78gradedIG173\-flash\-previewneutralS0096963\.373\.37\[2\.42,4\.70\]\[2\.42,\\ 4\.70\]\+2\.38\+2\.38gradedIG18flash\-liteneutralSBF00693693∗4\.304\.30\[4\.01,4\.60\]\[4\.01,\\ 4\.60\]\+0\.36\+0\.36gradedIG193\-flash\-previewstrictSB001001002\.872\.87\[1\.95,4\.10\]\[1\.95,\\ 4\.10\]\+0\.09\+0\.09gradedIG203\-flash\-previewlenientSB001001006\.996\.99\[5\.76,8\.43\]\[5\.76,\\ 8\.43\]\+6\.60\+6\.60gradedIG21flash\-literigorousSB00101210125\.525\.52\[5\.25,5\.79\]\[5\.25,\\ 5\.79\]\+2\.79\+2\.79gradedIG22flash\-liteexactingSB00102110215\.965\.96\[5\.68,6\.25\]\[5\.68,\\ 6\.25\]\+3\.18\+3\.18gradedIG233\.1\-pro\-previewstrictSB00103810383\.673\.67\[3\.43,3\.94\]\[3\.43,\\ 3\.94\]−1\.00\-1\.00gradedIN01claude\-opus\-5neutralSB—103810384\.374\.37\[4\.13,4\.63\]\[4\.13,\\ 4\.63\]−2\.64\-2\.64gradedIO01gpt\-5\.5neutralSB00103810383\.543\.54\[3\.33,3\.77\]\[3\.33,\\ 3\.77\]\+1\.23\+1\.23gradedIO02gpt\-5\.5strictSB00103810383\.703\.70\[3\.48,3\.93\]\[3\.48,\\ 3\.93\]−1\.38\-1\.38gradedIO03gpt\-5\.5lenientSB00103810386\.296\.29\[6\.02,6\.57\]\[6\.02,\\ 6\.57\]\+5\.83\+5\.83gradedIO11gpt\-5\.4neutralSB00103810384\.304\.30\[4\.07,4\.55\]\[4\.07,\\ 4\.55\]−1\.95\-1\.95gradedIO12gpt\-5\.4strictSB00103810385\.115\.11\[4\.85,5\.37\]\[4\.85,\\ 5\.37\]−3\.61\-3\.61gradedIO13gpt\-5\.4lenientSB00103810388\.308\.30\[7\.96,8\.64\]\[7\.96,\\ 8\.64\]\+7\.94\+7\.94graded*Open\-weights \(IA\-\) configurations*IA01\-G32\-nglm4\-32bneutralSB00103810388\.038\.03\[7\.69,8\.37\]\[7\.69,\\ 8\.37\]\+7\.29\+7\.29gradedIA01\-G9\-nglm4\-9bneutralSB001038103813\.1513\.15\[12\.60,13\.71\]\[12\.60,\\ 13\.71\]\+12\.10\+12\.10gradedIA01\-GE27\-ngemma3\-27bneutralSB00103810388\.338\.33\[8\.00,8\.67\]\[8\.00,\\ 8\.67\]\+6\.83\+6\.83gradedIA01\-L8\-nllama31\-8bneutralSB001037103713\.1613\.16\[12\.61,13\.71\]\[12\.61,\\ 13\.71\]\+10\.27\+10\.27gradedIA01\-M24\-nmistral\-small\-24bneutralSB00103810388\.968\.96\[8\.59,9\.34\]\[8\.59,\\ 9\.34\]\+8\.16\+8\.16gradedIA01\-Q14\-nqwen25\-coder\-14bneutralSB00103810388\.618\.61\[8\.25,8\.98\]\[8\.25,\\ 8\.98\]\+7\.40\+7\.40gradedIA01\-Q30\-nqwen3\-coder\-30b\-a3bneutralSB001038103811\.1211\.12\[10\.64,11\.59\]\[10\.64,\\ 11\.59\]\+10\.44\+10\.44gradedIA01\-Q32\-nqwen25\-coder\-32bneutralSB00103810384\.574\.57\[4\.33,4\.83\]\[4\.33,\\ 4\.83\]−0\.21\-0\.21gradedIA01\-Q7\-nqwen25\-coder\-7bneutralSB00103810386\.886\.88\[6\.56,7\.20\]\[6\.56,\\ 7\.20\]\+3\.77\+3\.77gradedIA02\-G32\-sglm4\-32bstrictSB001038103812\.1812\.18\[11\.68,12\.69\]\[11\.68,\\ 12\.69\]−10\.68\-10\.68gradedIA02\-G9\-sglm4\-9bstrictSB001038103816\.6816\.68\[16\.15,17\.20\]\[16\.15,\\ 17\.20\]−16\.52\-16\.52collapseIA02\-GE27\-sgemma3\-27bstrictSB00103810387\.617\.61\[7\.23,7\.99\]\[7\.23,\\ 7\.99\]−6\.25\-6\.25gradedIA02\-L8\-sllama31\-8bstrictSB001038103829\.5829\.58\[28\.70,30\.45\]\[28\.70,\\ 30\.45\]−29\.58\-29\.58refusalIA02\-M24\-smistral\-small\-24bstrictSB001038103817\.3217\.32\[16\.77,17\.87\]\[16\.77,\\ 17\.87\]−17\.20\-17\.20collapseIA02\-Q14\-sqwen25\-coder\-14bstrictSB001038103811\.0611\.06\[10\.61,11\.50\]\[10\.61,\\ 11\.50\]−10\.46\-10\.46gradedIA02\-Q30\-sqwen3\-coder\-30b\-a3bstrictSB00103810385\.585\.58\[5\.31,5\.87\]\[5\.31,\\ 5\.87\]−1\.55\-1\.55gradedIA02\-Q32\-sqwen25\-coder\-32bstrictSB001038103813\.2413\.24\[12\.79,13\.69\]\[12\.79,\\ 13\.69\]−13\.07\-13\.07gradedIA02\-Q7\-sqwen25\-coder\-7bstrictSB00103810387\.457\.45\[7\.11,7\.80\]\[7\.11,\\ 7\.80\]−5\.67\-5\.67gradedIA11deepseek\-coder\-v2\-liteneutralS001038103816\.4816\.48\[15\.82,17\.15\]\[15\.82,\\ 17\.15\]\+14\.96\+14\.96collapseIA12deepseek\-coder\-v2\-litestrictS00103810389\.129\.12\[8\.70,9\.54\]\[8\.70,\\ 9\.54\]−5\.19\-5\.19gradedIA13deepseek\-coder\-v2\-liteneutralSB001038103816\.4116\.41\[15\.76,17\.07\]\[15\.76,\\ 17\.07\]\+14\.89\+14\.89collapseIA14deepseek\-coder\-v2\-litestrictSB00103810389\.229\.22\[8\.83,9\.62\]\[8\.83,\\ 9\.62\]\+2\.42\+2\.42gradedIA15deepseek\-coder\-v2\-litelenientSB001038103816\.2916\.29\[15\.64,16\.95\]\[15\.64,\\ 16\.95\]\+14\.96\+14\.96collapseIA16deepseek\-coder\-v2\-literigorousSB001038103814\.2314\.23\[13\.64,14\.83\]\[13\.64,\\ 14\.83\]\+12\.29\+12\.29gradedIA17deepseek\-coder\-v2\-liteexactingSB001038103815\.5015\.50\[14\.87,16\.13\]\[14\.87,\\ 16\.13\]\+13\.90\+13\.90gradedIA18gemma3\-12bneutralSB001038103810\.3610\.36\[9\.96,10\.76\]\[9\.96,\\ 10\.76\]\+9\.13\+9\.13gradedIA19gemma3\-12bstrictSB00103810386\.796\.79\[6\.49,7\.10\]\[6\.49,\\ 7\.10\]\+0\.09\+0\.09gradedIA20gemma3\-12blenientSB001038103813\.7013\.70\[13\.22,14\.18\]\[13\.22,\\ 14\.18\]\+13\.17\+13\.17gradedIA21gemma3\-12brigorousSB00103810388\.518\.51\[8\.16,8\.86\]\[8\.16,\\ 8\.86\]\+6\.02\+6\.02gradedIA22gemma3\-12bexactingSB00103810388\.718\.71\[8\.36,9\.06\]\[8\.36,\\ 9\.06\]\+6\.48\+6\.48gradedIA23gemma3\-27bneutralS00103810388\.098\.09\[7\.77,8\.42\]\[7\.77,\\ 8\.42\]\+6\.55\+6\.55gradedIA24gemma3\-27bstrictS00103810387\.357\.35\[6\.98,7\.72\]\[6\.98,\\ 7\.72\]−5\.88\-5\.88gradedIA25gemma3\-27bmpstrictSB00103810386\.776\.77\[6\.43,7\.12\]\[6\.43,\\ 7\.12\]−4\.90\-4\.90gradedIA26gemma3\-27bmprigorousSB00103810386\.316\.31\[5\.98,6\.64\]\[5\.98,\\ 6\.64\]−4\.05\-4\.05gradedIA27gemma3\-27bmpfairSB00103810385\.935\.93\[5\.62,6\.25\]\[5\.62,\\ 6\.25\]−3\.35\-3\.35gradedIA28gemma3\-27bmpnoclauseSB00103810386\.486\.48\[6\.16,6\.79\]\[6\.16,\\ 6\.79\]−2\.35\-2\.35gradedIA29gemma3\-27bmpnharshSB00103810387\.787\.78\[7\.47,8\.10\]\[7\.47,\\ 8\.10\]\+5\.74\+5\.74gradedIA30gemma3\-27blenientSB001038103818\.5418\.54\[18\.02,19\.06\]\[18\.02,\\ 19\.06\]\+18\.47\+18\.47collapseIA31gemma3\-27brigorousSB00103810386\.186\.18\[5\.91,6\.46\]\[5\.91,\\ 6\.46\]\+3\.18\+3\.18gradedIA32gemma3\-27bexactingSB00103810385\.955\.95\[5\.69,6\.23\]\[5\.69,\\ 6\.23\]\+1\.83\+1\.83gradedIA33glm4\-32bneutralS00103810387\.517\.51\[7\.19,7\.84\]\[7\.19,\\ 7\.84\]\+6\.75\+6\.75gradedIA34glm4\-32bstrictS001038103818\.4418\.44\[17\.86,19\.03\]\[17\.86,\\ 19\.03\]−17\.63\-17\.63collapseIA35glm4\-32bmpstrictSB00103810387\.297\.29\[6\.92,7\.67\]\[6\.92,\\ 7\.67\]−4\.76\-4\.76gradedIA36glm4\-32bmprigorousSB00103810387\.117\.11\[6\.74,7\.50\]\[6\.74,\\ 7\.50\]−4\.74\-4\.74gradedIA37glm4\-32bmpfairSB00103810386\.746\.74\[6\.38,7\.11\]\[6\.38,\\ 7\.11\]−3\.85\-3\.85gradedIA38glm4\-32bmpnoclauseSB00103810385\.605\.60\[5\.29,5\.92\]\[5\.29,\\ 5\.92\]−3\.40\-3\.40gradedIA39glm4\-32bmpnharshSB00103810386\.836\.83\[6\.52,7\.15\]\[6\.52,\\ 7\.15\]\+5\.82\+5\.82gradedIA40glm4\-32blenientSB001038103815\.3515\.35\[14\.90,15\.80\]\[14\.90,\\ 15\.80\]\+15\.23\+15\.23gradedIA41glm4\-32brigorousSB00103810387\.477\.47\[7\.15,7\.80\]\[7\.15,\\ 7\.80\]\+6\.64\+6\.64gradedIA42glm4\-32bexactingSB00103810385\.575\.57\[5\.28,5\.86\]\[5\.28,\\ 5\.86\]\+2\.97\+2\.97gradedIA43glm4\-9blenientSB001038103825\.9825\.98\[25\.22,26\.74\]\[25\.22,\\ 26\.74\]\+25\.96\+25\.96collapseIA44glm4\-9brigorousSB001038103811\.8011\.80\[11\.29,12\.31\]\[11\.29,\\ 12\.31\]\+10\.49\+10\.49gradedIA45glm4\-9bexactingSB001038103814\.7114\.71\[14\.11,15\.33\]\[14\.11,\\ 15\.33\]\+14\.05\+14\.05gradedIA46llama31\-8bmpframeSB001038103811\.5411\.54\[11\.07,12\.02\]\[11\.07,\\ 12\.02\]\+7\.78\+7\.78gradedIA47llama31\-8bmps2onlySB001038103829\.3929\.39\[28\.52,30\.25\]\[28\.52,\\ 30\.25\]−29\.37\-29\.37near\-refusalIA48llama31\-8bmpnoclauseSB001038103825\.8725\.87\[25\.08,26\.65\]\[25\.08,\\ 26\.65\]−25\.84\-25\.84collapseIA49llama31\-8bmpharshSB001038103829\.5829\.58\[28\.70,30\.45\]\[28\.70,\\ 30\.45\]−29\.58\-29\.58refusalIA50llama31\-8blenientSB001037103727\.7327\.73\[26\.92,28\.54\]\[26\.92,\\ 28\.54\]\+27\.69\+27\.69collapseIA51llama31\-8brigorousSB00103810389\.019\.01\[8\.61,9\.42\]\[8\.61,\\ 9\.42\]\+3\.07\+3\.07gradedIA52llama31\-8bexactingSB001038103811\.9611\.96\[11\.45,12\.48\]\[11\.45,\\ 12\.48\]\+9\.15\+9\.15gradedIA53mistral\-small\-24bmpstrictSB001038103812\.4912\.49\[12\.03,12\.96\]\[12\.03,\\ 12\.96\]−12\.14\-12\.14gradedIA54mistral\-small\-24bmprigorousSB00103810388\.818\.81\[8\.43,9\.21\]\[8\.43,\\ 9\.21\]−7\.67\-7\.67gradedIA55mistral\-small\-24bmpfairSB00103810388\.638\.63\[8\.24,9\.02\]\[8\.24,\\ 9\.02\]−7\.41\-7\.41gradedIA56mistral\-small\-24bmpnoclauseSB00103810386\.376\.37\[6\.08,6\.67\]\[6\.08,\\ 6\.67\]−0\.67\-0\.67gradedIA57mistral\-small\-24bmpnharshSB00103810388\.098\.09\[7\.74,8\.45\]\[7\.74,\\ 8\.45\]\+6\.94\+6\.94gradedIA58mistral\-small\-24bmpframeSB00103810386\.586\.58\[6\.27,6\.90\]\[6\.27,\\ 6\.90\]\+4\.10\+4\.10gradedIA59mistral\-small\-24bmps2onlySB001038103826\.0326\.03\[25\.29,26\.77\]\[25\.29,\\ 26\.77\]−26\.02\-26\.02collapseIA60mistral\-small\-24bmpharshSB001037103717\.2917\.29\[16\.73,17\.84\]\[16\.73,\\ 17\.84\]−17\.17\-17\.17collapseIA61mistral\-small\-24blenientSB001038103815\.5615\.56\[15\.08,16\.05\]\[15\.08,\\ 16\.05\]\+15\.44\+15\.44gradedIA62mistral\-small\-24brigorousSB00103810385\.895\.89\[5\.61,6\.19\]\[5\.61,\\ 6\.19\]\+2\.96\+2\.96gradedIA63mistral\-small\-24bexactingSB00103810385\.705\.70\[5\.43,5\.98\]\[5\.43,\\ 5\.98\]\+2\.41\+2\.41gradedIA64qwen25\-coder\-32bneutralS00103810384\.454\.45\[4\.21,4\.71\]\[4\.21,\\ 4\.71\]\+0\.13\+0\.13gradedIA65qwen25\-coder\-32bstrictS001038103813\.4313\.43\[12\.97,13\.89\]\[12\.97,\\ 13\.89\]−13\.25\-13\.25gradedIA66qwen25\-coder\-32blenientSB00103810386\.966\.96\[6\.66,7\.26\]\[6\.66,\\ 7\.26\]\+5\.56\+5\.56gradedIA67qwen25\-coder\-32bmpstrictSB00103810389\.659\.65\[9\.26,10\.04\]\[9\.26,\\ 10\.04\]−9\.30\-9\.30gradedIA68qwen25\-coder\-32bmprigorousSB001038103810\.1910\.19\[9\.80,10\.59\]\[9\.80,\\ 10\.59\]−9\.87\-9\.87gradedIA69qwen25\-coder\-32bmpfairSB001038103810\.4010\.40\[10\.00,10\.81\]\[10\.00,\\ 10\.81\]−10\.09\-10\.09gradedIA70qwen25\-coder\-32bmpnoclauseSB00103810389\.459\.45\[9\.05,9\.85\]\[9\.05,\\ 9\.85\]−9\.09\-9\.09gradedIA71qwen25\-coder\-32bmpnharshSB00103810384\.674\.67\[4\.42,4\.93\]\[4\.42,\\ 4\.93\]−0\.64\-0\.64gradedIA72qwen25\-coder\-32brigorousSB00103810384\.734\.73\[4\.47,5\.00\]\[4\.47,\\ 5\.00\]−1\.47\-1\.47gradedIA73qwen25\-coder\-32bexactingSB00103810385\.435\.43\[5\.14,5\.73\]\[5\.14,\\ 5\.73\]−3\.73\-3\.73gradedIA74qwen25\-coder\-32bmpframeSB00103810384\.784\.78\[4\.52,5\.05\]\[4\.52,\\ 5\.05\]−1\.71\-1\.71gradedIA75qwen25\-coder\-32bmps2onlySB00103810386\.686\.68\[6\.35,7\.02\]\[6\.35,\\ 7\.02\]−5\.67\-5\.67gradedIA76qwen25\-coder\-32bmpharshSB001038103813\.2413\.24\[12\.80,13\.70\]\[12\.80,\\ 13\.70\]−13\.07\-13\.07gradedIA77qwen25\-coder\-7bneutralS00103810387\.557\.55\[7\.20,7\.89\]\[7\.20,\\ 7\.89\]\+4\.92\+4\.92gradedIA78qwen25\-coder\-7blenientSB00103810388\.148\.14\[7\.79,8\.50\]\[7\.79,\\ 8\.50\]\+6\.50\+6\.50gradedIA79qwen3\-coder\-30b\-a3bneutralS00103810389\.139\.13\[8\.73,9\.54\]\[8\.73,\\ 9\.54\]\+7\.80\+7\.80gradedIA80qwen3\-coder\-30b\-a3blenientSB001038103817\.1617\.16\[16\.58,17\.74\]\[16\.58,\\ 17\.74\]\+17\.02\+17\.02collapseIA91qwen25\-coder\-32bneutralB00103310334\.734\.73\[4\.49,4\.98\]\[4\.49,\\ 4\.98\]\+1\.20\+1\.20gradedIA92qwen25\-coder\-32bneutralSB0\.50\.541414\.654\.65\[2\.78,7\.37\]\[2\.78,\\ 7\.37\]\+4\.51\+4\.51gradedIA93qwen25\-coder\-32bneutralSBR00103310334\.594\.59\[4\.34,4\.85\]\[4\.34,\\ 4\.85\]−0\.24\-0\.24gradedIA94qwen25\-coder\-7bneutralB00103810387\.517\.51\[7\.16,7\.88\]\[7\.16,\\ 7\.88\]\+5\.57\+5\.57gradedIA95qwen3\-coder\-30b\-a3bneutralB001013101311\.9611\.96\[11\.50,12\.40\]\[11\.50,\\ 12\.40\]\+11\.53\+11\.53gradedIA96qwen3\-coder\-30b\-a3bneutralSBR001017101711\.0911\.09\[10\.61,11\.57\]\[10\.61,\\ 11\.57\]\+10\.40\+10\.40gradedIA101glm45\-airneutralSB00103710375\.605\.60\[5\.31,5\.89\]\[5\.31,\\ 5\.89\]−1\.34\-1\.34gradedIA102glm45\-airstrictSB001038103818\.4918\.49\[17\.96,19\.02\]\[17\.96,\\ 19\.02\]−18\.31\-18\.31collapseIA103glm45\-airlenientSB001038103815\.3015\.30\[14\.74,15\.87\]\[14\.74,\\ 15\.87\]\+15\.17\+15\.17gradedIA104glm45\-airrigorousSB00103710376\.266\.26\[5\.95,6\.58\]\[5\.95,\\ 6\.58\]−3\.97\-3\.97gradedIA105glm45\-airexactingSB00102210229\.959\.95\[9\.55,10\.36\]\[9\.55,\\ 10\.36\]−9\.15\-9\.15gradedIA106llama33\-70bneutralS00101610164\.294\.29\[4\.05,4\.55\]\[4\.05,\\ 4\.55\]\+0\.01\+0\.01gradedIA107llama33\-70bstrictS001013101314\.1314\.13\[13\.62,14\.64\]\[13\.62,\\ 14\.64\]−13\.64\-13\.64gradedIA108llama33\-70bmpstrictSB00103810389\.569\.56\[9\.16,9\.96\]\[9\.16,\\ 9\.96\]−9\.05\-9\.05gradedIA109llama33\-70bmprigorousSB00103810388\.558\.55\[8\.18,8\.94\]\[8\.18,\\ 8\.94\]−7\.86\-7\.86gradedIA110llama33\-70bmpfairSB00103810388\.138\.13\[7\.76,8\.50\]\[7\.76,\\ 8\.50\]−7\.36\-7\.36gradedIA111llama33\-70bmpnoclauseSB00103810386\.126\.12\[5\.82,6\.43\]\[5\.82,\\ 6\.43\]−4\.87\-4\.87gradedIA112llama33\-70bmpnharshSB00103810384\.604\.60\[4\.36,4\.85\]\[4\.36,\\ 4\.85\]\+1\.61\+1\.61gradedIA113llama33\-70bneutralSB00103810384\.704\.70\[4\.45,4\.96\]\[4\.45,\\ 4\.96\]\+1\.90\+1\.90gradedIA114llama33\-70bstrictSB001038103810\.1710\.17\[9\.77,10\.58\]\[9\.77,\\ 10\.58\]−9\.63\-9\.63gradedIA115llama33\-70blenientSB001038103811\.7411\.74\[11\.34,12\.15\]\[11\.34,\\ 12\.15\]\+11\.55\+11\.55gradedIA116llama33\-70brigorousSB00103810384\.604\.60\[4\.36,4\.85\]\[4\.36,\\ 4\.85\]\+1\.71\+1\.71gradedIA117llama33\-70bexactingSB00103810385\.235\.23\[4\.95,5\.51\]\[4\.95,\\ 5\.51\]−3\.31\-3\.31gradedIA118qwen25\-72bneutralSB001038103810\.6410\.64\[10\.25,11\.03\]\[10\.25,\\ 11\.03\]\+10\.31\+10\.31gradedIA119qwen25\-72bstrictSB00103810385\.425\.42\[5\.15,5\.70\]\[5\.15,\\ 5\.70\]−1\.67\-1\.67gradedIA120qwen25\-72blenientSB001038103816\.9316\.93\[16\.44,17\.42\]\[16\.44,\\ 17\.42\]\+16\.89\+16\.89collapseIA121qwen25\-72brigorousSB00103810388\.508\.50\[8\.16,8\.85\]\[8\.16,\\ 8\.85\]\+7\.94\+7\.94gradedIA122qwen25\-72bexactingSB00103810388\.458\.45\[8\.09,8\.81\]\[8\.09,\\ 8\.81\]\+7\.80\+7\.80gradedIA123qwen3\-235b\-a22bneutralSB00103710375\.405\.40\[5\.13,5\.67\]\[5\.13,\\ 5\.67\]\+3\.73\+3\.73gradedIA124qwen3\-235b\-a22bstrictSB00103810385\.185\.18\[4\.89,5\.47\]\[4\.89,\\ 5\.47\]−3\.39\-3\.39gradedIA125qwen3\-235b\-a22blenientSB001038103817\.4617\.46\[16\.99,17\.93\]\[16\.99,\\ 17\.93\]\+17\.41\+17\.41collapseIA126qwen3\-235b\-a22brigorousSB00103810384\.534\.53\[4\.29,4\.78\]\[4\.29,\\ 4\.78\]\+2\.11\+2\.11gradedIA127qwen3\-235b\-a22bexactingSB00103810384\.584\.58\[4\.32,4\.84\]\[4\.32,\\ 4\.84\]−1\.58\-1\.58gradedIA128qwen3\-coder\-480bneutralSB001038103810\.1010\.10\[9\.66,10\.54\]\[9\.66,\\ 10\.54\]\+9\.52\+9\.52gradedIA129qwen3\-coder\-480bstrictSB00103810386\.046\.04\[5\.75,6\.34\]\[5\.75,\\ 6\.34\]\+3\.25\+3\.25gradedIA130qwen3\-coder\-480blenientSB001038103815\.3715\.37\[14\.87,15\.88\]\[14\.87,\\ 15\.88\]\+15\.29\+15\.29gradedIA131qwen3\-coder\-480brigorousSB00103810387\.897\.89\[7\.54,8\.25\]\[7\.54,\\ 8\.25\]\+6\.99\+6\.99gradedIA132qwen3\-coder\-480bexactingSB00103810387\.037\.03\[6\.69,7\.37\]\[6\.69,\\ 7\.37\]\+5\.04\+5\.04gradedIA133qwen3\-coder\-nextneutralSB00102110214\.474\.47\[4\.24,4\.72\]\[4\.24,\\ 4\.72\]\+1\.51\+1\.51gradedIA134qwen3\-coder\-nextstrictSB00100110018\.638\.63\[8\.26,9\.00\]\[8\.26,\\ 9\.00\]−8\.12\-8\.12gradedIA135qwen3\-coder\-nextlenientSB00102110217\.647\.64\[7\.32,7\.96\]\[7\.32,\\ 7\.96\]\+7\.10\+7\.10gradedIA136qwen3\-coder\-nextneutralSB0\.50\.541415\.045\.04\[3\.26,7\.76\]\[3\.26,\\ 7\.76\]\+5\.04\+5\.04gradedIA137qwen3\-coder\-nextneutralSBR00102110214\.474\.47\[4\.24,4\.72\]\[4\.24,\\ 4\.72\]\+1\.51\+1\.51gradedIA201qwen3\-coder\-30b\-a3bneutralSBF00103810385\.845\.84\[5\.56,6\.13\]\[5\.56,\\ 6\.13\]\+4\.56\+4\.56graded