Towards a Deterministic Math Solver for Clinical Language Models

# Towards a Deterministic Math Solver for Clinical Language Models
Source: [https://arxiv.org/html/2609.10728](https://arxiv.org/html/2609.10728)
###### Abstract

Large language models are unreliable at arithmetic, which is a problem for clinical calculators where a single numerical error changes the recommendation\. The standard response is to hardcode each calculator as a validated function, one at a time\. We test an alternative: the model does not calculate\. Instead, it writes case\-specific Python that a restricted local executor runs as a deterministic solver, and the model’s task reduces to deciding how to use it\. We evaluate this Program\-Solve interface on MedCalc\-Bench Verified \(1,100 cases, 55 calculators\) against direct model arithmetic and a hand\-written 22\-calculator library, using Qwen2\.5\-7B and Qwen2\.5\-32B\-AWQ, after auditing the benchmark’s formulas against current clinical guidelines and flagging 16 of 55 with version, use or coefficient concerns\. With formulas and gold variables supplied and both routes reading the whole note, handing off to the solver is not a reliable advantage at 7B \(75\.31% against 72\.02%, a paired\+3\.29\+3\.29points with a 95% calculator\-cluster interval of\[−3\.49,10\.38\]\[\-3\.49,10\.38\]\) but is one at 32B \(90\.53% against 83\.47%,\+7\.05\+7\.05\[0\.47,14\.60\]\[0\.47,14\.60\], clear of zero\)\. The hand\-written library is exact on its 440 supported cases but abstains elsewhere \(40\.0% overall\)\. Adding an executor thus helps some open\-weight models more than others even under matched formula, variable and note access, and is not a substitute for verified formulas or reliable variable extraction either way\. Code and data:[https://github\.com/felipeocampoos/Towards\-a\-Deterministic\-Math\-Solver\-for\-Clinical\-Language\-Models](https://github.com/felipeocampoos/Towards-a-Deterministic-Math-Solver-for-Clinical-Language-Models)\.

## 1Introduction

Automating clinical calculators such as APACHE II\[[1](https://arxiv.org/html/2609.10728#bib.bib1)\]with a language model requires selecting the formula, extracting variables from a free\-text note and executing the arithmetic, on which MedCalc\-Bench\[[2](https://arxiv.org/html/2609.10728#bib.bib2)\]shows models losing accuracy: Goodell and colleagues report incorrect answers in about one third of unaided ChatGPT trials across 48 calculation tasks\[[3](https://arxiv.org/html/2609.10728#bib.bib3)\]\. The standard remedy is a hand\-written function per calculator, each written, validated and maintained, with every case outside the set unanswerable: the evaluated 22\-calculator library implements 440 of 1,100 cases, 40\.0% full\-set accuracy under abstention\. We evaluate whether calculator\-specific execution can be replaced by a general interface: the model is given the case and writes a short Python program, and a restricted executor runs it and returns a number, a date or a gestational\-age tuple\. The executor contains no calculator\-specific functions or constants; the model translates the supplied clinical formula into code\. Clinically, execution fidelity is necessary but not sufficient: the formulas a calculator encodes are themselves versioned, and several in routine use \(race\-free eGFR, MELD 3\.0, PREVENT, the Sampson LDL equation\) have replaced predecessors that benchmarks may still reward\. Local serving avoids external API calls and data transmission; its unmeasured costs are in Section[4](https://arxiv.org/html/2609.10728#S4)\.

#### Related work and contribution\.

Program\-aided reasoning has the model emit code for an interpreter\[[4](https://arxiv.org/html/2609.10728#bib.bib4),[5](https://arxiv.org/html/2609.10728#bib.bib5)\]; chain\-of\-thought prompting\[[6](https://arxiv.org/html/2609.10728#bib.bib6)\]is the in\-context alternative\. Executing that code carries a security exposure distinct from whether it is correct\[[7](https://arxiv.org/html/2609.10728#bib.bib7)\]\. MedCalc\-Bench formalised calculator invocation\[[2](https://arxiv.org/html/2609.10728#bib.bib2)\]; MedRaC pairs retrieval with Python execution and scores formula selection, extraction and arithmetic separately\[[8](https://arxiv.org/html/2609.10728#bib.bib8)\]; RiskAgent selects among validated tools\[[9](https://arxiv.org/html/2609.10728#bib.bib9)\]; a clinical\-calculator chatbot routes to verifiable calculators\[[10](https://arxiv.org/html/2609.10728#bib.bib10)\]; MeNTi bridges calculators and agents through nested tool calling\[[11](https://arxiv.org/html/2609.10728#bib.bib11)\]; verifiable\-reward training raises the aggregate\[[12](https://arxiv.org/html/2609.10728#bib.bib12)\]; decomposition adds failure points when extraction is incomplete\[[13](https://arxiv.org/html/2609.10728#bib.bib13)\]; most calculator\-selection errors are comprehension errors, not arithmetic ones\[[14](https://arxiv.org/html/2609.10728#bib.bib14)\]\. A code\-interpreter arm compared against task\-specific calculator tools found the tools more accurate\[[3](https://arxiv.org/html/2609.10728#bib.bib3)\]\. AgentMD automates the tool curation we describe as a maintenance burden\[[15](https://arxiv.org/html/2609.10728#bib.bib15)\], and the coverage\-accuracy trade\-off a partial library exhibits is the abstention problem\[[16](https://arxiv.org/html/2609.10728#bib.bib16),[17](https://arxiv.org/html/2609.10728#bib.bib17)\]\. Our contribution is a controlled comparison of case\-specific program generation against direct arithmetic and the hand\-written alternative, under matched formula, variable and note access\. It establishes what execution does and does not fix on one benchmark; generalization to unseen formulas, languages or settings is outside its scope\.

Figure 1:\(a\)The model writes a program from note, formula and variables and an executor without calculator code runs it: all 55 calculators attempted, the library implements 22\.\(b\)Accuracy on 1,100 cases; open bar: Blind Program\-Solve\. Program\-Solve minus Open\-book arithmetic: calculator\-cluster interval \(Table[2](https://arxiv.org/html/2609.10728#S3.T2)\) crosses zero at 7B and clears it at 32B\.

## 2Method

#### Data and models\.

MedCalc\-Bench Verified, 1,100 test cases across 55 calculators, scored under benchmark\-defined tolerance\. Two open\-weight models, Qwen2\.5\-7B\-Instruct \(bf16\) and Qwen2\.5\-32B\-Instruct\-AWQ \(4\-bit\), served through vLLM on cloud H100 GPUs \(tensor\-parallel 1, one H100 each; other checkpoints likewise one H100 or, for Mistral, two under tensor parallelism\), at five seeds \(42 to 46\) over all 1,100 cases\. The worked one\-shot example comes from the benchmark’s separate one\-shot split; no test case’s gold answer or explanation enters any prompt\. MedCalc\-Bench Verified is CC\-BY\-SA 4\.0, both models Apache 2\.0\.

#### Executor\.

A fresh subprocess per program: a builtins allow\-list with no file, eval or exec primitives; imports limited to the standard math, date, time and calendar modules; static rejection of async and generator constructs; 256 MB and CPU limits, 5 s wall clock, an executed\-line cap\. The executor is restricted but is no sandbox: there is no container or syscall filter\.

#### Arms\.

Table[1](https://arxiv.org/html/2609.10728#S3.T1)states each arm’s formula access, variable access and execution method\.Note onlygets the note\.Page without formulaadds the calculator’s page with the formula suppressed andOpen\-book arithmeticadds the formula, both with variables given and the model doing the arithmetic\.Extract\-Solve libraryextracts variables and calls one of 22 hand\-written Python calculators;Gold\-Solve librarygives those same calculators the gold variables\. The 22 were not chosen by a stated criterion: all are laboratory, physical or date calculators and none is a point\-based score, an opportunistic set rather than a principled one\. Because the benchmark is balanced at 20 cases per calculator, a library’s coverage is a fixed function of how many calculators it implements, so its full\-set accuracy under abstention is bounded by that count \(Figure[2](https://arxiv.org/html/2609.10728#A1.F2)\)\.Program\-Solveis ours: the model writes a program that the executor runs, given the formula text and gold variables, the inputs Open\-book arithmetic receives;Blind Program\-Solvegets neither and extracts its own variables\. Calculator support, attempted\-answer rate, valid\-program rate and correct\-answer rate are distinct measures; Table[1](https://arxiv.org/html/2609.10728#S3.T1)reports correct\-answer rates only\.

#### Comparison design\.

Comparisons are distinguished by formula access, variable access and note length\. Note only \(120 words\) and Page without formula \(250 words\) cap the note; Open\-book arithmetic, Program\-Solve, Blind Program\-Solve and the Extract\-Solve library all read the whole note, so Program\-Solve against Open\-book arithmetic is matched on formula, variables and note length\. A 250\-word one\-shot arm without gold variables \(Table[20](https://arxiv.org/html/2609.10728#A1.T20)\) separates budget from access: with variable access fixed, the larger budget alone adds 4 to 12 points in every family\. Program\-Solve against the Extract\-Solve library is not input\-matched; the matched pairs are Program\-Solve/Gold\-Solve library and Blind Program\-Solve/Extract\-Solve library\. Decoding, token, note\-budget and executor settings are in Table[16](https://arxiv.org/html/2609.10728#A1.T16); every arm’s protocol is in Appendix[A\.1](https://arxiv.org/html/2609.10728#A1.SS1)\.

#### Statistics\.

Cases nest in 55 calculators and recur across five seeds, so the primary uncertainty is a cluster bootstrap over calculators \(10,000 draws, percentile 95% intervals, seeds kept together\); a case bootstrap, exact McNemar and a sign\-flip permutationppare secondary, Holm\-corrected within family, in the appendix\. Cluster intervals carry no multiplicity adjustment; the Holm correction applies to the case\-level tests only\. Cases cluster strongly within calculators for the library comparisons \(intraclass correlation 0\.68 to 0\.81\), so their effective sample size is 67 to 79 cases against 120 to 190 for the arithmetic comparisons \(Table[7](https://arxiv.org/html/2609.10728#A1.T7)\)\.

## 3Results

Table 1:Accuracy \(%\), five seeds\.Impl\.: the 440 cases of the 22 calculators the library implements;Full: all 1,100\. Gold\-Solve is correct on all 440 by audit and abstains elsewhere, so 40\.00 by construction; both library rows abstain rather than guess\. Both Open\-book arithmetic and every program arm now read the whole note \(Note only and Page without formula still cap at 120 and 250 words\)\. Syntax lines are a benchmark\-specific ablation\. Dash: no formula given\.### 3\.1Matched execution and partial\-library comparisons

Given the same formula text, gold variables and note access, Program\-Solve is not reliably more accurate than Open\-book arithmetic at 7B but is at 32B, clear of zero \(Table[2](https://arxiv.org/html/2609.10728#S3.T2)\)\. Program\-Solve returns no valid answer on 6\.7% and 0\.7% of case\-seed rows and a wrong answer on 18\.0% and 8\.8%, against none unanswered and 28\.0% and 16\.5% wrong for Open\-book arithmetic \(Table[14](https://arxiv.org/html/2609.10728#A1.T14)\)\.

The library comparison is a different story from the matched one above: most of the difference comes from coverage rather than from execution\. Against the Gold\-Solve library, correct on every case it implements, Program\-Solve leads by\+35\.31\+35\.31pp at 7B and\+50\.53\+50\.53pp at 32B on the full set \(Table[2](https://arxiv.org/html/2609.10728#S3.T2)\), because the library abstains on the 660 cases it does not implement; on the 440 it does, it reaches 100% against 84\.20% and 98\.64% for Program\-Solve, and answering the other 660 replaces abstentions with some wrong answers \(Table[8](https://arxiv.org/html/2609.10728#A1.T8)\)\. Restricted to the 39 audit\-clean calculators \(Table[12](https://arxiv.org/html/2609.10728#A1.T12)\), the gap is\+32\.26\+32\.26\(\[15\.51,48\.21\]\[15\.51,48\.21\]\) at 7B and\+47\.59\+47\.59\(\[32\.97,61\.79\]\[32\.97,61\.79\]\) at 32B\. A Gold\-first hybrid \(library on its 440, Program\-Solve elsewhere\) reaches 81\.64% and 91\.07%, above every single arm; with Open\-book arithmetic as fallback it reaches 78\.56% and 86\.89% \(\+3\.07\+3\.07/\+4\.18\+4\.18pp for the program route, both intervals crossing zero\), while an Extract\-first hybrid’s program fallback is worse \(−26\.44\-26\.44/−25\.84\-25\.84pp, both clear of zero; Tables[15](https://arxiv.org/html/2609.10728#A1.T15),[23](https://arxiv.org/html/2609.10728#A1.T23)\)\.

Two syntax and date lines added to the prompt \(Program\-Solve \+ syntax\) were selected on the test split and are exploratory \(Table[2](https://arxiv.org/html/2609.10728#S3.T2), lower block\)\. They lift 7B to 77\.95%,\+2\.64\+2\.64pp over Program\-Solve without them \(cluster CI\[−0\.64,6\.29\]\[\-0\.64,6\.29\]\); 32B moves little on top of its already\-clear advantage \(\+0\.44\+0\.44pp,\[−1\.38,2\.35\]\[\-1\.38,2\.35\]\)\. Across the four checkpoints from three model families their effect on the Program\-Solve/arithmetic gap ranges from−4\.5\-4\.5to\+2\.6\+2\.6pp \(Table[5](https://arxiv.org/html/2609.10728#A1.T5)\)\.

Table 2:Paired gaps \(pp\) on the full 1,100, five seeds, with 95% calculator\-cluster bootstrap intervals \(10,000 draws\), unadjusted for multiplicity\. Upper block: the original prompt\. Lower block: the syntax\-added prompt, selected on the test split\. Intervals crossing zero establish neither difference nor equivalence; paired estimates can differ slightly from rounded\-mean differences\.
### 3\.2Removing formula and gold\-variable access

Blind Program\-Solve reaches 28\.04% at 7B and 44\.71% at 32B, declines of 47\.27 and 45\.82pp from Program\-Solve \(Table[1](https://arxiv.org/html/2609.10728#S3.T1)\); formula and variable access change together, so the design does not separate recall from extraction\. Against the Extract\-Solve library the point estimate is lower at 7B and higher at 32B, but both intervals cross zero \(Table[2](https://arxiv.org/html/2609.10728#S3.T2)\)\. The observed failures are formula and variable errors, read from outputs without a controlled decomposition: an ideal\-body\-weight convention in Cockcroft\-Gault, potassium in a corrected anion gap, heart rate for respiratory rate\.

#### Other families\.

Table[1](https://arxiv.org/html/2609.10728#S3.T1)includes the Mistral and Phi results, full set, formula\-given code against arithmetic, both reading the whole note: secondary permutationp<0\.001p<0\.001andp=0\.010p=0\.010; the two move in opposite directions, Mistral’s code route 7\.6pp behind arithmetic and Phi\-3\.5’s 4\.2pp ahead\. Mistral’s Extract\-Solve library reaches 34\.89%, above either Program\-Solve arm\.

## 4Limitations

#### Experimental scope\.

Two Qwen checkpoints do not establish scaling; Mistral and Phi\-3\.5 move in opposite directions from each other and from both Qwen checkpoints\. Four delta\-gap calculators received the plain anion\-gap formula until the program audit found it, and two more \(an anion\-gap variant and a related osmolality calculator\) were found aliased onto the wrong quantity in a second pass; formula\-reading arms were rerun after each fix\. A completeness audit of all 55 supplied texts \(Table[21](https://arxiv.org/html/2609.10728#A1.T21)\) then found 28 wrong or incomplete, 10 unable to reproduce the benchmark’s number; every arm in this rerun, including Open\-book arithmetic and Program\-Solve, read the same, fully corrected texts\. Re\-auditing the corrected runs \(Table[18](https://arxiv.org/html/2609.10728#A1.T18)\), no sampled error traces to an under\-specified formula; the residual failures are unit conversions invented for supplied inputs and one\-tier slips inside multi\-band scoring tables, so Program\-Solve fails on arithmetic hygiene rather than on clinical knowledge\. Note only reads 120 words, understating a budget\-matched baseline by 4 to 12 points, and the comparator is a partial 22\-calculator library\. The residual failures are the errors a tired clinician makes, invented unit conversions and one\-tier slips in banded scores, and the errors a validated calculator never makes; the library’s 100% on its 440 covered cases shows what abstention is worth\.

#### Clinical validity\.

Every gold answer is the calculator’s own output, so final\-answer accuracy validates neither the program’s logic nor the formula\. A preliminary literature\-based audit of all 55 calculators \(Tables[11](https://arxiv.org/html/2609.10728#A1.T11)to[13](https://arxiv.org/html/2609.10728#A1.T13)\) flags 16 with a version, use or coefficient concern, four of them replaced by a current guideline, which the benchmark still rewards reproducing exactly\. On those four, Program\-Solve scores 71\.00 and 90\.00 against 28\.25 and 36\.50 for Open\-book arithmetic, its largest gap over arithmetic at both scales; removing them moves no headline gap outside its interval \(Table[12](https://arxiv.org/html/2609.10728#A1.T12)\)\. The larger point is that formula provenance and version should be explicit inputs to any calculator interface, human or automated, rather than assumptions inherited from the benchmark\.

#### Deployment\.

No Global South data, language or locale is evaluated; the benchmark is English with US conventions, and the cost of local serving is not measured\. Notes in other languages, laboratory values in mmol/L rather than mg/dL and day/month/year dates each open a further path to the unit\-conversion failures observed here; the exploratory date\-format prompt lines show how much locale the current result silently assumes\.

## 5Conclusion

With the formula, variables and note access matched, an open\-weight model writing a program is not reliably more accurate than the same model doing the arithmetic at 7B scale, but is at 32B scale \(\+7\.1pp, cluster CI clear of zero\); its advantage over a partial hand\-written library at either scale comes mostly from answering where the library abstains; where both answer, the library is the more accurate\. Which of these two patterns a given open\-weight checkpoint will show is not yet predictable from scale alone: Mistral\-7B and Phi\-3\.5\-mini move in opposite directions on the same comparison\. Pending that answer, the clinically defensible configuration is a verified library where one exists, program generation where it does not, and explicit abstention where neither can be trusted\. The next question is what separates them\. Code and data:[https://github\.com/felipeocampoos/Towards\-a\-Deterministic\-Math\-Solver\-for\-Clinical\-Language\-Models](https://github.com/felipeocampoos/Towards-a-Deterministic-Math-Solver-for-Clinical-Language-Models)\.

## Acknowledgments

This research was supported by Anthropic’s AI for Science program\. GPU compute was provided by NVIDIA through the Brev academic grant node and by the MIT Office of Research Computing and Data \(ORCD\) cluster\.

## References

- \[1\]William A\. Knaus, Elizabeth A\. Draper, Douglas P\. Wagner, and Jack E\. Zimmerman\.APACHE II: A severity of disease classification system\.Critical Care Medicine, 13\(10\):818–829, October 1985\.
- \[2\]Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, et al\.Medcalc\-bench: Evaluating large language models for medical calculations\.Advances in Neural Information Processing Systems, 37:84730–84745, 2024\.
- \[3\]Alex J Goodell, Simon N Chu, Dara Rouholiman, and Larry F Chu\.Large language model agents can use tools to perform clinical calculations\.NPJ digital medicine, 8\(1\):163, 2025\.
- \[4\]Luyu Gao, Aman Madaan, Shuyan Zhou, et al\.Pal: Program\-aided language models\.InInternational conference on machine learning, pages 10764–10799\. PMLR, 2023\.
- \[5\]Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen\.Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks\.Transactions on Machine Learning Research, 2023\.
- \[6\]Jason Wei, Xuezhi Wang, Dale Schuurmans, et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems, 35:24824–24837, 2022\.
- \[7\]Xingyao Wang, Yangyi Chen, Lifan Yuan, et al\.Executable code actions elicit better LLM agents\.InInternational Conference on Machine Learning\. PMLR, 2024\.
- \[8\]Benlu Wang, Iris Xia, Yifan Zhang, Junda Wang, Feiyun Ouyang, Shuo Han, Arman Cohan, Hong Yu, and Zonghai Yao\.From scores to steps: Diagnosing and improving LLM performance in evidence\-based medical calculations\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing \(EMNLP\), 2025\.arXiv:2509\.16584\.
- \[9\]Fenglin Liu, Jinge Wu, Hongjian Zhou, et al\.Riskagent: autonomous medical ai copilot for generalist risk prediction\.medRxiv 2025\.04\.03\.25323489; arXiv:2503\.03802, 2025\.
- \[10\]Niranjan Kumar, Farzaneh Seifi, Marisa Conte, and Allen J\. Flynn\.An LLM\-powered clinical calculator chatbot backed by verifiable clinical calculators and their metadata\.InAMIA Annual Symposium Proceedings, 2024\.PMID 41726491\.
- \[11\]Yakun Zhu, Shaohang Wei, Xu Wang, Kui Xue, Xiaofan Zhang, and Shaoting Zhang\.MeNTi: Bridging medical calculator and LLM agent with nested tool calling\.arXiv preprint arXiv:2410\.13610, 2024\.
- \[12\]Haotian Wang, Lian Yan, Xingzhi Yao, et al\.Medcalc\-r1: Knowledge\-guided reward framework for medical mathematical reasoning\.OpenReview preprint, 2026\.
- \[13\]Savyasachi V Shah\.Accuracy, consistency, and hallucination of large language models when analyzing unstructured clinical notes in electronic medical records\.JAMA Network Open, 7\(8\):e2425953, 2024\.
- \[14\]Nicholas Wan, Qiao Jin, Joey Chan, et al\.Humans and large language models in clinical decision support: A study with medical calculators\.arXiv preprint arXiv:2411\.05897, 2025\.
- \[15\]Qiao Jin, Zhizheng Wang, Yifan Yang, Qingqing Zhu, Donald Wright, Thomas Huang, Nikhil Khandekar, Nicholas Wan, Xuguang Ai, W John Wilbur, et al\.Agentmd: Empowering language agents for risk prediction with large\-scale clinical tool learning\.Nature Communications, 16\(1\):9377, 2025\.
- \[16\]Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin\.The art of abstention: Selective prediction and error regularization for natural language processing\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\), pages 1040–1051, 2021\.
- \[17\]Bingbing Wen, Jihan Yao, Shangbin Feng, et al\.Know your limits: A survey of abstention in large language models\.Transactions of the Association for Computational Linguistics, 13, 2025\.
- \[18\]David C\. Goff, Donald M\. Lloyd\-Jones, Glen Bennett, et al\.2013 ACC/AHA guideline on the assessment of cardiovascular risk\.Circulation, 129:S49–S73, 2014\.
- \[19\]Sadiya S\. Khan, Kunihiro Matsushita, Yingying Sang, et al\.Development and validation of the American Heart Association’s PREVENT equations\.Circulation, 149:430–449, 2024\.
- \[20\]Cynthia Delgado, Mukta Baweja, Deidra C\. Crews, et al\.A unifying approach for GFR estimation: Recommendations of the NKF\-ASN task force on reassessing the inclusion of race in diagnosing kidney disease\.American Journal of Kidney Diseases, 79:268–288, 2022\.
- \[21\]Lesley A\. Inker, Nwamaka D\. Eneanya, Josef Coresh, et al\.New creatinine\- and cystatin C\-based equations to estimate GFR without race\.New England Journal of Medicine, 385:1737–1749, 2021\.
- \[22\]W\. Ray Kim, Ajitha Mannalithara, Julie K\. Heimbach, et al\.MELD 3\.0: The model for end\-stage liver disease updated for the modern era\.Gastroenterology, 161:1887–1895, 2021\.
- \[23\]Mervyn Singer, Clifford S\. Deutschman, Christopher Warren Seymour, et al\.The third international consensus definitions for sepsis and septic shock \(Sepsis\-3\)\.JAMA, 315:801–810, 2016\.
- \[24\]Laura Evans, Andrew Rhodes, Waleed Alhazzani, et al\.Surviving sepsis campaign: International guidelines for management of sepsis and septic shock 2021\.Critical Care Medicine, 49\(11\):e1063–e1143, 2021\.PMID 34605781\.
- \[25\]Isabelle C\. Van Gelder, Michiel Rienstra, Karina V\. Bunting, et al\.2024 ESC guidelines for the management of atrial fibrillation developed in collaboration with the European Association for Cardio\-Thoracic Surgery \(EACTS\)\.European Heart Journal, 45:3314–3414, 2024\.
- \[26\]Jose A\. Joglar, Mina K\. Chung, Anastasia L\. Armbruster, et al\.2023 ACC/AHA/ACCP/HRS guideline for the diagnosis and management of atrial fibrillation: A report of the American College of Cardiology/American Heart Association joint committee on clinical practice guidelines\.Circulation, 149\(1\):e1–e156, 2024\.PMID 38033089\.
- \[27\]Noémie Desgagnés, James A\. King, Gregory A\. Kline, Isolde Seiden\-Long, and Alexander A\. Leung\.Use of albumin\-adjusted calcium measurements in clinical practice\.JAMA Network Open, 8\(1\):e2455251, 2025\.PMID 39836424\.
- \[28\]R\. B\. Payne, A\. J\. Little, R\. B\. Williams, and J\. R\. Milner\.Interpretation of serum calcium in patients with abnormal serum proteins\.British Medical Journal, 4:643–646, 1973\.
- \[29\]Scott M\. Grundy, Neil J\. Stone, Alison L\. Bailey, et al\.2018 AHA/ACC/AACVPR/AAPA/ABC/ACPM/ADA/AGS/APhA/ASPC/NLA/PCNA guideline on the management of blood cholesterol\.Circulation, 139:e1082–e1143, 2019\.
- \[30\]Seth S\. Martin, Michael J\. Blaha, Mohamed B\. Elshazly, et al\.Comparison of a novel method vs the Friedewald equation for estimating low\-density lipoprotein cholesterol levels from the standard lipid profile\.JAMA, 310:2061–2068, 2013\.
- \[31\]Maureen Sampson, Clarence Ling, Qian Sun, et al\.A new equation for calculation of low\-density lipoprotein cholesterol in patients with normolipidemia and/or hypertriglyceridemia\.JAMA Cardiology, 5:540–548, 2020\.
- \[32\]Pentti M\. Rautaharju, Borys Surawicz, Leonard S\. Gettes, et al\.AHA/ACCF/HRS recommendations for the standardization and interpretation of the electrocardiogram: Part IV: The ST segment, T and U waves, and the QT interval\.Circulation, 119\(10\):e241–e250, 2009\.PMID 19228821\.
- \[33\]Bert Vandenberk, Eline Vandael, Tomas Robyns, et al\.Which QT correction formulae to use for QT monitoring?Journal of the American Heart Association, 5:e003264, 2016\.
- \[34\]Manjunath P\. Pai and Frank P\. Paloucek\.The origin of the “ideal” body weight equations\.Annals of Pharmacotherapy, 34:1066–1069, 2000\.
- \[35\]Horacio J\. Adrogué and Nicolaos E\. Madias\.Hypernatremia\.New England Journal of Medicine, 342\(20\):1493–1499, 2000\.PMID 10816188\.
- \[36\]Teresa A\. Hillier, Robert D\. Abbott, and Eugene J\. Barrett\.Hyponatremia: evaluating the correction factor for hyperglycemia\.American Journal of Medicine, 106:399–403, 1999\.
- \[37\]Murray A\. Katz\.Hyperglycemia\-induced hyponatremia: calculation of expected serum sodium depression\.New England Journal of Medicine, 289:843–844, 1973\.
- \[38\]Jack E\. Zimmerman, Andrew A\. Kramer, Douglas S\. McNair, and Fern M\. Malila\.Acute physiology and chronic health evaluation \(APACHE\) IV: hospital mortality assessment for today’s critically ill patients\.Critical Care Medicine, 34:1297–1310, 2006\.
- \[39\]Joseph A Caprini\.Thrombosis risk assessment as a guide to quality patient care\.Disease\-a\-Month, 51\(2\-3\):70–78, 2005\.
- \[40\]MaryAnne Cronin, Nancy Dengler, Eugene S\. Krauss, Ayal Segal, Nancy Wei, et al\.Completion of the updated Caprini risk assessment model \(2013 version\)\.Clinical and Applied Thrombosis/Hemostasis, 25:1076029619838052, 2019\.PMID 30939900\.
- \[41\]Hude Quan, Bing Li, Chantal M\. Couris, et al\.Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries\.American Journal of Epidemiology, 173:676–682, 2011\.

## Appendix AAppendix

### A\.1Arm protocols and settings

Every arm calls the served model through one chat request per turn with only temperature and the output\-token limit set; Table[16](https://arxiv.org/html/2609.10728#A1.T16)lists every setting and executor limit, and the repository linked in the Conclusion holds the implementation of every arm\. All single\-call arms decode greedily \(temperature 0\); sampled candidates use temperature 0\.7\. Every arm runs five seeds \(42 to 46\); the vote and repair levers run on the covered 440 only\. The seed sets the case order and the run identity; it never reaches the request, so the main arms are five repeated greedy runs\. The worked example is a fixed benchmark case except in the note\-only arms, where the seed chooses it\. Repeated runs still differ through server batching: agreement across the five seeds is 78\.5 to 100% for the greedy arms and 55\.6 to 75\.2% where the example moves \(Table[24](https://arxiv.org/html/2609.10728#A1.T24)\)\. The benchmark’s test split circulates in three copies that disagree on a few dozen rows; every run read the Hugging Face parquet at revision 5488179, which the runner pins and downloads at start; the file itself is not redistributed with the code, and every number here is scored against that revision\.

#### Note only\.

One call: a step\-by\-step instruction, one worked example from another benchmark case \(its note cut to 24 words, its explanation to 40 words, and its answer; the example is chosen by the seed\), then the case with the first 120 words of its note\. Note only with gold variables \(the row labelled Note only with gold variables in Table[3](https://arxiv.org/html/2609.10728#A1.T3)\) adds the gold variable block to the same single call; the code has no second pass\.

#### Page without formula and Open\-book arithmetic\.

One greedy call with a zero\-shot chain\-of\-thought instruction, the gold variable block and the question; Page without formula reads the first 250 words of the note and omits the calculator’s formula text, Open\-book arithmetic reads the whole note and adds the formula text\.

#### Extract\-Solve and Gold\-Solve library\.

Extract\-Solve makes one greedy extraction call over the whole note, naming the calculator and listing the variables its hand\-written function needs, then runs that function; it abstains outside the 22 implemented calculators or when the extraction is invalid\. Gold\-Solve gives the same functions the gold variables\. Extract\-Solve library, recalled \(Table[3](https://arxiv.org/html/2609.10728#A1.T3)\) first samples three chain\-of\-thought answers at temperature 0\.7 from the Note only prompt \(one fixed worked example\); if at least two agree and none is empty, that answer is returned\. Otherwise, when the calculator is one of the 22, one greedy extraction call over the whole note feeds the hand\-written function; when it is not, the majority answer, or the first sample when there is no majority, is returned, so all arithmetic outside the library is the model’s\. It abstains only when a routed extraction is invalid or the function raises \(30 of 5,500 rows at 7B, 45 at 32B\) and costs three model calls per case, four when it routes \(35\.1% of rows at 7B, 29\.7% at 32B\)\.

#### Formulate\-Solve tree\.

One greedy call over the whole note, with no variables, calculator name or formula, asking for a structured reply with the calculation name, a one\-line formula, the numeric variables, a list of missing variables and an expression tree over basic arithmetic operators and comparisons; the tree is evaluated exactly\. It abstains when the tree or the variables are invalid, when a referenced variable is listed as missing, or when evaluation raises; the 60 date cases abstain before any call\.

#### Program\-Solve and Blind Program\-Solve\.

One greedy program call over the whole note, run by the executor at an output\-token budget of 2,048 for Program\-Solve and its syntax variant and 1,024 for Blind Program\-Solve and its vote/repair variants \(across the formula\-reading arms, at most 2\.1% of calls in any family end at the output limit, 0\.2% or fewer at 32B, and no truncated call scores correct\); the Program\-Solve prompt carries the formula text and the gold variables, the Blind Program\-Solve prompt neither\. The \+ syntax variants insert two lines before the instruction to return one code block, both about the language and the note, not the calculation: variable names must be valid Python identifiers \(lowercase words joined by underscores, never the calculation’s name\), and the notes write dates as month/day/year, so the program must build each date from the three numbers itself and add or subtract days with the standard date library\. Blind \+ vote samples five programs at temperature 0\.7, runs each, and returns the value that occurs at least twice \(ties go to the value sampled first\), else the first computed answer; it abstains if none runs\. Blind \+ repair makes one greedy program call and, if the executor returns no value, one corrective turn that shows the original prompt, the previous reply and the executor’s own error string, never anything about the calculation; one or two calls per case\. Blind \+ repair \+ vote combines both, five to eight calls per case\.

#### Scoring\.

A numeric answer is the first number on the reply’s final\-answer line and is correct when it lies in the benchmark’s own interval: gold plus or minus 5% for decimal outputs and exactly the gold for integers; a reply with no parseable number scores wrong\. Date answers are parsed to a calendar day and must match the gold day; gestational ages are reduced to their \(weeks, days\) integers and must match exactly\.

#### Covered\-subset runs\.

Blind Program\-Solve on the 440 covered cases is 39\.59 / 57\.55 in Table[1](https://arxiv.org/html/2609.10728#S3.T1), the covered subset of the full\-set run, and 39\.73 / 57\.59 in Table[4](https://arxiv.org/html/2609.10728#A1.T4), a separate run restricted to those cases with the same prompt; the per\-case answers agree on about 95% \(7B\) and at least 99\.8% \(32B\) of case\-seed pairs and the difference is re\-run noise\.

### A\.2Supplementary tables

Tables[3](https://arxiv.org/html/2609.10728#A1.T3)to[10](https://arxiv.org/html/2609.10728#A1.T10)and[14](https://arxiv.org/html/2609.10728#A1.T14)to[16](https://arxiv.org/html/2609.10728#A1.T16)are produced from the committed per\-case results by the analysis scripts in the repository; Tables[11](https://arxiv.org/html/2609.10728#A1.T11)to[13](https://arxiv.org/html/2609.10728#A1.T13)come from the calculator audit in the same repository\.

- •Table[3](https://arxiv.org/html/2609.10728#A1.T3): full ladders, four families, every arm that ran\.
- •Table[4](https://arxiv.org/html/2609.10728#A1.T4): blind Program\-Solve levers, vote and repair, on the covered 440\.
- •Table[5](https://arxiv.org/html/2609.10728#A1.T5): the syntax\-lines ablation\.
- •Table[6](https://arxiv.org/html/2609.10728#A1.T6): the 250\-word note\-budget ablation\.
- •Table[7](https://arxiv.org/html/2609.10728#A1.T7): every paired gap, with case and calculator\-cluster intervals\.
- •Table[8](https://arxiv.org/html/2609.10728#A1.T8): abstention accounting for the program arms\.
- •Table[9](https://arxiv.org/html/2609.10728#A1.T9): calculators the blind arm never gets right, with the formula text and the gold variables both withheld\.
- •Table[10](https://arxiv.org/html/2609.10728#A1.T10): one case, formula given against formula recalled\.
- •Table[11](https://arxiv.org/html/2609.10728#A1.T11): the 16 calculators with a concern, four of them replaced by a guideline, by concern type, with sources\.
- •Table[12](https://arxiv.org/html/2609.10728#A1.T12): the four headline gaps recomputed on the not\-replaced and no\-issue\-identified calculator sets\.
- •Table[13](https://arxiv.org/html/2609.10728#A1.T13): accuracy by audit group\.
- •Table[14](https://arxiv.org/html/2609.10728#A1.T14): right, wrong and no\-answer rates on the supported 440, the unsupported 660 and the full 1,100\.
- •Table[15](https://arxiv.org/html/2609.10728#A1.T15): library\-first hybrid baselines with paired intervals\.
- •Table[16](https://arxiv.org/html/2609.10728#A1.T16): reproducibility ledger, decoding, note budgets and executor limits\.
- •Table[20](https://arxiv.org/html/2609.10728#A1.T20): note budget against variable access\.
- •Table[21](https://arxiv.org/html/2609.10728#A1.T21): completeness audit of the 55 supplied formula texts\.
- •Table[23](https://arxiv.org/html/2609.10728#A1.T23): library\-first baselines with each fallback route\.
- •Table[24](https://arxiv.org/html/2609.10728#A1.T24): agreement across the five seeds, by arm\.

Figure 2:Coverage against full\-set accuracy for a library that abstains outside the calculators it implements\. The benchmark is balanced, so a library ofkkcalculators covers20​k20kof 1,100 cases and its full\-set accuracy cannot exceed that share; the 22\-calculator library sits at 40%\. Program\-Solve supports all 55 calculators and attempts every case, though it does not return a valid answer on every one \(Table[8](https://arxiv.org/html/2609.10728#A1.T8)\)\.Table 3:Accuracy in percent on the MedCalc\-Bench test split: mean over seeds, seed standard deviation in small type\. Unmarked cells are the full 1,100 cases;440marks the covered subset,1a single seed\.Table 4:Blind Program\-Solve levers on the covered 440, where every family has data\. Voting is over five sampled programs; the repair turn shows the executor error back to the model once\. Mean over seeds, seed standard deviation in small type\.Table 5:The two generic Python lines name no calculator, formula or clinical quantity: variable names must be valid identifiers, and the notes write dates as month/day/year\. Each column pairs the two arms on the same subset, the full 1,100 where both exist, else the covered 440 \(440\)\.Table 6:Note\-budget ablation on the full 1,100\. The extracting arms read the whole note by default, as the program arms and Open\-book arithmetic do; the ablation caps them at the 250 words Page without formula reads\. The one\-shot note arms read 120\. Table[20](https://arxiv.org/html/2609.10728#A1.T20)moves note budget and variable access separately\.GapnnppPerm\.ppHolmppCase CICluster CIICCneffn\_\{\\mathrm\{eff\}\}*Qwen2\.5\-7B*Prog\.\+syntax−\-Arith\.1,100\+5\.93<<0\.001<<0\.001\[3\.16,8\.69\]\[3\.16,\\,8\.69\]\[−0\.11,12\.55\]\[\-0\.11,\\,12\.55\]0\.23206Prog\.\+syntax−\-Gold lib\.1,100\+37\.95<<0\.001<<0\.001\[34\.56,41\.24\]\[34\.56,\\,41\.24\]\[25\.24,50\.38\]\[25\.24,\\,50\.38\]0\.6582Prog\.−\-Gold lib\.1,100\+35\.31<<0\.001<<0\.001\[31\.80,38\.73\]\[31\.80,\\,38\.73\]\[21\.87,48\.11\]\[21\.87,\\,48\.11\]0\.6879Prog\.−\-Arith\.1,100\+3\.290\.0210\.033\[0\.47,6\.13\]\[0\.47,\\,6\.13\]\[−3\.49,10\.38\]\[\-3\.49,\\,10\.38\]0\.25190Prog\.\+syntax−\-Prog\.1,100\+2\.640\.0160\.033\[0\.45,4\.80\]\[0\.45,\\,4\.80\]\[−0\.64,6\.29\]\[\-0\.64,\\,6\.29\]0\.10390Blind−\-Arith\.1,100\-43\.98<<0\.001<<0\.001\[−46\.95,−40\.93\]\[\-46\.95,\\,\-40\.93\]\[−53\.40,−34\.49\]\[\-53\.40,\\,\-34\.49\]0\.41124Blind−\-Extract lib\.1,100\-10\.80<<0\.001<<0\.001\[−14\.18,−7\.38\]\[\-14\.18,\\,\-7\.38\]\[−24\.35,2\.64\]\[\-24\.35,\\,2\.64\]0\.7374Blind−\-Recalled lib\.1,100\-18\.84<<0\.001<<0\.001\[−21\.85,−15\.84\]\[\-21\.85,\\,\-15\.84\]\[−29\.91,−8\.04\]\[\-29\.91,\\,\-8\.04\]0\.51103Blind\+vote−\-Blind440\+3\.82<<0\.0010\.002\[1\.64,6\.14\]\[1\.64,\\,6\.14\]\[0\.95,7\.05\]\[0\.95,\\,7\.05\]0\.05229*Qwen2\.5\-32B\-AWQ*Prog\.\+syntax−\-Gold lib\.1,100\+50\.96<<0\.001<<0\.001\[47\.95,53\.98\]\[47\.95,\\,53\.98\]\[38\.47,63\.09\]\[38\.47,\\,63\.09\]0\.8167Prog\.−\-Gold lib\.1,100\+50\.53<<0\.001<<0\.001\[47\.45,53\.55\]\[47\.45,\\,53\.55\]\[38\.00,62\.80\]\[38\.00,\\,62\.80\]0\.8167Prog\.\+syntax−\-Arith\.1,100\+7\.49<<0\.001<<0\.001\[5\.07,9\.98\]\[5\.07,\\,9\.98\]\[1\.00,15\.16\]\[1\.00,\\,15\.16\]0\.40128Prog\.−\-Arith\.1,100\+7\.05<<0\.001<<0\.001\[4\.71,9\.47\]\[4\.71,\\,9\.47\]\[0\.47,14\.60\]\[0\.47,\\,14\.60\]0\.43120Prog\.\+syntax−\-Prog\.1,100\+0\.440\.5250\.525\[−0\.84,1\.71\]\[\-0\.84,\\,1\.71\]\[−1\.38,2\.35\]\[\-1\.38,\\,2\.35\]0\.10390Blind−\-Arith\.1,100\-38\.76<<0\.001<<0\.001\[−41\.96,−35\.56\]\[\-41\.96,\\,\-35\.56\]\[−48\.40,−29\.40\]\[\-48\.40,\\,\-29\.40\]0\.43121Blind−\-Extract lib\.1,100\+5\.710\.0020\.004\[2\.11,9\.29\]\[2\.11,\\,9\.29\]\[−7\.96,18\.87\]\[\-7\.96,\\,18\.87\]0\.6978Blind−\-Recalled lib\.1,100\-5\.560\.0010\.003\[−8\.91,−2\.24\]\[\-8\.91,\\,\-2\.24\]\[−16\.33,4\.64\]\[\-16\.33,\\,4\.64\]0\.42123*Mistral\-7B\-v0\.3*Prog\.−\-Arith\.1,100\-7\.62<<0\.001<<0\.001\[−11\.20,−4\.05\]\[\-11\.20,\\,\-4\.05\]\[−16\.42,1\.40\]\[\-16\.42,\\,1\.40\]0\.27181Prog\.−\-Arith\.440\-4\.860\.1130\.113\[−10\.91,1\.05\]\[\-10\.91,\\,1\.05\]\[−19\.77,11\.09\]\[\-19\.77,\\,11\.09\]0\.3065Prog\.−\-Extract lib\.1,100\-3\.450\.0520\.104\[−7\.00,0\.04\]\[\-7\.00,\\,0\.04\]\[−15\.36,8\.16\]\[\-15\.36,\\,8\.16\]0\.50105Blind\+vote−\-Blind440\+8\.73<<0\.001<<0\.001\[6\.09,11\.41\]\[6\.09,\\,11\.41\]\[3\.36,14\.77\]\[3\.36,\\,14\.77\]0\.11144*Phi\-3\.5\-mini*Prog\.−\-Arith\.1,100\+4\.150\.0100\.010\[0\.96,7\.24\]\[0\.96,\\,7\.24\]\[−3\.56,12\.22\]\[\-3\.56,\\,12\.22\]0\.27181Prog\.−\-Arith\.440\-25\.05<<0\.001<<0\.001\[−30\.14,−20\.00\]\[\-30\.14,\\,\-20\.00\]\[−40\.09,−8\.27\]\[\-40\.09,\\,\-8\.27\]0\.4151Table 7:Every paired gap, solver minus reference\. Prog\. is Program\-Solve, Arith\. Open\-book arithmetic, Gold lib\., Extract lib\. and Recalled lib\. the Gold\-Solve, Extract\-Solve and recalled Extract\-Solve libraries, Blind the blind program arm\. Permutationppis a sign\-flip test on seed\-averaged per\-case differences, Holm\-adjusted within family; the cluster interval resamples calculators\.No valid answer, rowsAccuracy, %ArmSetSeedsNo prog\.RejectedRaisedBad kindUnread\.None %AllAtt\.Dates*Qwen2\.5\-7B*Blind Program\-SolveFull 1,1005384111591216813\.528\.0431\.9644\.33Program\-SolveFull 1,10057047245056\.775\.3180\.6233\.00Program\-Solve \+ syntaxFull 1,10050110128024\.477\.9581\.4759\.00Blind \+ voteCovered 44050001110\.643\.5543\.5750\.00Blind \+ repairCovered 4405001397607\.739\.7741\.8744\.67Blind \+ repair \+ voteCovered 4405000070\.343\.4143\.4148\.67*Qwen2\.5\-32B\-AWQ*Blind Program\-SolveFull 1,10053660290212\.644\.7145\.7571\.67Program\-SolveFull 1,10051530000\.790\.5391\.1293\.33Program\-Solve \+ syntaxFull 1,10050829000\.790\.9691\.58100\.00Blind \+ voteCovered 4405001010\.157\.7357\.7575\.33Blind \+ repairCovered 4405005050\.557\.4557\.5971\.67Blind \+ repair \+ voteCovered 4405000010\.157\.6857\.6873\.67*Mistral\-7B\-v0\.3*Blind Program\-SolveFull 1,100514734511594921839\.311\.1618\.2923\.67Program\-SolveFull 1,100596151128274531\.831\.4446\.0642\.33Program\-Solve \+ syntaxFull 1,10052421220711111245\.826\.9549\.5323\.33Blind \+ voteCovered 4405032010142\.121\.3621\.6929\.00Blind \+ repairCovered 4405349216129718\.414\.7718\.0323\.00Blind \+ repair \+ voteCovered 44050032140\.920\.9521\.0026\.67*Phi\-3\.5\-mini*Blind Program\-SolveFull 1,1005831585301358518\.023\.3627\.9725\.33Program\-SolveFull 1,10054127825414212815\.957\.1366\.1234\.00Program\-Solve \+ syntaxFull 1,10057821228113015615\.657\.2565\.6342\.33Blind \+ voteCovered 440501233292\.535\.4135\.8537\.00Blind \+ repairCovered 440501512441309\.631\.1433\.9130\.00Blind \+ repair \+ voteCovered 44050011411\.937\.4537\.4943\.67Table 8:Every case\-seed row of the program arms under the outcome taxonomy shared with Table[14](https://arxiv.org/html/2609.10728#A1.T14): no program written, sandbox rejection, raised error, wrong return kind, or a value the scorer could not read\. Counts pool the seeds shown\. Att\. is accuracy on attempted rows; Dates, on the 60 date cases\.CalculatorProgram rightBlind wrong%*Qwen2\.5\-7B*Delta Gap100100100Maintenance Fluids Calculations100100100Steroid Conversion Calculator1009595Adjusted Body Weight1009494QTc Rautaharju Calculator1009090QTc Fridericia Calculator1009999*Qwen2\.5\-32B\-AWQ*MDRD GFR Equation1009090Delta Gap1009595Albumin Corrected Anion Gap1009090CKD\-EPI Equations for Glomerular Filtration Rate1009090Albumin Corrected Delta Ratio100100100Delta Ratio100100100*Mistral\-7B\-v0\.3*Maintenance Fluids Calculations9696100QTc Fridericia Calculator827996QTc Bazett Calculator767599Estimated Date of Conception4444100QTc Framingham Calculator4343100Framingham Risk Score for Hard Coronary Heart Disease4242100*Phi\-3\.5\-mini*QTc Framingham Calculator969094Delta Ratio9494100Delta Gap908898Free Water Deficit747399Adjusted Body Weight686798Morphine Milligram Equivalents \(MME\) Calculator645992Table 9:Calculators the blind arm loses on every seed once the supplied formula and the gold variable list are both removed, execution held fixed: case\-seed pairs, five seeds, where the formula\-given program is correct and the blind one is not\. At least ten such pairs, top 6 per family\.Table 10:One case \(Creatinine Clearance \(Cockcroft\-Gault Equation\), seed 42, Qwen2\.5\-7B\), gold 40\.97\. The same model writes both programs and both implement Cockcroft\-Gault\. The upper row is given the calculator name, its formula, the gold variable list and two generic Python lines; the lower row none of them\.Table 11:Preliminary single\-annotator literature audit of the 55 calculators: the 16 with a concern, with formula, type and use, guideline body and date, concern and sources; the 39 others counted by use\. Gold answers are the calculator’s own output, so a replaced formula still scores correct\.Table 12:Exploratory: the four paired gaps on three calculator sets from Table[11](https://arxiv.org/html/2609.10728#A1.T11): all 55, 51 not replaced by a guideline, 39 with no issue\. Percentage points, five seeds, calculator\-cluster bootstrap; Gold\-Solve abstains outside its 440 cases\.Table 13:Exploratory: accuracy by audit group, full set, five seeds, for Open\-book arithmetic \(Arith\.\), Program\-Solve \(Prog\.\) and Blind Program\-Solve \(Blind\)\. Groups follow Table[11](https://arxiv.org/html/2609.10728#A1.T11): replaced by a guideline, any other concern, and no issue identified\. The replaced group is 4 calculators, so its numbers are indicative only\.Table 14:Outcome of every case\-seed evaluation, in percent, by whether the case’s calculator is one of the 22 the library implements\. None is no valid answer; the full\-set block splits it into no output, an executor failure, and a returned value the scorer could not read\. Five seeds pooled\.ReferenceAcc\.Gap, ppCase CICluster CIPerm\.pp*Qwen2\.5\-7B, Gold\-first, Program\-Solve fallback, accuracy 81\.64*Open\-book arithmetic72\.02\+9\.62\[6\.84,12\.40\]\[6\.84,\\,12\.40\]\[2\.78,17\.40\]\[2\.78,\\,17\.40\]<<0\.001Program\-Solve75\.31\+6\.33\[5\.00,7\.73\]\[5\.00,\\,7\.73\]\[2\.13,11\.93\]\[2\.13,\\,11\.93\]<<0\.001Gold\-Solve library40\.00\+41\.64\[38\.78,44\.49\]\[38\.78,\\,44\.49\]\[30\.98,52\.16\]\[30\.98,\\,52\.16\]<<0\.001*Qwen2\.5\-7B, Gold\-first, arithmetic fallback, accuracy 78\.56*Open\-book arithmetic72\.02\+6\.55\[5\.16,7\.98\]\[5\.16,\\,7\.98\]\[2\.27,11\.98\]\[2\.27,\\,11\.98\]<<0\.001Program\-Solve75\.31\+3\.25\[0\.38,6\.09\]\[0\.38,\\,6\.09\]\[−4\.45,11\.18\]\[\-4\.45,\\,11\.18\]0\.024Gold\-Solve library40\.00\+38\.56\[35\.76,41\.31\]\[35\.76,\\,41\.31\]\[28\.36,48\.96\]\[28\.36,\\,48\.96\]<<0\.001*Qwen2\.5\-7B, Extract\-first, Program\-Solve fallback, accuracy 51\.58*Open\-book arithmetic72\.02\-20\.44\[−23\.55,−17\.25\]\[\-23\.55,\\,\-17\.25\]\[−31\.15,−9\.76\]\[\-31\.15,\\,\-9\.76\]<<0\.001Program\-Solve75\.31\-23\.73\[−27\.11,−20\.31\]\[\-27\.11,\\,\-20\.31\]\[−35\.35,−11\.65\]\[\-35\.35,\\,\-11\.65\]<<0\.001Blind Program\-Solve28\.04\+23\.55\[21\.05,26\.07\]\[21\.05,\\,26\.07\]\[13\.98,33\.95\]\[13\.98,\\,33\.95\]<<0\.001Extract\-Solve library38\.84\+12\.75\[10\.95,14\.64\]\[10\.95,\\,14\.64\]\[6\.80,19\.87\]\[6\.80,\\,19\.87\]<<0\.001Extract\-Solve library, recalled46\.87\+4\.71\[2\.93,6\.56\]\[2\.93,\\,6\.56\]\[−0\.04,10\.27\]\[\-0\.04,\\,10\.27\]<<0\.001*Qwen2\.5\-7B, Extract\-first, arithmetic fallback, accuracy 78\.02*Open\-book arithmetic72\.02\+6\.00\[4\.62,7\.44\]\[4\.62,\\,7\.44\]\[1\.76,11\.35\]\[1\.76,\\,11\.35\]<<0\.001Blind Program\-Solve28\.04\+49\.98\[47\.02,52\.84\]\[47\.02,\\,52\.84\]\[40\.84,59\.15\]\[40\.84,\\,59\.15\]<<0\.001Extract\-Solve library38\.84\+39\.18\[36\.45,41\.98\]\[36\.45,\\,41\.98\]\[29\.09,49\.42\]\[29\.09,\\,49\.42\]<<0\.001*Qwen2\.5\-32B\-AWQ, Gold\-first, Program\-Solve fallback, accuracy 91\.07*Open\-book arithmetic83\.47\+7\.60\[5\.27,9\.98\]\[5\.27,\\,9\.98\]\[0\.93,15\.31\]\[0\.93,\\,15\.31\]<<0\.001Program\-Solve90\.53\+0\.55\[0\.18,1\.00\]\[0\.18,\\,1\.00\]\[0\.09,1\.27\]\[0\.09,\\,1\.27\]0\.032Gold\-Solve library40\.00\+51\.07\[48\.05,54\.02\]\[48\.05,\\,54\.02\]\[38\.78,63\.09\]\[38\.78,\\,63\.09\]<<0\.001*Qwen2\.5\-32B\-AWQ, Gold\-first, arithmetic fallback, accuracy 86\.89*Open\-book arithmetic83\.47\+3\.42\[2\.42,4\.53\]\[2\.42,\\,4\.53\]\[0\.40,7\.76\]\[0\.40,\\,7\.76\]<<0\.001Program\-Solve90\.53\-3\.64\[−5\.85,−1\.49\]\[\-5\.85,\\,\-1\.49\]\[−10\.49,2\.20\]\[\-10\.49,\\,2\.20\]<<0\.001Gold\-Solve library40\.00\+46\.89\[43\.91,49\.89\]\[43\.91,\\,49\.89\]\[35\.29,58\.53\]\[35\.29,\\,58\.53\]<<0\.001*Qwen2\.5\-32B\-AWQ, Extract\-first, Program\-Solve fallback, accuracy 60\.87*Open\-book arithmetic83\.47\-22\.60\[−25\.71,−19\.47\]\[\-25\.71,\\,\-19\.47\]\[−32\.15,−13\.05\]\[\-32\.15,\\,\-13\.05\]<<0\.001Program\-Solve90\.53\-29\.65\[−32\.71,−26\.69\]\[\-32\.71,\\,\-26\.69\]\[−39\.42,−20\.22\]\[\-39\.42,\\,\-20\.22\]<<0\.001Blind Program\-Solve44\.71\+16\.16\[14\.02,18\.36\]\[14\.02,\\,18\.36\]\[8\.35,24\.87\]\[8\.35,\\,24\.87\]<<0\.001Extract\-Solve library39\.00\+21\.87\[19\.45,24\.29\]\[19\.45,\\,24\.29\]\[14\.16,30\.25\]\[14\.16,\\,30\.25\]<<0\.001Extract\-Solve library, recalled50\.27\+10\.60\[8\.20,13\.04\]\[8\.20,\\,13\.04\]\[4\.89,17\.25\]\[4\.89,\\,17\.25\]<<0\.001*Qwen2\.5\-32B\-AWQ, Extract\-first, arithmetic fallback, accuracy 86\.71*Open\-book arithmetic83\.47\+3\.24\[2\.22,4\.35\]\[2\.22,\\,4\.35\]\[0\.31,7\.56\]\[0\.31,\\,7\.56\]<<0\.001Blind Program\-Solve44\.71\+42\.00\[38\.85,45\.09\]\[38\.85,\\,45\.09\]\[32\.85,51\.49\]\[32\.85,\\,51\.49\]<<0\.001Extract\-Solve library39\.00\+47\.71\[44\.71,50\.69\]\[44\.71,\\,50\.69\]\[36\.31,59\.27\]\[36\.31,\\,59\.27\]<<0\.001Table 15:Library\-first baselines assembled per case and seed\. Gold\-first answers with the Gold\-Solve library on its 440 cases; Extract\-first with the Extract\-Solve library whenever it answers\. Each comes with a Program\-Solve fallback and an Open\-book arithmetic fallback for the rest\. Full 1,100, five seeds; gap is baseline minus reference\.Table 16:Reproducibility ledger, run settings, from the run manifests\. One value spans the four families unless the row lists them; the model row names the Hugging Face id vLLM served\. Every arm makes one OpenAI\-compatible chat request per turn, setting only temperature and the output\-token cap\.Table 17:Reproducibility ledger, executor limits\. The executor runs each program in a fresh isolated interpreter process under the caps and allow\-lists listed; only the return value is read\.Category, case\-seed rowsSetRowsNeeds readingInvalid codeExecution failureAnswer\-format failureCorrect*Qwen2\.5\-7B*Supported 4402,200150019801,852Unsupported 6603,3008411174752,290*Qwen2\.5\-32B\-AWQ*Supported 4402,200502502,170Unsupported 6603,3004806502,809Table 18:Why Program\-Solve still fails when the formula and gold variables are given, audited on the corrected runs, five seeds: every case\-seed row by category\. Supported means the 440 library cases\.Annotated categoryQwen2\.5\-7BQwen2\.5\-32B\-AWQNeeds reading, sampled rows6060Incorrect formula translation54Wrong variables or units46 \(1\)32 \(8\)Wrong branch924*of which the supplied formula was underspecified*00Invalid code or execution failure, sampled rows2020Invalid code75Execution failure1315Table 19:The annotated sample behind the failure audit: a hash\-drawn stratified sample of the rows that needed reading and of the error rows, one category each, one annotator; uncertain counts in parentheses and sent to a second reader\.Accuracy %, gap ppHeld fixedComparisonLeftRightGapCase CICluster CIPerm\.pp*Qwen2\.5\-7B*budgetGold variables vs none, 250 words30\.0025\.53\+4\.47\[2\.95,6\.00\]\[2\.95,\\,6\.00\]\[1\.62,7\.80\]\[1\.62,\\,7\.80\]<<0\.001access250 vs 120 words, no variables25\.5319\.62\+5\.91\[4\.42,7\.49\]\[4\.42,\\,7\.49\]\[3\.02,9\.27\]\[3\.02,\\,9\.27\]<<0\.001budgetOpen\-book arithmetic, whole note, vs 250\-word one\-shot, no variables72\.0225\.53\+46\.49\[43\.75,49\.24\]\[43\.75,\\,49\.24\]\[38\.69,54\.29\]\[38\.69,\\,54\.29\]<<0\.001*Qwen2\.5\-32B\-AWQ*budgetGold variables vs none, 250 words48\.6940\.84\+7\.85\[6\.02,9\.69\]\[6\.02,\\,9\.69\]\[3\.87,12\.62\]\[3\.87,\\,12\.62\]<<0\.001access250 vs 120 words, no variables40\.8428\.67\+12\.16\[10\.29,14\.18\]\[10\.29,\\,14\.18\]\[8\.00,16\.58\]\[8\.00,\\,16\.58\]<<0\.001budgetOpen\-book arithmetic, whole note, vs 250\-word one\-shot, no variables83\.4740\.84\+42\.64\[39\.87,45\.45\]\[39\.87,\\,45\.45\]\[34\.40,51\.16\]\[34\.40,\\,51\.16\]<<0\.001*Mistral\-7B\-v0\.3*budgetGold variables vs none, 250 words18\.3317\.49\+0\.84\[−0\.64,2\.33\]\[\-0\.64,\\,2\.33\]\[−2\.31,3\.75\]\[\-2\.31,\\,3\.75\]0\.271access250 vs 120 words, no variables17\.4913\.35\+4\.15\[2\.89,5\.45\]\[2\.89,\\,5\.45\]\[1\.87,6\.78\]\[1\.87,\\,6\.78\]<<0\.001budgetOpen\-book arithmetic, whole note, vs 250\-word one\-shot, no variables39\.0517\.49\+21\.56\[18\.65,24\.47\]\[18\.65,\\,24\.47\]\[14\.18,28\.96\]\[14\.18,\\,28\.96\]<<0\.001*Phi\-3\.5\-mini*budgetGold variables vs none, 250 words27\.0722\.93\+4\.15\[2\.64,5\.67\]\[2\.64,\\,5\.67\]\[1\.55,7\.05\]\[1\.55,\\,7\.05\]<<0\.001access250 vs 120 words, no variables22\.9318\.13\+4\.80\[3\.29,6\.38\]\[3\.29,\\,6\.38\]\[2\.18,7\.62\]\[2\.18,\\,7\.62\]<<0\.001budgetOpen\-book arithmetic, whole note, vs 250\-word one\-shot, no variables52\.9822\.93\+30\.05\[27\.25,32\.98\]\[27\.25,\\,32\.98\]\[22\.47,37\.65\]\[22\.47,\\,37\.65\]<<0\.001Table 20:Note budget and variable access: rows one and two of each block move one factor, row three both; gap is left minus right\. Note only reads 120 or 250 words, with or without gold variables; the third row of each block instead sets Open\-book arithmetic, which always reads the whole note, against that 250\-word one\-shot arm without gold variables\. Full 1,100, five seeds; intervals as in Table[7](https://arxiv.org/html/2609.10728#A1.T7)\.Table 21:Completeness audit of the 55 supplied formula texts against the benchmark’s own worked solutions, part one: the 10 texts that could not reproduce the benchmark’s number, and the repair\. Clinical appropriateness is audited separately\.Table 22:Completeness audit, part two: the 18 texts that omitted a constant, unit, branch or convention the benchmark applies, and the repair; the 27 already complete are unchanged\.BaselineLibrary %ProgramArith\.Gap, ppCase CICluster CIPerm\.pp*Qwen2\.5\-7B*Gold\-first40\.081\.6478\.56\+3\.07\[0\.64,5\.55\]\[0\.64,\\,5\.55\]\[−2\.20,9\.15\]\[\-2\.20,\\,9\.15\]0\.014Extract\-first39\.451\.5878\.02\-26\.44\[−29\.05,−23\.78\]\[\-29\.05,\\,\-23\.78\]\[−34\.89,−18\.22\]\[\-34\.89,\\,\-18\.22\]<<0\.001*Qwen2\.5\-32B\-AWQ*Gold\-first40\.091\.0786\.89\+4\.18\[2\.11,6\.36\]\[2\.11,\\,6\.36\]\[−1\.60,10\.96\]\[\-1\.60,\\,10\.96\]<<0\.001Extract\-first39\.260\.8786\.71\-25\.84\[−28\.65,−23\.04\]\[\-28\.65,\\,\-23\.04\]\[−34\.25,−17\.91\]\[\-34\.25,\\,\-17\.91\]<<0\.001Table 23:The same library\-first baseline with each fallback, on the same cases and seeds: Program\-Solve minus Open\-book arithmetic\. Library share is the percentage of case\-seed rows the library answers; the fallback answers the rest\. Full 1,100, five seeds, intervals as in Table[7](https://arxiv.org/html/2609.10728#A1.T7)\.Table 24:The five seeds fix the case order, the checkpoint name and, in the marked arms, the worked example; no seed reaches the server and every arm here decodes greedily\. Agree: cases all five seeds score alike\. Pairwise: mean over the ten seed pairs\. Right, wrong: unanimous cases\.