Teaching LLMs to Recommend and Defer in Underrepresented Epilepsy Care

# Teaching LLMs to Recommend and Defer in Underrepresented Epilepsy Care
Source: [https://arxiv.org/html/2606.31036](https://arxiv.org/html/2606.31036)
Shreyas Rajesh1,\*,Kartik Sharma1,\*,Tonmoy Monsoor1,Mehmet Yigit Turali1,Richard Idro3,Juliana Kayaga3,Robert Sebunya4,Tracy Tushabe Namata3,Jessica Nichole Pasqua2,Vwani Roychowdhury1,Rajarshi Mazumder21Electrical and Computer Engineering, University of California, Los Angeles2Neurology, University of California, Los Angeles3Makerere University4St\. Francis Hospital Nsambya\*Equal contribution\.![[Uncaptioned image]](https://arxiv.org/html/2606.31036v1/figs/github_mark.png)GitHub:[github\.com/roychowdhuryresearch/Manana](https://github.com/roychowdhuryresearch/Manana)

###### Abstract

Specialist epilepsy expertise is scarce in resource\-constrained settings, making LLM\-based decision support attractive for frontline clinicians managing longitudinal treatment\. Such support systems must do more than apply medical knowledge: they must adapt to local prescribing practice and know when to defer\. Public medical AI benchmarks are dominated by high\-income clinical settings, leaving prescribing practices, medication availability, and follow\-up patterns in low\-resource contexts largely unrepresented\. We study this problem through a multidisciplinary collaboration in Ugandan pediatric epilepsy care\. The task is to predict anti\-seizure medication regimens from longitudinal unstructured notes collected by local clinicians across serial visits\. Standard prompting achieves non\-trivial agreement with physician prescriptions, but neurologists’ review of model reasoning traces shows that its errors stem from distribution\-miscalibrated prescribing defaults rather than the local care environment\. We introduceManana, a non\-parametric prompt\-learning framework that learns how to reason about local prescribing decisions from a small patient\-level training set\.Mananaturns observed prescription errors into an auditable prompt memory, instantiated in single\-agent and multi\-agent variants, and outperforms classical ML models, direct LLM prompting, and prompt\-optimization baselines across two independently collected Ugandan cohorts\. To make the system uncertainty\-aware, we propose Bayesian prompt averaging \(BPA\), a Bayesian model averaging procedure over the learned prompt trajectory\. This converts a sequence of learned prompts into prescription likelihoods and produces a deferral signal\. On the independently collected held\-out cohort, BPA improves visit\-level top\-3 prescription accuracy by4\-8 percentage pointsover the prompt\-optimization baselines\. More consequentially, it enables clinically meaningful selective prediction: the system can auto\-handle the most confident half of cases at95%precision, or the most confident quarter at99%precision, while deferring lower\-confidence cases for specialist review\. These results suggest a path toward locally adapted clinical LLM systems that learn from limited site\-specific data and reserve scarce specialist attention for the cases where uncertainty is highest\. Our code is available at: https://github\.com/roychowdhuryresearch/Manana

![Refer to caption](https://arxiv.org/html/2606.31036v1/x1.png)Figure 1:Mananaworkflow\. \(A\) During adaptation, each batch of patient records is processed with the current memory state: the Predictor proposes three regimens, the Inspector compares them with the physician prescription revealed as supervision, and the proposed candidate learnings accumulate in an append\-only buffer\. Once per batch, the Architect consolidates recurring cross\-case signals into the next shared memory, either as a Single correction\-rule list or a Multi specialist\-prompt population\. \(B\) At inference, the learned memory trajectory is treated as an ensemble of prompt states\. Bayesian prompt averaging converts their candidate regimen predictions into prescription probabilities, enabling high\-confidence assistance or deferral to a specialist\.## 1Introduction

Neurological conditions are a major and unevenly distributed source of global health burden\. More than three billion people live with neurological conditions worldwide, and over 80% of neurological deaths and health loss occur in low\- and middle\-income countries \(LMICs\)\(GBD 2021 Nervous System Disorders Collaborators,[2024](https://arxiv.org/html/2606.31036#bib.bib38)\)\. Epilepsy affects around 0\.8% of the global population \(∼\\sim50 million people\); nearly 80% live in LMICs, and many do not receive the treatment they need\(World Health Organization,[2024](https://arxiv.org/html/2606.31036#bib.bib39)\)\. Specialist epilepsy care remains scarce in many LMICs, where diagnosis and treatment fall to generalist providers\(World Health Organization and World Federation of Neurology,[2017](https://arxiv.org/html/2606.31036#bib.bib40)\)\. Longitudinal epilepsy medication management in these settings is challenging because clinicians must adjust treatment over time despite incomplete diagnostic information, limited EEG or imaging access, medication stock\-outs, comorbidities, adverse effects, and variable follow\-ups\. Frontline clinicians must decide when to initiate or switch anti\-seizure medications, escalate doses, manage breakthrough seizures or side effects, and identify patients who need referral to specialists in an overburdened system\. LLM\-based decision support is therefore attractive: a system that relies on local clinical notes and assists with treatment decisions could extend specialist\-informed reasoning to settings where such expertise is scarce\(Diessenet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib25)\)\.

Deployment, however, requires more than answering medical exam questions\. Medical LLM agents have largely been evaluated on standardized QA, diagnosis, or EHR\-operation benchmarks\(Kimet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib12); Jianget al\.,[2025](https://arxiv.org/html/2606.31036#bib.bib15)\)\. Moreover, medication recommendation systems typically use structured hospital data from high\-income countries with high\-resource settings such as MIMIC\(Shanget al\.,[2019](https://arxiv.org/html/2606.31036#bib.bib2); Yanget al\.,[2021b](https://arxiv.org/html/2606.31036#bib.bib3),[a](https://arxiv.org/html/2606.31036#bib.bib4); Sunet al\.,[2022](https://arxiv.org/html/2606.31036#bib.bib5); Fanet al\.,[2025](https://arxiv.org/html/2606.31036#bib.bib16); Johnsonet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib18)\)\. A realistic decision\-support system in an underrepresented clinic needs two additional properties\. First, its recommendations must be calibrated to local practice: drug availability, cost, and follow\-up norms\. Second, the system must surface uncertainty in medical decision\-making, because scarce specialist attention should be directed toward cases where the model is least reliable\.

We study these requirements in longitudinal epilepsy care in Uganda, where the task is to predict anti\-seizure medication regimens across serial clinical visits\.Here, we measure performance as agreement with the anti\-seizure medication \(ASM\) regimen selected by the treating physician\.Each prediction is grounded in the clinic notes available for that patient, rather than structured diagnostic codes or medication histories from a high\-resource EHR\. Within this setting, we test whether an LLM can recover local prescribing practice from clinician notes and perform reliably across serial visits\.

Direct prompting provides a useful baseline but not a deployable system: although a single\-agent LLM achieves non\-trivial agreement with physician prescriptions, neurologists’ audit shows systematic errors driven by Western prescribing priors rather than Ugandan clinical practice\. The problem is therefore not missing medical knowledge, but applying the wrong prior in the target health system\. We ask whether an LLM, without parametric weight updates, can learn deployment\-specific prescribing corrections from local cases and use the resulting adaptation trajectory to estimate uncertainty for deferral\. This constraint matters because new clinical deployments need adaptation to local data without the retraining, validation, and audit burden of weight updates, especially where data and compute are limited\.

We introduceManana, an error\-grounded prompt learning framework designed to address these questions\.Mananalearns from patient notes and physician prescriptions, without clinician\-written rules or expert review of model reasoning traces\.Mananais organized as a multi\-agent learning framework: one agent proposes medication regimens, a second agent analyzes errors against physician prescriptions, and a third agent consolidates recurring lessons into an interpretable prompt memory\. This loop turns observed failures into deployment\-specific reasoning guidance while retaining an auditable text representation that clinicians can review\. We present two variants:Manana\-Single, which learns a single shared set of learnings, andManana\-Multi, which instantiates specialist agents with learned prompts\.

To convert prompt learning into uncertainty\-aware decision support, we propose Bayesian prompt averaging \(BPA\): Bayesian model averaging over learned prompt trajectories\. Each learning round produces a distinct prompt state, and these prompt states form an ensemble of estimators over medication regimens\. Because each state reflects a different stage of evidence accumulation, the trajectory captures a range of plausible local prescribing rules learned from the calibration data\. Weighting these estimators by their posterior support yields prescription probabilities rather than only ranked predictions, enabling selective prediction: high\-confidence cases can proceed without additional specialist review, while low\-confidence cases can be deferred to clinicians for further review\. This concentrates scarce specialist attention on the cases where the system is least certain\. We summarize our contributions as follows:

1. 1\.We presentManana, a non\-parametric prompt\-learning framework that learns how to reason about local prescribing decisions from patient clinical notes and physician prescriptions, instantiated in single\-agent and multi\-agent variants\.
2. 2\.We show that, using only a small patient\-level training set,Mananaachieves strong held\-out performance across two independently collected Ugandan epilepsy cohorts, outperforming classical ML models, a single\-agent LLM, and existing prompt optimization methods\.
3. 3\.We further proposeBayesian prompt averaging \(BPA\), a Bayesian model averaging framework over learned prompt trajectories that produces prescription probabilities and deferral signals, allowing high\-confidence cases to proceed without additional specialist review while routing low\-confidence cases for further clinical review\.

## 2Manana

Mananaadapts an LLM without weight updates by learning an explicit memory state that conditions future predictions\. During the learning phase, the system observes a small set of local training cases, each pairing the patient history available before prescription with the regimen prescribed by the treating clinician\. In our experiments, this learning set contains 50 patients; Section[3](https://arxiv.org/html/2606.31036#S3)gives the full split and evaluation protocol\.

Mananais a multi\-agent learning system with three components\. The*Predictor*is the regimen\-producing LLM: given a patient history and the current memory state, it returns three candidate regimens over the 10\-drug clinic action space\. The*Inspector*compares those regimens with the clinician prescription and writes structured reports and candidate learnings\. The*Architect*aggregates evidence across candidate learnings and recent reports, then updates the memory state through the variant\-specific update rule\. The goal is to learn a sequence of memory statesm0,m1,…,mTm\_\{0\},m\_\{1\},\\ldots,m\_\{T\}that capture recurring deployment\-specific prescribing corrections from these examples\. All three components are LLM calls implemented with the same base model and role\-specific prompts, so learning proceeds through memory\-state updates rather than parameter updates\. Figure[1](https://arxiv.org/html/2606.31036#S0.F1)gives the end\-to\-end adaptation and inference workflow\.

### 2\.1Learning

We represent each clinic visit as\(xi,yi\)\(x\_\{i\},y\_\{i\}\), wherexix\_\{i\}is the cumulative patient history available before the prescribing decision andyiy\_\{i\}is the physician\-prescription\. Letmtm\_\{t\}denote the memory state after learning roundtt\. The learning process starts with an empty memory,m0=∅m\_\{0\}=\\emptyset\. In roundtt,Mananauses the current memorymt−1m\_\{t\-1\}to process a batch of clinic visitsℬt\\mathcal\{B\}\_\{t\}\. For each visit, Predictor returns

y^i,1:3=Pred​\(xi;mt−1\)\.\\hat\{y\}\_\{i,1:3\}=\\mathrm\{Pred\}\(x\_\{i\};m\_\{t\-1\}\)\.The Inspector then compares those three candidates with the physician prescription and outputs a structured reportρi\\rho\_\{i\}and one*candidate learning*cic\_\{i\}, a concise proposed lesson grounded in the note:

\(ρi,ci\)=Insp​\(xi,y^i,1:3,yi\)\.\(\\rho\_\{i\},c\_\{i\}\)=\\mathrm\{Insp\}\(x\_\{i\},\\hat\{y\}\_\{i,1:3\},y\_\{i\}\)\.Candidate learnings are not immediately added to memory\. They first enter an append\-only*evidence buffer*𝒞\\mathcal\{C\},

Δ​𝒞t=\(\(i,ci\)\)\(xi,yi\)∈ℬt,𝒞t=𝒞t−1∥Δ​𝒞t,𝒞0=∅,\\Delta\\mathcal\{C\}\_\{t\}=\\bigl\(\(i,c\_\{i\}\)\\bigr\)\_\{\(x\_\{i\},y\_\{i\}\)\\in\\mathcal\{B\}\_\{t\}\},\\qquad\\mathcal\{C\}\_\{t\}=\\mathcal\{C\}\_\{t\-1\}\\,\\\|\\,\\Delta\\mathcal\{C\}\_\{t\},\\qquad\\mathcal\{C\}\_\{0\}=\\emptyset,whereiiindexes the visit and∥\\\|denotes append\-only accumulation\. The current batch reports are

ℛt=\(\(i,ρi\)\)\(xi,yi\)∈ℬt\.\\mathcal\{R\}\_\{t\}=\\bigl\(\(i,\\rho\_\{i\}\)\\bigr\)\_\{\(x\_\{i\},y\_\{i\}\)\\in\\mathcal\{B\}\_\{t\}\}\.Unlike the evidence buffer,ℛt\\mathcal\{R\}\_\{t\}is not persistent: it is recomputed for each batch and gives the Architect local diagnostic context for the current batch\. The Architect then updates memory as

mt=Arch​\(mt−1,𝒞t,ℛt\),m\_\{t\}=\\mathrm\{Arch\}\(m\_\{t\-1\},\\mathcal\{C\}\_\{t\},\\mathcal\{R\}\_\{t\}\),with the exact update rule determined by the memory variant\. Section[2\.2](https://arxiv.org/html/2606.31036#S2.SS2)describes these update rules for the Single and Multi variants\.

The separation between candidate learnings and memory updates is one of our main distinctions from existing prompt\-optimization methods such as TextGrad\(Yuksekgonulet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib13)\): the difference is not textual feedback, but the unit of update\. TextGrad rewrites a mutable prompt variable via textual\-gradient descent, producing global rules whose support is not tracked rule\-by\-rule across casesMananaseparates error diagnosis from memory update: the Inspector writes candidate learnings, the evidence buffer preserves them with case provenance, and the Architect applies a constrained memory update defined by the chosen variant\. We return to the empirical implications of this design choice in Section[4](https://arxiv.org/html/2606.31036#S4)\.

### 2\.2Memory Updates

Mananauses the same Predictor–Inspector–Architect loop with two variants of Architect update\.

Single\.InManana\-Single, the memory state is a list ofLtL\_\{t\}learned rules,mt=\(r1,…,rLt\)m\_\{t\}=\(r\_\{1\},\\ldots,r\_\{L\_\{t\}\}\)\. The Architect either appends one new rule synthesized from recurring candidate learnings in the evidence buffer or leaves it unchanged:

mt∈\{mt−1,mt−1‖rnew\},\|mt\|−\|mt−1\|≤1\.m\_\{t\}\\in\\\{m\_\{t\-1\},\\,m\_\{t\-1\}\\\|r\_\{\\mathrm\{new\}\}\\\},\\qquad\|m\_\{t\}\|\-\|m\_\{t\-1\}\|\\leq 1\.Since the memory starts empty, this impliesLt≤tL\_\{t\}\\leq t\. The append action is allowed only when the proposed rule is supported by at leastN=2N=2learnings originating from distinct clinic visits\.

Multi\.InManana\-Multi, the memory state is a set of active specialist agents,

mt=\{sℓ,t:ℓ∈𝒜t\}\.m\_\{t\}=\\\{s\_\{\\ell,t\}:\\ell\\in\\mathcal\{A\}\_\{t\}\\\}\.Here𝒜t\\mathcal\{A\}\_\{t\}is the set of active specialist agents after learning roundtt, andsℓ,ts\_\{\\ell,t\}is the instruction for agentℓ\\ellat that round\. Each specialist agent is responsible for extracting one clinical signal from the patient history\. For a casexix\_\{i\}, the active specialist agents produce observations

oi,ℓ,t=sℓ,t​\(xi\),ℓ∈𝒜t\.o\_\{i,\\ell,t\}=s\_\{\\ell,t\}\(x\_\{i\}\),\\qquad\\ell\\in\\mathcal\{A\}\_\{t\}\.The Predictor receives these observations as case\-specific context and produces the regimen predictions\. The observations are not persistent memory; they are discarded after the case\. At each learning round, it may perform up to two actions chosen fromspawn,edit,prune, andNone\. Herespawnadds a new specialist agent,editrewrites an existing specialist instruction, andpruneremoves a specialist agent\. Spawn and edit actions require recurring support from distinct clinic visits, using candidate learnings in the evidence buffer together with the current batch reports\. Prune actions are used when current reports indicate that a specialist is redundant, misleading, or violating its observation\-only role\. If the specialist set is empty, the Architect may perform one bootstrapspawnafter the first batch so the Multi memory can begin\. In our experiments, we prompt the Architect with a target budget𝒜max=5\\mathcal\{A\}\_\{\\max\}=5to keep the specialist context small and inspectable, including an instruction to prune before spawning when the active set reaches this budget\. Full prompt templates are provided with the released code; representative learned artifacts are summarized in Appendix[M](https://arxiv.org/html/2606.31036#A13)\.

Both variants produce a memory trajectory rather than one final prompt\. Different memory states can make different correct predictions on different patients\. The next subsection uses this trajectory as the basis for uncertainty estimation\.

### 2\.3Bayesian Prompt Averaging

Clinical decision support requires a confidence score for each prediction, especially when specialist resources are sparse\. Bayesian model averaging \(BMA\) provides this by averaging predictions over candidate models according to posterior support\(MacKay,[1992](https://arxiv.org/html/2606.31036#bib.bib46); Hoetinget al\.,[1999](https://arxiv.org/html/2606.31036#bib.bib47)\)\. For candidate modelsm1,…,mTm\_\{1\},\\ldots,m\_\{T\}and data𝒟\\mathcal\{D\},

p​\(y∣x,𝒟\)=∑t=1Tp​\(y∣x,mt\)​p​\(mt∣𝒟\),p​\(mt∣𝒟\)=p​\(𝒟∣mt\)​p​\(mt\)∑q=1Tp​\(𝒟∣mq\)​p​\(mq\)\.p\(y\\mid x,\\mathcal\{D\}\)=\\sum\_\{t=1\}^\{T\}p\(y\\mid x,m\_\{t\}\)\\,p\(m\_\{t\}\\mid\\mathcal\{D\}\),\\qquad p\(m\_\{t\}\\mid\\mathcal\{D\}\)=\\frac\{p\(\\mathcal\{D\}\\mid m\_\{t\}\)p\(m\_\{t\}\)\}\{\\sum\_\{q=1\}^\{T\}p\(\\mathcal\{D\}\\mid m\_\{q\}\)p\(m\_\{q\}\)\}\.
BPA instantiates the candidate models as learned memory states\. The learning split𝒟train\\mathcal\{D\}\_\{\\mathrm\{train\}\}produces a trajectorym1,…,mTm\_\{1\},\\ldots,m\_\{T\}, where each state conditions the same Predictor and induces a different regimen predictor\. We use the held\-out validation set𝒟val\\mathcal\{D\}\_\{\\mathrm\{val\}\}to estimate posterior support over this finite learned trajectory, following validation\-based predictive averaging and stacking approaches that weight fixed predictors by held\-out or out\-of\-sample predictive performance\(Wolpert,[1992](https://arxiv.org/html/2606.31036#bib.bib48); Yaoet al\.,[2018](https://arxiv.org/html/2606.31036#bib.bib49)\)\. We retain the topK≤TK\\leq Tstates by validation marginal likelihood; settingK=TK=Trecovers the full trajectory\. After reindexing the retained states asm1,…,mKm\_\{1\},\\ldots,m\_\{K\}, we write

ℓk=log⁡p​\(𝒟val∣mk\),wk=exp⁡\{ℓk/τ\}∑q=1Kexp⁡\{ℓq/τ\},\\ell\_\{k\}=\\log p\(\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\mid m\_\{k\}\),\\qquad w\_\{k\}=\\frac\{\\exp\\\{\\ell\_\{k\}/\\tau\\\}\}\{\\sum\_\{q=1\}^\{K\}\\exp\\\{\\ell\_\{q\}/\\tau\\\}\},which is a temperature\-smoothed normalized posterior weight under a uniform prior over retained states\. BPA then computes

pBPA​\(y∣x\)=∑k=1Kwk​p​\(y∣x,mk\)\.p\_\{\\mathrm\{BPA\}\}\(y\\mid x\)=\\sum\_\{k=1\}^\{K\}w\_\{k\}\\,p\(y\\mid x,m\_\{k\}\)\.It remains to define the memory\-state predictive likelihoodp​\(y∣x,mk\)p\(y\\mid x,m\_\{k\}\)and the validation marginal likelihoodp​\(𝒟val∣mk\)p\(\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\mid m\_\{k\}\)\. For a new patient historyxx, the Predictor under memory statemkm\_\{k\}returns three candidate regimensy^k,1:3\\hat\{y\}\_\{k,1:3\}\. Since the LLM call does not return calibrated probabilities, we representp​\(y∣x,mk\)p\(y\\mid x,m\_\{k\}\)with an empirical\-Bayes\-style candidate\-position prior estimated from a small subset of the training set,π=\(0\.85,0\.11,0\.04\)\\pi=\(0\.85,0\.11,0\.04\)\(Robbins,[1956](https://arxiv.org/html/2606.31036#bib.bib50)\):

p​\(y∣x,mk\)=∑j=13πj​𝟏​\[y^k,j=y\]\.p\(y\\mid x,m\_\{k\}\)=\\sum\_\{j=1\}^\{3\}\\pi\_\{j\}\\mathbf\{1\}\[\\hat\{y\}\_\{k,j\}=y\]\.
Forp​\(𝒟val∣mk\)p\(\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\mid m\_\{k\}\), we run each retained memory state on every validation case and count position\-1 hitsck,1c\_\{k,1\}, position\-2\-or\-3 hitsck,\>1c\_\{k,\>1\}, and missesuku\_\{k\}, withhk=ck,1\+ck,\>1h\_\{k\}=c\_\{k,1\}\+c\_\{k,\>1\}andπ\>1=π2\+π3\\pi\_\{\>1\}=\\pi\_\{2\}\+\\pi\_\{3\}\. With aBeta​\(1,1\)\\mathrm\{Beta\}\(1,1\)prior on the top\-3 hit probability, the integrated validation likelihood is

p​\(𝒟val∣mk\)=π1ck,1​π\>1ck,\>1​B​\(hk\+1,uk\+1\)\.p\(\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\mid m\_\{k\}\)=\\pi\_\{1\}^\{c\_\{k,1\}\}\\pi\_\{\>1\}^\{c\_\{k,\>1\}\}B\(h\_\{k\}\+1,u\_\{k\}\+1\)\.Appendix[J](https://arxiv.org/html/2606.31036#A10)gives the derivation and reports alternatives that relax the shared candidate\-position prior\. Substituting the candidate\-position predictive likelihood gives

pBPA​\(y∣x\)=∑k=1Kwk​∑j=13πj​𝟏​\[y^k,j=y\]\.p\_\{\\mathrm\{BPA\}\}\(y\\mid x\)=\\sum\_\{k=1\}^\{K\}w\_\{k\}\\sum\_\{j=1\}^\{3\}\\pi\_\{j\}\\mathbf\{1\}\[\\hat\{y\}\_\{k,j\}=y\]\.For top\-1 evaluation, we report the regimen with maximum posterior predictive mass; for top\-3 evaluation, we report the three regimens with highest posterior predictive mass\. For selective prediction, the posterior mass assigned to a reported regimen is its confidence score, allowing low\-confidence cases to be routed for specialist review\.

## 3Experimental Setup

Clinical cohorts and preprocessing\.We study two independently collected pediatric epilepsy cohorts from Ugandan referral centers staffed by specialist pediatric neurologists: Cohort A contains 332 patients and 1,040 visits, and Cohort B contains 367 patients and 1,509 visits\. The records are clinician\-authored narrative outpatient notes from longitudinal epilepsy care, not structured high\-resource EHR records such as MIMIC\-IV\(Johnsonet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib18)\)\. They document serial medication\-management trajectories, including continuation, stopping, switching, adjunctive therapy, and escalation to polytherapy, rather than isolated one\-time drug choices\. Because regimen complexity varies across visits, the main analyses separatemonotherapyvisits, where the physician\-selected active regimen contains one ASM, frompolytherapyvisits, where it contains two or more ASMs\. The cohorts share a low\-resource clinical context but differ in documentation format, clinicians, patient characteristics, and prescribing patterns, supporting a transfer test across related\-but\-shifted environments\. LLM\-assisted preprocessing separated each visit into pre\-prescription clinical input and prescribed\-regimen target under a no\-leakage rule, with manual audit; Appendix[B](https://arxiv.org/html/2606.31036#A2)gives preprocessing details, cohort characteristics, regimen\-complexity distributions, and the 10\-drug clinic action space\. Main experiments use visits 1–3, where follow\-up remains sufficiently populated for stable cohort\-level estimates\.

Prediction task and metrics\.For each visit, each method receives the current pre\-prescription clinic note, prior visit notes, and prior prescribed regimens, then returns three possible anti\-seizure medication \(ASM\) regimens\. An ASM regimen is the active post\-visit set of anti\-seizure medications; evaluation uses only normalized active drug sets\. Thus, performance is measured as agreement with the ASM regimen selected by the treating physician on the ground\. The primary metric is exact match@3 \(EM@3\): a visit is correct if any proposed ASM regimen exactly matches the physician\-prescribed regimen after medication\-name normalization\. For ranked outputs, Top\-1 evaluates only the highest\-ranked regimen, while Top\-3/EM@3 evaluates whether any of the three returned regimens matches the physician\-prescribed regimen\. We also report maximum Jaccard similarity as partial\-credit set overlap, and stratify by physician\-prescribed monotherapy versus polytherapy because aggregate scores can hide different behavior on single\-drug and combination regimens\.

Baselines and comparators\.We compareMananaagainst standard prompting baselines, classical predictors trained on extracted clinical features, the EpiPick epilepsy rule\-based comparator for monotherapy, and prompt\-optimization baselines including TextGrad, ExpeL, and DSPy\(Asadi\-Pooyaet al\.,[2020](https://arxiv.org/html/2606.31036#bib.bib41);[EpiPick,](https://arxiv.org/html/2606.31036#bib.bib42); Yuksekgonulet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib13); Zhaoet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib11); Khattabet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib9)\); we optimize the DSPy program with the GEPA optimizer\(Agrawalet al\.,[2025](https://arxiv.org/html/2606.31036#bib.bib10)\)\. Full baseline definitions and implementation details are in Appendix[C](https://arxiv.org/html/2606.31036#A3)\.

Implementation details\.All learning methods train and validate on Cohort A, with Cohort B held out for transfer evaluation\. We use a fixed 70\-patient Cohort A learning pool; each of five seeds stratifies it into 50 training and 20 validation patients, with patient\-level separation\. Training processes 150 visits in 15 update rounds, and reported seeded results are mean±\\pmstandard deviation over seed\-level test scores\. Unless otherwise stated, experiments use OpenAI gpt\-oss\-120b\(OpenAI,[2025](https://arxiv.org/html/2606.31036#bib.bib65)\)as the base LLM\. Model inference ran on four NVIDIA RTX A6000 GPUs, with AWS Bedrock used for additional inference needs\. For BPA, validation scoring retainsK=5K=5memory states and usesτ=5\\tau=5; duplicate regimens are merged by posterior predictive mass\. All LLM\-based methods use the same output schema and regimen parser\. Full prompts are provided with the released code; additional model experiments are in Appendices[G](https://arxiv.org/html/2606.31036#A7)and[F](https://arxiv.org/html/2606.31036#A6)\.

## 4Results and Discussion

We evaluateMananaon the longitudinal ASM regimen prediction task defined in Section[3](https://arxiv.org/html/2606.31036#S3): methods learn from Cohort A training patients and are tested on held\-out Cohort A patients and the independently collected Cohort B\.Physician\-regimen agreement \(EM@3\)\(Table[1](https://arxiv.org/html/2606.31036#S4.T1)\):Mananaimproves over the Base Prompt, the doctor\-engineered Single\-agent prompt, and classical baselines across most visits and regimen strata, while remaining competitive with or stronger than prompt\-optimization methods\. The dominant difficulty is regimen complexity: monotherapy is consistently easier than polytherapy because exact\-match evaluation for a single drug only requires recovering one active medication, whereas polytherapy requires recovering the full drug combination\. Appendix[D](https://arxiv.org/html/2606.31036#A4)adds two interpretation checks: a previous\-regimen copy baseline tests whether high EM@3 mainly reflects unchanged prescriptions, and an outcome\-supported subset asks whether physician\-regimen agreement remains strong on visits associated with documented seizure reduction or seizure freedom\.Regularization effect ofManana: the empirical gap betweenMananaand prompt\-rewrite baselines is consistent with a difference in the learned artifact, not only a difference in the underlying LLM\. Candidate lessons must recur across patients before the Architect commits them, soManana\-Single learns compact evidence\-gated prescribing rules, whileManana\-Multi can spawn bounded specialist agents that surface recurring clinical signals\. Appendix[M](https://arxiv.org/html/2606.31036#A13)shows these artifacts side by side with TextGrad and ExpeL: TextGrad learns detailed but weakly grounded global rules, ExpeL learns useful but more generic memories, andMananalearns auditable rules or specialist signal extractors\.

Table 1:Main EM@3 results by cohort and regimen complexity\. Values are top\-3 exact match percentages; prompt\-optimization rows report mean±\\pmstd over independent seeds where applicable\. Light green rows markMananavariants; darker cells mark the best value in each column\.Uncertainty and deferral\(Tables[2](https://arxiv.org/html/2606.31036#S4.T2)and[3](https://arxiv.org/html/2606.31036#S4.T3), Figure[2](https://arxiv.org/html/2606.31036#S4.F2)\): We next evaluate whether the learned trajectory can support uncertainty\-aware decision support rather than only higher EM@3\. On the sameManana\-Multi trajectory, BPA separates ensemble coverage from uncertainty quality: majority vote achieves the same Top\-3 accuracy, but BPA improves Top\-1 accuracy and produces larger confidence gaps between correct and incorrect top\-1 predictions \(Table[2](https://arxiv.org/html/2606.31036#S4.T2)\)\. This confidence signal becomes a practical deferral policy in Table[3](https://arxiv.org/html/2606.31036#S4.T3): a clinic can choose a deferral operating point based on available specialist capacity, trading coverage for physician\-regimen agreement among non\-deferred cases\. For example, on Cohort B,Manana\-Multi reaches 99% top\-1 agreement on the most confident 25% of cases and 95% agreement on the most confident 50%\. Figure[2](https://arxiv.org/html/2606.31036#S4.F2)shows the corresponding confidence calibration and deferral curve\.

Table 2:Majority vote versus Beta\-Binomial BPA on theManana\-Multi trajectory\. Majority vote achieves the same Top\-3 accuracy, while BPA improves Top\-1 accuracy and confidence separation for selective prediction\.Table 3:Selective prediction fromManana\-Multi Beta\-Binomial BPA confidence\. Precision denotes top\-1 exact\-match accuracy within the retained subset\. Higher confidence thresholds produce higher precision at lower coverage, giving a practical deferral signal\.The reliability panels in Figure[2](https://arxiv.org/html/2606.31036#S4.F2)bin cases by BPA posterior confidence and compare each bin’s empirical top\-1 accuracy against the diagonal of perfect calibration\. BPA confidence tracks empirical accuracy across both cohorts and both learned\-prompt variants, with high\-confidence bins showing substantially higher physician\-regimen agreement than low\-confidence bins\. The right panel translates this separation into a selective\-prediction operating curve: precision is computed on the non\-deferred subset, while increasing deferral sends lower\-confidence cases to clinician review\.

![Refer to caption](https://arxiv.org/html/2606.31036v1/x2.png)Figure 2:BPA confidence and deferral behavior\. Dashed and solid curves denoteManana\-Single andManana\-Multi\. Left and middle: BPA confidence, defined as the final BPA probability assigned to the top\-ranked active ASM regimen, tracks empirical top\-1 physician\-regimen agreement; the dotted diagonal indicates perfect calibration\. Right: deferral curve obtained by deferring lower\-confidence cases to clinician review\. The marked Cohort B point shows that at 50% deferral,Manana\-Multi reaches 95% top\-1 agreement on the remaining cases\.Ablations, expert\-designed comparison, and clinician review\.We run extensive ablations to test whether the result depends on a single component or model choice: Appendix[H](https://arxiv.org/html/2606.31036#A8)ablates theMananaloop components, Appendix[F](https://arxiv.org/html/2606.31036#A6)tests transfer from the 120B model to the 20B model, Appendix[G](https://arxiv.org/html/2606.31036#A7)repeats the learning setup across LLM backbones, Appendix[J](https://arxiv.org/html/2606.31036#A10)varies the BPA weighting model, and Appendix[K](https://arxiv.org/html/2606.31036#A11)instantiates the method on MIMIC\-IV as a public\-data reproducibility check\. We also includeConsiliumas a strong expert\-designed reference system: neurologist review of single\-agent failures was used to define specialist clinical lenses, which were then implemented as a hand\-built multi\-agent comparator\. Appendix[E](https://arxiv.org/html/2606.31036#A5)describes the audit and specialist\-agent design, Appendix[L](https://arxiv.org/html/2606.31036#A12)tests whether the council is reducible, and Table[10](https://arxiv.org/html/2606.31036#A5.T10)comparesConsiliumwithMananaand BPA\. Appendix[I](https://arxiv.org/html/2606.31036#A9)adds a qualitative neurologist review ofManana\-Multi againstConsilium, focused on clinical coherence rather than another aggregate accuracy number\. Together, these comparisons show that specialist feedback can produce a strong hand\-designed system, butConsiliumdoes not learn from site\-specific supervision and does not by itself yield posterior regimen probabilities or a confidence\-based deferral policy\.

## 5Related Work

LLMs for clinical decision support\.Clinical LLM systems have been evaluated on medical QA, simulated EHR workflows, medication prediction, and epilepsy\-variable extraction\(Singhalet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib19); Noriet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib20); Kimet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib12); Jianget al\.,[2025](https://arxiv.org/html/2606.31036#bib.bib15); Kraljevicet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib21); Williamset al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib22); Fanget al\.,[2025](https://arxiv.org/html/2606.31036#bib.bib24)\), while ASM\-response and medication\-recommendation models typically use coded or structured EHR features\(Hakeemet al\.,[2022](https://arxiv.org/html/2606.31036#bib.bib6); Shanget al\.,[2019](https://arxiv.org/html/2606.31036#bib.bib2); Yanget al\.,[2021b](https://arxiv.org/html/2606.31036#bib.bib3); Sunet al\.,[2022](https://arxiv.org/html/2606.31036#bib.bib5)\)\. These systems do not learn site\-specific prescribing rules from a small clinic dataset, recover complete serial ASM regimens from narrative outpatient notes, or decide when to defer to scarce specialist review\.

Prompt optimization and self\-improving agents\.Prompt search, textual\-gradient methods, and reflection\-style agents adapt LLM behavior through text artifacts built from feedback\(Zhouet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib27); Yanget al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib28); Khattabet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib9); Yuksekgonulet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib13); Shinnet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib31); Zhaoet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib11)\); we compare directly against DSPy, TextGrad, and ExpeL\. For clinical prescription learning, the failure mode is the unit of update: salient individual cases can become brittle rules, and self\-critiques can reinforce wrong priors\(Sclaret al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib35); Huanget al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib36)\)\.Mananainstead separates error diagnosis from memory commitment, allowing the Architect to commit only recurring cross\-patient patterns\.

Uncertainty and selective prediction\.Prior work on calibration, Bayesian ensembling, conformal prediction, and learning\-to\-defer studies when models should abstain or hand off to humans\(Guoet al\.,[2017](https://arxiv.org/html/2606.31036#bib.bib43); Hoetinget al\.,[1999](https://arxiv.org/html/2606.31036#bib.bib47); Lakshminarayananet al\.,[2017](https://arxiv.org/html/2606.31036#bib.bib51); Shafer and Vovk,[2008](https://arxiv.org/html/2606.31036#bib.bib59); Madraset al\.,[2018](https://arxiv.org/html/2606.31036#bib.bib61)\)\. BPA uses a different source of uncertainty: the learned prompt trajectory itself, treating prompt states across rounds as local estimators whose posterior concentration gives a practical deferral signal\. We do not claim prospective conformal validity; we test whether these trajectories can route uncertain prescriptions to specialist review\. A more expansive version of the related work expanding on each of these areas is provided in Appendix[A](https://arxiv.org/html/2606.31036#A1)\.

## 6Limitations and Future Work

While our results show promise for using LLM systems to learn local prescribing patterns in low\-resource clinical settings, several limitations of the current study guide future work\. First, BPA gives a retrospective confidence score, not a deployment\-ready deferral rule\. The deferral thresholds in this paper are chosen on held\-out data to show the coverage–accuracy tradeoff\. In a clinic, the threshold would need to be chosen before use, checked over time as the patient mix changes, and tied to a clear review workflow for deferred cases\. The confidence scores and learned memories produced here would help such a workflow by identifying low\-confidence visits for specialist review and exposing the local prescribing rules the system is using\.

Second, our endpoint is agreement with the ASM regimen selected by the treating physician, not proof that the regimen is clinically optimal\. Local prescriptions reflect clinical judgment, but also formulary limits, availability, affordability, and follow\-up constraints\. A model that matches local practice may therefore also learn local resource constraints\. Future work should include a thorough clinical analysis of the memories and specialist agents learned byManana, as well as the reasoning produced by the resulting models\. A complementary direction, and part of our ongoing work, is to predict seizure freedom or seizure reduction directly from the longitudinal record, either alongside physician\-regimen agreement or as an outcome\-conditioned extension of the current framework\.

## Acknowledgments

This work was supported by Global Health Seed Grant, David Geffen School of Medicine, UCLA\. RM was supported by the Fogarty International Center of the National Institutes of Health under Award Number K01TW012178\. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health\.

## References

- L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang, C\. Potts, K\. Sen, A\. G\. Dimakis, I\. Stoica, D\. Klein, M\. Zaharia, and O\. Khattab \(2025\)GEPA: reflective prompt evolution can outperform reinforcement learning\.arXiv preprint arXiv:2507\.19457\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2507.19457)Cited by:[Appendix C](https://arxiv.org/html/2606.31036#A3.SS0.SSS0.Px6.p1.1),[§3](https://arxiv.org/html/2606.31036#S3.p3.1)\.
- A pragmatic algorithm to select appropriate antiseizure medications in patients with epilepsy\.Epilepsia61\(8\),pp\. 1668–1677\.External Links:[Document](https://dx.doi.org/10.1111/epi.16610)Cited by:[Appendix C](https://arxiv.org/html/2606.31036#A3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.31036#S3.p3.1)\.
- S\. Bates, A\. N\. Angelopoulos, L\. Lei, J\. Malik, and M\. I\. Jordan \(2021\)Distribution\-free, risk\-controlling prediction sets\.Journal of the ACM68\(6\),pp\. 1–34\.External Links:[Document](https://dx.doi.org/10.1145/3478535)Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- H\. Chenet al\.\(2025\)Large language models and global health equity: a roadmap for equitable adoption in LMICs\.The Lancet Regional Health – Western Pacific63,pp\. 101707\.External Links:[Document](https://dx.doi.org/10.1016/j.lanwpc.2025.101707)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1)\.
- X\. Chen, Y\. Jin, X\. Mao, L\. Wang, S\. Zhang, and T\. Chen \(2024\)RareAgents: autonomous multi\-disciplinary team for rare disease diagnosis and treatment\.arXiv preprint arXiv:2412\.12475\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2412.12475)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1)\.
- C\. K\. Chow \(1970\)On optimum recognition error and reject tradeoff\.IEEE Transactions on Information Theory16\(1\),pp\. 41–46\.External Links:[Document](https://dx.doi.org/10.1109/TIT.1970.1054406)Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- P\. V\. T\. Dat, L\. Doan, and H\. T\. T\. Binh \(2024\)HSEvo: elevating automatic heuristic design with diversity\-driven harmony search and genetic algorithm using LLMs\.arXiv preprint arXiv:2412\.14995\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2412.14995)Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1)\.
- E\. v\. Diessen, R\. A\. van Amerongen, M\. Zijlmans, and W\. M\. Otte \(2024\)Potential merits and flaws of large language models in epilepsy care: a critical review\.Epilepsia65\(4\),pp\. 873–886\.External Links:[Document](https://dx.doi.org/10.1111/epi.17907)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§1](https://arxiv.org/html/2606.31036#S1.p1.1)\.
- R\. El\-Yaniv and Y\. Wiener \(2010\)On the foundations of noise\-free selective classification\.Journal of Machine Learning Research11,pp\. 1605–1641\.Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- \[10\]EpiPickEpiPick: antiseizure medication selection tool\.Note:[https://epipick\.org/\#/](https://epipick.org/#/)Accessed May 6, 2026Cited by:[Appendix C](https://arxiv.org/html/2606.31036#A3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.31036#S3.p3.1)\.
- C\. Fan, C\. Gao, W\. Shi, Y\. Gong, Z\. Zhao, and F\. Feng \(2025\)Fine\-grained list\-wise alignment for generative medication recommendation\.arXiv preprint arXiv:2505\.20218\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.20218)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§1](https://arxiv.org/html/2606.31036#S1.p2.1)\.
- S\. Fang, B\. Holgate, A\. Shek, J\. S\. Winston, M\. McWilliam, P\. F\. Viana, J\. T\. Teo, and M\. P\. Richardson \(2025\)Extracting epilepsy\-related information from unstructured clinic letters using large language models\.Epilepsia66\(9\),pp\. 3369–3384\.External Links:[Document](https://dx.doi.org/10.1111/epi.18475)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p1.1)\.
- GBD 2021 Nervous System Disorders Collaborators \(2024\)Global, regional, and national burden of disorders affecting the nervous system, 1990–2021: a systematic analysis for the global burden of disease study 2021\.The Lancet Neurology23\(4\),pp\. 344–381\.External Links:[Document](https://dx.doi.org/10.1016/S1474-4422%2824%2900038-3)Cited by:[§1](https://arxiv.org/html/2606.31036#S1.p1.1)\.
- Y\. Geifman and R\. El\-Yaniv \(2017\)Selective classification for deep neural networks\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,pp\. 1321–1330\.Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p3.1)\.
- H\. Hakeem, W\. Feng, Z\. Chen, J\. Choong, M\. J\. Brodie, S\. Fong, K\. Lim, J\. Wu, X\. Wang, N\. Lawn, G\. Ni, X\. Gao, M\. Luo, Z\. Chen, Z\. Ge, and P\. Kwan \(2022\)Development and validation of a deep learning model for predicting treatment response in patients with newly diagnosed epilepsy\.JAMA Neurology79\(10\),pp\. 986–996\.External Links:[Document](https://dx.doi.org/10.1001/jamaneurol.2022.2514)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p1.1)\.
- J\. A\. Hoeting, D\. Madigan, A\. E\. Raftery, and C\. T\. Volinsky \(1999\)Bayesian model averaging: a tutorial\.Statistical Science14\(4\),pp\. 382–417\.External Links:[Document](https://dx.doi.org/10.1214/ss/1009212519)Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1),[§2\.3](https://arxiv.org/html/2606.31036#S2.SS3.p1.2),[§5](https://arxiv.org/html/2606.31036#S5.p3.1)\.
- B\. Holgate, S\. Fang, A\. Shek, M\. McWilliam, P\. Viana, J\. S\. Winston, J\. T\. Teo, and M\. P\. Richardson \(2024\)Extracting epilepsy patient data with Llama 2\.InProceedings of the 23rd Workshop on Biomedical Natural Language Processing,pp\. 526–535\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.bionlp-1.43),[Link](https://aclanthology.org/2024.bionlp-1.43/)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1)\.
- J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. W\. Yu, X\. Song, and D\. Zhou \(2024\)Large language models cannot self\-correct reasoning yet\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p2.1)\.
- Y\. Jiang, K\. C\. Black, G\. Geng, D\. Park, J\. Zou, A\. Y\. Ng, and J\. H\. Chen \(2025\)MedAgentBench: a realistic virtual EHR environment to benchmark medical LLM agents\.arXiv preprint arXiv:2501\.14654\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2501.14654)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§1](https://arxiv.org/html/2606.31036#S1.p2.1),[§5](https://arxiv.org/html/2606.31036#S5.p1.1)\.
- A\. E\. W\. Johnson, L\. Bulgarelli, L\. Shen, A\. Gayles, A\. Shammout, S\. Horng, T\. J\. Pollard, S\. Hao, B\. Moody, B\. Gow, L\. H\. Lehman, L\. A\. Celi, and R\. G\. Mark \(2023\)MIMIC\-IV, a freely accessible electronic health record dataset\.Scientific Data10\(1\),pp\. 1\.External Links:[Document](https://dx.doi.org/10.1038/s41597-022-01899-x)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§1](https://arxiv.org/html/2606.31036#S1.p2.1),[§3](https://arxiv.org/html/2606.31036#S3.p1.1)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Dodds, N\. DasSarma, E\. Tran\-Johnson,et al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. Potts \(2023\)DSPy: compiling declarative language model calls into self\-improving pipelines\.arXiv preprint arXiv:2310\.03714\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.03714)Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1),[§3](https://arxiv.org/html/2606.31036#S3.p3.1),[§5](https://arxiv.org/html/2606.31036#S5.p2.1)\.
- Y\. Kim, C\. Park, H\. Jeong, Y\. S\. Chan, X\. Xu, D\. McDuff, H\. Lee, M\. Ghassemi, C\. Breazeal, and H\. W\. Park \(2024\)MDAgents: an adaptive collaboration of LLMs for medical decision\-making\.arXiv preprint arXiv:2404\.15155\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2404.15155)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§1](https://arxiv.org/html/2606.31036#S1.p2.1),[§5](https://arxiv.org/html/2606.31036#S5.p1.1)\.
- B\. Kompa, J\. Snoek, and A\. L\. Beam \(2021\)Second opinion needed: communicating uncertainty in medical machine learning\.npj Digital Medicine4\(1\),pp\. 4\.External Links:[Document](https://dx.doi.org/10.1038/s41746-020-00367-3)Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- Z\. Kraljevic, J\. Au Yeung, D\. Bean, J\. Teo, and R\. J\. B\. Dobson \(2024\)Large language models for medical forecasting – Foresight 2\.arXiv preprint arXiv:2412\.10848\.Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p1.1)\.
- G\. Kumar \(2021\)Evaluation and management of drug resistant epilepsy in children\.Current Problems in Pediatric and Adolescent Health Care51\(7\),pp\. 101035\.External Links:[Document](https://dx.doi.org/10.1016/j.cppeds.2021.101035)Cited by:[§D\.2](https://arxiv.org/html/2606.31036#A4.SS2.p2.1)\.
- P\. Kwan, A\. Arzimanoglou, A\. T\. Berg, M\. J\. Brodie, W\. A\. Hauser, G\. Mathern, S\. L\. Moshé, E\. Perucca, S\. Wiebe, and J\. French \(2010\)Definition of drug resistant epilepsy: consensus proposal by the ad hoc task force of the ILAE commission on therapeutic strategies\.Epilepsia51\(6\),pp\. 1069–1077\.External Links:[Document](https://dx.doi.org/10.1111/j.1528-1167.2009.02397.x)Cited by:[§D\.2](https://arxiv.org/html/2606.31036#A4.SS2.p2.1)\.
- B\. Lakshminarayanan, A\. Pritzel, and C\. Blundell \(2017\)Simple and scalable predictive uncertainty estimation using deep ensembles\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p3.1)\.
- D\. J\. C\. MacKay \(1992\)A practical bayesian framework for backpropagation networks\.Neural Computation4\(3\),pp\. 448–472\.External Links:[Document](https://dx.doi.org/10.1162/neco.1992.4.3.448)Cited by:[§2\.3](https://arxiv.org/html/2606.31036#S2.SS3.p1.2)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. Clark \(2023\)Self\-refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1)\.
- D\. Madras, T\. Pitassi, and R\. Zemel \(2018\)Predict responsibly: improving fairness and accuracy by learning to defer\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p3.1)\.
- B\. A\. Mateen, V\. Menon, A\. Agweyu, R\. Korom, E\. Omoluabi, D\. McAfee, N\. Shimelash, S\. Rutunda, C\. Rugege, G\. Williams, M\. Emmanuel\-Fabula, A\. K\. Denniston, X\. Liu, and M\. Miles \(2025\)Trials for LLM\-supported clinical decisions in african primary healthcare\.Nature Medicine31,pp\. 2833–2835\.Note:CorrespondenceExternal Links:[Document](https://dx.doi.org/10.1038/s41591-025-03815-3)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1)\.
- H\. Mozannar and D\. Sontag \(2020\)Consistent estimators for learning to defer to an expert\.InProceedings of the 37th International Conference on Machine Learning,pp\. 7076–7087\.Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- H\. Nori, Y\. T\. Lee, S\. Zhang, D\. Carignan, R\. Edgar, N\. Fusi, N\. King, J\. Larson, Y\. Li, W\. Liu, R\. Luo, S\. M\. McKinney, R\. O\. Ness, H\. Poon, T\. Qin, N\. Usuyama, C\. White, and E\. Horvitz \(2023\)Can generalist foundation models outcompete special\-purpose tuning? Case study in medicine\.arXiv preprint arXiv:2311\.16452\.Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p1.1)\.
- OpenAI \(2025\)gpt\-oss\-120b & gpt\-oss\-20b model card\.Note:[https://openai\.com/index/gpt\-oss\-model\-card/](https://openai.com/index/gpt-oss-model-card/)Accessed May 6, 2026Cited by:[§3](https://arxiv.org/html/2606.31036#S3.p4.3)\.
- K\. Opsahl\-Ong, M\. J\. Ryan, J\. Purtell, D\. Broman, C\. Potts, M\. Zaharia, and O\. Khattab \(2024\)Optimizing instructions and demonstrations for multi\-stage language model programs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 9340–9366\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.525),[Link](https://aclanthology.org/2024.emnlp-main.525/)Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1)\.
- Y\. Ovadia, E\. Fertig, J\. Ren, Z\. Nado, D\. Sculley, S\. Nowozin, J\. V\. Dillon, B\. Lakshminarayanan, and J\. Snoek \(2019\)Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- R\. Pryzant, D\. Iter, J\. Li, Y\. Lee, C\. Zhu, and M\. Zeng \(2023\)Automatic prompt optimization with “gradient descent” and beam search\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 7957–7968\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.494),[Link](https://aclanthology.org/2023.emnlp-main.494/)Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1)\.
- M\. Raghu, K\. Blumer, R\. Sayres, Z\. Obermeyer, B\. Kleinberg, S\. Mullainathan, and J\. Kleinberg \(2019\)Direct uncertainty prediction for medical second opinions\.InProceedings of the 36th International Conference on Machine Learning,pp\. 5281–5290\.Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- H\. Robbins \(1956\)An empirical bayes approach to statistics\.InProceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Volume I: Contributions to the Theory of Statistics,pp\. 157–163\.Cited by:[§2\.3](https://arxiv.org/html/2606.31036#S2.SS3.p2.13)\.
- M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. Suhr \(2024\)Quantifying language models’ sensitivity to spurious features in prompt design or: how I learned to start worrying about prompt formatting\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p2.1)\.
- G\. Shafer and V\. Vovk \(2008\)A tutorial on conformal prediction\.Journal of Machine Learning Research9,pp\. 371–421\.Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p3.1)\.
- J\. Shang, C\. Xiao, T\. Ma, H\. Li, and J\. Sun \(2019\)GAMENet: graph augmented memory networks for recommending medication combination\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.33,pp\. 1126–1133\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v33i01.33011126)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§1](https://arxiv.org/html/2606.31036#S1.p2.1),[§5](https://arxiv.org/html/2606.31036#S5.p1.1)\.
- N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p2.1)\.
- K\. Singhal, S\. Azizi, T\. Tu, S\. S\. Mahdavi, J\. Wei, H\. W\. Chung, N\. Scales, A\. Tanwani, H\. Cole\-Lewis, S\. Pfohl, P\. Payne, M\. Seneviratne, P\. Gamble, C\. Kelly, A\. Babiker, N\. Schärli, A\. Chowdhery, P\. Mansfield, D\. Demner\-Fushman, B\. Agüera y Arcas, D\. Webster, G\. S\. Corrado, Y\. Matias, K\. Chou, J\. Gottweis, N\. Tomasev, Y\. Liu, A\. Rajkomar, J\. Barral, C\. Semturs, A\. Karthikesalingam, and V\. Natarajan \(2023\)Large language models encode clinical knowledge\.Nature620,pp\. 172–180\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06291-2)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p1.1)\.
- H\. Sun, S\. Xie, S\. Li, Y\. Chen, J\. Wen, and R\. Yan \(2022\)Debiased, longitudinal and coordinated drug recommendation through multi\-visit clinic records\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 27837–27849\.Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§1](https://arxiv.org/html/2606.31036#S1.p2.1),[§5](https://arxiv.org/html/2606.31036#S5.p1.1)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning \(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 5433–5442\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.330)Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- F\. Tonolini, N\. Aletras, J\. Massiah, and G\. Kazai \(2024\)Bayesian prompt ensembles: model uncertainty estimation for black\-box large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 12229–12272\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.728)Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- B\. Van Calster, D\. J\. McLernon, M\. van Smeden, L\. Wynants, and E\. W\. Steyerberg \(2019\)Calibration: the achilles heel of predictive analytics\.BMC Medicine17\(1\),pp\. 230\.External Links:[Document](https://dx.doi.org/10.1186/s12916-019-1466-7)Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2023\)Voyager: an open\-ended embodied agent with large language models\.arXiv preprint arXiv:2305\.16291\.Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1)\.
- C\. Y\. K\. Williams, B\. Y\. Miao, A\. E\. Kornblith, and A\. J\. Butte \(2024\)Evaluating the use of large language models to provide clinical recommendations in the emergency department\.Nature Communications15,pp\. 8236\.External Links:[Document](https://dx.doi.org/10.1038/s41467-024-52415-1)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p1.1)\.
- D\. H\. Wolpert \(1992\)Stacked generalization\.Neural Networks5\(2\),pp\. 241–259\.External Links:[Document](https://dx.doi.org/10.1016/S0893-6080%2805%2980023-1)Cited by:[§2\.3](https://arxiv.org/html/2606.31036#S2.SS3.p2.6)\.
- World Health Organization and World Federation of Neurology \(2017\)Atlas: country resources for neurological disorders\.2 edition,World Health Organization,Geneva\.External Links:ISBN 9789241565509,[Link](https://iris.who.int/handle/10665/258947)Cited by:[§1](https://arxiv.org/html/2606.31036#S1.p1.1)\.
- World Health Organization \(2024\)Epilepsy\.Note:Fact sheetPublished 7 February 2024\. Accessed 2 May 2026External Links:[Link](https://www.who.int/news-room/fact-sheets/detail/epilepsy)Cited by:[§1](https://arxiv.org/html/2606.31036#S1.p1.1)\.
- M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. Hooi \(2024\)Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs\.InInternational Conference on Learning Representations,Cited by:[§A\.3](https://arxiv.org/html/2606.31036#A1.SS3.p1.1)\.
- C\. Yang, C\. Xiao, L\. Glass, and J\. Sun \(2021a\)Change matters: medication change prediction with recurrent residual networks\.InProceedings of the Thirtieth International Joint Conference on Artificial Intelligence,pp\. 3728–3734\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2021/513)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§1](https://arxiv.org/html/2606.31036#S1.p2.1)\.
- C\. Yang, C\. Xiao, F\. Ma, L\. Glass, and J\. Sun \(2021b\)SafeDrug: dual molecular graph encoders for recommending effective and safe drug combinations\.InProceedings of the Thirtieth International Joint Conference on Artificial Intelligence,pp\. 3735–3741\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2021/514)Cited by:[§A\.1](https://arxiv.org/html/2606.31036#A1.SS1.p1.1),[§1](https://arxiv.org/html/2606.31036#S1.p2.1),[§5](https://arxiv.org/html/2606.31036#S5.p1.1)\.
- C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. Chen \(2024\)Large language models as optimizers\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p2.1)\.
- Y\. Yao, A\. Vehtari, D\. Simpson, and A\. Gelman \(2018\)Using stacking to average bayesian predictive distributions\.Bayesian Analysis13\(3\),pp\. 917–1007\.External Links:[Document](https://dx.doi.org/10.1214/17-BA1091)Cited by:[§2\.3](https://arxiv.org/html/2606.31036#S2.SS3.p2.6)\.
- M\. Yuksekgonul, F\. Bianchi, J\. Boen, S\. Liu, Z\. Huang, C\. Guestrin, and J\. Zou \(2024\)TextGrad: automatic “differentiation” via text\.arXiv preprint arXiv:2406\.07496\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2406.07496)Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1),[§2\.1](https://arxiv.org/html/2606.31036#S2.SS1.p2.1),[§3](https://arxiv.org/html/2606.31036#S3.p3.1),[§5](https://arxiv.org/html/2606.31036#S5.p2.1)\.
- A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\. Liu, and G\. Huang \(2024\)ExpeL: LLM agents are experiential learners\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19632–19642\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v38i17.29936)Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1),[§3](https://arxiv.org/html/2606.31036#S3.p3.1),[§5](https://arxiv.org/html/2606.31036#S5.p2.1)\.
- Y\. Zhou, A\. I\. Muresanu, Z\. Han, K\. Paster, S\. Pitis, H\. Chan, and J\. Ba \(2023\)Large language models are human\-level prompt engineers\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§A\.2](https://arxiv.org/html/2606.31036#A1.SS2.p1.1),[§5](https://arxiv.org/html/2606.31036#S5.p2.1)\.

## Appendix

Our appendix is structured as follows:

1. 1\.Appendix[A](https://arxiv.org/html/2606.31036#A1): Extended Related Work\.
2. 2\.Appendix[B](https://arxiv.org/html/2606.31036#A2): Clinical Cohorts and Distribution Shift\.
3. 3\.Appendix[C](https://arxiv.org/html/2606.31036#A3): Baseline Definitions\.
4. 4\.Appendix[D](https://arxiv.org/html/2606.31036#A4): Interpreting the Main\-Results Evaluation\.
5. 5\.Appendix[E](https://arxiv.org/html/2606.31036#A5): Single\-Agent Audit and Expert SystemConsilium\.
6. 6\.Appendix[F](https://arxiv.org/html/2606.31036#A6): Cross\-Model Transfer Learning\.
7. 7\.Appendix[G](https://arxiv.org/html/2606.31036#A7): Additional Open Source Model Experiments\.
8. 8\.Appendix[H](https://arxiv.org/html/2606.31036#A8):MananaComponent Ablations\.
9. 9\.Appendix[I](https://arxiv.org/html/2606.31036#A9): Clinician Review ofManana\.
10. 10\.Appendix[J](https://arxiv.org/html/2606.31036#A10):MananaBayesian Prompt Averaging Ablations\.
11. 11\.Appendix[K](https://arxiv.org/html/2606.31036#A11): MIMIC\-IV\.
12. 12\.Appendix[L](https://arxiv.org/html/2606.31036#A12):ConsiliumCouncil Ablations\.
13. 13\.Appendix[M](https://arxiv.org/html/2606.31036#A13): Learned Artifacts Across Optimization Methods\.

## Appendix AExtended Related Work

This appendix expands the related\-work discussion of Section[5](https://arxiv.org/html/2606.31036#S5), covering the broader literature in clinical LLM decision support, prompt optimization and self\-improving agents, and uncertainty quantification\.

### A\.1Clinical LLM Decision Support

Large language models have shown strong performance on medical QA, biomedical reasoning, and patient\-facing dialogue\[Singhalet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib19), Noriet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib20)\], and recent clinical\-agent systems extend this work to multi\-agent reasoning and simulated EHR workflows\[Kimet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib12), Jianget al\.,[2025](https://arxiv.org/html/2606.31036#bib.bib15), Chenet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib14)\]\. These systems are usually evaluated on benchmark answers, diagnostic tasks, or simulated actions rather than longitudinal prescribing decisions in a specific clinic\. More directly related work predicts medications from MIMIC clinical text\[Kraljevicet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib21), Johnsonet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib18)\], evaluates LLM recommendations from emergency\-department notes\[Williamset al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib22)\], or extracts epilepsy variables from clinic letters\[Holgateet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib23), Fanget al\.,[2025](https://arxiv.org/html/2606.31036#bib.bib24)\]; recent reviews discuss the potential uses and risks of LLMs in epilepsy care\[Diessenet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib25)\]\. Parallel ASM\-response and medication\-recommendation models predict seizure freedom, single\-drug response, or multi\-drug sets from structured clinical features or coded EHRs\[Hakeemet al\.,[2022](https://arxiv.org/html/2606.31036#bib.bib6), Shanget al\.,[2019](https://arxiv.org/html/2606.31036#bib.bib2), Yanget al\.,[2021b](https://arxiv.org/html/2606.31036#bib.bib3),[a](https://arxiv.org/html/2606.31036#bib.bib4), Sunet al\.,[2022](https://arxiv.org/html/2606.31036#bib.bib5), Fanet al\.,[2025](https://arxiv.org/html/2606.31036#bib.bib16)\]\. Recent global\-health CDS work motivates evaluation in underrepresented settings\[Mateenet al\.,[2025](https://arxiv.org/html/2606.31036#bib.bib26), Chen and others,[2025](https://arxiv.org/html/2606.31036#bib.bib37)\], but leaves open the regime studied here: learning site\-specific prescribing rules from a small clinic dataset, recovering complete serial ASM regimens from narrative outpatient notes, and identifying cases that should be deferred to scarce specialist review\.

### A\.2Prompt Optimization and Self\-Improving Agents

There has been extensive work on adapting LLM behavior through text artifacts created from feedback, rather than through weight updates\. Prompt\-optimization methods search or rewrite prompts against task feedback, including APE\[Zhouet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib27)\], OPRO\[Yanget al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib28)\], ProTeGi\[Pryzantet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib29)\], DSPy and MIPRO\[Khattabet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib9), Opsahl\-Onget al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib34)\], and TextGrad\[Yuksekgonulet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib13)\]\. Agent\-learning methods such as Reflexion and Self\-Refine\[Shinnet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib31), Madaanet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib33)\], Voyager\[Wanget al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib32)\], and ExpeL\[Zhaoet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib11)\]accumulate verbal lessons or skills from execution feedback\. Related evolutionary program\-search work studies a similar optimization pressure at the population level: HSEvo preserves candidate diversity when objective\-driven search would otherwise collapse onto narrow solutions\[Datet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib30)\]\. These systems show that textual feedback can improve future LLM calls, and we compare directly against DSPy, TextGrad, and ExpeL\. For clinical prescription learning, the central failure mode is the unit of update: prompt\-search and reflection methods can turn salient individual cases into broad rules, and self\-generated critiques may reinforce incorrect priors when the model is not externally grounded\[Sclaret al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib35), Huanget al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib36)\]\.Mananainstead separates error diagnosis from memory commitment: Inspectors write case\-level failures, but the Architect commits only recurring cross\-patient patterns, an evidence\-gated update that targets the clinical risk that a single unusual patient becomes a brittle prescribing rule\.

### A\.3Uncertainty Quantification and Selective Prediction

Reliable clinical decision support requires a model to know when not to act\. Neural models can be accurate while poorly calibrated, especially under dataset shift\[Guoet al\.,[2017](https://arxiv.org/html/2606.31036#bib.bib43), Ovadiaet al\.,[2019](https://arxiv.org/html/2606.31036#bib.bib44)\], and clinical prediction work treats calibration as a deployment requirement\[Van Calsteret al\.,[2019](https://arxiv.org/html/2606.31036#bib.bib45)\]\. Standard uncertainty methods marginalize over plausible models, including Bayesian model averaging and deep ensembles\[Hoetinget al\.,[1999](https://arxiv.org/html/2606.31036#bib.bib47), Lakshminarayananet al\.,[2017](https://arxiv.org/html/2606.31036#bib.bib51)\], while recent LLM work studies confidence elicitation, self\-evaluation, and prompt ensembles for black\-box models\[Kadavathet al\.,[2022](https://arxiv.org/html/2606.31036#bib.bib52), Tianet al\.,[2023](https://arxiv.org/html/2606.31036#bib.bib53), Xionget al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib54), Tonoliniet al\.,[2024](https://arxiv.org/html/2606.31036#bib.bib55)\]\. Selective prediction, reject\-option classification, conformal prediction, risk control, and learning\-to\-defer formalize when a system should abstain or hand off to a human expert\[Chow,[1970](https://arxiv.org/html/2606.31036#bib.bib56), El\-Yaniv and Wiener,[2010](https://arxiv.org/html/2606.31036#bib.bib57), Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2606.31036#bib.bib58), Shafer and Vovk,[2008](https://arxiv.org/html/2606.31036#bib.bib59), Bateset al\.,[2021](https://arxiv.org/html/2606.31036#bib.bib60), Madraset al\.,[2018](https://arxiv.org/html/2606.31036#bib.bib61), Mozannar and Sontag,[2020](https://arxiv.org/html/2606.31036#bib.bib62), Raghuet al\.,[2019](https://arxiv.org/html/2606.31036#bib.bib63), Kompaet al\.,[2021](https://arxiv.org/html/2606.31036#bib.bib64)\]\. BPA uses a different source of uncertainty: the learned prompt trajectory itself\. It treats prompt states across learning rounds as an empirical distribution over local estimators, combines their prescription distributions, and uses posterior concentration as a practical deferral signal\. We therefore do not claim prospective conformal validity; instead, we test whether self\-learned prompt trajectories can route uncertain medication predictions to specialist review\.

## Appendix BClinical Cohorts and Distribution Shift

#### Cohort description\.

This appendix provides source\-format, preprocessing, and distribution\-shift details for the two Ugandan cohorts introduced in Section[3](https://arxiv.org/html/2606.31036#S3)\. Cohort A contains 332 patients and 1,040 visits and was provided as a clinic spreadsheet with demographics, notes, medication fields, and follow\-up documentation distributed across columns\. Cohort B contains 367 patients and 1,509 visits and was provided as clinic PDF records\.

#### Preprocessing\.

Records were de\-identified before analysis\. The raw records did not consistently separate clinical observations from treatment plans\. We therefore used LLM\-assisted preprocessing to split each visit into clinical context and prescription text\. The splitter kept history, examination findings, seizure descriptions, current medications, treatment response, developmental information, and investigation results on the input side\. It assigned prescriptions, dose changes, drug starts or stops, investigation orders, referrals, and follow\-up plans to the output side\. The exact preprocessing prompts are provided with the released code documentation\.

#### Drug\-label extraction\.

We normalized each prescription into structured labels over the 10\-drug action space\. A pharmacist\-style extraction prompt mapped brand names, abbreviations, spelling variants, and local documentation variants to canonical drug names, and returned drugs prescribed and drugs stopped\. A separate cleaning pass merged duplicated prescription information across free\-text plans and structured medication fields while preserving the original clinical wording\.

#### Leakage controls\.

All preprocessing prompts used an explicit no\-leakage rule: treatment decisions from the current visit must not appear in the model input\. Prior prescriptions are included only for visits before the prediction visit, because they are part of the longitudinal history available to the physician\. We manually audited 200 patients of preprocessed records of Cohort A against the source notes and found the input\-output splits consistent with the intended prediction task\.

Table 4:Regimen complexity by cohort and visit for the main evaluation visits\. “None” indicates that no medication from the 10\-drug action space was extracted as prescribed at that visit\.Table[4](https://arxiv.org/html/2606.31036#A2.T4)summarizes regimen complexity for the main evaluation visits\. Visits in the “None” row have no extracted active ASM in the physician\-prescribed regimen and are excluded from EM@3 and Jaccard calculations\. Monotherapy and polytherapy strata are defined by the size of the physician\-prescribed active ASM set\. Table[5](https://arxiv.org/html/2606.31036#A2.T5)summarizes documented clinical characteristics in the two cohorts\. Figure[3](https://arxiv.org/html/2606.31036#A2.F3)reports the medication\-use fingerprint and documented\-feature distribution shift between cohorts\.

Table 5:Clinical characteristics of the two Ugandan epilepsy cohorts\. Entries are percentages computed from extracted cohort\-profile features\. Seizure\-type and seizure\-frequency rows use documented nonunknown values as denominators; missingness rows report the percentage of visits without the corresponding documented feature\.![Refer to caption](https://arxiv.org/html/2606.31036v1/x3.png)

![Refer to caption](https://arxiv.org/html/2606.31036v1/x4.png)

Figure 3:Medication\-use and documented\-feature distribution shift between cohorts\. Top: percentage of patients prescribed each anti\-seizure medication in Cohort A and Cohort B\. Bottom: symmetric KL divergence between Cohort A and Cohort B for clinical features \(purple\) and medication\-use variables \(orange\)\. Larger values indicate larger between\-cohort distribution shift\.Together, Table[4](https://arxiv.org/html/2606.31036#A2.T4)and Figure[3](https://arxiv.org/html/2606.31036#A2.F3)show that the two cohorts are related but shifted clinical environments\. Cohort B has a heavier polytherapy profile and a different medication\-use fingerprint, whereas Cohort A is more concentrated in carbamazepine\- and valproate\-centered prescribing\. We therefore report cohort\-specific results where relevant rather than treating the combined dataset as homogeneous\.

## Appendix CBaseline Definitions

This appendix first defines the comparator methods used in Table[1](https://arxiv.org/html/2606.31036#S4.T1)\. The baselines are ordered by what they test: non\-agentic clinical prediction, direct LLM prompting, and frozen\-model prompt optimization\. This sequence establishes how far simpler approaches go before the specialist audit and expert\-derived reference system described in Appendix[E](https://arxiv.org/html/2606.31036#A5)\. LLM prompts used for note cleaning, preprocessing, drug\-label extraction, and baseline feature extraction are provided with the released code documentation\.

#### EpiPick\.

EpiPick is an external web\-based antiseizure\-medication selection tool for seizure classification and medication monotherapy\[Asadi\-Pooyaet al\.,[2020](https://arxiv.org/html/2606.31036#bib.bib41),[EpiPick,](https://arxiv.org/html/2606.31036#bib.bib42)\]\. The app is designed for patients whose seizures begin at age 10 years or older; since no pediatric regimen\-selection baseline exists for this setting, we use it as the closest epilepsy\-specific rule\-based comparator\. To run EpiPick on our visits, we reimplemented the public client\-side rule bundle locally, then used an LLM extractor to produce the exact fields required by the algorithm: seizure type, age, gender, menopausal status, and documented modifiers for daily medication use, contraception, tumor, hepatic or renal failure, obesity, diabetes, bleeding risk, neutropenia, renal stones, drug allergy, depression, aggressive behavior, and migraine\. The local rule engine maps these extracted variables to EpiPick’s ranked medication groups\. We evaluate physician\-prescribed monotherapy visits only, restrict scoring to the 10 tracked drugs, and count a hit when the prescribed drug appears in EpiPick’s Group 1 recommendations\.

#### Naive Bayes\.

The Naive Bayes baseline is a trigram frequency\-table model over structured visit features and prescribed medication sets\. Before fitting the table, an LLM converts each visit into JSON fields for age, gender, seizure\-onset age, seizure frequency, cognitive priority, seizure type, and current\-medication flags for the 10 tracked anti\-seizure medications\. The trigram table conditions on seizure type and seizure frequency\. For each training\-cell, we estimate a smoothed Bernoulli probability for each drug using\(c\+0\.5\)/\(n\+1\.0\)\(c\+0\.5\)/\(n\+1\.0\), whereccis the number of visits in that cell whose physician\-prescribed regimen includes the drug andnnis the number of training visits in the cell\. At prediction time, the model uses the most specific cell with at least five training visits, backing off from seizure type plus seizure frequency to seizure type alone, then to the unigram table, and finally to marginal drug probabilities\. Let𝒟\\mathcal\{D\}be the 10 tracked drugs and letpdp\_\{d\}be the selected table’s probability for drugdd\. For every candidate regimenS⊆𝒟S\\subseteq\\mathcal\{D\}, the model assigns

score​\(S\)=∑d∈Slog⁡pd\+∑d∈𝒟∖Slog⁡\(1−pd\)\.\\mathrm\{score\}\(S\)=\\sum\_\{d\\in S\}\\log p\_\{d\}\+\\sum\_\{d\\in\\mathcal\{D\}\\setminus S\}\\log\(1\-p\_\{d\}\)\.It enumerates all2102^\{10\}candidate regimens and returns the three highest\-scoring sets\.

#### XGBoost clinical\.

The XGBoost clinical baseline tests whether a supervised tabular model can use the same structured visit representation without free\-text reasoning\. We first run the LLM feature extractor on each visit and retain six clinical variables: age, gender, seizure\-onset age, seizure frequency, cognitive priority, and seizure type\. Missing values are imputed by the feature mean\. On the same patient\-level Cohort A training split used by the learning baselines, we train 10 independent binary XGBoost classifiers, one per tracked drug\. For drugdd, the label isyd=1y\_\{d\}=1if the physician\-prescribed regimen includesddand0otherwise\. The implementation uses fixed parameters: 200 trees, maximum depth 3, learning rate 0\.05, and 0\.8 row and column subsampling\. At test time, the classifiers produce probabilitiespd=P​\(yd=1∣x\)p\_\{d\}=P\(y\_\{d\}=1\\mid x\)for all 10 drugs\. We then score every candidate regimenS⊆𝒟S\\subseteq\\mathcal\{D\}with the same independent\-Bernoulli objective used for Naive Bayes,

score​\(S\)=∑d∈Slog⁡pd\+∑d∈𝒟∖Slog⁡\(1−pd\),\\mathrm\{score\}\(S\)=\\sum\_\{d\\in S\}\\log p\_\{d\}\+\\sum\_\{d\\in\\mathcal\{D\}\\setminus S\}\\log\(1\-p\_\{d\}\),and return the three highest\-scoring regimens\.

#### Base Prompt \(0\-learning\)\.

The Base Prompt is the unadapted predictor prompt used to initializeMananaand the prompt\-optimization baselines; the full template is provided with the released code\. This row evaluates that initialization with the learning slot empty: no Inspector feedback, Architect updates, learned rules, specialist analyses, or optimized instructions are supplied\. Each visit uses the same longitudinal patient\-history input format as the adapted prompt methods\. All prompt\-based methods are parsed with the same regimen parser; EM@3 counts the visit as correct if any of the three returned drug sets exactly matches the physician\-prescribed set\. This row isolates what the shared task scaffold achieves before adaptation\.

#### Single\-agent\.

The Single\-agent baseline is the hand\-written direct\-prompting comparator for this task; the full prompt is provided with the released code\. It was written and refined with clinician input, uses the same longitudinal patient\-history input and 10\-drug regimen output space as the other prompt\-based methods, and performs no learning or optimization\.

#### DSPy\.

DSPy is a prompt\-compilation baseline\. We instantiate the same clinical task as a DSPy signature with one input field for the longitudinal visit history and three regimen\-output fields over the 10 tracked drugs, then optimize the DSPy program with the GEPA optimizer\[Agrawalet al\.,[2025](https://arxiv.org/html/2606.31036#bib.bib10)\]\. The optimizer uses the same Cohort A training and validation split and the same top\-3 exact\-match objective used for the other prompt\-based methods\. DSPy therefore tests whether framework\-level prompt compilation can improve the shared task scaffold without an explicit clinical memory, Inspector–Architect loop, or cross\-patient evidence buffer\.

#### ExpeL\.

ExpeL is an experience\-learning agent baseline run through the official ExpeL framework with aConsiliumtask environment\. It uses the same 50\-patient Cohort A training split, 20\-patient validation split, 10\-drug action space, and held\-out evaluation protocol asManana\. Each episode asks the agent to reason over a visit and finish with three ranked regimens\. ExpeL then converts prior episodes into natural\-language insights that condition later predictions\. We report mean±\\pmstandard deviation across repeated official\-code runs\.

#### TextGrad\.

TextGrad is a textual\-gradient prompt\-optimization baseline initialized from the same 0\-learning predictor scaffold\. The clinical role, visit input, 10\-drug output space, and regimen format are fixed; only the learned\-instruction slot is optimized\. For each batch, TextGrad runs the predictor, asks an LLM error\-diagnosis prompt to compare the prediction with the physician regimen, and applies textual\-gradient descent to rewrite the instruction variable\. We use the same Cohort A 50/20 split, batch size 10, 15 update rounds, validation gating, and five\-seed reporting protocol asManana\.

## Appendix DInterpreting the Main\-Results Evaluation

This appendix reports two interpretive checks on the Table[1](https://arxiv.org/html/2606.31036#S4.T1)headline numbers\. Section[D\.1](https://arxiv.org/html/2606.31036#A4.SS1)asks whether longitudinal EM@3 gains reflect more than carrying the previous regimen forward, and Section[D\.2](https://arxiv.org/html/2606.31036#A4.SS2)asks whether physician agreement is most clinically meaningful when corroborated by the patient’s seizure trajectory\.

### D\.1Is the model just copying the previous prescription?

A natural concern with longitudinal regimen prediction is that physician\-agreement metrics conflate*continuation*\(recognizing that the current regimen should be carried forward\) and*revision*\(deciding when and how to change it\)\. Because V2/V3 EM@3 is consistently higher than V1 and every method sees the prior visit’s prescription, a simple explanation is that models copy the previous regimen rather than learn a clinically meaningful prescribing policy\.

We test this directly by introducing a*previous\-regimen\-copy*baseline that ignores the clinical note entirely and predicts the active drug set from the prior visit\. We then evaluate every method on two slices: \(i\) all V2/V3 visits, and \(ii\) the*change\-visit*subset, defined as V2/V3 visits where the physician\-prescribed active drug set differs from the previous visit\. A method that has only learned continuation should match the copy baseline on \(i\) and collapse to near zero on \(ii\)\.

Table 6:Longitudinal continuity baseline and change\-visit performance\. Previous\-regimen copy uses the prior visit’s active drug set\. Change visits are V2/V3 cases where the physician\-prescribed active drug set differs from the previous visit\.Table[6](https://arxiv.org/html/2606.31036#A4.T6)shows that the previous\-regimen\-copy baseline is strong at the aggregate level — 85\.1% on Cohort A and 76\.6% on Cohort B — so any single\-number later\-visit EM@3 must be read against this continuation floor\. YetManana\-Multi reaches 81\.7% \(Cohort A\) and 83\.5% \(Cohort B\) overall while recovering 39\.0% and 55\.6% of regimens on change visits, where copy accuracy is 0% by construction\. On Cohort B, the independently collected held\-out cohort, it is also the only method to beat the copy baseline outright\. Methods that look competitive on aggregate EM@3 \(e\.g\. TextGrad,Manana\-Single\) have markedly lower change\-visit accuracy, especially on Cohort B \(41\.7% and 47\.3% vs\. 55\.6%\)\. We read this as evidence thatManana\-Multi has learned a prescribing policy with a non\-trivial revision component, not only a continuation prior\.

### D\.2Clinical validity of EM@3 under outcome\-stratified evaluation

Physician\-agreement metrics treat the prescribed regimen as ground truth, but the clinical validity of that label varies across visits\. When seizure burden decreases under the current regimen, the next prescription is outcome\-validated by the patient’s trajectory\. When seizures increase, the prior regimen has failed and the revised regimen is an unvalidated re\-attempt; high agreement may indicate agreement with a clinical guess rather than clinical correctness\. We therefore interpret EM@3 as a quality signal only where the physician’s prescription is followed by confirmed reduction in seizure burden\.

We use the seizure\-frequency annotations available in Cohort A \(‘Seizure frequency’ for V2 and for V3\) to identify this outcome\-validated subset\. We define*Improved*as V2/V3 visits annotated ‘Reduced’ or ‘Seizure\-free’ — visits where the physician’s choice at this encounter is corroborated by the patient’s clinical trajectory\. This subset contains 342 visits, or 92% of the V2/V3 visits in Cohort A for which a seizure\-frequency annotation is recorded\. Visits annotated ‘Unchanged’ or ‘Other’ are similarly excluded as ambiguous\. Drug\-resistant epilepsy — patients who fail to achieve seizure freedom after adequate trials of two tolerated, appropriately chosen and used antiepileptic drug schedules\[Kwanet al\.,[2010](https://arxiv.org/html/2606.31036#bib.bib7)\]— is well documented in pediatric epilepsy populations\[Kumar,[2021](https://arxiv.org/html/2606.31036#bib.bib8)\], and a proper evaluation on these patients requires longer follow\-up, multi\-trial outcome data, and likely a reformulated label\.

Table 7:EM@3 on the outcome\-validated Cohort A V2/V3 subset \(‘Improved’ = ‘Reduced’ or ‘Seizure\-free’ on the per\-visit seizure\-frequency annotation\)\.Table[7](https://arxiv.org/html/2606.31036#A4.T7)shows that, on this outcome\-validated subset,Manana\-Multi achieves 86\.0% EM@3, outperforming the frozen single\-agent baseline \(80\.7%\) and the two prompt\-learning baselines \(TextGrad 79\.7%,Manana\-Single 82\.3%\)\. This is the cleanest physician\-agreement claim available from this dataset: when the prescribed regimen is corroborated by reduced seizure burden,Manana\-Multi recovers it most often\. We do not claim a corresponding ranking on visits where seizures worsened, because agreement on those visits is not interpretable as a quality signal\.

## Appendix ESingle\-Agent Audit and Expert SystemConsilium

This appendix explains the bridge from direct prompting to the expert\-designedConsiliumreference system\. Section[E\.1](https://arxiv.org/html/2606.31036#A5.SS1)describes the neurologist audit of single\-agent failures, and Section[E\.2](https://arxiv.org/html/2606.31036#A5.SS2)describes the resultingConsiliumcomparator\. The single\-agent baseline reached non\-trivial accuracy, but neurologist review showed that its errors were not random: they recurred along specific clinical lenses\. We used that audit to buildConsiliumas the expert\-designed comparator that tests whether expert decomposition can correct those errors\.

### E\.1Single\-Agent Failure Audit

The single\-agent LLM was a useful starting point because it often produced plausible epilepsy reasoning, but its errors exposed a pattern that direct prompting did not address\. Neurologist review showed that the misses clustered around local and longitudinal details: active\-medication timelines, seizure classification, pediatric context, drug access, escalation to polytherapy, interaction risk, and infectious triggers\.Consiliumfollows from that review: each recurring failure mode becomes a specialist lens, and the final prescription is synthesized across those lenses rather than produced by one monolithic prompt\.

#### Case sampling\.

To understand why standard prompting failed, we intentionally enriched the neurologist audit set for model errors\. The goal was not to estimate population\-level accuracy, which is reported separately in Table[1](https://arxiv.org/html/2606.31036#S4.T1), but to identify whether the errors reflected recurring, patchable reasoning failures rather than irreducible clinical ambiguity or missing medical knowledge\. We sampled 20 longitudinal Cohort A patients, covering 60 visits total\. The sampling criteria were not shown to the neurologists\.

- •10 polytherapy patients whose regimens were predicted incorrectly\.
- •5 polytherapy patients whose regimens were predicted correctly\.
- •3 monotherapy patients whose regimens were predicted incorrectly\.
- •2 monotherapy patients whose regimens were predicted correctly\.

#### Neurologist audit protocol and descriptive results\.

Two pediatric neurologists independently reviewed the sampled cases using the audit interface shown in Figure[4](https://arxiv.org/html/2606.31036#A5.F4)\. For each visit, the interface showed the*Model Input*,*Doctor’s Actual Output*,*Model Thinking*, and*Regimen Options*panels\. The neurologists then selected one of four*Feedback*choices—“Agreed with both LLM and physician,” “Agreed with LLM only,” “Agreed with physician only,” or “Agreed with neither”—with an optional written comment\. For inter\-rater agreement, we used the common binary endpoint available for both neurologists: whether the LLM was accepted in the Feedback choice\.

![Refer to caption](https://arxiv.org/html/2606.31036v1/figs/gui1.png)

![Refer to caption](https://arxiv.org/html/2606.31036v1/figs/gui2.png)

Figure 4:Neurologist audit interface\. The paper figure is partially redacted: patient identifiers, Model Input text, and Doctor’s Actual Output text are blurred\. In the review interface, neurologists saw the Model Input, Doctor’s Actual Output, Model Thinking, and Regimen Options panels, then selected one of four Feedback choices and could add an optional written comment\.Table[8](https://arxiv.org/html/2606.31036#A5.T8)shows representative comments from these reviews\. These comments illustrate why aggregate exact\-match performance was not enough: the neurologists often agreed that a prediction was clinically plausible while still identifying a recurring reasoning gap, such as timeline drift, missed seizure classification, pediatric dosing, local formulary constraints, or unsafe combinations\.

Table 8:Representative neurologist audit comments\. Comments are reproduced verbatim from the audit records; case identifiers are omitted\. The final two rows correspond to the redacted case shown in Figure[4](https://arxiv.org/html/2606.31036#A5.F4)\.For the binary endpoint, “Agreed with both LLM and physician” and “Agreed with LLM only” were coded as LLM accepted; “Agreed with physician only” and “Agreed with neither” were coded as LLM not accepted\. Binary LLM acceptability was summarized across 120 neurologist\-visit judgments:

By neurologist, the LLM was accepted at similar rates:

Inter\-rater agreement on binary LLM acceptability across the 60 shared visits was:

Because the LLM was accepted on most sampled visits, Cohen’sκ\\kappais conservative under class imbalance\. Gwet’s AC1 and PABAK provide prevalence\-robust agreement summaries and show high agreement on the shared binary endpoint\. We therefore use the Feedback choices and written comments descriptively, primarily to identify recurrent failure modes rather than to claim clinical superiority\. We then characterized the written comments into recurring failure modes, summarized in Table[9](https://arxiv.org/html/2606.31036#A5.T9)\.

Table 9:Recurring failure modes from the neurologist audit\. Rows group single\-agent reasoning failures from 20 Cohort A patients \(60 visits\) and the specialist lens later encoded inConsilium\.The audit pointed to a design hypothesis: the model did not need only more generic epilepsy knowledge; it needed more reliable separation of clinical lenses\.Consiliumtests that hypothesis by assigning recurring reasoning gaps to specialist agents, asking for independent reports, and requiring a final synthesis step\.

### E\.2Expert\-DesignedConsiliumComparator

![Refer to caption](https://arxiv.org/html/2606.31036v1/figs/consilium_flowchat.png)Figure 5:Consiliumexpert\-system workflow\. The orchestrator sends the same visit information to specialist clinical agents, the epileptologist synthesizes their independent reports into ranked regimens, and the pharmacologist reviews safety before final output\.Consiliumuses the same visit input as the single\-agent baseline, but separates the reasoning before synthesis \(Figure[5](https://arxiv.org/html/2606.31036#A5.F5)\)\. An orchestrator routes the visit to specialist agents, each specialist writes an independent report, and an epileptologist combines those reports into three ranked regimens\. A pharmacologist then checks dosing, safety, and interaction risks; if it raises concerns, the critique is returned to the epileptologist before the final regimens are returned\. This hand\-built system serves as a clinical reference point forManana, not as the scalable method proposed in the main paper\.

#### Aggregate comparison\.

Table[10](https://arxiv.org/html/2606.31036#A5.T10)focuses on the methods needed to interpret the move from direct prompting toConsilium\. The Base Prompt row shows the unadapted task scaffold\. TheMananarows show what can be learned from patient\-level supervision without specialist trace feedback\. The doctor\-designed rows then ask a narrower question: after clinicians helped write a strong single\-agent prompt, does explicit specialist decomposition add value beyond that prompt?

Table 10:Aggregate EM@3 and Jaccard results for prompt\-learning and doctor\-designed prompt systems\. EM@3 is the top\-3 exact\-match rate; Jaccard is the maximum set overlap between the three predicted regimens and the physician\-prescribed regimen\. Pre\-BPAMananarows are averaged over five seeds;Manana\+ BPA uses the seed\-42 trajectory with Beta\-Binomial posterior weighting, and its Jaccard is the maximum over the top\-3 posterior\-weighted regimens\.Aggregation\.In Table[10](https://arxiv.org/html/2606.31036#A5.T10), aggregate EM@3 is computed over evaluated visit\-level cases\. Equivalently, it is the visit\-count\-weighted average of the corresponding monotherapy and polytherapy strata within each cohort and visit\. Visits with no extracted active ASM are excluded unless stated otherwise\.

#### All\-agents\-combined\.

All\-agents\-combined is the control for prompt content versus council structure\. It concatenates the specialist roles used inConsiliuminto one direct prompt, effectively asking a single model call to reason as all specialists at once and return the final ranked regimens in one forward pass\. The gap between this row andConsiliumtests whether the council structure adds value beyond listing the same clinical lenses in a larger prompt\.

#### Ablations test whether the council is reducible\.

Appendix[L](https://arxiv.org/html/2606.31036#A12)reports leave\-one\-out and only\-one ablations forConsilium\. The full council is strongest overall, but the leave\-one\-out rows show that different specialists matter in different cohort–visit slices rather than contributing a uniform gain everywhere\.

## Appendix FCross\-Model Transfer Learning

A learned prompt memory is useful only if it captures reusable task knowledge rather than idiosyncrasies of the model that produced it\. We therefore ask whether the lessons learned by a stronger model can improve a smaller model at inference time\. In the native 20B condition, the 20B model both learns the prompt memory and uses that memory for evaluation\. In the 120B→\\to20B condition, the memory is learned by the 120B model, then inserted unchanged into the 20B inference prompt\. The evaluator is always the 20B model; only the source of the learned memory changes\. This isolates whether the learned text acts as portable clinical guidance rather than model\-specific prompt phrasing\.

Table[11](https://arxiv.org/html/2606.31036#A6.T11)reports the cross\-model transfer experiment\. TransferredMananamemory improves over native 20B learning on five of six cohort–visit pairs\. The largest gains occur in Cohort A, where 120B\-derived memory improves all three visits\. Cohort B shows the same pattern at later visits, with a small drop at V1 but gains at V2 and V3\. TextGrad also benefits from 120B→\\to20B transfer, but its transferred memories remain less consistently aligned with the clinical task structure thanManana\.

These results support the view thatMananalearns an interpretable, reusable correction artifact\. The transferred memory acts like portable clinical guidance: it distills recurring prescribing corrections from the source model’s adaptation trajectory and makes them available to a smaller model that would otherwise learn weaker memories from the same supervision\. This matters for deployment because larger models may be useful for offline adaptation, while smaller models may be preferable for lower\-cost or lower\-latency inference\.

Table 11:Cross\-model transfer: all variants are evaluated on the 20B model\. Native means optimized on 20B; 120B→\\to20B means optimized on 120B and applied to 20B\. Entries are mean±\\pmstd over 5 seeds\.
## Appendix GAdditional Open Source Model Experiments

The main paper reports results withopenai\.gpt\-oss\-120bas the underlying LLM\. To check that theMananaloop is not specific to a single model family, we replicate the training on additional Bedrock\-available LLM backbones spanning a range of scales and architectures\. Each backbone is trained with the same single\-buffer and multi\-agent pipelines, identical 50/20 patient split, and 15 rounds\. Test\-set inference is run on the round selected by held\-out eval accuracy\.

### G\.1Models Evaluated

- •google\.gemma\-3\-12b\-it— 12B dense, instruction\-tuned\.
- •mistral\.ministral\-3\-14b\-instruct— 14B dense\.
- •openai\.gpt\-oss\-20b— smaller MoE from the same family\.
- •google\.gemma\-3\-27b\-it— 27B dense, instruction\-tuned\.
- •qwen\.qwen3\-32b— 32B dense\.
- •openai\.gpt\-oss\-120b— main paper reference \(120B, MoE, native reasoning traces\)\.

### G\.2Test\-Set Performance Across Models

Table[12](https://arxiv.org/html/2606.31036#A7.T12)reports the held\-out test performance for each LLM backbone andMananavariant\.

Table 12:EM@3 \(%\) of single\-buffer and multi\-agentMananaacross LLM backbones, on Cohort A and Cohort B test sets\. Best\-per\-backbone round selected on held\-out eval accuracy\.
### G\.3Model Capacity and Specialist Decomposition

For smaller models such as Gemma\-3\-12B and Ministral\-3\-14B, the multi\-agent variant often struggled to create useful specialists and write clear, role\-specific system prompts\. This suggests a capacity\-dependent tradeoff: specialist decomposition helps only when the underlying model can reliably manage the extra prompt\-construction and coordination steps\. Table[12](https://arxiv.org/html/2606.31036#A7.T12)reflects this pattern, with Single performing more consistently than Multi on the smallest evaluated models\.

## Appendix HMananaComponent Ablations

To isolate which components of theMananaloop drive performance, we run paired ablations against the full pipeline\. Each ablation removes a single component while keeping the rest of the system fixed\. We report no\-buffer and no\-inspector results on both the single\-buffer and multi\-agent variants ofManana, evaluated on Cohort A and Cohort B with the same train/eval split \(50 train, 20 eval\) and 15 rounds\. We additionally define a no\-architect ablation for the single\-buffer variant below\. Results are reported in Table[13](https://arxiv.org/html/2606.31036#A8.T13)\.

#### No\-buffer\.

The append\-only candidate buffer is removed while the learned prompt state persists across rounds \(shared learnings in the single\-buffer variant; the agent population in the multi\-agent variant\)\. The Architect receives the current prompt state and the current batch’s Inspector reports, but no candidate learnings from previous rounds\. This isolates the contribution of cross\-round candidate memory from the learned state itself\.

#### No\-inspector\.

The Inspector is removed\. The Architect receives raw per\-case triples\(patient notes, predictor’s three options, ground\-truth prescription\)instead of structured Inspector reports, and must perform error attribution itself\. This isolates whether the diagnostic decomposition \(match status, error type, root cause, missed signal\) is what makes the Architect’s updates useful, or whether raw signal suffices\.

#### No\-architect\.

In the single\-buffer variant, the Architect is removed and each Inspector\-producedCANDIDATE\_LEARNINGis appended directly to the shared rule list\. This ablation keeps supervised error diagnosis but removes the consolidation layer: there is no quorum check, deduplication, rewriting, or cross\-case synthesis before a lesson enters the Predictor prompt\. We do not define an analogous multi\-agent no\-architect row because the Architect is the mechanism that creates, edits, and prunes specialist agents; removing it from an initially empty multi\-agent state would leave no learned agents and collapse the method to the base Predictor rather than a meaningful multi\-agent ablation\.

#### No\-quorum\.

The Inspector, candidate buffer, and Architect are all retained, but the explicit quorum rule is stripped from the Architect’s prompt\. Concretely, the Architect still consumes structured Inspector reports and the cross\-round candidate buffer and still issues prompt\-state updates, but it is no longer instructed to require evidence from multiple cases before promoting a candidate learning into the rule list \(single\-buffer\) or before creating, editing, or retiring an agent \(multi\-agent\)\. This ablation isolates the contribution of the multi\-case evidence threshold from the rest of the consolidation layer \(deduplication, rewriting, cross\-case synthesis\), which the Architect continues to perform\.

#### In\-context learning\.

We also evaluate a non\-learning in\-context baseline for the single\-buffer variant\. In this setting, the self\-learning loop is disabled entirely: there is no Inspector, Architect, buffer, or learned rule consolidation\. Instead, the Predictor receives raw training examples directly in its prompt, where each example consists of a patient record paired with the doctor’s ground\-truth prescription\. This ablation asks whether performance comes simply from exposing the model to many labeled cases at inference time, rather than from distilling those cases into compact, reusable clinical learnings\. We report this only for the single\-buffer setting because the intervention is directly analogous to replacing the shared learned rule list with labeled in\-context examples; there is no corresponding multi\-agent state to populate without an Architect\.

Table 13:Component ablations on the single\-buffer and multi\-agent variants ofManana\. EM@3 \(%\) on the held\-out test sets\. The “Full” row reproduces the corresponding row of Table[10](https://arxiv.org/html/2606.31036#A5.T10)\.

## Appendix IClinician Review ofManana

We conducted an additional neurologist audit comparing the expert\-designedConsiliumsystem with the self\-learningManana\-Multi system\. The reviewer examined 20 longitudinal patients, with three visits per patient and outputs from both systems, yielding20×3×2=12020\\times 3\\times 2=120system\-visit reviews\. The review asked whether each output correctly identified seizure type, assessed seizure activity, accounted for medication context, and gave clinically sound drug\-selection reasoning; each question was answered asYes,Partially, orNo\. The reviewer also rated overall usefulness asVery useful,Somewhat useful, orNot useful\.

Qualitative findings\.Because the neurologist’s ratings were near ceiling across the answered fields, we use the review primarily as a qualitative audit\. We therefore focus on the neurologist’s comments, which describe how the systems behave as practical decision\-support tools\. The clearest theme was option coherence:Manana\-Multi often proposed related variants of the same clinical plan, whileConsiliumsometimes proposed individually plausible but directionally different options\. In Cohort A, the reviewer wrote that “System B reasoning is easier to follow,” and in another case noted that “all three recommendations are concordant,” contrasting this with System A’s “appropriate but different choices\.” The comments make the practical issue clearer: whenConsiliumoffered plausible but different options, the reviewer treated this as less useful than recommendations that preserved the same treatment direction\.

These observations motivate a deeper system\-level audit of when model recommendations are clinically coherent, which we leave as a planned next step\.

## Appendix JMananaBayesian Prompt Averaging Ablations

### J\.1Beta\-Binomial BPA Derivation

The main text uses a Beta\-Binomial marginal likelihood to score each retained memory state\. For a validation case evaluated under memory statemkm\_\{k\}, there are three relevant outcomes: the physician regimen appears in candidate position 1, appears in candidate positions 2–3, or is absent from the three candidates\. Letck,1c\_\{k,1\},ck,\>1c\_\{k,\>1\}, anduku\_\{k\}denote the corresponding counts, and lethk=ck,1\+ck,\>1h\_\{k\}=c\_\{k,1\}\+c\_\{k,\>1\}be the number of top\-3 hits\.

We factor this outcome into a state\-specific top\-3 hit probabilityθk\\theta\_\{k\}and a conditional candidate\-position distributionπ\\pi\. Thus a case is a top\-3 hit with probabilityθk\\theta\_\{k\}; conditional on a hit, its position is drawn fromπ\\pi\. Withπ\>1=π2\+π3\\pi\_\{\>1\}=\\pi\_\{2\}\+\\pi\_\{3\}, the validation likelihood is

p​\(𝒟val∣θk,π,mk\)=π1ck,1​π\>1ck,\>1​θkhk​\(1−θk\)uk\.p\(\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\mid\\theta\_\{k\},\\pi,m\_\{k\}\)=\\pi\_\{1\}^\{c\_\{k,1\}\}\\pi\_\{\>1\}^\{c\_\{k,\>1\}\}\\theta\_\{k\}^\{h\_\{k\}\}\(1\-\\theta\_\{k\}\)^\{u\_\{k\}\}\.We estimateπ=\(0\.85,0\.11,0\.04\)\\pi=\(0\.85,0\.11,0\.04\)once from a small subset of the training set and treat it as a fixed empirical\-Bayes\-style plug\-in estimate\. For the state\-specific hit probability, we use the conjugate priorθk∼Beta​\(1,1\)\\theta\_\{k\}\\sim\\mathrm\{Beta\}\(1,1\)and integrate outθk\\theta\_\{k\}:

p​\(𝒟val∣mk\)=π1ck,1​π\>1ck,\>1​∫01θkhk​\(1−θk\)uk​𝑑θk=π1ck,1​π\>1ck,\>1​B​\(hk\+1,uk\+1\)\.p\(\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\mid m\_\{k\}\)=\\pi\_\{1\}^\{c\_\{k,1\}\}\\pi\_\{\>1\}^\{c\_\{k,\>1\}\}\\int\_\{0\}^\{1\}\\theta\_\{k\}^\{h\_\{k\}\}\(1\-\\theta\_\{k\}\)^\{u\_\{k\}\}\\,d\\theta\_\{k\}=\\pi\_\{1\}^\{c\_\{k,1\}\}\\pi\_\{\>1\}^\{c\_\{k,\>1\}\}B\(h\_\{k\}\+1,u\_\{k\}\+1\)\.This is the score used for the Beta\-Binomial BPA weights in the main text\. The Dirichlet\-Multinomial variant below relaxes the shared fixedπ\\piassumption by estimating a separate candidate\-position distribution for each memory state under a conjugate Dirichlet prior\.

### J\.2Weighting Ablations

Table[14](https://arxiv.org/html/2606.31036#A10.T14)compares the Beta\-Binomial BPA used in the main text against two alternatives\. Linear weighting uses validation top\-3 rates directly\. Dirichlet\-Multinomial BPA relaxes the shared candidate\-position prior assumption by estimating a separate candidate\-position distribution for each prompt\. The three variants produce similar predictions, which suggests that the deferral signal is not an artifact of a fragile weighting rule\. Gap is the difference in mean confidence between correct and incorrect top\-1 predictions; P@25% and P@50% report top\-1 precision after retaining the most confident 25% or 50% of cases\. We use Beta\-Binomial BPA in the main text because it gives the clearest closed\-form likelihood and the strongest or near\-strongest calibration on the full test set\.

Table 14:BPA weighting ablations on full held\-out test sets\.

## Appendix KMIMIC\-IV

Note on task mismatch\.MIMIC\-IV is not the same clinical setting as our primary Ugandan epilepsy cohorts\. Our main task uses longitudinal outpatient epilepsy\-care notes written before or during serial clinic visits, where the model must infer a locally appropriate anti\-seizure medication \(ASM\) regimen from sparse visit documentation\. The MIMIC data are retrospective hospital discharge summaries from a high\-resource US hospital system\. These notes are written after the admission has ended, so they often contain the treatment course, response to therapy, discharge intent, and medication reconciliation\. Therefore, unprocessed MIMIC discharge notes can leak the answer\.

The MIMIC cohort is still useful because it is the closest public, credentialed\-access source containing epilepsy\-related clinical notes and medication records\. We use it as a reproducibility and robustness setting, not as a replacement for the Ugandan serial\-care task\. The cohort is adult and hospital\-based: median age is 49 years, IQR 34–60, with 55\.3% female admissions\. Across the final MIMIC cohort, patients have broad inpatient medication exposure: median 16 unique prescribed drug strings during the admission, and median 8 active scheduled medications at discharge\. The anti\-seizure component is only part of that medication burden\.

Cohort construction\.We filter MIMIC\-IV admissions to those with a primary epilepsy ICD code \(345\.x or G40\.x atseq\_num=1\) and Neurology as the first managing service \(NMED\), keeping the longest discharge summary per admission\. After ground\-truth \(GT\) extraction, action\-space restriction, and removal of manually audited leaky admissions, the final cohort contains 1,977 admissions from 1,257 patients \(31\.6% monotherapy, 68\.4% polytherapy; max GT regimen size 4\)\. Full filter logic and per\-stage counts are in the released code\.

Ground\-truth extraction\.Ground truth is extracted from the prescription table, not the note: a prescription qualifies as a target ASM if it is oral/enteral, scheduled \(not PRN\), active at discharge, and newly started during the admission \(fosphenytoin excluded\)\. The MIMIC action space is the 15 ASMs meeting the 4% frequency threshold in this cohort: levetiracetam, lamotrigine, lacosamide, zonisamide, phenytoin, valproate, oxcarbazepine, lorazepam, gabapentin, clobazam, topiramate, clonazepam, carbamazepine, pregabalin, phenobarbital\.

Cleaning rationale\.MIMIC discharge summaries are written after the admission and encode the prescribed regimen through hospital\-course text, medication reconciliation, and discharge planning\. The cleaner removes those sections so the input contains pre\-admission and admission\-evidence text without the discharge\-side decisions\.

LLM cleaning\.Cleaning is performed by an LLM following the released editing prompt, with an additional regex pass that strips residual discharge\-tail headers \(e\.g\.,*Discharge Diagnosis*,*Discharge Medications*,*Follow\-up Instructions*\) and excludes manually audited leaky admissions\. Results are reported on this configuration; the saved test predictions and prompt are released for reproducibility\.

Held\-out test results\.Table[15](https://arxiv.org/html/2606.31036#A11.T15)reports held\-out test performance from the MIMIC\-IV test split \(split seed 42\)\. Each learning system uses the same 50\-patient training split and 20\-patient validation split;Manana\-Single,Manana\-Multi, TextGrad, and ExpeL are evaluated from their best learned checkpoints\. The Base Prompt row evaluates the unadapted single\-agent template against MIMIC\.

Table 15:MIMIC\-IV held\-out test results\. EM@1 and EM@3 are top\-1 and top\-3 exact\-match rates over the 15\-ASM MIMIC action space\. Mono@3 and Poly@3 stratify EM@3 by ground\-truth regimen size\. EpiPick is evaluated on all monotherapy admissions, Poly@3 does not apply\.
## Appendix LConsiliumCouncil Ablations

Table[16](https://arxiv.org/html/2606.31036#A12.T16)reports leave\-one\-out and only\-one ablations forConsilium\.

Table 16:Ablations of theConsiliumcouncil\. Entries are EM@3 percentages across visits in each cohort\.
## Appendix MLearned Artifacts Across Optimization Methods

We selected representative artifacts from high\-performing runs to illustrate the form of each method’s learned text state\. The full artifacts are provided with the released code documentation; here we summarize the qualitative differences that matter for interpreting the results\.

TextGrad: detailed prose, weak grounding\.TextGrad learns by rewriting a global instruction variable\. In this setting, the learned artifact can read like a guideline document, but the supervision signal is only a prescribed drug set\. In representative runs, it commits to dose ranges, laboratory schedules, formulation choices, pill\-burden constraints, stocking assumptions, and cost hierarchy details that are not recoverable from the training labels alone\.

ExpeL: useful but generic rules\.ExpeL produces a ranked experience memory\. Its lessons capture plausible prescribing heuristics, such as prioritizing continuation, selecting locally common first\-line agents, and using weight\-based dosing, but they remain at the level of generic best practice rather than naming the recurring drug\-specific or seizure\-specific local patterns that drive cohort\-level errors\.

DSPy\-GEPA: a sharper interface, no clinical memory\.DSPy\-GEPA produces an optimized instruction for the prediction program rather than a persistent clinical memory\. It improves the task interface by emphasizing explicit\-directive parsing, output formatting, dose\-adequacy reasoning, and constraints against unnecessary drug additions, but it has no slot for cross\-patient lessons to accumulate over rounds\.

Manana\-Single: compact, evidence\-gated rules\.The Single variant learns a short list of global correction rules that survive the Architect’s quorum check across cases\. The representative rules name specific drugs and clinical situations, such as continuing a tolerated regimen when no modification directive is present, adding levetiracetam before changing tolerated carbamazepine in partially controlled focal epilepsy, and avoiding escalation after a solitary fever\-related breakthrough seizure\.

Manana\-Multi: specialists, not prescribers\.The Multi variant turns recurring errors into specialist signal extractors\. The learned agents surface clinical observations for the Predictor rather than directly writing prescriptions, making the memory easier to audit as clinical signal extraction instead of a hidden decision policy\.