Can I Take Another Dose? Evaluating LLM Decision-Making Under Temporal Uncertainty in OTC Dosing QA

# Can I Take Another Dose? Evaluating LLM Decision-Making Under Temporal Uncertainty in OTC Dosing QA
Source: [https://arxiv.org/html/2606.04262](https://arxiv.org/html/2606.04262)
Yibo Hu Illinois Institute of Technology Chicago, IL, USA yhu89@illinoistech\.edu

###### Abstract

Large language models \(LLMs\) are increasingly used for everyday health questions, including whether a user can safely take another dose of an over\-the\-counter \(OTC\) medication\. Yet this common safety\-relevant setting remains underexplored in existing medical QA evaluations, where correct answers require tracking dose timing, computing rolling 24\-hour intake, following product\-label constraints, and handling incomplete medication histories\. We introduceDoseBench, a focused benchmark of 81 curated OTC dosing scenarios focused on adult acetaminophen and ibuprofen use, with manually annotated gold references\. We evaluate four LLMs across repeated runs using metrics for decision correctness, consistency, explanation verifiability, failure types, and confidence\-related signals, resulting in 1,620 model responses\. Our results show that models frequently struggle with rolling\-window reasoning and ambiguity\-sensitive cases and that stable or confident\-looking responses can still violate dosing constraints\. These findings suggest that OTC dosing QA provides a narrow yet practical testbed for evaluating temporal reasoning, constraint\-following, and safety\-relevant uncertainty handling in medical QA\.111Code and data available for review at[https://github\.com/yibo\-hu\-lab/DoseBench](https://github.com/yibo-hu-lab/DoseBench)

Can I Take Another Dose? Evaluating LLM Decision\-Making Under Temporal Uncertainty in OTC Dosing QA

Maroof KousarIllinois Institute of TechnologyChicago, IL, USAmkousar@hawk\.illinoistech\.eduYibo HuIllinois Institute of TechnologyChicago, IL, USAyhu89@illinoistech\.edu

## 1Introduction

Large language models \(LLMs\) are increasingly used for everyday health\-related questions, including whether a user can safely take another dose of an over\-the\-counter \(OTC\) medicationSinghal et al\. \([2025](https://arxiv.org/html/2606.04262#bib.bib18)\)\. Common OTC drugs such as acetaminophen and ibuprofen are widely available and frequently discussed in search engines, online forums, and AI assistants\. Although these questions may appear simple, safe answers often require applying dosage\-label constraints to incomplete, user\-provided medication histories\.

This difficulty can arise even in seemingly simple user prompts, as illustrated in Figure[1](https://arxiv.org/html/2606.04262#S1.F1)\. The user describes three 650 mg acetaminophen doses totaling only 1,950 mg, yet the model confidently rejects another dose without resolving the missing temporal context\.

![Refer to caption](https://arxiv.org/html/2606.04262v1/figures/teaser_image.png)Figure 1:OTC dosing scenarios expose stable but medically unsupported LLM reasoning under rolling 24\-hour and ambiguity\-sensitive constraints\.Despite the practical importance of this setting, existing medical question\-answering benchmarks primarily focus on clinical knowledge, professional exams, or general medical reasoningAbacha et al\. \([2019](https://arxiv.org/html/2606.04262#bib.bib1)\); Suri et al\. \([2021](https://arxiv.org/html/2606.04262#bib.bib19)\)\. Relatively little work isolates OTC dosing as a focused safety\-relevant reasoning problem written as realistic consumer\-facing scenarios\. Unlike many open\-ended clinical QA tasks, OTC dosing decisions are often grounded in public dosage\-label constraints that can be systematically checked\. In practice, users may forget prior doses, misunderstand timing intervals, combine medications, or ask vague questions while under discomfort\. These characteristics make OTC medication reasoning both practically important and challenging for current LLMs\.

In this work, we introduceDoseBench, a benchmark of realistic adult OTC medication dosing scenarios focused on acetaminophen and ibuprofen\. DoseBench is designed as a controlled diagnostic setting for evaluating LLM reliability in consumer\-facing medication questions\. The benchmark includes user\-style scenarios involving dosage timing, cumulative intake, multi\-medication use, repeated dosing, and incomplete medication histories\.

We evaluate four instruction\-tuned LLMs using structured decision outputs \(Yes,No, andAmbiguous\) together with reasoning explanations\. Beyond correctness, we analyze repeated\-run consistency, reasoning verifiability, failure patterns, and confidence\-related behavior across repeated generations\.

Across evaluated models, we observe that LLMs frequently produce stable or confident\-looking responses that are not medically supported\. Errors often arise from failures in temporal reasoning, ambiguity handling, or constraint following rather than from output\-formatting issues alone\. These findings suggest that OTC dosing provides a narrow but practical testbed for studying safety\-oriented reliability in consumer\-facing medical QA\.

Our contributions are summarized as follows:

- •We introduceDoseBench, a realistic OTC dosing benchmark focused on adult acetaminophen and ibuprofen dosing scenarios, grounded in public dosage\-label constraints\.
- •We design a structured evaluation protocol that measures decision correctness, repeated\-run consistency, explanation verifiability, failure types, and confidence\-related behavior\.
- •We show that current LLMs can produce stable and confident\-looking responses that remain incorrect, unverifiable, or unsupported under safety\-relevant OTC dosing constraints\.

![Refer to caption](https://arxiv.org/html/2606.04262v1/figures/overview_image_updated.png)Figure 2:Overview of theDoseBenchevaluation framework\. We construct curated acetaminophen and ibuprofen reasoning scenarios and evaluate four LLMs across repeated runs\. Each scenario is annotated with a gold decision label, reasoning complexity type, and key dosing facts\. Model outputs are analyzed for correctness, consistency, verifiability, confidence mismatch, and failure patterns\.
## 2Related Work

### 2\.1Medical Question Answering

Medical question\-answering benchmarks have become a central tool for evaluating LLMs in healthcare\. Exam\-style resources such as MedQA and MedMCQA primarily assess whether models can answer professional medical questions requiring factual and clinical reasoningJin et al\. \([2021](https://arxiv.org/html/2606.04262#bib.bib8)\); Pal et al\. \([2022](https://arxiv.org/html/2606.04262#bib.bib16)\); Kung et al\. \([2023](https://arxiv.org/html/2606.04262#bib.bib12)\)\. Broader benchmark collections such as MultiMedQA further evaluate models across multiple medical QA settings, including professional examinations and consumer health questionsSinghal et al\. \([2025](https://arxiv.org/html/2606.04262#bib.bib18)\)\. These benchmarks have been valuable for measuring general medical knowledge, but they do not isolate medication dosing as a temporally constrained decision\-making task\.

Other datasets move closer to consumer\-facing and medication\-related information needs\. MedicationQA focuses on answering real consumer medication questions using trusted sourcesAbacha et al\. \([2019](https://arxiv.org/html/2606.04262#bib.bib1)\), while MedExQA and MedExpQA evaluate medical QA with explanations and multilingual coverageKim et al\. \([2024](https://arxiv.org/html/2606.04262#bib.bib11)\); Alonso et al\. \([2024](https://arxiv.org/html/2606.04262#bib.bib3)\)\. Additional work has examined medical dialogue QA, realistic clinical questions, and healthcare retrieval settingsSuri et al\. \([2021](https://arxiv.org/html/2606.04262#bib.bib19)\); Kell et al\. \([2025](https://arxiv.org/html/2606.04262#bib.bib10)\); Zhu et al\. \([2019](https://arxiv.org/html/2606.04262#bib.bib22)\)\. These resources emphasize realistic health information needs, but they generally evaluate broad medical or clinical QA rather than repeated OTC dosing decisions grounded in explicit public\-label constraints\.

DoseBench is complementary to these prior benchmarks\. Instead of testing broad medical knowledge, it focuses on whether models can make safe consumer\-facing OTC dosing decisions from limited medication histories\. This narrower setting allows us to study decision correctness, ambiguity handling, and reasoning verifiability under explicit dosage rules for adult acetaminophen and ibuprofen\.

### 2\.2LLM Confidence and Reliability Evaluation

Prior work has shown that LLM confidence is not always aligned with correctnessHu and Khan \([2021](https://arxiv.org/html/2606.04262#bib.bib6)\); Kadavath et al\. \([2022](https://arxiv.org/html/2606.04262#bib.bib9)\); Lin et al\. \([2022](https://arxiv.org/html/2606.04262#bib.bib13)\); Geng et al\. \([2024](https://arxiv.org/html/2606.04262#bib.bib5)\)\. Studies on calibration and uncertainty estimation have examined whether models can express what they know, how verbalized confidence relates to accuracy, and how token\-level probabilities can be used as uncertainty signals\. In medical QA, token\-probability\-based methods have also been explored as a way to identify overconfident or unreliable answersBentegeac et al\. \([2025](https://arxiv.org/html/2606.04262#bib.bib4)\)\.

Repeated sampling provides another perspective on reliability\. Self\-consistency methods show that multiple generations can improve reasoning performance in some settingsWang et al\. \([2023](https://arxiv.org/html/2606.04262#bib.bib21)\), while hallucination\-detection work such as SelfCheckGPT uses disagreement across generations as a signal of unreliabilityManakul et al\. \([2023](https://arxiv.org/html/2606.04262#bib.bib14)\)\. These studies suggest that repeated generations can reveal instability, but stable agreement does not necessarily imply correctness, especially in safety\-sensitive settings\.

Our work uses confidence and consistency signals diagnostically rather than proposing a new calibration method\. DoseBench enables us to compare majority\-vote correctness, repeated\-run agreement, verbal confidence, internal decision confidence, and reasoning verifiability within a constrained medication\-safety task where final decisions can be checked against explicit OTC dosing rules\.

## 3Benchmark Construction

### 3\.1Benchmark Scope and Design

DoseBenchfocuses on adult OTC medication dosing scenarios involving acetaminophen and ibuprofen\. We intentionally restrict the benchmark to OTC dose\-decision questions rather than broader clinical diagnosis, pediatric dosing, prescription medication interactions, or chronic disease management\. This constrained scope allows us to evaluate model behavior in a setting where safe answers can be checked against explicit public dosage guidance\.

The benchmark is designed around realistic consumer\-facing questions in which users ask whether they can take another dose, how long they should wait, or whether prior medication use affects the next decision\. Rather than requiring access to patient records or expert\-only clinical annotations, each scenario is grounded in information provided within the question and in publicly available OTC dosing guidance\.

For many common OTC products, relevant interval and cumulative dosage rules are explicitly stated in public Drug Facts labels and DailyMed materials\.222[https://dailymed\.nlm\.nih\.gov/dailymed/](https://dailymed.nlm.nih.gov/dailymed/)This makes the setting suitable for evaluating whether models can follow concrete dosage constraints and provide checkable reasoning without relying on patient records, expert\-only clinical annotations, or open\-ended diagnostic reasoning\.

### 3\.2Dataset Collection

Table 1:Overview of reasoning complexity categories inDoseBench\. Each question was manually annotated according to its most prominent reasoning challenge\.To improve realism and reduce author\-specific phrasing bias, we intentionally used a small set of survey\-inspired seed questions during dataset construction\. Participants were asked only to provide example OTC medication questions as they might naturally ask an AI assistant or a search engine\. We did not collect personal health information, demographic information, or private medical histories\. Responses were manually filtered to remove overly generic, duplicate, or medically irrelevant questions\.

The final dataset contains 81 manually curated OTC medication reasoning scenarios focused primarily on adult acetaminophen and ibuprofen usage scenarios\.

### 3\.3Question Refinement

Questions were manually refined to preserve realistic conversational phrasing while ensuring coverage across the benchmark’s predefined reasoning categories\. During refinement, we standardized medication names, dosage strengths, timing expressions, and scenario wording when needed, while keeping the questions close to natural user\-style phrasing\.

Refinement also ensured that each scenario contained enough information to support a gold decision label or, when appropriate, clearly required anAmbiguouslabel due to missing or uncertain information\. Annotation decisions were guided by conservative, safety\-oriented reasoning grounded in public Drug Facts labels and DailyMed dosage guidanceU\.S\. National Library of Medicine \([2025](https://arxiv.org/html/2606.04262#bib.bib20)\)\.

### 3\.4Reasoning Complexity Categories

To better characterize the reasoning challenge in each scenario, each question was manually reviewed and annotated with its most prominent reasoning complexity type\.

Table[1](https://arxiv.org/html/2606.04262#S3.T1)summarizes the resulting reasoning categories and their distribution inDoseBench\.

These categories are intended to support analysis of model performance by reasoning type, rather than to represent mutually exclusive medical conditions\. When a question involved multiple challenges, annotators assigned the category corresponding to the most prominent reasoning requirement needed to reach the gold decision\.

### 3\.5Gold Annotation

Table 2:Representative examples of conservative safety\-oriented gold annotation decisions, including timing\-sensitive, ambiguity\-sensitive, and multi\-medication OTC medication scenarios\.Each question inDoseBenchwas manually annotated with a gold decision label \(Yes,No, orAmbiguous\), an annotation rationale, a reasoning complexity category, and key dosing facts needed to support the decision\. Gold labels were assigned using structured annotation guidelines based on public OTC dosing rules and DailyMed guidanceU\.S\. National Library of Medicine \([2025](https://arxiv.org/html/2606.04262#bib.bib20)\)\. When product formulations differed, conservative standardized limits were applied consistently across the benchmark\.

Gold labels and reasoning annotations were reviewed by two annotators using a shared rubric\. Annotators reviewed decision correctness, ambiguity handling, key dosing facts, and reasoning verifiability\. Disagreements were resolved through discussion\. Because the task is grounded in public dosage\-label constraints rather than open\-ended clinical diagnosis, this adjudication process was sufficient for the controlled benchmark setting\. Detailed annotation instructions and scoring criteria are provided in Appendix[B](https://arxiv.org/html/2606.04262#A2)\.

Table[2](https://arxiv.org/html/2606.04262#S3.T2)presents representative examples of conservative safety\-oriented gold annotation decisions\. Ambiguous labels were assigned when the available information was insufficient for a medically safe recommendation without making unsupported assumptions\.

Figure[2](https://arxiv.org/html/2606.04262#S1.F2)summarizes the overall benchmark construction and evaluation workflow\.

## 4Experimental Setup and Evaluation

### 4\.1Models

We evaluated four instruction\-tuned large language models: Qwen2\.5\-7B\-InstructQwen Team \([2024](https://arxiv.org/html/2606.04262#bib.bib17)\), Meta\-Llama\-3\-8B\-InstructAI@Meta \([2024](https://arxiv.org/html/2606.04262#bib.bib2)\), Mistral\-7B\-Instruct\-v0\.3Jiang et al\. \([2023](https://arxiv.org/html/2606.04262#bib.bib7)\), and GPT\-4o\-miniOpenAI \([2024](https://arxiv.org/html/2606.04262#bib.bib15)\)\. The first three are open\-source models deployed locally with HuggingFace Transformers using 4\-bit quantization on one NVIDIA A100, while GPT\-4o\-mini was accessed through the OpenAI API\.

### 4\.2Prompting Framework

All models were evaluated using a structured JSON\-based prompting framework designed to encourage explicit reasoning and standardized decision outputs\. Each response included a free\-text reasoning explanation, a final decision label, and a verbal self\-reported confidence score\.

The decision space was restricted toYes,No, andAmbiguous\. TheAmbiguouslabel was used for cases where the available information was insufficient to safely determine whether an additional medication dose could be recommended without making unsupported assumptions\.

To improve evaluation consistency, prompts instructed models to produce valid JSON\-only outputs using a predefined schema containing reasoning, decision, and confidence fields\. The full prompt template is provided in Appendix[A](https://arxiv.org/html/2606.04262#A1)\.

### 4\.3Repeated Sampling Protocol

EachDoseBenchscenario was independently generated five times per model to analyze repeated\-run stability and response consistency under stochastic generation conditions\. Repeated sampling has previously been used to study self\-consistency and reasoning stability in LLMsWang et al\. \([2023](https://arxiv.org/html/2606.04262#bib.bib21)\)\. Across four evaluated models and 81 benchmark scenarios, this produced a total of 1,620 model generations for analysis\.

Repeated sampling enabled evaluation of single\-run correctness and inter\-run stability\. For each question, we computed the majority decision across five runs, whether that majority decision matched the gold label, and the agreement rate among repeated generations\. Consistency was computed as the proportion of repeated generations matching the majority decision for a given question\.

This framework allows evaluation of whether models remain stable and reliable across repeated generations rather than relying solely on single\-response correctness\.

### 4\.4Evaluation Metrics

We evaluate model behavior using complementary dimensions that capture both final\-answer quality and reliability\-related behavior: correctness, repeated\-run consistency, reasoning verifiability, failure patterns, and confidence\-related signals\.

#### Correctness and Consistency

Correctness evaluates whether the model’s predicted decision label matches the gold decision label for a given scenario\. For each model and question, we generated five independent responses and parsed each final decision into one ofYes,No, orAmbiguous\. We then computed the majority decision across the five runs\. Majority\-vote correctness was assigned a value of 1 when the majority decision matched the gold label and 0 otherwise\.

Consistency measures repeated\-run stability for the same question\. We compute consistency as the proportion of the five generations that match the majority decision\. For example, if a model produces threeYesresponses, oneNo, and oneAmbiguous, the majority decision isYesand the consistency rate is 3/5 = 0\.6\. This agreement rate is reported as the consistency metric in model\-level results\. This metric captures whether a model repeatedly gives the same answer, regardless of whether that answer is correct\.

#### Verifiability

Verifiability evaluates whether the model’s reasoning can be explicitly checked using the question context and OTC dosing guidance\. Responses were manually annotated using a three\-level scale: 2 \(fully verifiable\), 1 \(partially verifiable\), and 0 \(non\-verifiable or incorrect reasoning\)\. This metric captures whether the explanation is medically grounded and logically checkable, rather than only whether the final decision label is correct\.

#### Failure Analysis

To characterize dominant reasoning failures, incorrect and partially correct responses were assigned one primary failure type: \(1\)Factual Error, where the response contains incorrect calculations, medically inaccurate statements, or unsafe recommendations; \(2\)Incomplete Reasoning, where the response is partially reasonable but lacks key dosage or timing justification; and \(3\)Ambiguous Reasoning, where the response makes unsupported assumptions or fails to handle missing information appropriately\.

#### Confidence\-Related Signals

In addition to verbal self\-reported confidence scores, we extracted probability\-based confidence signals from model generation probabilities, including decision confidence, entropy\-based uncertainty, and decision margin for the final predicted label\. Prior work has used token\-level probabilities and entropy\-based measures to analyze confidence and calibration behavior in LLMsKadavath et al\. \([2022](https://arxiv.org/html/2606.04262#bib.bib9)\); Lin et al\. \([2022](https://arxiv.org/html/2606.04262#bib.bib13)\)\. We use these signals as diagnostic indicators rather than directly calibrated probabilities, since probability outputs and APIs differ across models\.

## 5Results

### 5\.1Overall Benchmark Performance

Table 3:Overall benchmark performance across evaluated models\. Accuracy is reported as majority\-vote accuracy over 81 curated OTC reasoning scenarios\. Consistency measures repeated\-run agreement across five generations per scenario\. The consistency gap measures the difference between repeated\-run consistency and majority\-vote accuracy, highlighting that models can repeatedly generate the same incorrect recommendation across multiple runs\. Verifiability is averaged over manually annotated responses on a 0–2 scale\. Internal confidence is computed from token\-level decision probabilities averaged across generated runs\.Table[3](https://arxiv.org/html/2606.04262#S5.T3)summarizes overall performance onDoseBenchusing majority\-vote accuracy, repeated\-run consistency, manual verifiability, and confidence\-related signals\. The benchmark contains 81 curated OTC dosing scenarios and produces 1,620 model generations across four models and five repeated runs per scenario\. Unless otherwise stated, accuracy is computed from the majority decision across repeated runs, while consistency is reported as the average agreement rate with the majority decision\. Confidence\-related statistics are averaged across generated responses\.

All evaluated models showed substantial difficulty on realistic OTC dosing questions\. GPT\-4o\-mini achieved the strongest overall performance across several metrics, while Qwen2\.5\-7B was the strongest open\-source model in terms of majority\-vote accuracy\. However, no model achieved high reliability across all evaluation dimensions, indicating that OTC dosing remains challenging even in a constrained decision space\.

The results also show that correctness and consistency capture different aspects of model behavior\. Verifiability analysis further showed that stronger final\-answer accuracy did not always correspond to stronger reasoning quality\. Some models produced interpretable reasoning despite lower accuracy, while others generated correct decisions supported by incomplete or weak explanations\. This suggests that final\-answer accuracy alone is insufficient for evaluating safety\-relevant OTC medication reasoning\.

### 5\.2Performance by Reasoning Complexity

![Refer to caption](https://arxiv.org/html/2606.04262v1/x1.png)Figure 3:Mean majority\-vote accuracy across reasoning complexity categories\. Error bars show standard deviation across evaluated models\. Timing interval and repeated\-dosing scenarios showed lower average performance and larger variation, suggesting unstable temporal and cumulative dosing reasoning behavior\.Figure[3](https://arxiv.org/html/2606.04262#S5.F3)summarizes majority\-vote accuracy across reasoning complexity categories\. Performance varied substantially by category, indicating that model reliability depends on the type of dosing reasoning required rather than overall model accuracy alone\.

Timing interval and repeated\-dosing scenarios showed lower average accuracy and larger variation across models\. Rolling 24\-hour cases also remained challenging for several systems, while multi\-medication scenarios produced comparatively stronger and more stable performance\. Missing\-information scenarios showed notable cross\-model variation, suggesting that models differ in how cautiously they handle incomplete medication histories\.

These results show thatDoseBenchcaptures distinct reasoning challenges rather than a single uniform difficulty pattern\. Detailed per\-model category results are provided in Table[9](https://arxiv.org/html/2606.04262#A6.T9)\.

### 5\.3Confidence, Consistency, and Reliability

Beyond final\-answer accuracy, we evaluated whether confidence and repeated\-run agreement reliably reflected correct OTC medication reasoning behavior\. A central finding is that models often remained highly consistent and confident even when producing medically incorrect or unsupported recommendations\.

Repeated\-run consistency substantially exceeded majority\-vote accuracy across all evaluated models \(Table[3](https://arxiv.org/html/2606.04262#S5.T3)\)\. In many cases, models repeatedly generated the same incorrect recommendation across multiple runs, indicating that repeated agreement alone does not necessarily correspond to reliable reasoning behavior\. For example, GPT\-4o\-mini achieved 84\.9% consistency despite only 55\.6% majority\-vote accuracy, while Llama3\-8B produced 73\.8% consistency with only 43\.2% accuracy\. Across questions, this consistency\-correctness gap was statistically significant \(Wilcoxon signed\-rank test,p<10−4p<10^\{\-4\}\), and the same pattern held across all evaluated models\.

Confidence signals showed a similar reliability gap\. Figure[4](https://arxiv.org/html/2606.04262#S5.F4)compares average internal confidence for correct and incorrect responses across evaluated models\. Several systems assigned high confidence to incorrect recommendations, with confidence levels approaching or exceeding those of correct responses\. This suggests that internal confidence alone is not a sufficient indicator of safe or correct OTC dosing reasoning\.

![Refer to caption](https://arxiv.org/html/2606.04262v1/x2.png)Figure 4:Average internal decision confidence for correct and incorrect generated responses across evaluated models\. Several models remained highly confident even when producing incorrect medication recommendations, indicating weak alignment between confidence and correctness\.Figure[5](https://arxiv.org/html/2606.04262#S5.F5)additionally shows a substantial mismatch between verbal self\-reported confidence and internal token\-level confidence signals\. Several models remained internally overconfident even when generating medically incorrect or weakly supported recommendations\.

![Refer to caption](https://arxiv.org/html/2606.04262v1/x3.png)Figure 5:Distribution of mismatch between internal token\-level confidence and verbal self\-reported confidence across evaluated models\. Positive values indicate higher internal confidence relative to verbal confidence, revealing cases where models remained internally overconfident despite weaker or incorrect recommendations\.Several observed failures also appeared systematic rather than random\. In some ambiguity\-sensitive questions, models repeatedly produced overly conservative or unsupported decisions, such as selectingAmbiguouswhen the available information supported aNodecision or generating high\-confidenceNoanswers for cases labeledYes\. These patterns suggest that OTC dosing failures may reflect systematic safety\-reasoning errors rather than simple generation variability\.

### 5\.4Verifiability and Failure Analysis

#### Verifiability

Final\-answer correctness did not always correspond to medically grounded or explicitly verifiable reasoning\. Manual verifiability scores provide a complementary view of model reliability beyond final\-answer correctness\. A response may select the correct decision label but still provide reasoning that is vague, incomplete, or unsupported\. Conversely, an incorrect decision may sometimes include partially relevant reasoning but fail at a key dosage or timing step\.

Figure[6](https://arxiv.org/html/2606.04262#S5.F6)summarizes the distribution of manually annotated reasoning\-quality categories across evaluated models\. These categories are separate from the numeric verifiability score: the verifiability score measures how checkable the reasoning is on a 0–2 scale, while the category label identifies the dominant reasoning outcome or failure type\. The quality of reasoning varied substantially across systems, suggesting that final\-answer accuracy alone does not fully capture whether the generated recommendation is medically grounded or explicitly checkable\. Several responses received low verifiability scores because they omitted key dosage information, failed to apply the relevant timing constraint, or made unsupported assumptions about missing medication history\.

![Refer to caption](https://arxiv.org/html/2606.04262v1/x4.png)Figure 6:Distribution of manually annotated reasoning\-quality and failure categories across evaluated models\. Models frequently produced incomplete, unsupported, or weakly verifiable reasoning despite fluent explanations\.
#### Failure Types

Beyond numeric verifiability scores, we examined the dominant failure types observed in weak or incorrect responses\. Common failures included incorrect cumulative dosage calculations, incorrect next\-dose timing recommendations, incomplete reasoning, and unsupported assumptions under ambiguous medication histories\. Some models also generated confident but medically unsupported dosage statements, such as misstating daily limits or inventing timing rules not grounded in OTC guidelines\. These patterns show why reasoning verifiability is important: fluent explanations may still be medically unreliable\.

#### Failure Example

> Question: “I took 2 Tylenol 325mg caplets at 8:00 AM, 12:00 PM, and 4:00 PM, then 2 ibuprofen tablets at 6:00 PM, and later 2 Tylenol again at 10:00 PM and 2:00 AM\. It’s 11:00 AM now\. Can I take more Tylenol?” Model response \(Mistral7B\): “The last dose of Tylenol was taken at 2:00 AM, and based on typical dosing guidelines for acetaminophen, it is recommended to wait at least 4 hours between doses\. As it’s only 11:00 AM now, another dose of Tylenol should be avoided until at least 6:00 PM\.” Model decision:No, confidence = 9 Gold reasoning: The final recommendation is medically unsupported\. The model incorrectly interprets the dosing interval despite more than 9 hours having passed since the last Tylenol dose at 2:00 AM\. The response also fails to correctly reason about cumulative rolling 24\-hour intake and produces a highly confident but internally inconsistent recommendation\.

Importantly, some partially correct or even correct final decisions were still supported by weak reasoning\. In several cases, models selected the correct decision label while providing explanations that were vague, incomplete, or difficult to verify\. These findings suggest that final\-answer correctness alone may overestimate model reliability when the reasoning process remains medically unsupported or weakly justified\.

## 6Discussion

Our findings show thatDoseBenchexposes reliability weaknesses that are not captured by final\-answer accuracy alone\. Many failures were not simple arithmetic mistakes alone but involved unsupported assumptions, weak handling of incomplete medication histories, or fluent explanations that were difficult to verify against the provided dosing context\.

Repeated\-run evaluation further showed that stability does not necessarily imply reliability\. Across evaluated systems, models often produced consistent and confident responses even when the majority decision was incorrect\. This suggests that repeated agreement and internal confidence should be interpreted cautiously in safety\-sensitive medication reasoning tasks\.

These results highlight the need for evaluation methods that jointly consider correctness, consistency, verifiability, and uncertainty behavior\. Future OTC medication QA systems may benefit from explicit rule\-grounding, temporal state tracking, or tool\-assisted dosage checking rather than relying only on parametric model reasoning\.

## 7Conclusion

We introducedDoseBench, a focused OTC medication reasoning benchmark for adult acetaminophen and ibuprofen dosing under public dosage\-label constraints\. Through repeated\-run evaluation across four LLMs, we analyzed temporal dosage tracking, rolling 24\-hour reasoning, consistency, confidence, and reasoning verifiability\. Our results show that models can remain stable and highly confident even when producing medically incorrect or weakly supported recommendations, suggesting that repeated agreement and confidence alone are insufficient indicators of reliable medical reasoning\. Overall, OTC dosing provides a narrow but reproducible setting for studying reliability\-oriented evaluation in consumer\-facing medical QA\.

## Limitations

DoseBenchfocuses on a narrow OTC setting: adult acetaminophen and ibuprofen dosing under public Drug Facts constraints\. This limits generalizability to broader clinical settings, but enables a controlled and reproducible testbed for temporal dosage reasoning and reliability analysis\.

The benchmark is also modest in size compared with large\-scale medical QA datasets\. Our goal is not leaderboard\-scale coverage, but detailed reliability evaluation of repeated\-run stability, reasoning verifiability, and confidence\-related behavior in a constrained medication\-safety task\.

We do not yet include deterministic rule\-based or tool\-augmented baselines grounded in explicit dosage rules\. Manual verifiability and failure annotations may also retain some subjectivity despite structured guidelines\. Future work may expandDoseBenchto additional OTC medications, multilingual scenarios, and larger collections of realistic consumer medication questions\.

## Ethical Considerations

This work evaluates LLM behavior in OTC medication reasoning scenarios and is intended solely for research\.DoseBenchis designed to study dosage\-rule following, uncertainty handling, and safety\-relevant failure patterns, not to provide medical advice or treatment recommendations\.

The benchmark does not contain personally identifiable information, clinical records, or protected health information\. To diversify question phrasing, we used survey\-inspired seed questions in which contributors provided example OTC medication questions as they might ask an AI assistant or search engine\. All benchmark scenarios were manually filtered, rewritten, and curated as synthetic evaluation items grounded in public Drug Facts and DailyMed materials\.

Because medication\-use examples could be misinterpreted as guidance, all model outputs are presented strictly as evaluation artifacts\. Real medication decisions should follow product labels and professional medical advice\.

DoseBenchfocuses on adult acetaminophen and ibuprofen dosing and should not be generalized to broader clinical settings such as pediatric dosing, pregnancy, chronic disease management, or prescription drug interactions\. We plan to release the curated scenarios, annotation guidelines, prompts, and evaluation scripts, subject to final safety and institutional review\.

## Acknowledgments

This work used Jetstream2 at Indiana University through ACCESS allocation CIS260684 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support \(ACCESS\) program, which is supported by U\.S\. National Science Foundation grants \#2138259, \#2138286, \#2138307, \#2137603, and \#2138296\. We thank the Jetstream2 and ACCESS support teams for providing the computational infrastructure used in this work\.

We thank Jinsung Han for assistance with annotation and evaluation review\. We also thank the survey participants who contributed realistic OTC medication questions used duringDoseBenchconstruction\. We are grateful to colleagues and early readers who provided feedback on the benchmark design, evaluation setup, and manuscript\.

## References

- Abacha et al\. \(2019\)Asma Ben Abacha, Yassine Mrabet, Mark Sharp, Travis R\. Goodwin, Sonya E\. Shooshan, and Dina Demner\-Fushman\. 2019\.[Bridging the gap between consumers’ medication questions and trusted answers](https://doi.org/10.3233/SHTI190176)\.In*MEDINFO 2019: Health and Wellbeing e\-Networks for All*, pages 25–29\. IOS Press\.
- AI@Meta \(2024\)AI@Meta\. 2024\.[The llama 3 herd of models](https://arxiv.org/abs/2407.21783)\.*Preprint*, arXiv:2407\.21783\.
- Alonso et al\. \(2024\)Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri\. 2024\.[Medexpqa: Multilingual benchmarking of large language models for medical question answering](https://doi.org/10.1016/j.artmed.2024.102938)\.*Artificial Intelligence in Medicine*, 155:102938\.
- Bentegeac et al\. \(2025\)Raphaël Bentegeac, Bastien Le Guellec, Grégory Kuchcinski, Philippe Amouyel, and Aghiles Hamroun\. 2025\.[Token probabilities to mitigate large language models overconfidence in answering medical questions: Quantitative study](https://doi.org/10.2196/64348)\.*Journal of Medical Internet Research*, 27:e64348\.
- Geng et al\. \(2024\)Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych\. 2024\.[A survey of confidence estimation and calibration in large language models](https://doi.org/10.18653/v1/2024.naacl-long.366)\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 6577–6595, Mexico City, Mexico\. Association for Computational Linguistics\.
- Hu and Khan \(2021\)Yibo Hu and Latifur Khan\. 2021\.Uncertainty\-aware reliable text classification\.In*Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining*, pages 628–636\.
- Jiang et al\. \(2023\)Albert Q\. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie\-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed\. 2023\.[Mistral 7b](https://arxiv.org/abs/2310.06825)\.*arXiv preprint arXiv:2310\.06825*\.
- Jin et al\. \(2021\)Di Jin, Eileen Pan, Nassim Oufattole, Wei\-Hung Weng, Hanyi Fang, and Peter Szolovits\. 2021\.[What disease does this patient have? a large\-scale open domain question answering dataset from medical exams](https://doi.org/10.3390/app11146421)\.*Applied Sciences*, 11\(14\):6421\.
- Kadavath et al\. \(2022\)Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield\-Dodds, Nova DasSarma, Eli Tran\-Johnson, Scott Johnston, Sheer El\-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others\. 2022\.[Language models \(mostly\) know what they know](https://arxiv.org/abs/2207.05221)\.*arXiv preprint arXiv:2207\.05221*\.
- Kell et al\. \(2025\)Gregory Kell, Angus Roberts, Serge Umansky, Yuti Khare, Najma Ahmed, Nikhil Patel, Chloe Simela, Jack Coumbe, Julian Rozario, Ryan\-Rhys Griffiths, and Iain J\. Marshall\. 2025\.Realmedqa: A pilot biomedical question answering dataset containing realistic clinical questions\.In*AMIA Annual Symposium Proceedings*, volume 2024, pages 590–598\.
- Kim et al\. \(2024\)Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu\. 2024\.[Medexqa: Medical question answering benchmark with multiple explanations](https://doi.org/10.18653/v1/2024.bionlp-1.14)\.In*Proceedings of the 23rd Workshop on Biomedical Natural Language Processing*, pages 167–181, Bangkok, Thailand\. Association for Computational Linguistics\.
- Kung et al\. \(2023\)Tiffany H\. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz\-Candido, James Maningo, and Victor Tseng\. 2023\.[Performance of chatgpt on usmle: Potential for ai\-assisted medical education using large language models](https://doi.org/10.1371/journal.pdig.0000198)\.*PLOS Digital Health*, 2\(2\):e0000198\.
- Lin et al\. \(2022\)Stephanie Lin, Jacob Hilton, and Owain Evans\. 2022\.[Teaching models to express their uncertainty in words](https://openreview.net/forum?id=8s8K2UZGTZ)\.*Transactions on Machine Learning Research*\.
- Manakul et al\. \(2023\)Potsawee Manakul, Adian Liusie, and Mark Gales\. 2023\.[Selfcheckgpt: Zero\-resource black\-box hallucination detection for generative large language models](https://doi.org/10.18653/v1/2023.emnlp-main.557)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 9004–9017, Singapore\. Association for Computational Linguistics\.
- OpenAI \(2024\)OpenAI\. 2024\.[Gpt\-4o system card](https://arxiv.org/abs/2410.21276)\.*arXiv preprint arXiv:2410\.21276*\.
- Pal et al\. \(2022\)Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu\. 2022\.[Medmcqa: A large\-scale multi\-subject multi\-choice dataset for medical domain question answering](https://proceedings.mlr.press/v174/pal22a.html)\.In*Proceedings of the Conference on Health, Inference, and Learning*, volume 174 of*Proceedings of Machine Learning Research*, pages 248–260\. PMLR\.
- Qwen Team \(2024\)Qwen Team\. 2024\.[Qwen2\.5 technical report](https://arxiv.org/abs/2412.15115)\.*arXiv preprint arXiv:2412\.15115*\.
- Singhal et al\. \(2025\)Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R\. Pfohl, Heather Cole\-Lewis, Darlene Neal, Qazi Mamunur Rashid, Mike Schaekermann, Amy Wang, Dev Dash, Jonathan H\. Chen, Nigam H\. Shah, Sami Lachgar, Philip Andrew Mansfield, and 16 others\. 2025\.[Toward expert\-level medical question answering with large language models](https://doi.org/10.1038/s41591-024-03423-7)\.*Nature Medicine*, 31\(3\):943–950\.
- Suri et al\. \(2021\)Huqun Suri, Qi Zhang, Wenhua Huo, Yan Liu, and Chunsheng Guan\. 2021\.[Mediaqa: A question answering dataset on medical dialogues](https://arxiv.org/abs/2108.08074)\.*arXiv preprint arXiv:2108\.08074*\.
- U\.S\. National Library of Medicine \(2025\)U\.S\. National Library of Medicine\. 2025\.[Dailymed](https://dailymed.nlm.nih.gov/dailymed/)\.
- Wang et al\. \(2023\)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V\. Le, Ed H\. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou\. 2023\.[Self\-consistency improves chain of thought reasoning in language models](https://openreview.net/forum?id=1PL1NIMMrw)\.In*International Conference on Learning Representations*\.
- Zhu et al\. \(2019\)Ming Zhu, Aman Ahuja, Wei Wei, and Chandan K\. Reddy\. 2019\.[A hierarchical attention retrieval model for healthcare question answering](https://doi.org/10.1145/3308558.3313699)\.In*The World Wide Web Conference*, pages 2472–2482\. Association for Computing Machinery\.

## Appendix APrompt Template

We used a structured prompt to obtain both free\-text reasoning and a constrained decision output\. In the prompt, models selected one of three decision letters: A, B, or C\. These letters were then mapped toYes,No, andAmbiguous, respectively, for evaluation\. TheAmbiguouslabel was used when the available information was insufficient for a safe recommendation\.

Youareansweringanover\-the\-countermedicationdosingquestion\.

Medicationquestion:

\{question\_text\}

ReturnyouranswerONLYinvalidJSONusingthisschema:

\{

"reasoning":"1\-2shortsentencesexplainingyourreasoning",

"decision":"A",

"confidence":1

\}

Decisionmapping:

A=YES

B=NO

C=AMBIGUOUS

Rules:

\-Thevalueof"decision"mustbeexactlyoneof:

"A","B",or"C"\.

\-Thevalueof"confidence"mustbeaninteger

from1to10\.

\-UseCiftheinformationisincomplete,

conflicting,orinsufficient\.

\-Donotincludemarkdown,codefences,

oranytextoutsidetheJSONobject\.

For token\-level confidence extraction, we additionally used a constrained decision\-only prompt:

Youareansweringanover\-the\-countermedicationdosingquestion\.

Medicationquestion:

\{question\_text\}

Chooseexactlyonedecision\.

Decisionmapping:

A=YES

B=NO

C=AMBIGUOUS

Returnonlyoneletter:A,B,orC\.

## Appendix BAnnotation Guidelines

Gold labels and reasoning\-quality annotations inDoseBenchfollowed structured guidelines grounded in public Drug Facts and DailyMed dosage constraints\. Annotators reviewed each benchmark scenario using conservative safety\-oriented reasoning rules for OTC acetaminophen and ibuprofen dosing\.

### B\.1Gold Decision Labels

EachDoseBenchscenario was assigned one of three decision labels:

- •Yes: available dosage and timing information supported taking an additional dose within public\-label constraints\.
- •No: available information indicated that another dose would violate timing, cumulative dosage, or safety constraints\.
- •Ambiguous: available information was incomplete, conflicting, or insufficient for a medically safe recommendation without unsupported assumptions\.

Gold annotations emphasized conservative handling of uncertainty\-sensitive questions involving missing dosage history, uncertain medication strength, or incomplete timing information\.

### B\.2Reasoning Verifiability

Model reasoning quality was manually reviewed using a three\-level verifiability rubric:

- •Score = 2 \(Clearly Verifiable\): reasoning was medically grounded, logically clear, and easy to verify from the question context and public dosage guidance\.
- •Score = 1 \(Partially Verifiable\): reasoning was partially correct but vague, incomplete, or insufficiently justified\.
- •Score = 0 \(Not Verifiable\): reasoning was factually incorrect, contradictory, unsupported, or medically unreliable\.

Annotations focused primarily on reasoning quality rather than final\-answer correctness alone\.

### B\.3Failure\-Type Annotation

For each response, annotators assigned one primary reasoning\-quality category:

- •Verifiable Reasoning: reasoning was medically grounded and explicitly checkable\.
- •Factual Error: response contained incorrect dosage limits, interval rules, or medically unsupported statements\.
- •Incomplete Reasoning: response omitted key dosage calculations or timing constraints\.
- •Ambiguous Reasoning: response made unsupported assumptions or failed to handle missing information appropriately\.

Failure\-type annotations were assigned to problematic responses, and representative case\-level trends were summarized across repeated generations\.

Table 4:Representative survey\-inspired OTC medication questions used during dataset construction\.

## Appendix CDataset Construction Examples

### C\.1Survey\-Inspired Seed Questions

To improve realism and reduce author\-specific phrasing bias, we collected survey\-inspired OTC medication questions written in natural conversational language\. Contributors were asked to provide example OTC medication questions as they might naturally ask an AI assistant or search engine\. Responses were manually reviewed and refined before inclusion in the benchmark\.

Table[4](https://arxiv.org/html/2606.04262#A2.T4)presents representative survey\-inspired seed questions covering several OTC medication reasoning categories\.

### C\.2Dataset Refinement and Annotation Examples

Table 5:Examples of manual refinement applied to survey\-inspired OTC medication questions during benchmark construction\.Collected survey\-inspired seed questions were manually reviewed to remove duplicate, incomplete, medically irrelevant, or non\-OTC medication entries\.

Table[5](https://arxiv.org/html/2606.04262#A3.T5)presents representative examples of the manual refinement process used during benchmark construction\.

FinalizedDoseBenchscenarios were manually annotated with gold decision labels, reasoning complexity categories, key dosage facts, and annotation rationales using structured guidelines grounded in public OTC dosage guidance\. Annotation emphasized conservative safety\-oriented reasoning, particularly for ambiguity\-sensitive cases involving incomplete medication histories or uncertain dosage information\.

Table[6](https://arxiv.org/html/2606.04262#A3.T6)presents representative examples of finalized benchmark annotations\.

Table 6:Representative examples of finalizedDoseBenchannotations, including gold decision labels, reasoning complexity categories, key dosage facts, and annotation rationales\.

## Appendix DDetailed Experimental Configuration

All experiments used five repeated generations per benchmark scenario\. Open\-source models were deployed locally using HuggingFace Transformers with 4\-bit quantization on one NVIDIA A100 GPU, while GPT\-4o\-mini was accessed through the OpenAI API\.

Table[7](https://arxiv.org/html/2606.04262#A4.T7)summarizes the primary inference and evaluation settings used during benchmark generation\.

Table 7:Primary runtime and inference settings used during evaluation\.Table 8:Average entropy and decision\-margin statistics computed from case\-level summaries across 81 benchmark scenarios\. Case\-level values were averaged over five repeated generations\.Confidence\-related metrics were computed from token\-level decision probabilities extracted using constrained single\-token decision generation\. Entropy was computed over normalized decision\-label probabilities, while decision margin was defined as the probability difference between the top two candidate decisions\.

## Appendix EAdditional Confidence Results

![Refer to caption](https://arxiv.org/html/2606.04262v1/x5.png)Figure 7:Average decision entropy for correct and incorrect responses across evaluated models\. Higher entropy indicates greater uncertainty during decision selection\.Figure[7](https://arxiv.org/html/2606.04262#A5.F7)compares entropy values between correct and incorrect responses across evaluated models\. Llama3\-8B exhibited substantially higher entropy overall, indicating weaker separation between confident and uncertain decisions\.

Table[8](https://arxiv.org/html/2606.04262#A4.T8)reports average entropy and decision\-margin statistics computed from case\-level token\-probability summaries\.

## Appendix FAdditional Evaluation Examples

Table 9:Detailed majority\-vote accuracy \(%\) across OTC medication reasoning complexity categories for each evaluated model\. Mean and standard deviation are computed across models for each category\.Table[9](https://arxiv.org/html/2606.04262#A6.T9)reports detailed majority\-vote accuracy across reasoning complexity categories for each evaluated model\.

Table 10:Representative examples of manual verifiability and failure\-type annotations\. Verifiability was scored on a 0–2 scale, where 2 indicates clearly checkable and medically grounded reasoning\.Table[10](https://arxiv.org/html/2606.04262#A6.T10)presents representative examples of manual verifiability and failure\-type annotations assigned during analysis\. Verifiability scores indicate whether model reasoning was medically grounded and explicitly checkable, while failure types describe the dominant reasoning problem observed in incorrect or weakly supported responses\.

Table 11:Representative repeated\-run failure case showing stable but incorrect reasoning across five generations\.Table[11](https://arxiv.org/html/2606.04262#A6.T11)illustrates a representative repeated\-run failure case in which the model consistently produced the same unsupported recommendation across all five generations despite incomplete medication information\.

## Appendix GExampleDoseBenchScenarios

Table 12:Representative OTC reasoning scenarios sampled from the final benchmark\.Table[12](https://arxiv.org/html/2606.04262#A7.T12)presents representative OTC reasoning scenarios covering several reasoning categories in the final dataset\.

## Appendix HRepresentative Model Outputs

Table[13](https://arxiv.org/html/2606.04262#A8.T13)presents representative examples of successful and failure\-prone OTC medication reasoning behavior\.

Table 13:Representative examples of successful and failure\-prone OTC medication reasoning\.Successful ExampleQuestion:I took 2 extra strength Tylenol caplets \(500 mg each\) at 2:00 PM and 6:00 PM\. I still have a headache\. Can I take another dose at 12:00 AM?Gold Decision:YESModel:GPT\-4o\-miniModel Output:\{ "reasoning": "The doses were spaced 4\-\-6 hours apart, and another dose at midnight would remain within standard timing recommendations\.", "decision": "YES", "confidence": 9 \}Annotation Summary:Correct recommendation with explicit temporal reasoning and medically grounded justification\.Failure ExampleQuestion:I took Tylenol several times today but forgot exactly how many\. Is it okay if I take 2 more before sleeping?Gold Decision:AMBIGUOUSModel:Mistral7BModel Output:\{ "reasoning": "You can probably take another dose if your earlier doses were not too large\.", "decision": "YES", "confidence": 8 \}Annotation Summary:The model produced an unsafe recommendation by making unsupported assumptions despite missing cumulative dosage information\.