LongMedBench: Benchmarking Medical Agents for Long-Horizon Clinical Decision-Making

# LongMedBench: Benchmarking Medical Agents for Long-Horizon Clinical Decision-Making
Source: [https://arxiv.org/html/2607.09322](https://arxiv.org/html/2607.09322)
11institutetext:Zhejiang University, Hangzhou, China
11email:\{zihan1\.22,yanzhen\.22,xiaocheng\.22,zhiting\.23\}@intl\.zju\.edu\.cn11email:einstein@zju\.edu\.cn, zuozhuliu@intl\.zju\.edu\.cn22institutetext:Alibaba Group, Hangzhou, China
22email:zhaiweiqi\.zwq@alibaba\-inc\.com33institutetext:Transvascular Implantation Devices Research Institute, Hangzhou, ChinaYanzhen Chen\*Xiaocheng Zhang\*Zhiting FanWeiqi ZhaiHongxia XuZuozhu Liu†\\dagger

###### Abstract

In this work, we introduce LongMedBench, a real\-world EHR\-based benchmark for long\-horizon clinical decision\-making\. Prior evaluations of LLM\-based medical agents have largely emphasized short\-context knowledge QA and tool use\. However, real\-world medical care is inherently longitudinal, and clinicians must aggregate evidence across repeated visits, tests, and evolving treatments\. Therefore, long\-horizon interaction is essential for realistic assessment\. LongMedBench is constructed via a reproducible pipeline that integrates MIMIC\-IV admission records and clinical notes into time\-series event streams and long\-context memory datasets, enabling long\-horizon, multi\-session interactions between agents and a clinical environment\. It comprises 335 patients, with 19\.72 inpatient visits per patient on average and 44\.91 medical events per visit\. Guided by the long\-horizon decision process, we propose an evaluation taxonomy with three suites: fact\-based QA, temporal reasoning, and long\-horizon decision\-making\. This taxonomy measures how agents understand and leverage historical patient information over extended horizons\. Our experiments show that while recent LLMs can make good use of explicit timestamps, they have challenges in implicit time inference; The RAG and agent memory system can improve the performance of information retrieval tasks, but the performance of decision\-making tasks is highly dependent on the model’s immediate context\.

11footnotetext:\*These authors contributed equally to the work\.22footnotetext:†\\daggerCorresponding author: Zuozhu Liu\.## 1Introduction

In clinical practice, the diagnostic process is inherently longitudinal and time\-dependent\. Clinicians do not merely react to isolated symptoms; instead, they must synthesize evidence across multiple visits, diagnostic tests, and evolving treatment responses over years\[[4](https://arxiv.org/html/2607.09322#bib.bib10)\]\. This capacity oflong\-horizon clinical reasoningis fundamental to high\-quality care\. As Large Language Models \(LLMs\) transition to autonomous medical agents, their ability to navigate complex trajectories in real\-world Electronic Health Records \(EHR\)\[[7](https://arxiv.org/html/2607.09322#bib.bib24)\]has become the critical benchmark for clinical readiness\.

Table 1:Comparison of Medical Agent Benchmarks\.BenchmarkLong\.aCtx\.bEHR\.cDec\.dTemp\.eFocusMedAgentBench\[[6](https://arxiv.org/html/2607.09322#bib.bib14)\]×\\times×\\times✓\\checkmark✓\\checkmark×\\timesEHR Tool IntegrationAgentClinic\[[16](https://arxiv.org/html/2607.09322#bib.bib33)\]×\\times×\\times×\\times✓\\checkmark×\\timesSimulated InteractionDiagBench\[[14](https://arxiv.org/html/2607.09322#bib.bib20)\]×\\times×\\times✓\\checkmark✓\\checkmark×\\timesDiagnostic TrajectoryMedBench v4\[[3](https://arxiv.org/html/2607.09322#bib.bib16)\]×\\times✓\\checkmark✓\\checkmark×\\times×\\timesMedical Knowledge QAReflecTool\[[9](https://arxiv.org/html/2607.09322#bib.bib12)\]×\\times×\\times×\\times✓\\checkmark×\\timesReflective Tool\-UseEHRSQL\[[8](https://arxiv.org/html/2607.09322#bib.bib7)\]×\\times×\\times✓\\checkmark×\\times×\\timesRelational QueryingLongMedBench✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmarkLong\-horizon ReasoningaLongitudinal: Reasoning across multiple discrete clinical visits\.bContext: Whether the benchmark environment contains multi\-turn context\.cEHR: The dataset built upon real\-world EHR\.dDecision\-making: Evaluating proactive clinical planning beyond static retrieval or SQL\-based querying\.eTemporal Sensitivity: Evaluating understanding of clinical timeline and urgency\.Although general\-purpose benchmarks have emerged to evaluate long\-horizon or multi\-turn interactions\[[18](https://arxiv.org/html/2607.09322#bib.bib5),[17](https://arxiv.org/html/2607.09322#bib.bib6),kočiský2017narrativeqareadingcomprehensionchallenge,[19](https://arxiv.org/html/2607.09322#bib.bib3),[5](https://arxiv.org/html/2607.09322#bib.bib2),[3](https://arxiv.org/html/2607.09322#bib.bib16)\], these benchmarks primarily emphasizeexplicit reasoning based on information retrieval—the ability to locate specific facts within a long context\[[10](https://arxiv.org/html/2607.09322#bib.bib32)\]\. They test a model’s capacity to find a “needle in a haystack", but overlook the temporal dynamics that are essential to clinical reasoning\. In real\-world medical scenarios, the challenge extends beyond retrieving isolated facts to understanding how a patient’s state evolves over dozens of visits and how this temporal progression informs treatment strategies\.

In parallel with these general limitations, current medical evaluation paradigms also fall short of this requirement, as shown in Table[1](https://arxiv.org/html/2607.09322#S1.T1)\. Despite introducing simulated interactive environments, recent frameworks\[[14](https://arxiv.org/html/2607.09322#bib.bib20),[6](https://arxiv.org/html/2607.09322#bib.bib14),[16](https://arxiv.org/html/2607.09322#bib.bib33),[3](https://arxiv.org/html/2607.09322#bib.bib16),[9](https://arxiv.org/html/2607.09322#bib.bib12),[8](https://arxiv.org/html/2607.09322#bib.bib7)\]areconstrained by limited context windows and session counts, emphasizing the agent’s tool\-call abilities and immediate prediction rather than understanding of complete clinical trajectories\. Consequently, they fail to assess how models utilize extensive medical history for future clinical decisions\.

To address these limitations, we introduce LongMedBench, a MIMIC\-IV\[[7](https://arxiv.org/html/2607.09322#bib.bib24)\]\-based benchmark that overcomes constrained context lengths and simple factual recall by curating extensive longitudinal trajectories and designing reasoning tasks that challenge agent’s temporal sensitivity\. By converting 335 patient records into longitudinal event streams \(averaging 19\.72 visits per patient\), we construct a temporally dense environment and a framework explicitly targeting long\-horizon clinical reasoning\. Our contributions are: \(1\)A tri\-level memory dataset architecturefor different granularity history evaluation\. \(2\)A progressive evaluation taxonomythat spans three hierarchical tasks:factual QAbased on timestamp or relative positioning, targeting the fact retrieval limitation of general benchmarks;temporal reasoningfor multi\-visit and event\-level ordering, addressing the lack of time\-sensitive evaluation in existing medical frameworks; andlong\-horizon decision\-makingwhich directly challenges the agent’s ability to navigate extensive histories and autonomously plan next\-step clinical actions\. \(3\)Experimental findings: we showed that while state\-of\-the\-art LLMs can exploit explicit timestamps, they struggle with the implicit temporal reasoning required for visit\-level understanding\. Although RAG and memory systems improve fact retrieval performance, decision\-making accuracy remains highly dependent on immediate context, highlighting a profound limitation in reasoning over long\-term clinical trajectories\.

## 2Methodology

### 2\.1EHR Data Processing Pipeline

LongMedBench is built on MIMIC\-IV\[[7](https://arxiv.org/html/2607.09322#bib.bib24)\], a public database that contains medical records for\>100,000\>100,000patients\. To convert static EHRs into an interactive agent environment, we design a three\-stage pipeline \(Figure[1](https://arxiv.org/html/2607.09322#S2.F1)\)\.

![Refer to caption](https://arxiv.org/html/2607.09322v1/x1.png)Figure 1:Data processing pipeline for LongMedBench#### 2\.1\.1Event & Patient Filtering

To reduce redundancy while preserving clinical signals, we retain only abnormal lab results and patients with complete admission/discharge records\. Filtering for patients with≥15\\geq 15hospitalizations yields 355 patients and 6,999 visits\. With an average of 19\.72 visits per patient \(median=18\.00, SD=5\.72\), this dense multi\-session structure rigorously evaluates the agent’s ability in cross\-session memorization and temporal reasoning\.

#### 2\.1\.2Event Stream Construction

A complete event stream will be constructed for each patient𝒫\\mathcal\{P\}\. The event stream𝒮\\mathcal\{S\}contains a patient’s visit records𝒮:=\{𝒱1,𝒱2,…,𝒱n\}\\mathcal\{S\}:=\\\{\\mathcal\{V\}\_\{1\},\\mathcal\{V\}\_\{2\},\\dots,\\mathcal\{V\}\_\{n\}\\\}, where𝒱i\\mathcal\{V\}\_\{i\}is theithi\_\{\\text\{th\}\}visit of the patient andn≥15n\\geq 15\. On average, each visit𝒱i\\mathcal\{V\}\_\{i\}contains 44\.91 medical events \(median=25\.00, SD=70\.32\)\.

A visit𝒱i\\mathcal\{V\}\_\{i\}begins with an admission eventEia​d​mE^\{adm\}\_\{i\}, followed by a series of specific medical events, and ends with a discharge eventEid​i​sE^\{dis\}\_\{i\}\. Thekthk\_\{\\text\{th\}\}medical event in𝒱i\\mathcal\{V\}\_\{i\}is denoted asEik=\{aik,tik,pik,oik\}E^\{k\}\_\{i\}=\\\{a^\{k\}\_\{i\},t^\{k\}\_\{i\},p^\{k\}\_\{i\},o^\{k\}\_\{i\}\\\}, whereaika^\{k\}\_\{i\}is an action within a typical clinic action space𝒜\\mathcal\{A\}, which includesimaging, lab tests, medication,*etc*\.tikt^\{k\}\_\{i\}is the event timestamp, andpikp^\{k\}\_\{i\}is the corresponding action argument,such as the modality \(CT, X\-ray, MRI\) of an imaging event\.oiko^\{k\}\_\{i\}is the event result or clinical observation,like the radiology report in an imaging event\.𝒱i\\mathcal\{V\}\_\{i\}can therefore be expressed as a union of sequenced events:𝒱i=\{Eia​d​m,Ei1,Ei2,…,Eid​i​s\}\\mathcal\{V\}\_\{i\}=\\\{E^\{adm\}\_\{i\},E^\{1\}\_\{i\},E^\{2\}\_\{i\},\\dots,E^\{dis\}\_\{i\}\\\}\.

In addition to the structured hospitalization data, each visit also includes two types of notes: one is radiology notes, which are parsed as imaging events; the other is discharge notes, which are summariesNiN\_\{i\}from visits𝒱i\\mathcal\{V\}\_\{i\}, and can be parsed into admission infoNia​d​mN\_\{i\}^\{adm\}and discharge infoNid​i​sN\_\{i\}^\{dis\}according to logic\.

#### 2\.1\.3Memory Dataset Generation

To evaluate how agents utilize long\-context information, we design three memory modules with progressively finer granularity\. From the beginning history visit𝒱i\\mathcal\{V\}\_\{i\}, current visit𝒱j\\mathcal\{V\}\_\{j\}and reasoning timestampTT, the agent’s memory access is strictly bounded to prevent event leakage\.

1\. Note Memory:ℳN\(i,j\)=\{Ni,Ni\+1,…,Nj−1\}\\mathcal\{M\}^\{\(i,j\)\}\_\{N\}=\\\{N\_\{i\},N\_\{i\+1\},\\dots,N\_\{j\-1\}\\\}, which contains visit\-level summaries from preceding𝒱i\\mathcal\{V\}\_\{i\}to𝒱j−1\\mathcal\{V\}\_\{j\-1\}\. The summary is high\-level compressed and emphasizing global trajectory rather than detailed event recall\.

2\. Event Memory:ℳE\(i,j\)=⋃k=ij−1𝒱k\{\\textstyle\\mathcal\{M\}^\{\(i,j\)\}\_\{E\}=\\bigcup\_\{k=i\}^\{j\-1\}\\mathcal\{V\}\_\{k\}\}, which contains all clinical events from𝒱i\\mathcal\{V\}\_\{i\}to𝒱j−1\\mathcal\{V\}\_\{j\-1\}\. Events are flattened into a single chronological sequence, discarding visit\-level boundaries, and forming a unified patient trajectory\.

3\. Contextual Memory:ℳC\(j,T\)=\{\(r0,m0\),\(r1,m1\),…,\(rP,mP\)\}\\mathcal\{M\}\_\{C\}^\{\(j,T\)\}=\\\{\(r\_\{0\},m\_\{0\}\),\(r\_\{1\},m\_\{1\}\),\\dots,\(r\_\{P\},m\_\{P\}\)\\\}, whererkr\_\{k\}andmkm\_\{k\}denote the dialog role and message content\. It is constructed by rewriting the event stream of current𝒱j\\mathcal\{V\}\_\{j\}into an LLM\-style trajectory\. Only eventsEjkE\_\{j\}^\{k\}withtjk<Tt\_\{j\}^\{k\}<Tare retained\. EachEjkE\_\{j\}^\{k\}is transformed into a action–feedback pair using LLM:\{\(assistant,\{ajk,tjk,pjk\}\),\(user,\{fjk,tjk′\}\)\}\\\{\(\\texttt\{assistant\},\\\{a^\{k\}\_\{j\},t^\{k\}\_\{j\},p^\{k\}\_\{j\}\\\}\),\(\\texttt\{user\},\\\{f^\{k\}\_\{j\},\{t^\{k\}\_\{j\}\}^\{\\prime\}\\\}\)\\\}, wherefjkf^\{k\}\_\{j\}andtjk′\{t^\{k\}\_\{j\}\}^\{\\prime\}denote simulated clinic feedback inferred fromojko\_\{j\}^\{k\}and corresponding timestamp\. This memory simulates the instant interaction status for a medical agent\.

### 2\.2Benchmark Question Generation

Based on the memory modules\{ℳN\(i,j\),ℳE\(i,j\),ℳC\(j,T\)\}\\\{\\mathcal\{M\}^\{\(i,j\)\}\_\{N\},\\mathcal\{M\}^\{\(i,j\)\}\_\{E\},\\mathcal\{M\}\_\{C\}^\{\(j,T\)\}\\\}, we construct three progressively challenging task families, from*factual QA*,*temporal reasoning*, to*long\-horizon decision making*, reflecting increasingly global memory dependency\.

![Refer to caption](https://arxiv.org/html/2607.09322v1/x2.png)Figure 2:Example of the three evaluation tasks in LongMedBench\.#### 2\.2\.1Factual QA

This task evaluates precise retrieval and temporal alignment within the agent’s event memory\. From visit𝒱i\\mathcal\{V\}\_\{i\}, a ground truth eventEik∈𝒱iE^\{k\}\_\{i\}\\in\\mathcal\{V\}\_\{i\}is randomly sampled, and is unified as questions using the following two formats:Explicittasks includetikt^\{k\}\_\{i\}in the query text, evaluating direct retrieval capacity; whileRelativequestions provide only relative temporal relationships without timestamp\(e\.g\. the medicine name in the 2nd medication event of𝒱i\\mathcal\{V\}\_\{i\}\)\. The questions require the agent to recover full factual detailspikp\_\{i\}^\{k\}from a redundant memory windowℳE\(i−m,i\+m\+1\)\\mathcal\{M\}\_\{E\}^\{\(i\-m,i\+m\+1\)\}provided, wherem≥0m\\geq 0is a hyperparameter to adjust the window size\. We place𝒱i\\mathcal\{V\}\_\{i\}in the middle of the window to evaluate retrieval from non\-recency\-favored regions of long\-context memory\[[10](https://arxiv.org/html/2607.09322#bib.bib32)\]\.

#### 2\.2\.2Temporal Reasoning

This task evaluates the agent’s ability to reconstruct chronological order at different memory granularities\. The evaluation score is measured by Kendall’sτ\\tau\. Given a target event stream𝒮\\mathcal\{S\}or its visit𝒱i\\mathcal\{V\}\_\{i\}, we design three sub\-tasks using the original event streams as ground truth, each targeting at a distinct level of temporal reasoning\.Visit Clozetests event\-level reasoning: several intervention eventsEik∈𝒱iE\_\{i\}^\{k\}\\in\\mathcal\{V\}\_\{i\}with timestamps preserved are masked and provided to the agent as a shuffled list\. The agent must insert each event back to its original position\.Visit Sortingevaluates visit\-level ordering: five shuffled visit summariesN∈ℳN\(i,i\+5\)N\\in\\mathcal\{M\}\_\{N\}^\{\(i,i\+5\)\}must be ordered solely by clinical progression cues with their timestamps removed\.Joint Sortingassesses integration: five visits are split into ten admission/discharge clips\. The agent must correctly pair the clips from one visit and order them in time sequence\.

#### 2\.2\.3Long\-Horizon Decision Making

This task evaluates agent’s next\-step planning under long\-term historical context, a more comprehensive evaluation\. From𝒱n\\mathcal\{V\}\_\{n\}, we sample a ground\-truth eventEiG​TE\_\{i\}^\{GT\}, and define the agent’s contextual memory as its working contextSS\. The long\-horizon note or event memory is also provided as a referenceMM\. When generating questionQQ, we randomly select one of the following three sub\-task formats:Next Action Prediction \(T3\-N\)requires the agent to predictaiG​Ta\_\{i\}^\{GT\},Argument Prediction \(T3\-A\)evaluates inference forpiG​Tp\_\{i\}^\{GT\}, andDischarge Decision \(T3\-D\)queries if the patient can be discharged within66h\. For twopredictiontasks, as shown in Figure[2](https://arxiv.org/html/2607.09322#S2.F2), a time\-decay scoring mechanism is applied to recognize actionsℰ24​h\\mathcal\{E\}\_\{24h\}, events that occur within24​h24haftertiG​Tt\_\{i\}^\{GT\}\. Unlike prior tasks, this task requires integrating immediate stateSSwith long\-term memoryMMto produce clinically coherent decisions\. It evaluates true long\-horizon reasoning rather than isolated memory access\.

## 3Experiments and Results

### 3\.1Experiment Environments

To verify the robustness of medical agents in long\-term clinical decision, we selected representative long\-context LLMs as the agent’s backbone, including the closed\-source frontier modelgpt\-5\-mini\[[12](https://arxiv.org/html/2607.09322#bib.bib27)\], the lightweight oneqwen\-turbo\[[15](https://arxiv.org/html/2607.09322#bib.bib29)\], and the open\-source high\-performance onedeepseek\-v3\.2\[[2](https://arxiv.org/html/2607.09322#bib.bib30)\]\. All LLMs are deployed with default configurations\.

Referring to common retrieval methods in long\-horizon tasks\[[18](https://arxiv.org/html/2607.09322#bib.bib5),[11](https://arxiv.org/html/2607.09322#bib.bib4)\], we set up three memory architectures: \(1\)Naive Long\-Context\.No external tools is used; memories form preceding visits are greedily injected into the LLM’s context window\. \(2\)RAG\.text\-embedding\-3\-small\[[13](https://arxiv.org/html/2607.09322#bib.bib28)\]is used to vectorize memory entries; the agent can retrieve Top\-K similar memory fragments when answering the question\. \(3\)Agent Memory System\. A dynamically updated external storage layer is included\. When answering questions, the agent can spontaneously search based on the request and recall appropriate memories\. We introduce Mem0\[[1](https://arxiv.org/html/2607.09322#bib.bib31)\], a product\-ready agent memory architecture, in the following experiments\.

### 3\.2Results

#### 3\.2\.1Factual QA

We evaluateNaive Long\-Context LLMsusingqwen\-turbo, comparing window sizes\{3,5,7\}\\\{3,5,7\\\}\(*i\.e\.*,m∈\{1,2,3\}m\\in\\\{1,2,3\\\}inℳE\\mathcal\{M\}\_\{E\}\) and full history \(m=∞m=\\infty\) against theAgent Memory system\(Mem0\)\. Results in Table[2](https://arxiv.org/html/2607.09322#S3.T2)reportLab\-T\(recall\) andLab\-F\(rejection\) to assess hallucination impact\. Table[2](https://arxiv.org/html/2607.09322#S3.T2)reveals thatnaive\-LLM’s performance decays severely as history grows, dropping from 0\.882 to 0\.423 in explicit retrieval, with low Lab\-F scores \(≤\\leq0\.570\) highlighting a pervasive hallucination bias\.The performance of agent memory system is strongly correlated with the specific task type; while Mem0 achieves near\-optimal results in explicit Lab\-T \(0\.993\) and Medication \(0\.983\), its relative performance \(Overall 0\.331\) remains inferior\. This gap reveals that: first, agents struggle to generate precise queries for imaging reports even with few\-shot prompting; interestingly, when falling back to vector\-based search, relative semantic queries outperform explicit ID/timestamp matching, as the latter lacks sufficient embedding density in vector space; second, the agent’s memory architecture lacks a systematic indexing of relationships between events\.

Table 2:Performance on Factual Retrieval under Explicit and Relative Settings\.Table 3:Temporal reasoning performance \(Kendall’sτ\\tau\) on LongMedBench\.
#### 3\.2\.2Temporal Reasoning

Table[3](https://arxiv.org/html/2607.09322#S3.T3)presents Kendall’sτ\\taufor temporal reasoning across three progressive tasks\. In event\-levelvisit cloze, top models achieve near\-perfect performance \(gpt\-5\-mini: 0\.925\), while weaker models struggle \(qwen\-turbo: 0\.046\)\. Enabling thinking mode substantially boosts reasoning \(deepseek\-v3\.2\-thinking: 0\.969 vs\.deepseek\-v3\.2: 0\.316\)\. This confirms thatstrong LLMs can effectively utilize explicit timestamp cuesfor event\-level ordering, regradless of the option numbers \(the length of the event stream\), as illustrated in Figure[3\(a\)](https://arxiv.org/html/2607.09322#S3.F3.sf1)\.

Moving toVisit Sorting, where timestamps are absent so that models must sort five complete visit summaries based solely on clinical progression, performance drops sharply \(best:deepseek\-v3\.2\-thinking0\.423\)\. This reveals thatimplicit temporal reasoning in visit\-level remains challenging\.

![Refer to caption](https://arxiv.org/html/2607.09322v1/x3.png)\(a\)Mean Kendall’sτ\\tauvs\. number of options invisit\_cloze\.
![Refer to caption](https://arxiv.org/html/2607.09322v1/x4.png)\(b\)Comparison betweenjoint\_sortingandvisit\_sorting\.

Figure 3:Temporal reasoning analysis\.The difficulty further escalates inJoint Sorting, where each visit summary is split into admission and discharge fragments, requiring simultaneous event\-level pairing and visit\-level sorting\. Performance declines to 0\.330, demonstrating the compounding complexity\. Figure[3\(b\)](https://arxiv.org/html/2607.09322#S3.F3.sf2)shows this clear descending trend, confirming that when crucial information is fragmented at both event and visit level,implicit temporal reasoning in realistic multi\-visit scenarios remains a major challenge\.

Table 4:Long\-context LLM performance on Long\-Horizon Decision Making\.
#### 3\.2\.3Long\-Horizon Decision Making

We first evaluate the performance of Naive Long\-Context\. All models are limited to a maximum context of 128K tokens\. As shown in Table[4](https://arxiv.org/html/2607.09322#S3.T4), structured event memory outperforms note memory in most models in decision making\.gpt\-5\-miniperforms best\. Notably, fordeepseek\-v3\.2, thinking patterns did not significantly improve decision performance\. This is due to thinking mode makes the agent more conservative, attempting to ask question to get instant information rather than referring to past memories\.

To analyze the Lost\-in\-Middle\[[10](https://arxiv.org/html/2607.09322#bib.bib32)\]effect, usingqwen\-turboas the baseline model, we \(1\) adjusted the number of injected historical visits and \(2\) implement RAG and Mem0 architecture as retrial source in separate experiments\. As shown in Table[5](https://arxiv.org/html/2607.09322#S3.T5), although the difference is subtle, close to that under full memory conditions, increasing the context length makes the agent perform worse\. The performance differences between Mem0 and RAG are minimal, both close to the baseline performance, demonstrating that existing memory augmentation architectures offer limited gains for long\-term decision tasks\.

Table 5:Ablation study\. Left: visit injection \(nn\)\. Right: memory architectures\.Visit InjectionMemory ArchitecturennMemoryT3\-AT3\-DT3\-NAvgArchMemoryT3\-AT3\-DT3\-NAvg0 \(Base\)j–0\.490\.610\.320\.45RAGevent0\.450\.510\.290\.402event0\.490\.600\.320\.44Mem0event0\.470\.620\.310\.445event0\.480\.600\.310\.43RAGnote0\.480\.600\.310\.442note0\.480\.600\.320\.44Mem0note0\.460\.560\.290\.415note0\.480\.600\.320\.44jBaseline with no memory content injected\.Therefore, under the current LongMedBench setting, decision quality is not strongly correlated with the amount of retrieved historical information, whilethe agent performance depends more on the model’s clinical reasoning capacity within the immediate context\.Accordingly, we need to design tasks that amplify deeper reasoning requirements and higher decision interdependencies in an attempt to better reveal the limitations of current models and more effectively assess their long\-term clinical reasoning capabilities\.

## 4Conclusion

We introduce LongMedBench, a benchmark using real\-world EHR data to evaluate medical agents in long\-horizon clinical reasoning\. By converting MIMIC\-IV records into multi\-session event streams, we assess agents across three memory types with varying tasks\. Experiments reveal that while state\-of\-the\-art models handle explicit timestamps well, implicit temporal reasoning remains a significant bottleneck\. As history scales, models increasingly rely on immediate context rather than cross\-session integration\. Furthermore, while memory augmentation improves retrieval, decision\-making performance is highly task\-sensitive and remains limited by the model’s instant reasoning capacity\. Our results highlight a critical gap in handling time\-dependent clinical tasks for medical agents, necessitating future optimization of cross\-session memory augmentation architectures\.

## References

- \[1\]P\. Chhikara, D\. Khant, S\. Aryan, T\. Singh, and D\. Yadav\(2025\)Mem0: building production\-ready ai agents with scalable long\-term memory\.arXiv preprint arXiv:2504\.19413\.Cited by:[§3\.1](https://arxiv.org/html/2607.09322#S3.SS1.p2.1)\.
- \[2\]DeepSeek\-AI\(2025\)DeepSeek\-v3\.2: pushing the frontier of open large language models\.Cited by:[§3\.1](https://arxiv.org/html/2607.09322#S3.SS1.p1.1.3)\.
- \[3\]J\. Ding, L\. Lu, C\. Ding, M\. Bian, J\. Chen, W\. Pang, R\. Chen, X\. Peng, R\. Lu, S\. Ren, G\. Zhu, X\. Wu, Z\. Liu, R\. Zhang, L\. Jiang, B\. Han, Y\. Wang, and J\. Xu\(2025\)MedBench v4: a robust and scalable benchmark for evaluating chinese medical language models, multimodal models, and intelligent agents\.External Links:2511\.14439,[Link](https://arxiv.org/abs/2511.14439)Cited by:[Table 1](https://arxiv.org/html/2607.09322#S1.T1.25.25.6.1.1),[§1](https://arxiv.org/html/2607.09322#S1.p2.1),[§1](https://arxiv.org/html/2607.09322#S1.p3.1)\.
- \[4\]E\. J\. Gong, C\. S\. Bang, J\. J\. Lee, and G\. H\. Baik\(2025\)Knowledge\-practice performance gap in clinical large language models: systematic review of 39 benchmarks\.\.Journal of medical Internet research27,pp\. e84120\.External Links:[Document](https://dx.doi.org/10.2196/84120),ISSN 1438\-8871 \(Electronic\)Cited by:[§1](https://arxiv.org/html/2607.09322#S1.p1.1)\.
- \[5\]C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg\(2024\)RULER: what’s the real context size of your long\-context language models?\.External Links:2404\.06654,[Link](https://arxiv.org/abs/2404.06654)Cited by:[§1](https://arxiv.org/html/2607.09322#S1.p2.1)\.
- \[6\]Y\. Jiang, K\. C\. Black, G\. Geng, D\. Park, J\. Zou, A\. Y\. Ng, and J\. H\. Chen\(2025\)MedAgentBench: a virtual ehr environment to benchmark medical llm agents\.NEJM AI,pp\. AIdbp2500144\.Cited by:[Table 1](https://arxiv.org/html/2607.09322#S1.T1.10.10.6.1.1),[§1](https://arxiv.org/html/2607.09322#S1.p3.1)\.
- \[7\]A\. Johnson, L\. Bulgarelli, T\. Pollard, B\. Gow, B\. Moody, S\. Horng, L\. A\. Celi, and R\. Mark\(2024\-10\)MIMIC\-IV\.PhysioNet\.Note:Version 3\.1External Links:[Document](https://dx.doi.org/10.13026/kpb9-mt58),[Link](https://doi.org/10.13026/kpb9-mt58)Cited by:[§1](https://arxiv.org/html/2607.09322#S1.p1.1),[§1](https://arxiv.org/html/2607.09322#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.09322#S2.SS1.p1.1)\.
- \[8\]G\. Lee, H\. Hwang, S\. Bae, Y\. Kwon, W\. Shin, S\. Yang, M\. Seo, J\. Kim, and E\. Choi\(2022\)EHRSQL: a practical text\-to\-sql benchmark for electronic health records\.Advances in Neural Information Processing Systems35,pp\. 15589–15601\.Cited by:[Table 1](https://arxiv.org/html/2607.09322#S1.T1.35.35.6.1.1),[§1](https://arxiv.org/html/2607.09322#S1.p3.1)\.
- \[9\]Y\. Liao, S\. Jiang, Y\. Wang, and Y\. Wang\(2024\)ReflecTool: towards reflection\-aware tool\-augmented clinical agents\.External Links:2410\.17657,[Link](https://arxiv.org/abs/2410.17657)Cited by:[Table 1](https://arxiv.org/html/2607.09322#S1.T1.30.30.6.1.1),[§1](https://arxiv.org/html/2607.09322#S1.p3.1)\.
- \[10\]N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang\(2023\)Lost in the middle: how language models use long contexts\.External Links:2307\.03172,[Link](https://arxiv.org/abs/2307.03172)Cited by:[§1](https://arxiv.org/html/2607.09322#S1.p2.1),[§2\.2\.1](https://arxiv.org/html/2607.09322#S2.SS2.SSS1.p1.8),[§3\.2\.3](https://arxiv.org/html/2607.09322#S3.SS2.SSS3.p2.1)\.
- \[11\]A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. Fang\(2024\)Evaluating very long\-term conversational memory of llm agents\.arXiv preprint arXiv:2402\.17753\.Cited by:[§3\.1](https://arxiv.org/html/2607.09322#S3.SS1.p2.1)\.
- \[12\]OpenAI\(2026\)GPT\-5 mini model\.Note:[https://developers\.openai\.com/api/docs/models/gpt\-5\-mini](https://developers.openai.com/api/docs/models/gpt-5-mini)Accessed 22 Feb 2026Cited by:[§3\.1](https://arxiv.org/html/2607.09322#S3.SS1.p1.1.1)\.
- \[13\]OpenAI\(2026\)Text\-embedding\-3\-small model\.Note:[https://developers\.openai\.com/api/docs/models/text\-embedding\-3\-small](https://developers.openai.com/api/docs/models/text-embedding-3-small)Accessed 22 Feb 2026Cited by:[§3\.1](https://arxiv.org/html/2607.09322#S3.SS1.p2.1.3)\.
- \[14\]P\. Qiu, C\. Wu, J\. Liu, Q\. Zheng, Y\. Liao, H\. Wang, Y\. Yue, Q\. Fan, S\. Zhen, J\. Wang, J\. Gu, Y\. Wang, Y\. Zhang, and W\. Xie\(2026\)Evolving interactive diagnostic agents in a virtual clinical environment\.External Links:2510\.24654,[Link](https://arxiv.org/abs/2510.24654)Cited by:[Table 1](https://arxiv.org/html/2607.09322#S1.T1.20.20.6.1.1),[§1](https://arxiv.org/html/2607.09322#S1.p3.1)\.
- \[15\]QwenTeam\(2024\)Qwen3: think deeper, act faster\.Note:[https://qwen\.ai/blog?id=qwen3](https://qwen.ai/blog?id=qwen3)Accessed: 22 Feb 2026Cited by:[§3\.1](https://arxiv.org/html/2607.09322#S3.SS1.p1.1.2)\.
- \[16\]S\. Schmidgall, R\. Ziaei, C\. Harris, E\. Reis, J\. Jopling, and M\. Moor\(2025\)AgentClinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments\.External Links:2405\.07960,[Link](https://arxiv.org/abs/2405.07960)Cited by:[Table 1](https://arxiv.org/html/2607.09322#S1.T1.15.15.6.1.1),[§1](https://arxiv.org/html/2607.09322#S1.p3.1)\.
- \[17\]Y\. Shen, Z\. Huang, Z\. Wang, M\. Tian, Z\. Guo, C\. Zhang, S\. Zhou, Z\. Hu, D\. Li, J\. Xu, K\. Wang, W\. Liu, T\. Li, F\. Yue, F\. Hong, C\. Liu, and K\. Zeng\(2026\)TRIP\-bench: a benchmark for long\-horizon interactive agents in real\-world scenarios\.External Links:2602\.01675,[Link](https://arxiv.org/abs/2602.01675)Cited by:[§1](https://arxiv.org/html/2607.09322#S1.p2.1)\.
- \[18\]D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. Yu\(2024\)LongMemEval: benchmarking chat assistants on long\-term interactive memory\.https://arxiv\.org/abs/2410\.10813\.Cited by:[§1](https://arxiv.org/html/2607.09322#S1.p2.1),[§3\.1](https://arxiv.org/html/2607.09322#S3.SS1.p2.1)\.
- \[19\]Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning\(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§1](https://arxiv.org/html/2607.09322#S1.p2.1)\.