MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstracts
Summary
MedConclusion introduces a large-scale benchmark of 5.7 million PubMed structured abstracts for evaluating LLMs on biomedical conclusion generation from structured scientific evidence. The study finds that conclusion writing is behaviorally distinct from summarization and that current automatic metrics cluster strong models closely together.
View Cached Full Text
Cached at: 04/21/26, 07:20 AM
Paper page - MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstracts
Source: https://huggingface.co/papers/2604.06505
Abstract
A large-scale dataset of 5.7 million PubMed structured abstracts is introduced for biomedical conclusion generation, enabling evaluation of large language models’ ability to reason from structured scientific evidence.
Large language models(LLMs) are widely explored for reasoning-intensive research tasks, yet resources for testing whether they can infer scientific conclusions from structured biomedical evidence remain limited. We introduce MedConclusion, a large-scale dataset of 5.7M PubMedstructured abstractsforbiomedical conclusion generation. Each instance pairs the non-conclusion sections of an abstract with the original author-written conclusion, providing naturally occurring supervision forevidence-to-conclusion reasoning. MedConclusion also includes journal-level metadata such as biomedical category and SJR, enabling subgroup analysis across biomedical domains. As an initial study, we evaluate diverse LLMs under conclusion and summary prompting settings and score outputs with bothreference-based metricsandLLM-as-a-judge. We find that conclusion writing is behaviorally distinct from summary writing, strong models remain closely clustered under current automatic metrics, and judge identity can substantially shift absolute scores. MedConclusion provides a reusable data resource for studying scientificevidence-to-conclusion reasoning. Our code and data are available at: https://github.com/Harvard-AI-and-Robotics-Lab/MedConclusion.
View arXiv pageView PDFGitHub1Add to collection
Get this paper in your agent:
hf papers read 2604\.06505
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2604.06505 in a model README.md to link it from this page.
Datasets citing this paper2
#### harvardairobotics/MedConclusion-Compact Viewer• Updated1 day ago • 140k • 1 #### harvardairobotics/MedConclusion Viewer• Updated1 day ago • 5.83M • 2
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2604.06505 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Can AI Agents Synthesize Scientific Conclusions?
This paper introduces SciConBench, a large-scale benchmark with 9.11K questions and expert-written conclusions for evaluating AI agents' ability to synthesize scientific conclusions from open-domain evidence. The study finds that even the best agent achieves only a factual F1 of 0.337 in clean-room settings, highlighting that reliable synthesis remains an open challenge.
MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction
MedicalBench is a new benchmark for evaluating large language models on medical concept extraction from electronic health records, focusing on implicit reasoning and evidence grounding. It includes 823 expert-annotated examples and shows that current models perform modestly, highlighting the difficulty of extracting implicitly stated medical concepts.
MEDSYN: Benchmarking Multi-Evidence Synthesis in Complex Clinical Cases for Multimodal Large Language Models
MEDSYN is a multilingual multimodal benchmark for evaluating MLLMs on complex clinical cases with up to 7 distinct visual evidence types per case. The study reveals that while frontier models match human experts on differential diagnosis generation, all MLLMs show significant gaps in final diagnosis selection due to poor synthesis of heterogeneous clinical evidence.
MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs
This paper introduces MedAction, a framework for training LLMs on active, multi-turn clinical diagnosis by simulating iterative test ordering and hypothesis updates. It presents a new dataset, MedAction-32K, and demonstrates state-of-the-art performance for open-source models on medical benchmarks.
CalBrief: A Pilot Diagnostic Benchmark for Evidence-Calibrated Scientific Briefing with Large Language Models
The paper presents CalBrief, a pilot diagnostic benchmark of 16 evidence packages and 96 human-verified takeaways for evaluating whether large language models can generate evidence-calibrated scientific briefings. The study finds that structured organization improves reasoning but explicit strength-calibration policies are overly conservative, with most conservatism arising from expanded label spaces rather than signal injection.