SciIR: A Large-scale Training Dataset and Benchmark for Scientific Image Reasoning Generation
Summary
SciIR introduces a large-scale dataset (SciIR-82k) and benchmark (SciIR-Bench) to enhance scientific reasoning in text-to-image models, with fine-tuning on Qwen leading to a significant performance improvement.
View Cached Full Text
Cached at: 07/02/26, 03:49 PM
Paper page - SciIR: A Large-scale Training Dataset and Benchmark for Scientific Image Reasoning Generation
Source: https://huggingface.co/papers/2606.30124
Abstract
Scientific image generation faces challenges in semantic alignment and logical reasoning, prompting the creation of SciIR-82k dataset and SciIR-Bench evaluation framework to improve scientific reasoning capabilities in text-to-image models.
WhileText-to-Image(T2I) models have shown remarkable success in generating photorealistic visual content, they still struggle with the rigorous semantic alignment and logical reasoning required for scientific imagery. Inspired byPeirce’s Semiotic Triad, we introduce Scientific Image Reasoning (SciIR), a comprehensive resource for training and evaluation ofscientific image generation. We formalize scientific reasoning into three core dimensions:Entity Structure(Icon),Scientific Process(Index), andScientific Law(Symbol). Specifically, to overcome the scarcity of training data inscientific image generation, we elaborately createSciIR-82k, a large-scale dataset containing over 80,000 high-quality scientific image-text pairs from cutting-edge publications. The dataset is hierarchically organized according to the semiotic dimensions and incorporates aScientific Reasoning Chain-of-Thought(Sci-RCoT) to explicitly model underlying visual logic. For evaluation, we proposeSciIR-Bench, which aligns with these three semiotic levels and employs anAtomic Checklistto convert the outcome-oriented scientific accuracy into process-oriented, verifiable, fine-grained questions. Our extensive experiments reveal significant deficiencies in current models’ scientific reasoning capabilities. Furthermore, by fine-tuning on theSciIR-82kdataset, we developed theQwen-Image-SciIRmodel, which achieves a substantial improvement on theSciIR-Bench, increasing the final score from 35\% to 43\%, laying a solid foundation for future advances inscientific image generation.
View arXiv pageView PDFProject pageGitHub2Add to collection
Get this paper in your agent:
hf papers read 2606\.30124
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.30124 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.30124 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.30124 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
SciR is a new controllable benchmark for evaluating LLMs on scientific reasoning including deduction, induction, and causal abduction, with parametric control over extraction and inference difficulty. Tests show both axes degrade performance across models, with reasoning models like DeepSeek-R1 outperforming instruct models on inference.
A Very Big Video Reasoning Suite
This paper introduces the Very Big Video Reasoning (VBVR) dataset and benchmark, a large-scale resource with over one million video clips across 200 reasoning tasks, enabling systematic study of spatiotemporal reasoning and showing early signs of emergent generalization.
@drfeifei: I’m very excited by this new benchmark dataset for visual generation that is suitable for the modern era of large scale…
Introducing GPIC (Giant Permissive Image Corpus), a large-scale dataset of 100M VLM-captioned image-text pairs for training and 1M pairs for benchmarking, fully permissive for research and commercial use.
Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models
Introduces SciDraw-Bench, a benchmark for evaluating scientific figure generation by text-to-image and multimodal models, with a four-dimensional evaluation protocol. Findings show domain-specific systems outperform general-purpose models, with text fidelity remaining the hardest challenge.
WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark
Introduces WorldBench, a visually diverse multimodal reasoning benchmark that reveals significant limitations in current multimodal large language models' visual understanding.