CellTrack-Bench: Microscopic Cell Tracking Benchmark for Vision-Language Models
Summary
CellTrack-Bench introduces a microscopic cell tracking benchmark evaluating 22 vision-language models against specialized trackers across 15 datasets and 4 imaging modalities, revealing significant reasoning gaps in frontier models.
View Cached Full Text
Cached at: 10/28/26, 01:52 AM
Paper page - CellTrack-Bench: Microscopic Cell Tracking Benchmark for Vision-Language Models
Source: https://huggingface.co/papers/2610.31566
Abstract
A new benchmark evaluates vision-language models on cell tracking against specialized algorithms, revealing reasoning gaps across diverse imaging modalities.
Large language models have shown outstanding performance on text-first tasks such as coding, math, and writing. Recent efforts are shifting models’ training to include scientific problems that often live in highly specialized domains beyond text, such as protein structure and biological images. However, in these scenarios, their general-purpose accuracy compared with specialized algorithms remains in question. One such domain iscell tracking, which requiresspatiotemporal awarenessof cell dynamics. To evaluate this, we created a new benchmark,CellTrack-Bench, which assessesVision-Language Models’ (VLMs) cell-tracking-related performance against specialized trackers. The full benchmark draws from 15 standard cell-tracking datasets across fourimaging modalitiesand comprises 1,537 problems corresponding to 9,041 cell-linking decisions. We selected a hard subset requiring greater reasoning effort by identifying failures shared across distinctalgorithmic trackers. With this proposed dataset, we: (i) evaluated different environment harnesses for VLMs; (ii) benchmarked 9 frontier closed-source models and 13 open-weight models; and (iii) provided a detailed analysis of their failure modes. Code, the stored model outputs, and a pipeline that regenerates the benchmark from the source datasets are available at https://github.com/royerlab/celltrack-bench.
View arXiv pageView PDFGitHub2Add to collection
Get this paper in your agent:
hf papers read 2610\.31566
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2610.31566 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2610.31566 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2610.31566 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models
SpaceCast-Bench introduces the first benchmark for predictive spatial reasoning in vision-language models, evaluating 21 models and finding a large gap between the best model (58.0%) and human performance (87.2%), while showing fine-tuning can significantly lift performance.
Benchmarking Visual State Tracking in Multimodal Video Understanding
Introduces VSTAT, a benchmark for evaluating visual state tracking in multimodal large language models (MLLMs) using 834 clips and 1,500 questions. Current MLLMs perform poorly compared to humans, failing at visual perception rather than reasoning.
PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models
This paper introduces PlanBench-V, the first comprehensive benchmark for evaluating Vision-Language Models on spatial planning map interpretation, including an expert-annotated dataset and a four-dimension evaluation framework. Experiments show significant progress but highlight persistent challenges in implementation-oriented tasks.
CT-$\Delta$Bench: A Benchmark for Longitudinal 3D Medical Imaging Difference Reporting with Vision-Language Models
Introduces CT-ΔBench, a benchmark for longitudinal 3D CT imaging difference reporting with vision-language models, along with change-aware metrics and a baseline model DeltaMed.
Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models
Introduces Blind-Spots-Bench, a benchmark designed to expose persistent failures in modern multimodal AI models on tasks that are trivial for humans. Evaluates a range of models, revealing performance gaps and that no single model dominates across all task types.