An Exam for Active Observers

# An Exam for Active Observers
Source: [https://arxiv.org/html/2607.16165](https://arxiv.org/html/2607.16165)
Muzi Tao\*Shangshang Wang\*Ollie LiuXuezhe MaWillie Neiswanger University of Southern California \*Equal contribution

###### Abstract

Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot\. Decades of psychophysics and cognitive science have argued that this*active observation*is essential for a wide range of tasks\. Whether today’s multimodal large language models \(MLLMs\) exercise active observation is an empirical question that current vision\-language benchmarks do not answer\. We introduceActiveVision, a benchmark that makes active observation measurable for MLLMs, comprising 17 tasks across 3 categories\. Tasks are designed to force repeated visual perception rather than a single static description\. Frontier MLLMs collapse onActiveVision: the highest\-scoring model we evaluate, GPT\-5\.5 at the highest exposed reasoning\-effort tier, solves only 10\.6% of items and scores zero on 11 of the 17 tasks, and even Claude Fable 5, despite topping most reasoning and coding leaderboards, solves just 3\.5%, far behind three human participants who average 96\.1%\. Furthermore, much of the gap persists even when models write and run their own vision code\. Such code is unreliable on realistic imagery, and catching its failures itself requires the active perception the models lack\. Together, these results indicate that current MLLMs lack robust active visual observation, motivating architectures and training objectives that close the perception–reasoning loop\.

[Website](https://activevision.dev/)\|[GitHub](https://github.com/saccharomycetes/ActiveVision)\|[Dataset](https://huggingface.co/datasets/activevisionai/ActiveVision)

![Refer to caption](https://arxiv.org/html/2607.16165v1/x1.png)Figure 1:Seeing is not always a single\-glance task\.Many visual problems require an observer to coordinate attention and memory across an image: scanning exhaustively for distributed evidence, following connected structures without losing track, and comparing fine\-grained attributes across distant regions\.ActiveVisionturns these abilities into a controlled test of whether MLLMs can keep visual evidence in the reasoning loop\.## 1Introduction

Try the three tasks in Figure[1](https://arxiv.org/html/2607.16165#S0.F1)\. Count the separated regions of grassland in the aerial photo on the left\. In the middle, trace the rope from the green “S” through all crossings to the far end, reading off each labeled point that you pass\. On the right, count every beige fabric patch on the linen whose silhouette matches the one shown on the kraft\-paper template\. None can be solved at a glance\. Each requires several seconds of iterative, hypothesis\-driven inspection: fixate, predict, return\. The visual system executes this loop effortlessly, and the conscious mind barely notices it\. Decades of research in psychophysics, cognitive science, and computer vision have argued that this iterative looking, known as*active observation*, is not an ornament on top of perception but, for a wide range of tasks, an essential part of perception itself\[[2](https://arxiv.org/html/2607.16165#bib.bib2),[5](https://arxiv.org/html/2607.16165#bib.bib5),[6](https://arxiv.org/html/2607.16165#bib.bib6),[13](https://arxiv.org/html/2607.16165#bib.bib13),[18](https://arxiv.org/html/2607.16165#bib.bib18),[34](https://arxiv.org/html/2607.16165#bib.bib34),[48](https://arxiv.org/html/2607.16165#bib.bib48)\]\.

Multimodal large language models \(MLLMs\) have advanced rapidly, yet it remains unclear whether their benchmark gains reflect progress in this capability\. From the GPT\-4 family\[[1](https://arxiv.org/html/2607.16165#bib.bib1)\]onward, frontier models such as Claude Fable 5\[[4](https://arxiv.org/html/2607.16165#bib.bib4)\], GPT\-5\.5\[[32](https://arxiv.org/html/2607.16165#bib.bib32)\], and Gemini 3\.1 Pro\[[16](https://arxiv.org/html/2607.16165#bib.bib16)\]have achieved strong results on benchmarks commonly used to distinguish among frontier models\. Performance on multimodal benchmarks such as MMMU\-Pro\[[51](https://arxiv.org/html/2607.16165#bib.bib51)\]and CharXiv\[[47](https://arxiv.org/html/2607.16165#bib.bib47)\]is nearing saturation, leaving little separation among frontier models\.

Yet benchmark saturation does not mean that vision\-language understanding is solved\. Applications in robotics\[[30](https://arxiv.org/html/2607.16165#bib.bib30)\], design\[[25](https://arxiv.org/html/2607.16165#bib.bib25)\], manufacturing\[[19](https://arxiv.org/html/2607.16165#bib.bib19),[52](https://arxiv.org/html/2607.16165#bib.bib52)\], computer use\[[50](https://arxiv.org/html/2607.16165#bib.bib50)\], spatial understanding\[[35](https://arxiv.org/html/2607.16165#bib.bib35)\], and scientific discovery\[[20](https://arxiv.org/html/2607.16165#bib.bib20),[28](https://arxiv.org/html/2607.16165#bib.bib28)\]require models to revisit visual evidence as they reason, a capability current MLLMs do not reliably exhibit\. Existing evaluations leave this requirement largely unmeasured\. Public leaderboards remain dominated by captioning, single\-image visual question answering, and multiple\-choice tasks that can often be answered from a static description of the image\. Few benchmarks assess whether models can revisit visual evidence to form, test, and refine hypotheses as reasoning unfolds\.

Current MLLMs appear to be passive perceivers: the image is encoded once as a fixed sequence of visual tokens, with no explicit perception–action loop\. Yet this need not preclude active observation\. During autoregressive reasoning, the model may shift attention among visual tokens as hypotheses evolve, letting earlier visual findings guide later steps\[[54](https://arxiv.org/html/2607.16165#bib.bib54)\]\. The question is therefore behavioral rather than architectural:

Do MLLMs perform*active observation*, returning to the image to form and test hypotheses as they reason?

We answer the question withActiveVision, a benchmark that makes iterative visual perception measurable\. It consists of 17 tasks spanning three cognitive demands of active vision:Distributed Scanningfor exhaustive coverage of many local signals,Sequential Traversalfor ordered stepping along a connected structure, andVisual Attribute Transferfor fine\-grained comparison across regions\. Across these tasks, the underlying configurations resist concise language description, so a model cannot easily summarize an image once in language and answer from that summary alone; it must keep returning to the pixels as its reasoning unfolds\.

To create this structural complexity and render it realistically, each task follows a two\-stage construction pipeline\. First, a procedural scaffold specifies the underlying geometry exactly, including arbitrary positions, shapes, and curves\. Second, GPT\-image\-2\[[33](https://arxiv.org/html/2607.16165#bib.bib33)\]re\-renders the scaffold as a photorealistic image while preserving positions, counts, labels, and topology \(§[3\.3](https://arxiv.org/html/2607.16165#S3.SS3)\)\. Voronoi regions become aerial fields and rivers, arrow chains become stones linked by footprints, and tangled loops become ropes on driftwood\. The resulting images introduce realistic textures and visual complexity absent from sparse, cartoon\-like renderings\.

Every frontier model we evaluate fails on the overwhelming majority of the 85 released items\. We test each model at every reasoning\-effort tier exposed by its API\. Across all settings, the best result is 10\.6%, achieved by GPT\-5\.5*xhigh*, which still scores zero on 11 of the 17 tasks\. Claude Fable 5, despite topping most reasoning and coding leaderboards, solves just 3\.5%\. Three human participants, by contrast, average 96\.1%, roughly nine times the best model’s accuracy\. Increasing reasoning effort barely narrows this gap\.

Tool use narrows but does not close the gap\. We evaluate Codex and two Claude Code agents, which write and run their own vision code and reach only 24\.7–50\.6% accuracy\. Gains concentrate where tasks admit reliable code\-based solutions, but traversal remains difficult\. On realistic imagery, models often miss tool failures\. Tool use therefore shifts the active\-observation bottleneck to verification in the agent loop\.

In summary, we \(i\) introduceActiveVision, a controlled benchmark that isolates active observation through 17 diverse tasks spanning three cognitive axes \(§[3](https://arxiv.org/html/2607.16165#S3)\); \(ii\) systematically evaluate frontier MLLMs across their full reasoning\-effort range and, as a tool\-use test, autonomous coding agents built on them, finding a large and consistent human–model gap that neither additional reasoning nor tool use closes \(§[4](https://arxiv.org/html/2607.16165#S4)\); and \(iii\) trace the residual failures to a perceptual bottleneck unresolved by additional reasoning or tooling, positioning active vision as a distinct capability that current models lack but real\-world applications require \(§[5](https://arxiv.org/html/2607.16165#S5)\)\.

## 2Related Work

ActiveVisiondraws on three lines of work: a long\-running argument from cognitive science and computer vision that perception is intrinsically active, evidence that MLLMs struggle with basic visual perception, and existing benchmarks constructed to stress vision rather than language\. We elaborate on each below\.

### 2\.1Active Vision in Cognitive Science and Computer Vision

*Eye Movements and Vision*\[[48](https://arxiv.org/html/2607.16165#bib.bib48)\]showed that the same painting elicits different scanpaths under different task prompts, establishing that gaze is allocated by the task rather than the image alone\. Later works\[[2](https://arxiv.org/html/2607.16165#bib.bib2),[5](https://arxiv.org/html/2607.16165#bib.bib5)\]formalized the consequence in computer vision: inverse problems such as shape from shading, structure from motion, and optical flow are ill\-posed for a passive observer but become well\-posed once the observer can actively control its sensors as part of inference\.

*Animate vision*\[[6](https://arxiv.org/html/2607.16165#bib.bib6)\]reframed the same argument in terms of computational economy, using gaze\-targeted local computation rather than global computation\. The*sensorimotor account*\[[34](https://arxiv.org/html/2607.16165#bib.bib34)\]advanced the strongest version of this position, arguing that seeing*is*the mastery of how visual input changes under one’s own movements\. Eye\-tracking of everyday tasks supports this account beyond the laboratory\[[18](https://arxiv.org/html/2607.16165#bib.bib18),[13](https://arxiv.org/html/2607.16165#bib.bib13)\]\. The shared prediction is sharp: without iterative sensor redirection, a vision system will fail on perceptual problems that humans solve routinely\.ActiveVisionoperationalizes that prediction for multimodal large language models\.

### 2\.2Vision\-Centric Benchmarks for MLLMs

A first line of work set out to isolate visual perception from linguistic priors\. Early diagnoses found frontier models leaning on memorized priors and overlooking queried visual details\[[46](https://arxiv.org/html/2607.16165#bib.bib46),[54](https://arxiv.org/html/2607.16165#bib.bib54)\], and benchmarks such as MMVP\[[43](https://arxiv.org/html/2607.16165#bib.bib43)\], MMStar\[[10](https://arxiv.org/html/2607.16165#bib.bib10)\], and CV\-Bench\[[42](https://arxiv.org/html/2607.16165#bib.bib42)\]were built to exclude items solvable from text alone and reward reading the image itself, alongside a parallel push on vision\-centric model design\[[31](https://arxiv.org/html/2607.16165#bib.bib31),[23](https://arxiv.org/html/2607.16165#bib.bib23),[40](https://arxiv.org/html/2607.16165#bib.bib40),[22](https://arxiv.org/html/2607.16165#bib.bib22),[53](https://arxiv.org/html/2607.16165#bib.bib53)\]\. These benchmarks are now largely saturated, indicating that current MLLMs can often read the image when linguistic shortcuts are controlled, while leaving their remaining perceptual limitations unresolved\.

A second line instead targets the human–model gap directly\. BLINK\[[14](https://arxiv.org/html/2607.16165#bib.bib14)\]collects tasks a person answers in the blink of an eye, BlindTest\[[36](https://arxiv.org/html/2607.16165#bib.bib36)\]poses questions so simple that a sighted human never misses them, ArtQA\[[41](https://arxiv.org/html/2607.16165#bib.bib41)\]asks painting questions that require looking, and BabyVision\[[9](https://arxiv.org/html/2607.16165#bib.bib9)\], ChildBench\[[11](https://arxiv.org/html/2607.16165#bib.bib11)\], and KidGym\[[49](https://arxiv.org/html/2607.16165#bib.bib49)\]draw on tasks an infant or child can already solve\. What these share is a framing by*human\-model gap*alone: they establish that models fall short of people without isolating*which*capability is missing; we recognize active vision as a major part of that missing capability, one that deserves to be measured on its own terms\.

A third line is application\-driven: examples include OSWorld\[[50](https://arxiv.org/html/2607.16165#bib.bib50)\]for real\-world computer\-use tasks and BenchCAD\[[52](https://arxiv.org/html/2607.16165#bib.bib52)\]for parametric CAD generation\. These usefully track practical deployment value, but each task bundles coding, reasoning, and visual perception together, so a failure cannot be attributed to perception\.ActiveVisionisolates exactly this capability\. It targets*active observation*, the iterative looking loop that prior benchmarks do not explicitly name\. Its tasks resist shortcuts from any single language description and are rendered photorealistically to resemble real\-world inputs encountered in downstream applications\.

## 3ActiveVision

![Refer to caption](https://arxiv.org/html/2607.16165v1/x2.png)Figure 2:Overview of all 17 tasks inActiveVision, grouped into three task families that probe distinct dimensions of active observation\. For each task, we show one photorealistic instance produced by the rendering pipeline described in §[3\.3](https://arxiv.org/html/2607.16165#S3.SS3), together with a brief description of the required operation\.ActiveVisioncollects tasks where a single one\-shot glance is unlikely to suffice and the answer benefits from continuously returning to the pixels\. §[3\.1](https://arxiv.org/html/2607.16165#S3.SS1)grounds the design in three elemental operations of human vision, following the definitions of three task families that instantiate them in Figure[2](https://arxiv.org/html/2607.16165#S3.F2)\. §[3\.2](https://arxiv.org/html/2607.16165#S3.SS2)states the design principle every generator follows: discriminative visual state whose information content exceeds what one language description can losslessly carry\. §[3\.3](https://arxiv.org/html/2607.16165#S3.SS3)introduces each of the 17 tasks and describes the photorealistic rendering pipeline that turns each procedural scaffold into the natural\-looking image models actually see\.

### 3\.1Task Families: Grounding the Cognitive Foundations

ActiveVisionis built around three*elemental operations*\. Specifically,*exhaustive enumeration*beyond the subitizing range\[[24](https://arxiv.org/html/2607.16165#bib.bib24),[44](https://arxiv.org/html/2607.16165#bib.bib44)\],*curve tracing*along contours\[[21](https://arxiv.org/html/2607.16165#bib.bib21),[38](https://arxiv.org/html/2607.16165#bib.bib38)\], and*fine\-grained comparison*under visual working memory limits\[[29](https://arxiv.org/html/2607.16165#bib.bib29),[7](https://arxiv.org/html/2607.16165#bib.bib7)\]\. Psychophysics and neurophysiology characterize these operations as serial, attention\-demanding routines and are carried out*actively*through repeated shifts of gaze\[[45](https://arxiv.org/html/2607.16165#bib.bib45),[37](https://arxiv.org/html/2607.16165#bib.bib37)\]\. We instantiate them below as three task families, each targeting a distinct dimension of active observation\.

Distributed Scanning\.The image contains many spatially distributed local signals, including dots, strokes, regions, and graph faces\. Each must be found and accumulated\. Difficulty scales with the number of signals and how evenly they cover the canvas\. The characteristic failure has two forms\. In*partial coverage*, the model counts only five or six of ten items and stops before completing a full scan\. In faulty*individuation*, it fails to recognize each signal as a discrete element or distinguish adjacent, similar signals\. Items may then be merged, split, or confused with the background before counting begins\.

Sequential Traversal\.The image encodes a connected structure, such as an arrow chain, a tangled curve through colored regions, or a winding tube\. The model must follow it step by step while maintaining its current position, direction, and running tally\. Difficulty scales with path length, crossing density, and the visual similarity of decoys to the correct next step\. The characteristic failure is*gestalt interpolation*\. The model guesses the endpoint from the start without traversing the intermediate steps\.

Visual Attribute Transfer\.Fine\-grained comparison across regions\. The model extracts a visual property from a reference region\. This may be length, curvature, thickness, color arrangement, dot pattern, or orientation\. It then matches or compares that property against candidates elsewhere in the image\. Difficulty scales with the subtlety of the attribute distinction and the number of candidates\. The characteristic failure is*prior substitution*\. Instead of measuring both regions, the model applies a learned linguistic prior\.

### 3\.2Task Design Principle: Forcing Iterative Perception

ActiveVisionis built around a single design principle: every task instance carries discriminative visual state whose information content exceeds what a single language description can losslessly carry\. An observer that compresses the image once \(e\.g\.,*“six red circles in the top\-left, three blue squares in the bottom\-right”*\) and then reasons over that summary will, by construction, lose the information the answer depends on; solving the task requires keeping the image itself in the loop\. The principle is realized via three properties as follows\.

We realize this principle through three design properties:

- •Arbitrary positions\.Items are placed at continuous, sampled coordinates rather than on a grid or at named anchors\. Twenty dots scattered over the canvas carry 190 pairwise spatial relations and twenty real\-valued coordinate pairs, far beyond what any single linguistic summary can capture\.
- •Arbitrary shapes\.Region boundaries, contours, blob silhouettes, and tile\-motif outlines are synthesized fresh for every instance rather than drawn from a fixed library of named shapes\. Each is either a closed contour whose radius is modulated by a random number of Fourier harmonics with sampled amplitudes and phases or a periodic spline through jittered ring waypoints\. Consequently, the silhouette space is continuous and high\-variance, and no two instances repeat a shape\. Each boundary is defined by its exact polyline rather than by a named shape such as a triangle\.
- •Arbitrary traces\.The routes the tasks ask the observer to follow, including paths, arrow chains, tangled loops, and the connecting curve through color zones, are smooth random splines through sampled control points, with dozens of meaningful inflection points that no single description preserves\.

Together, these properties make iterative perception the only natural solution path\. Each instance carries more visual state than a concise description can preserve, so the tasks remain readily tractable for human observers who can revisit the image yet consistently difficult for a one\-pass observer to solve reliably\.

### 3\.3Task Instantiation: From Synthetic Scaffolds to Real\-world Images

We instantiate this design with 17 task generators grouped by the three families above\. Each produces a synthetic scaffold, question, and ground\-truth answer from a deterministic seed, making instances reproducible\. Answers span broad ranges with flat per\-task distributions, preventing the modal answer or task identity from serving as a reliable shortcut\. Table[1](https://arxiv.org/html/2607.16165#S3.T1)summarizes the resulting task set\.

Table 1:The 17 task generators inActiveVision, grouped by cognitive axis\. Each row gives the task name and a one\-line description of what the model is asked to do\.These tasks are not merely abstract puzzles\. Each isolates a perceptual operation that people use in everyday life and across practical settings\. For example,*Region Counting*resembles counting countries on a political map or zones in a thermal scan, while*Signal Difference Spotting*mirrors comparing ECGs or reviewing seismic traces\. Other tasks capture topological reasoning, path tracing, and attribute matching relevant to wiring inspection, floor\-plan analysis, microscopy, manufacturing inspection, and satellite change detection\.

![Refer to caption](https://arxiv.org/html/2607.16165v1/x3.png)Figure 3:The four\-stage generation pipeline, illustrated on*Tangled Loop Counting*\. A deterministic Python generator \(1\) emits the geometric scaffold \(2\): random smooth closed curves with full ground truth\. A task\-specific GPT\-image\-2 prompt \(3\) re\-renders the scaffold into a photorealistic image \(4\) without altering positions, counts, or topology\. The benchmark question and ground\-truth answer apply unchanged to both renderings; only the photorealistic image is served to the model at evaluation time\.We place these operations in realistic settings via a two\-stage pipeline \(Figure[3](https://arxiv.org/html/2607.16165#S3.F3)\)\. Each task generator first produces a procedural Matplotlib scaffold with full ground truth attached\. To make the images noisy and realistic rather than cartoon\-clean, we re\-render every scaffold with GPT\-image\-2 using a task\-specific prompt that maps the procedural primitives to a real\-world setting\. Only the rendered image is shown to the model\.

The pipeline serves three purposes\. First, it removes the cartoon\-input confound: the perceptual difficulty seen by models is dominated by the task’s discriminative structure rather than by an unfamiliar rendering style\. Second, the photorealistic regime is where classical computer\-vision baselines degrade sharply, in line with their known brittleness on scanned documents, ultrasound speckle, low\-resolution phone snapshots, and similar real\-world inputs, so the same images that test MLLMs also bound what a tool\-using script can recover\. Third, the imagery resembles what downstream applications actually face, making the active\-vision diagnosis externally meaningful rather than an artifact of toy renderings\.

## 4Benchmark Evaluation

We first report the headline accuracy of frontier MLLMs onActiveVision\(§[4\.1](https://arxiv.org/html/2607.16165#S4.SS1)\), then dive deeper with two ablation studies on the effect of reasoning effort \(§[4\.2](https://arxiv.org/html/2607.16165#S4.SS2)\) and of agentic tool use \(§[4\.4](https://arxiv.org/html/2607.16165#S4.SS4)\)\.

From each of the 17 generators, we sample a fixed evaluation split ofN=5N=5instances, yielding8585items in total\. Every instance passes through the photorealistic rendering pipeline described in §[3\.3](https://arxiv.org/html/2607.16165#S3.SS3)\. We use exact\-match accuracy as the primary metric\. Unless stated otherwise, evaluations use pure CoT with no external tools or code execution\. Each item is presented as a single user message containing the question verbatim from the released manifest, followed by the image\. No system prompt is used\. The model is instructed to wrap its final answer in<answer\>tags\.

We score the last such block by exact match after normalizing case, whitespace, and separators\. The human baseline comprises three participants, each of whom completed the full 85\-item split unaided through a self\-paced web interface and was scored by the same criterion\. Table[2](https://arxiv.org/html/2607.16165#S4.T2)reports frontier models at their highest reasoning effort, while Figure[4](https://arxiv.org/html/2607.16165#S4.F4)reports the full reasoning\-effort range for GPT\-5\.5\[[32](https://arxiv.org/html/2607.16165#bib.bib32)\], Claude Fable 5\[[3](https://arxiv.org/html/2607.16165#bib.bib3)\], Claude Opus 4\.8\[[17](https://arxiv.org/html/2607.16165#bib.bib17)\], Gemini 3\.1 Pro\[[16](https://arxiv.org/html/2607.16165#bib.bib16)\], and Gemini 3\.5 Flash\[[17](https://arxiv.org/html/2607.16165#bib.bib17)\]\.

Table 2:Per\-task exact\-match accuracy onActiveVision\. Model entries report correct responses out of five\. Human entries report the mean across three participants on the same scale \(e\.g\., 14/15 is reported as 4\.7/5\)\. The Overall row reports scores out of 85; individual human accuracies are 97\.6%, 96\.5%, and 94\.1%\.### 4\.1Main Results: Every Frontier Model Fails Most Tasks

Table[2](https://arxiv.org/html/2607.16165#S4.T2)reports per\-task accuracy for six frontier models at their highest effort settings alongside a three\-participant human baseline\. The highest\-scoring model, GPT\-5\.5, solves only 10\.6% \(9 of 85\); even Claude Fable 5, despite its strong performance on reasoning and coding leaderboards, solves just 3\.5%\. On 11 of 17 tasks, GPT\-5\.5 scoreszero\. The human mean is 96\.1%, roughly nine times the accuracy of the best model\. The six models have only weakly overlapping success sets\. No item is solved by all six, suggesting that the gap is not specific to a single model\. As a shortcut check, a GPT\-5\.5 question\-only control with the image omitted solves only 2 of 85 items, for an accuracy of 2\.4%\. This result matches the run with both image and question at the same*none*effort, indicating that prompt priors alone do not explain the reported performance\.

### 4\.2Does More Reasoning Help?

![Refer to caption](https://arxiv.org/html/2607.16165v1/x4.png)Figure 4:Accuracy versus API cost per item \(log scale, public list prices\) on the 85\-itemActiveVisionsplit\. Points are complete pure\-CoT runs, with marker size encoding the reasoning\-effort tier; lines connect runs of the same model\. The dashed line and band are the human mean and range \(N=3N\{=\}3\)\.![Refer to caption](https://arxiv.org/html/2607.16165v1/x5.png)Figure 5:Predicted versus ground\-truth counts across eight counting tasks at each model’s highest reasoning effort\. Solid lines are least\-squares fits; gray dashed lines show the identityy=xy=x\. Slopes below one reveal increasing undercounting as the true count grows\.
![Refer to caption](https://arxiv.org/html/2607.16165v1/x6.png)Figure 6:Prefix survival on three ordered\-walk tasks \(over 18 pure\-CoT runs\): the share of walks whose firstkksteps are all correct\. Dashed line: shortest walk length \(k=6k\{=\}6\); no walk is completed exactly\.

The gap is not a deliberation deficit\. Figure[4](https://arxiv.org/html/2607.16165#S4.F4)plots accuracy against API cost per item for every pure\-CoT run across reasoning\-effort tiers\. Every model remains in a narrow low\-accuracy band, far below the human baseline\. Scaling GPT\-5\.5’s reasoning effort from*none*to*xhigh*increases its per\-item cost nearly a hundredfold but improves accuracy only from 2\.4% to 10\.6%\. Task\-level failures persist across tiers\. The other frontier models show the same pattern\. Spending31×31\\timesmore per item on Fable 5 does not make it more accurate, and differences between effort tiers are within sampling noise on 85 items\. The models do not run out of reasoning steps; they fail to pull the right visual evidence out of the image\.

### 4\.3What Do Models Do When They Fail?

Pooling the 18 pure\-CoT runs of the evaluation sweep, we examine the response behavior of each task family: what the models count, how far they trace, and when they declare a difference\.

Counting falls behind as scenes grow crowded\.Figure[6](https://arxiv.org/html/2607.16165#S4.F6)plots predicted count against ground\-truth count on the eight counting tasks, with a least\-squares fit per model\. All fitted slopes fall well below the perfect\-counting diagonal, and the fits drop further below it as the true count grows\. The models are conservative counters: the more there is to count, the larger the share left uncounted, as if the model glimpses the image rather than scanning it exhaustively\.

Tracing loses its way at the very start\.Figure[6](https://arxiv.org/html/2607.16165#S4.F6)shows, for the three ordered\-walk tasks, the share of walks whose firstkksteps are all correct\. Survival collapses within the first one or two moves, and not a single walk in the pool is completed exactly\. The models cannot reliably trace along a structure in the image; failure is not gradual drift late in a long walk but the loss of the walk’s frame at its very beginning\.

“Same” becomes the safe default in visual comparison\.Table[3](https://arxiv.org/html/2607.16165#S4.T3)reports per\-run error rates on the two panel\-naming difference tasks: the share of real differences missed and the share of identical panels falsely flagged\. Miss rates are high across the board while false alarms stay low, and the highlighted runs answer “none” on nearly every item, missing all real differences\. Unable to resolve the fine\-grained comparison, the models fall back on “same” because it is the safe answer, a response bias rather than an act of perception\.

Table 3:Per\-run miss and false\-alarm rates on the difference tasks\. Highlighted runs answer “none” on nearly every item, missing all real differences\.![Refer to caption](https://arxiv.org/html/2607.16165v1/x7.png)\(a\)Per\-agent accuracy: even the strongest agent \(Fable 5, 50\.6%\) stays far below the human band\.
![Refer to caption](https://arxiv.org/html/2607.16165v1/x8.png)\(b\)Per\-task accuracy \(correct out of 5\): attribute\-transfer tasks largely yield to code, the traversal family resists every agent except Fable 5, and*Tangled Loop Counting*resists all three agents completely\.

Figure 7:Agentic tool use on the 85\-itemActiveVisionsplit\. \(a\) Per\-agent accuracy against the human reference \(N=3N\{=\}3\); \(b\) per\-task breakdown across the three agents\.
### 4\.4Can Active Vision Be Substituted by Tool Use?

Our main result shows that frontier models lack active observation\. We next ask whether computation can substitute for it\. Code offers a strong substitution channel: brightness thresholding can segment regions, while connected\-component analysis can return their count, bypassing direct visual inspection\. We test this by running three autonomous coding agents per item: Codex \(GPT\-5\.5\), Claude Code \(Opus 4\.8\), and Claude Code \(Fable 5\)\. All operate at*xhigh*effort in fresh sandboxes containing only the image and question\.

![Refer to caption](https://arxiv.org/html/2607.16165v1/x9.png)Figure 8:Four examples illustrating two failure modes of agentic tool use, shown through the agents’ own intermediate artifacts and verbatim rollout excerpts\. For each failure mode, the upper example is from Codex and the lower example is from Claude Code\.#### 4\.4\.1How Far Does Substitution Carry?

Agentic tools improve performance substantially but do not close the human gap \(Figure[7\(a\)](https://arxiv.org/html/2607.16165#S4.F7.sf1)\)\. The Fable 5 agent solves43 of 85 items \(50\.6%\), Codex32 \(37\.6%\), and the Opus 4\.8 agent21 \(24\.7%\), compared with 10\.6% for the best tool\-free model\. The gains are highly task dependent \(Figure[7\(b\)](https://arxiv.org/html/2607.16165#S4.F7.sf2)\)\. All three agents perform best onVisual Attribute Transfer\(43–66%\), where template cropping and measurement admit clean algorithmic reductions\. In contrast,Sequential Traversalremains difficult: Codex solves 1/25 items, Claude Code with Opus 4\.8 solves 3/25, and Claude Code with Fable 5 solves 10/25\. Failure is not limited to traversal; all three agents score zero on*Tangled Loop Counting*\. Overall, tool use helps most when perception can be reduced to a reliable computation\. When no such reduction is available, the gap persists\.

Table 4:Accuracy and mean per\-item resource use on the 85\-item split\. Parentheses retain counts and run\-level details; dashes indicate quantities that do not apply\.
#### 4\.4\.2Why Does Substitution Stop There?

Figure[8](https://arxiv.org/html/2607.16165#S4.F8)shows the two recurring failure modes\. First, tools are not robust on realistic textures \(Figure[8](https://arxiv.org/html/2607.16165#S4.F8), top\)\. On*Bounded Face Counting*, the agent’s color mask dissolves into the cobblestone texture, so nodes and edges are mis\-detected and its Euler\-formula count comes out wrong \(12 vs\. a truth of 10\); on*Contour Difference Spotting*, shadows and paper creases bake into the regenerated silhouettes and a badly scaled crop distorts the true contours, netting out to one wrong “same\.”

Second, weak perception cannot catch the tool’s errors \(Figure[8](https://arxiv.org/html/2607.16165#S4.F8), bottom\)\. On*Color Zone Sequencing*, the agent’s tracer jumps onto a crossing strand and halts early, and the agent audits neither error, submitting the truncated route; on*Maze Path Tracing*, the binarized mask reads a shadowed hedge as corridor and fuses two openings, and the agent trusts the flood\-fill’s incorrect path\. In both cases, a glance at the overlay it had just produced would falsify the answer\. In short, a model that cannot see cannot tell when its code is wrong\.

#### 4\.4\.3How Costly Is Substitution?

Substitution is also expensive and slow\. Table[4](https://arxiv.org/html/2607.16165#S4.T4)reports the per\-item cost of the three tracks: each item costs an agent $2\.74–$7\.63 of compute and 12–15 minutes of wall\-clock time, spent across dozens of tool calls and tens of thousands of output tokens, about 25 times longer than the half minute a human needs, plus compute a human never spends\. Even the most expensive agent, Fable 5, solves only about half the benchmark; the human participants answer 96\.1% with no tools and no code\.

## 5Discussion

### 5\.1Interpreting the Agentic Evaluation

Tool\-use gains and remaining bottlenecks\.ActiveVisionevaluates models as pure visual observers\. The three agentic tool use runs \(§[4\.4](https://arxiv.org/html/2607.16165#S4.SS4)\) show why this distinction matters\. Classical computer vision primitives such asfindContours, Canny, and template matching work best on clean inputs\. OnActiveVision’s photorealistic images, they fragment contours, merge objects, and fail under variations in style\. A human can look back at the image and recognize that two masks belong to the same shape or that a dashed route continues beyond where the tracer stopped, while the script may return a confident but incorrect answer\. Agentic tools therefore depend on active perception for robustness rather than replacing it\. This challenge is greater in real deployments, where medical scans, satellite images, phone snapshots, and scanned documents contain more varied and less predictable noise than fixed pipelines can anticipate\.

Scope of the agentic evaluation\.The agentic ablation instead measures tool\-orchestrated visual problem solving: decomposing a visual question into checkable operations, writing extraction code that works on noisy imagery, and recognizing when an output requires visual verification\. In effect, the agent offloads iterative visual reasoning to code and reduces verification to simpler visual checks, a lower bar than solving the task through vision alone\. Even so, the strongest agent remains far less accurate and efficient than an unaided human\. BecauseActiveVisionis organized around cognitive axes rather than real\-world scenarios, fixed pipelines transfer unevenly across tasks\. Figure[7\(b\)](https://arxiv.org/html/2607.16165#S4.F7.sf2)shows this split\. The attribute transfer tasks are largely solved, while the traversal tasks remain near zero for two of the three agents, and all agents fail*Tangled Loop Counting*\. We therefore report the agentic evaluation as an ablation rather than a standalone track and leave a dedicated benchmark for agentic visual problem solving to future work\.

### 5\.2Limitations and Broader Implications

Limitations\.Our images are synthetic even with photorealistic re\-rendering: GPT\-image\-2 produces realistic\-looking outputs from a controlled prompt rather than samples of the natural\-image distribution\. We accept this in exchange for exact ground truth and controlled task structure\. External validity therefore rests on the elemental visual operations the tasks measure, including scanning, tracing, and comparing, rather than on the renderings themselves \(§[3\.1](https://arxiv.org/html/2607.16165#S3.SS1)\)\. Furthermore, our tasks are built so that no short language description carries the answer; as models get better at describing images, this property can erode, and future versions of the benchmark would need to tighten the tasks to restore it\.

Outlook\.ActiveVisionreveals and quantifies a wide gap between human and machine perception that current vision\-language benchmarks do not surface, turning “active vision” from a rhetorical claim into a number that future architectures and training objectives can be held to\. The gap matters beyond the benchmark because the three operations it isolates \(§[3\.1](https://arxiv.org/html/2607.16165#S3.SS1)\) are the documented substance of high\-stakes visual work\. Exhaustive scanning underlies radiology, cell counting, inventory inspection, and aerial\-image search\. In chest radiography, about 30% of missed nodules are never fixated at all\[[27](https://arxiv.org/html/2607.16165#bib.bib27)\]\. Sequential tracing is central to connectomics, where proofreaders follow neurites through electron microscopy volumes\[[39](https://arxiv.org/html/2607.16165#bib.bib39)\], as well as to vessel and schematic tracing\. Fine\-grained comparison underlies latent\-print examination, pathology, and industrial quality control\[[8](https://arxiv.org/html/2607.16165#bib.bib8),[26](https://arxiv.org/html/2607.16165#bib.bib26),[12](https://arxiv.org/html/2607.16165#bib.bib12)\]\. Eye\-tracking across these professions shows that expertise largely*is*knowing where and how to look\[[15](https://arxiv.org/html/2607.16165#bib.bib15)\]\. A model that cannot count, trace, or compare reliably in a controlled image has no path to reliability in these settings; closing the gap is a prerequisite for the broader goal that motivates this work: reliable MLLMs and the multimodal agents built on top of them\.

## 6Conclusion

We introduceActiveVision, a benchmark that isolates active observation, the ability to keep returning to an image, forming and checking hypotheses, as reasoning unfolds\. Its tasks are built so that no single language description carries the answer, and its generation pipeline preserves the exact underlying geometry while rendering it as a realistic, noisy image\. InActiveVision, humans solve nearly all items in about half a minute each, whereas frontier MLLMs fail most items\. Increasing reasoning effort does little to close this gap\. Code and tools provide larger but uneven gains, concentrated on tasks amenable to reliable algorithmic reduction; even the strongest agent solves only half of the items while requiring far more time and compute than unaided human participants\. Closing this gap is a prerequisite for many real\-world applications such as robotics, healthcare, manufacturing, and scientific discovery, andActiveVisionturns the gap into a number that future architectures and training objectives can be held to\.

## References

- Achiam et al\. \[2023\]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al\.Gpt\-4 technical report\.*arXiv preprint arXiv:2303\.08774*, 2023\.
- Aloimonos et al\. \[1988\]John Aloimonos, Isaac Weiss, and Amit Bandyopadhyay\.Active vision\.*International journal of computer vision*, 1\(4\):333–356, 1988\.
- Anthropic \[2026a\]Anthropic\.Introducing claude opus 4\.8\.[https://www\.anthropic\.com/news/claude\-opus\-4\-8](https://www.anthropic.com/news/claude-opus-4-8), 2026a\.Accessed: 2026\-07\-16\.
- Anthropic \[2026b\]Anthropic\.Introducing claude fable 5 and claude mythos 5\.[https://www\.anthropic\.com/news/claude\-fable\-5\-mythos\-5](https://www.anthropic.com/news/claude-fable-5-mythos-5), 2026b\.Accessed: 2026\-07\-01\.
- Bajcsy \[1988\]Ruzena Bajcsy\.Active perception\.*Proceedings of the IEEE*, 76\(8\):966–1005, 1988\.
- Ballard \[1991\]Dana H\. Ballard\.Animate vision\.*Artificial Intelligence*, 48\(1\):57–86, 1991\.
- Ballard et al\. \[1995\]Dana H\. Ballard, Mary M\. Hayhoe, and Jeff B\. Pelz\.Memory representations in natural tasks\.*Journal of Cognitive Neuroscience*, 7\(1\):66–80, 1995\.
- Busey et al\. \[2011\]Thomas Busey, Chen Yu, Dean Wyatte, John Vanderkolk, Francisco Parada, and Ruj Akavipat\.Consistency and variability among latent print examiners as revealed by eye tracking methodologies\.*Journal of Forensic Identification*, 61\(1\):60–91, 2011\.
- Chen et al\. \[2026\]Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, Hans Zhao, et al\.Babyvision: Visual reasoning beyond language\.*arXiv preprint arXiv:2601\.06521*, 2026\.
- Chen et al\. \[2024\]Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al\.Are we on the right way for evaluating large vision\-language models?*arXiv preprint arXiv:2403\.20330*, 2024\.
- ChildBench Team \[2025\]ChildBench Team\.Easy for children, hard for ai: The limits of mllms in early learning tasks recognition\.[https://github\.com/Jderder/ChildBench](https://github.com/Jderder/ChildBench), 2025\.
- Drury \[1992\]Colin G\. Drury\.Inspection performance\.In Gavriel Salvendy, editor,*Handbook of Industrial Engineering*, pages 2282–2314\. Wiley, 2nd edition, 1992\.
- Findlay and Gilchrist \[2003\]John M\. Findlay and Iain D\. Gilchrist\.*Active Vision: The Psychology of Looking and Seeing*\.Oxford University Press, 2003\.
- Fu et al\. \[2024\]Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei\-Chiu Ma, and Ranjay Krishna\.Blink: Multimodal large language models can see but not perceive\.*arXiv preprint arXiv:2404\.12390*, 2024\.
- Gegenfurtner et al\. \[2011\]Andreas Gegenfurtner, Erno Lehtinen, and Roger Säljö\.Expertise differences in the comprehension of visualizations: A meta\-analysis of eye\-tracking research in professional domains\.*Educational Psychology Review*, 23\(4\):523–552, 2011\.
- Google DeepMind \[2026a\]Google DeepMind\.Gemini 3\.1 pro: A smarter model for your most complex tasks\.[https://blog\.google/innovation\-and\-ai/models\-and\-research/gemini\-models/gemini\-3\-1\-pro/](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/), 2026a\.Accessed: 2026\-05\-05\.
- Google DeepMind \[2026b\]Google DeepMind\.Gemini 3\.5: Frontier intelligence with action\.[https://blog\.google/innovation\-and\-ai/models\-and\-research/gemini\-models/gemini\-3\-5/](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-5/), 2026b\.Accessed: 2026\-07\-16\.
- Hayhoe and Ballard \[2005\]Mary Hayhoe and Dana Ballard\.Eye movements in natural behavior\.*Trends in Cognitive Sciences*, 9\(4\):188–194, 2005\.
- Jian et al\. \[2026\]Xiangru Jian, Hao Xu, Wei Pang, Xinjian Zhao, Chengyu Tao, Qixin Zhang, Xikun Zhang, Chao Zhang, Guanzhi Deng, Alex Xue, Juan Du, Tianshu Yu, Garth Tarr, Linqi Song, Qiuzhuang Sun, and Dacheng Tao\.FORGE: Fine\-grained multimodal evaluation for manufacturing scenarios\.*arXiv preprint arXiv:2604\.07413*, 2026\.
- Jiang et al\. \[2025\]Mohan Jiang, Jin Gao, Jiahao Zhan, and Dequan Wang\.MAC: A live benchmark for multimodal large language models in scientific understanding\.*arXiv preprint arXiv:2508\.15802*, 2025\.
- Jolicoeur et al\. \[1986\]Pierre Jolicoeur, Shimon Ullman, and Marilynne Mackay\.Curve tracing: A possible basic operation in the perception of spatial relations\.*Memory & Cognition*, 14\(2\):129–140, 1986\.
- Kar et al\. \[2024\]Oğuzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari\.BRAVE: Broadening the visual encoding of vision\-language models\.*arXiv preprint arXiv:2404\.07204*, 2024\.
- Karamcheti et al\. \[2024\]Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh\.Prismatic VLMs: Investigating the design space of visually\-conditioned language models\.*arXiv preprint arXiv:2402\.07865*, 2024\.
- Kaufman et al\. \[1949\]E\. L\. Kaufman, M\. W\. Lord, T\. W\. Reese, and J\. Volkmann\.The discrimination of visual number\.*The American Journal of Psychology*, 62\(4\):498–525, 1949\.
- Kolodiazhnyi et al\. \[2026\]Maksim Kolodiazhnyi, Denis Tarasov, Dmitrii Zhemchuzhnikov, Alexander Nikulin, Ilya Zisman, Anna Vorontsova, Anton Konushin, Vladislav Kurenkov, and Danila Rukhovich\.cadrille: Multi\-modal CAD reconstruction with reinforcement learning\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.arXiv:2505\.22914\.
- Krupinski et al\. \[2006\]Elizabeth A\. Krupinski, Allison A\. Tillack, Lynne Richter, Jeffrey T\. Henderson, Achyut K\. Bhattacharyya, Katherine M\. Scott, Anna R\. Graham, Michael R\. Descour, John R\. Davis, and Ronald S\. Weinstein\.Eye\-movement study and human performance using telepathology virtual slides: Implications for medical education and differences with experience\.*Human Pathology*, 37\(12\):1543–1556, 2006\.
- Kundel et al\. \[1978\]Harold L\. Kundel, Calvin F\. Nodine, and Dennis Carmody\.Visual scanning, pattern recognition and decision\-making in pulmonary nodule detection\.*Investigative Radiology*, 13\(3\):175–181, 1978\.
- Laurent et al\. \[2024\]Jon M\. Laurent, Joseph D\. Janizek, Michael Ruzo, Michaela M\. Hinks, Michael J\. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D\. White, and Samuel G\. Rodriques\.LAB\-Bench: Measuring capabilities of language models for biology research\.*arXiv preprint arXiv:2407\.10362*, 2024\.
- Luck and Vogel \[1997\]Steven J\. Luck and Edward K\. Vogel\.The capacity of visual working memory for features and conjunctions\.*Nature*, 390\(6657\):279–281, 1997\.
- Luo et al\. \[2025\]Yulin Luo, Chun\-Kai Fan, Menghang Dong, Jiayu Shi, Mengdi Zhao, Bo\-Wen Zhang, Cheng Chi, Jiaming Liu, Gaole Dai, Rongyu Zhang, Ruichuan An, Kun Wu, Zhengping Che, Shaoxuan Xie, Guocai Yao, Zhongxia Zhao, Pengwei Wang, Guang Liu, Zhongyuan Wang, Tiejun Huang, and Shanghang Zhang\.RoboBench: A comprehensive evaluation benchmark for multimodal large language models as embodied brain\.*arXiv preprint arXiv:2510\.17801*, 2025\.
- McKinzie et al\. \[2024\]Brandon McKinzie, Zhe Gan, Jean\-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al\.Mm1: Methods, analysis & insights from multimodal llm pre\-training\.*arXiv preprint arXiv:2403\.09611*, 2024\.
- OpenAI \[2026a\]OpenAI\.Introducing gpt\-5\.5\.[https://openai\.com/index/introducing\-gpt\-5\-5/](https://openai.com/index/introducing-gpt-5-5/), 2026a\.Accessed: 2026\-05\-05\.
- OpenAI \[2026b\]OpenAI\.ChatGPT images 2\.0 \(GPT\-image\-2\)\.[https://openai\.com/index/introducing\-chatgpt\-images\-2\-0/](https://openai.com/index/introducing-chatgpt-images-2-0/), 2026b\.
- O’Regan and Noë \[2001\]J\. Kevin O’Regan and Alva Noë\.A sensorimotor account of vision and visual consciousness\.*Behavioral and Brain Sciences*, 24\(5\):939–973, 2001\.
- Petersson et al\. \[2025\]Lukas Petersson, Axel Backlund, Axel Wennstöm, Hanna Petersson, Callum Sharrock, and Arash Dabiri\.Blueprint\-Bench: Comparing spatial intelligence of LLMs, agents and image models\.*arXiv preprint arXiv:2509\.25229*, 2025\.
- Rahmanzadehgervi et al\. \[2024\]Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen\.Vision language models are blind\.*arXiv preprint arXiv:2407\.06581*, 2024\.
- Roelfsema \[2005\]Pieter R\. Roelfsema\.Elemental operations in vision\.*Trends in Cognitive Sciences*, 9\(4\):226–233, 2005\.
- Roelfsema et al\. \[1998\]Pieter R\. Roelfsema, Victor A\. F\. Lamme, and Henk Spekreijse\.Object\-based attention in the primary visual cortex of the macaque monkey\.*Nature*, 395\(6700\):376–381, 1998\.
- Scheffer et al\. \[2020\]Louis K\. Scheffer, C\. Shan Xu, Michal Januszewski, et al\.A connectome and analysis of the adult Drosophila central brain\.*eLife*, 9:e57443, 2020\.
- Shi et al\. \[2024\]Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De\-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al\.Eagle: Exploring the design space for multimodal llms with mixture of encoders\.*arXiv preprint arXiv:2408\.15998*, 2024\.
- Tao and Xie \[2024\]Muzi Tao and Saining Xie\.What does a visual formal analysis of the world’s 500 most famous paintings tell us about multimodal llms?In*The Second Tiny Papers Track at ICLR 2024*, 2024\.
- Tong et al\. \[2024a\]Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al\.Cambrian\-1: A fully open, vision\-centric exploration of multimodal llms\.*arXiv preprint arXiv:2406\.16860*, 2024a\.
- Tong et al\. \[2024b\]Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie\.Eyes wide shut? exploring the visual shortcomings of multimodal llms\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 9568–9578, 2024b\.
- Trick and Pylyshyn \[1994\]Lana M\. Trick and Zenon W\. Pylyshyn\.Why are small and large numbers enumerated differently? A limited\-capacity preattentive stage in vision\.*Psychological Review*, 101\(1\):80–102, 1994\.
- Ullman \[1984\]Shimon Ullman\.Visual routines\.*Cognition*, 18\(1–3\):97–159, 1984\.
- Vo et al\. \[2025\]An Vo, Khai\-Nguyen Nguyen, Mohammad Reza Taesiri, Vy Tuong Dang, Anh Totti Nguyen, and Daeyoung Kim\.Vision language models are biased\.*arXiv preprint arXiv:2505\.23941*, 2025\.
- Wang et al\. \[2024\]Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen\.CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs\.In*Advances in Neural Information Processing Systems \(NeurIPS\) – Datasets and Benchmarks Track*, volume 37, pages 113569–113697, 2024\.URL[https://arxiv\.org/abs/2406\.18521](https://arxiv.org/abs/2406.18521)\.
- Yarbus \[1967\]Alfred L\. Yarbus\.*Eye Movements and Vision*\.Plenum Press, New York, 1967\.
- Ye et al\. \[2026\]Hengwei Ye, Yuanting Guan, Yuxuan Ge, Tianying Zhu, Zhenhan Guan, Yijia Zhong, et al\.Children’s intelligence tests pose challenges for mllms? kidgym: A 2d grid\-based reasoning benchmark for mllms\.*arXiv preprint arXiv:2603\.20209*, 2026\.
- Yuan et al\. \[2026\]Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong, Weiming Wu, Jiayang Sun, Jiamin Song, Tianbao Xie, Tao Yu, et al\.OSWorld2\.0: Benchmarking computer use agents on long\-horizon real\-world tasks\.*arXiv preprint arXiv:2606\.29537*, 2026\.
- Yue et al\. \[2025\]Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig\.MMMU\-Pro: A more robust multi\-discipline multimodal understanding benchmark\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 15050–15070\. Association for Computational Linguistics, 2025\.URL[https://aclanthology\.org/2025\.acl\-long\.736/](https://aclanthology.org/2025.acl-long.736/)\.
- Zhang et al\. \[2026\]Haozhe Zhang, Kaichen Liu, Miaomiao Chen, Lei Li, Shaojie Yang, Cheng Peng, and Hanjie Chen\.BenchCAD: A comprehensive, industry\-standard benchmark for programmatic CAD\.*arXiv preprint arXiv:2605\.10865*, 2026\.
- Zhang et al\. \[2024\]Jiarui Zhang, Ollie Liu, Tianyu Yu, Jinyi Hu, and Willie Neiswanger\.Euclid: Supercharging multimodal llms with synthetic high\-fidelity visual descriptions\.*arXiv preprint arXiv:2412\.08737*, 2024\.
- Zhang et al\. \[2025\]Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski\.MLLMs know where to look: Training\-free perception of small visual details with multimodal llms\.*arXiv preprint arXiv:2502\.17422*, 2025\.