Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models

# Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models
Source: [https://arxiv.org/html/2607.08317](https://arxiv.org/html/2607.08317)
Matteo SantelmoXiuying Wei11footnotemark:1Israa Fakih11footnotemark:1Felix Bauer11footnotemark:1 Juan Garcia Giraldo11footnotemark:1Chengkun Li11footnotemark:1Etienne Bamas22footnotemark:2Emmanuel Abbé22footnotemark:2 École Polytechnique Fédérale de Lausanne \(EPFL\), SwitzerlandEqual contribution\. Correspondence to<matteo\.santelmo@epfl\.ch\>,<xiuying\.wei@epfl\.ch\>\. Matteo led the evaluation and Xiuying led dataset cleaning and labeling\. Dagger†denotes advising roles\.

###### Abstract

Modern AI models achieve strong performance on many established benchmarks, yet they still fail on tasks that humans find almost trivial, such as manipulating a string or drawing a dog with five legs\. These examples suggest that existing benchmarks may under\-measure persistent blind spots in current systems\. We introduceblind\-spots\-bench, a benchmark designed to expose such blind spots through tasks that appear simple for humans but remain challenging for modern AI\. We collect raw questions from students in an AI course, clean and annotate them with structured reference solutions, and propose a task taxonomy tailored to the resulting dataset of 235 samples\. We further develop an automated grading pipeline to evaluate a wide range of models, including open\-weight and closed\-source language, vision\-language, and image\-generation models\. Our analysis onblind\-spots\-benchreveals that closed\-source frontier models can substantially outperform open\-weight models with even≈\\approx10% gap, even when they attain comparable performance on existing benchmarks\. A more fine\-grained analysis shows that no single model dominates across all task types, and that some tasks remain challenging for all evaluated models\. These results highlight the value ofblind\-spots\-benchas a diagnostic stress test for identifying concrete weaknesses in current modern models\.

![Refer to caption](https://arxiv.org/html/2607.08317v1/x3.png)Figure 1:Left: Accuracy onblind\-spots\-benchvs\. Artificial Analysis Intelligence Index score fortext\-onlyproblems\.Right:Performance of four VLM models on several sub\-tasks\.## 1Introduction

The rapid ascent of large language models \(LLMs\) and multimodal models has been marked by strong performance across a wide range of tasks, including mathematics, coding, reasoning, and vision\. Recent frontier models\[[10](https://arxiv.org/html/2607.08317#bib.bib58),[38](https://arxiv.org/html/2607.08317#bib.bib67),[16](https://arxiv.org/html/2607.08317#bib.bib63)\]have achieved impressive results, often nearly saturating several carefully designed benchmarks\[[40](https://arxiv.org/html/2607.08317#bib.bib80),[47](https://arxiv.org/html/2607.08317#bib.bib82),[6](https://arxiv.org/html/2607.08317#bib.bib85),[19](https://arxiv.org/html/2607.08317#bib.bib83),[11](https://arxiv.org/html/2607.08317#bib.bib84)\], in some cases rivaling or surpassing human experts\. Similar progress has also been observed on vision\-language benchmarks, where modern systems demonstrate strong capabilities\[[29](https://arxiv.org/html/2607.08317#bib.bib89),[28](https://arxiv.org/html/2607.08317#bib.bib88),[67](https://arxiv.org/html/2607.08317#bib.bib86)\]\.

In practice, however, these models still exhibit surprising failures on tasks that are straightforward for humans\[[72](https://arxiv.org/html/2607.08317#bib.bib36),[51](https://arxiv.org/html/2607.08317#bib.bib37),[55](https://arxiv.org/html/2607.08317#bib.bib40),[13](https://arxiv.org/html/2607.08317#bib.bib14),[8](https://arxiv.org/html/2607.08317#bib.bib43)\]\. For instance, a model may struggle to generate a string of a specified length\[[64](https://arxiv.org/html/2607.08317#bib.bib93)\], produce an image of a cat with exactly four eyes\[[63](https://arxiv.org/html/2607.08317#bib.bib17),[7](https://arxiv.org/html/2607.08317#bib.bib20)\], draw a clock showing a requested time\[[50](https://arxiv.org/html/2607.08317#bib.bib92)\], or solve a relatively simple Sudoku puzzle\[[56](https://arxiv.org/html/2607.08317#bib.bib91)\]\. These failures point to persistent blind spots across a range of abilities like spatial reasoning\[[62](https://arxiv.org/html/2607.08317#bib.bib74)\], logical consistency\[[26](https://arxiv.org/html/2607.08317#bib.bib94)\], and character\-level manipulation\[[64](https://arxiv.org/html/2607.08317#bib.bib93)\]\.

To systematically study which questions remain easy for humans yet difficult for today’s state\-of\-the\-art models, we introduceblind\-spots\-bench, a benchmark of 235 manually curated questions across multiple input\-output format, accompanied by a reproducible evaluation framework\. The questions are collected from students in a graduate\-level AI course, who were asked in October 2025 to propose problems frontier AI chatbots failed to answer correctly\. Each problem is annotated with a structured reference solution, question format, and task label\. To support fine\-grained analysis, we introduce a task taxonomy with three broad categories and finer\-grained subtask labels: object\-centric tasks, such as counting, recognition, and spatial reasoning; abstract reasoning tasks, mainly involving mathematical and logic\-oriented problems; and language\-and\-knowledge tasks, involving linguistic and knowledge\-based reasoning\. We further develop an end\-to\-end evaluation pipeline to automatically grade answers against the reference solutions\. Under this framework, we evaluate a broad range of frontier systems, including text\-only LLMs, vision\-language models \(VLMs\), and specialized image\-generation models, covering 38 models in total\.

Our results provide a systematic comparison across model variants and offer two complementary analyses\. At the model level, the best\-performing closed\-source systems achieve approximately 10 per cent higher accuracy onblind\-spots\-benchthan the highest\-scoring open\-weight models\. This difference remains marked even between models with comparable performance on established benchmarks \(see[Fig\. 1](https://arxiv.org/html/2607.08317#S0.F1)\)\. In contrast, open\-weight models can be more effective under the same evaluation budget\. We also find that tool use is not uniformly beneficial and can sometimes lead to lower accuracy\. At the task level, our taxonomy supports detailed analyses of model capabilities\. Fine\-grained visual perception tasks, such as counting objects in images, remain challenging for all models with no more than 60% accuracy results\. No single model remains top\-1 across all tasks\. For example, the strongestGPTandGeminimodels show different strengths across different tasks \(e\.g\., 88\.6% ofGPTon arithmetic reasoning\)\. Meanwhile, other models, can still excel in specific categories, such asDeepSeek\-V4on character\-level manipulation andKimi\-2\.5on arithmetic reasoning\. Finally, we find that scaling model size within the same family does not always improve performance consistently\.

We summarize our contributions as follows:

1. 1\.We introduceblind\-spots\-bench, a curated dataset of 235 open\-ended problems designed to expose blind spots in multimodal AI systems through tasks that are easy for humans but challenging for today’s models\. We apply a detailed cleaning and annotation process and provide structured reference solutions to support reliable evaluation\.
2. 2\.We develop an end\-to\-end evaluation pipeline, covering response generation, AI\-based automatic grading, and manual auditing of grading reliability\. Using this pipeline, we perform a comprehensive evaluation of 32 LLMs/VLMs, and 6 specialized image\-generation models\.
3. 3\.We propose a task taxonomy and conduct fine\-grained analyses of model behavior across task types\. The taxonomy consists of three high\-level categories and 12 subcategories, enabling us to examine question composition and sources of difficulty\. Our analysis reveals both shared weaknesses, such as object counting, and model\-specific strengths and failure patterns\.

## 2Related Works

#### General and Multimodal Evaluation Benchmarks\.

A growing body of work has emphasized the importance of moving beyond a single aggregate benchmark score toward more structured characterizations of model reasoning errors\. Recent benchmarks focusing on broad multidisciplinary reasoning performance, such as RBench\[[18](https://arxiv.org/html/2607.08317#bib.bib45)\], BIG\-Bench\[[53](https://arxiv.org/html/2607.08317#bib.bib97)\], GPQA\[[47](https://arxiv.org/html/2607.08317#bib.bib82)\], and MMMU\[[67](https://arxiv.org/html/2607.08317#bib.bib86)\], aggregate metrics across different categories, but fail to understand the differences in model behavior across specific reasoning components and identify residual weaknesses on simple but precise skills\. In contrast,blind\-spots\-benchintroduces a structured framework that decomposes reasoning into fine\-grained sub\-tasks, allowing for a more interpretable assessment beyond aggregate scores, revealing that different models exhibit distinct strengths and weaknesses

#### Taxonomies and Failure Analysis\.

Moving beyond aggregate benchmarks, prior work has explored characterizing model behaviors through behavior testing, diagnostic challenge sets, and structured task taxonomies and failure analysis\. In NLP, checklist\[[48](https://arxiv.org/html/2607.08317#bib.bib100)\]introduced capability\-level behavioral testing, while diagnostic benchmarks such as HANS\[[32](https://arxiv.org/html/2607.08317#bib.bib101)\]showed that models with high benchmark accuracy can still rely on brittle heuristics\. Dynamic and contrastive evaluation frameworks further highlight the value of constructing targeted examples around model failures\[[21](https://arxiv.org/html/2607.08317#bib.bib102),[14](https://arxiv.org/html/2607.08317#bib.bib103)\]\. In multimodal evaluation, benchmarks such as MME\[[12](https://arxiv.org/html/2607.08317#bib.bib104)\], ME\-Reasoning\[[66](https://arxiv.org/html/2607.08317#bib.bib44)\], MM\-Vet\[[65](https://arxiv.org/html/2607.08317#bib.bib105)\], GLUE\[[61](https://arxiv.org/html/2607.08317#bib.bib99)\], SIV\-Bench\[[23](https://arxiv.org/html/2607.08317#bib.bib98)\]organize evaluation around structured capabilities including perception, OCR, spatial reasoning, object\-attribute understanding, social reasoning, multimodal logical reasoning, and distinctions between informal and formal reasoning or embodied settings\[[52](https://arxiv.org/html/2607.08317#bib.bib48)\]\. Although these work provide useful high\-level analyses of model behavior, they are either too coarse or too broad to support fine\-grained analysis\. As a result, they are difficult to apply to our benchmark setting for reasoning failure analysis\. In this work, we instead propose a structured decomposition or reasoning problems into fine\-grained sub\-tasks and introduce failure modes for questions with more than one sub\-task, where the type of failure is determined based on the model’s responses\. This enables a more detailed and contextual analysis of errors\.

#### Robustness Evaluation

Another line of work studies whether high benchmark scores reflect robust reasoning\. GSM\-Plus\[[25](https://arxiv.org/html/2607.08317#bib.bib24)\]and Math\-Perturb\[[3](https://arxiv.org/html/2607.08317#bib.bib4)\]generate perturbed versions of problems from well established reasoning benchmarks\. GSM\-Symbolic\[[34](https://arxiv.org/html/2607.08317#bib.bib2)\]and VarBench\[[43](https://arxiv.org/html/2607.08317#bib.bib26)\]focus on creating parametric templates to test performance invariance on equivalent problems\. These benchmarks complementblind\-spots\-bench, but differ from it in their purpose\. Perturbation benchmarks create controlled variants of existing problems, whereasblind\-spots\-benchis built from naturally occurring failures of frontier models focusing on underrepresented tasks\.

## 3Benchmark construction and Composition analysis

![Refer to caption](https://arxiv.org/html/2607.08317v1/x4.png)Figure 2:Representative examples from the dataset\. While these tasks are generally easy for human, we find that they remain challenging for frontier models\. Each example is annotated with its solution \(an abbreviated version here\), question format, task and sub\-task categories, illustrating the diversity of skills evaluated in the benchmark\. Full taxonomy and additional examples in[Appendix A](https://arxiv.org/html/2607.08317#A1)\.### 3\.1Data collection

We collected raw questions from students in a graduate\-level AI\-related course\. Each student was asked to propose five questions that appeared easy to humans but were failed by frontier models available around October 2025\. This course\-wide collection yielded approximately 287 raw questions\. We then transformed the raw collection into a standardized benchmark through a systematic cleaning and annotation pipeline\. This process included filtering out overly difficult or duplicate entries, normalizing question formats, and ensuring each problem is clear and well\-posed\. We further added key annotations as described below to facilitate verification and analysis\. The resulting dataset,blind\-spots\-bench, contains 235 samples and is publicly available on Hugging Face\.111[https://huggingface\.co/datasets/matsant01/blind\-spots\-bench](https://huggingface.co/datasets/matsant01/blind-spots-bench)Representative examples are shown in[Fig\. 2](https://arxiv.org/html/2607.08317#S3.F2)\.

#### Solution and Question format annotation

For each question, we manually curate structured reference solutions to support reliable evaluation of model outputs\. These solutions are designed to support downstream automatic verifiers, which require explicit and operational correctness criteria\. Each reference solution specifies the expected answer, the necessary conditions for correctness, and common failure modes\. The common failure modes are identified from two sources: errors observed in students’ submitted interactions with AI agents during data collection, and additional error cases proposed by at least three annotators for each question\.

We also annotate each example with its question format to facilitate model comparison, since different models support different input\-output formats\. The benchmark includes three question\-format categories:text\-only, where both input and output are text;image generation, where the output is an image; andmulti\-to\-text, where the input contains both image and text and the output is text\. These labels allow us to assign examples to appropriate models, compare systems within their applicable settings, and identify which formats current models struggle with most\.

#### Task taxonomy

Unlike many task\-specific benchmarks\[[5](https://arxiv.org/html/2607.08317#bib.bib76),[31](https://arxiv.org/html/2607.08317#bib.bib75),[62](https://arxiv.org/html/2607.08317#bib.bib74)\],blind\-spots\-benchis designed to challenge AI systems and uncover reasoning blind spots across diverse question types, spanning a broad range of skills\. To better understand the sources of difficulty and support subsequent analysis of model performance, we introduce a taxonomy tailored to our dataset\. Our taxonomy consists of three main categories:*Object\-centric*,*Abstract reasoning*, and*Language and knowledge*\. Each category is further decomposed into sub\-tasks targeting specific abilities\. For example, as shown in[Fig\. 2](https://arxiv.org/html/2607.08317#S3.F2),*Object\-centric*tasks cover abilities such as attribute and pattern recognition, spatial reasoning, perceptual counting, and generative counting\. We provide detailed descriptions of all categories and sub\-tasks in[Table 4](https://arxiv.org/html/2607.08317#A1.T4), and discuss the overall distribution shown in[Fig\. 3](https://arxiv.org/html/2607.08317#S3.F3)in the next section\.

#### Review and Quality Control

The dataset underwent a three\-stage review process\. First, all the questions were manually verified to confirm that each task is well defined and easy to interpret\. Then we performed difficulty thresholding, removing questions that are easily solved by models or overly difficult for humans so that the benchmark focuses on blind spots\. Finally, some questions were refined to clarify linguistic ambiguities and to ensure the correctness and completeness of the ground truth annotations\.

![Refer to caption](https://arxiv.org/html/2607.08317v1/x5.png)

![Refer to caption](https://arxiv.org/html/2607.08317v1/x6.png)

Figure 3:Left:Question format composition\.Right:Task category composition\. Some questions \(about 15\) involve multiple subtask categories; for these cases, we count one occurrence for each applicable subtask\. The bar chart reports the total number of occurrences for each fine\-grained task category, grouped into three major categories\.

### 3\.2Dataset Composition

Following the construction and annotation phase, we conduct a composition analysis ofblind\-spots\-benchdataset for both question format and task\-category types\. As illustrated in[Fig\. 3](https://arxiv.org/html/2607.08317#S3.F3), text\-only questions constitute the largest portion of the dataset \(46\.2%\), followed by image\-gen \(35\.6%\) then multi\-to\-text \(18\.2%\) which are relatively fewer\.

Beyond the question format distribution, we further analyze the task and sub\-task category composition in[Fig\. 3](https://arxiv.org/html/2607.08317#S3.F3)\. Since the prompts were designed to challenge frontier models, frequent sub\-tasks can reveal sources of difficulty to some extent, rather than merely reflecting dataset statistics\. They also align with known limitations of current models, suggesting that our taxonomy is both data\-driven and grounded in prior findings\. For example, spatial reasoning is the most frequent sub\-task, appearing 53 times, consistent with prior observations that vision\-language models struggle with spatial relations and visual transformations\[[54](https://arxiv.org/html/2607.08317#bib.bib78),[71](https://arxiv.org/html/2607.08317#bib.bib79)\]\. Perceptual counting and generative counting are also common, reflecting the difficulty of counting visual instances\[[42](https://arxiv.org/html/2607.08317#bib.bib77)\], generating outputs with precise numerical constraints\[[7](https://arxiv.org/html/2607.08317#bib.bib20)\], and binding objects to novel quantities\[[63](https://arxiv.org/html/2607.08317#bib.bib17)\]\. Logical reasoning often requires multi\-step inference, where errors can accumulate when models make locally plausible decisions without sufficient global verification\[[49](https://arxiv.org/html/2607.08317#bib.bib49)\]\. Character\-level manipulation is another challenge requiring exact length control or repeated token sequences, which may be partly explained by limitations of subword tokenization\[[69](https://arxiv.org/html/2607.08317#bib.bib13),[13](https://arxiv.org/html/2607.08317#bib.bib14)\]\. We provide a more detailed discussion in[Subsec\. A\.1](https://arxiv.org/html/2607.08317#A1.SS1)\.

While these individual sub\-tasks have been studied separately in prior work, our contribution is to organize them into a unified taxonomy tailored to our dataset\. This taxonomy connects dataset composition, known model weaknesses, and later model performance analysis\.

## 4Experiments

### 4\.1Experimental Setup

#### Evaluation Pipeline

A harness was designed to automate the evaluation and scale it to a wide range of AI models onblind\-spots\-bench\. The evaluation pipeline consists of two steps: First, asolvermodel is prompted to answer questions fromblind\-spots\-bench, without neither in\-context examples nor chain\-of\-thought prompting\. Subsequently, questions and responses are processed by a second model, that we refer to asgrader, to produce binary outputs\. As the questions are designed to have objectively correct and incorrect answers, no partial grades are needed\. The prompt for thegraderincludes the original question, the solver’s attempt, and the solution as described in[Subsec\. 3\.1](https://arxiv.org/html/2607.08317#S3.SS1)\. The fullgraderprompts are reported in Appendix[Appendix B](https://arxiv.org/html/2607.08317#A2)\.

The pipeline is built using the Inspect AI\[[57](https://arxiv.org/html/2607.08317#bib.bib1)\]framework \(MIT License\)\. As agradermodel we usegemini\-3\-flashwhich offers a good trade\-off between grading quality, speed and cost\. We enable thegraderwith code execution to enhance its precision in verifying hard constraints, such as counting or finding sub\-strings\. The code of the evaluation pipeline is made public\.222[https://github\.com/matteosantelmo/reasoning\-blind\-spots](https://github.com/matteosantelmo/reasoning-blind-spots)

#### Grader Validation

Since the evaluation relies on an AIgrader, we manually validate its agreement with human judgments\. Specifically, we sample more than 100 generated responses, equally drawn from differentsolvermodels, score them using our pipeline, and in parallel manually annotate their correctness\. This validation set is designed to include roughly equal numbers of responses graded as ”correct” and ”incorrect”, preventing class imbalance from biasing agreement metrics\. Full human\-graderagreement scores are reported in[Table 5](https://arxiv.org/html/2607.08317#A3.T5), with an accuracy of 96\.6% on tasks with textual outputs, and 90\.9% on the more challenging image\-generation tasks\.

Since our evaluation usesgemini\-3\-flashas the grader, we additionally assess it exhibits bias that inflates scores for models from the same provider \(Gemini and Gemma\)\. Results are reported in[Table6](https://arxiv.org/html/2607.08317#A3.T6)\. Overall, we find no substantial evidence of a pro\-Google grading bias\. For both tasks with textual outputs and tasks with image outputs, agreement is comparable between Google and non\-Google models\. The false positive rate is instead higher for non\-Google models, meaning that incorrect answers are more likely to be promoted to ”correct” for non\-Google outputs than for Google model generations\.

#### Evaluated Models Choice and Configuration

We evaluate a diverse set of state\-of\-the\-art LLMs, VLMs and image generation models, covering a range of sizes and capabilities\. The following model families are included:Qwen3andQwen3\.5\[[44](https://arxiv.org/html/2607.08317#bib.bib51),[45](https://arxiv.org/html/2607.08317#bib.bib52)\],GLM 4\.7, 5, 5\.1, 5\.2\[[68](https://arxiv.org/html/2607.08317#bib.bib54)\],Kimi K2\.5and2\.6\[[22](https://arxiv.org/html/2607.08317#bib.bib56)\],DeepSeek V4\[[10](https://arxiv.org/html/2607.08317#bib.bib58)\],Gemma\-4\[[17](https://arxiv.org/html/2607.08317#bib.bib59)\], andGPT\-OSS\[[2](https://arxiv.org/html/2607.08317#bib.bib60)\]\. For frontier models, we consider Google’sGemini\-2\.5, 3, 3\.1\[[9](https://arxiv.org/html/2607.08317#bib.bib61),[15](https://arxiv.org/html/2607.08317#bib.bib62),[16](https://arxiv.org/html/2607.08317#bib.bib63)\]and OpenAIGPT\-5, 5\.2, 5\.4, 5\.5\[[36](https://arxiv.org/html/2607.08317#bib.bib64),[37](https://arxiv.org/html/2607.08317#bib.bib66),[38](https://arxiv.org/html/2607.08317#bib.bib67)\]\. When possible, we include multiple sizes of the same model to analyze the impact of scaling model size\. VLMs are tested on both multi\-to\-text and text\-only questions, while LLMs are evaluated on text\-only questions\. The evaluation is repeatedk=4k=4times for tasks with textual output, and only once for image generation tasks, due to their significantly higher cost\.

All models are evaluated withthinkingmode enabled, andmediumeffort when these options are available, with the maximum number of output tokens set to 32,768\. Unless stated otherwise, no model has access to tools\. Generation requests are retried at most five times in case of failure, and are counted as incorrect if the model fails to produce any response\. All open\-weight LLMs and VLMs were accessed through EPFL Research Computing Platform \(RCP\) AI\-Inference\-as\-a\-Service \(AIaaS\) which deploys and operates AI models, using using vLLM\[[24](https://arxiv.org/html/2607.08317#bib.bib106)\]as backbone and exposing OpenAI\-comapatible APIs\.

For image generation tasks we evaluate different versions of theGPT\-ImageandGemini\-Imagemodels, which are specifically designed for generative image tasks\. Due to technical constraints, no open\-weight image generation models are included\.

A detailed list of the models along with their configuration is provided in[Appendix D](https://arxiv.org/html/2607.08317#A4), where the reported open models pricing refers to the offering by RCP AIaaS\.

### 4\.2Main Results

Takeaways•Closed source models are more accuratethan open\-weight ones, even at the same level of capabilities on other established benchmarks\.•Open models are more cost\-effective: GLM\-5\.2, Qwen3\.5 and DeepSeek\-V4 perform better than frontier modelswith the same evaluation cost\.•Tool\-use is not always beneficial: it reduces token\-use but sometimes decreases accuracy\.

#### Leaderboard

The complete evaluation results are reported in[Table 1](https://arxiv.org/html/2607.08317#S4.T1),[Subsec\. 4\.2](https://arxiv.org/html/2607.08317#S4.SS2.SSS0.Px4)\. We track performance by question type, reporting average accuracy \(mean@k\) and success withinkattempts \(pass@k, defined as the probability of obtaining at least one correct solution out ofkindependent attempts\), as well as average inference cost and the number of generated tokens\.

Table 1:LLM and VLM model results by evaluation subset\. Accuracy and output length are reported asmean±stderr\\texttt\{mean\}\_\{\\pm\\texttt\{stderr\}\}\. Evaluation cost in reported in USD\-per\-100\-samples\.Text\-onlyMulti\-to\-textModelmean@4pass@4out\-tkscostmean@4pass@4out\-tkscostText\-only modelsGLM\-4\.761\.3±4\.161\.3\_\{\\pm 4\.1\}75\.9±4\.175\.9\_\{\\pm 4\.1\}5346±4195346\_\{\\pm 419\}0\.91\-\-\-\-GLM\-564\.4±464\.4\_\{\\pm 4\}77\.8±477\.8\_\{\\pm 4\}3404±2163404\_\{\\pm 216\}1\.11\-\-\-\-GLM\-5\.167\.4±4\.167\.4\_\{\\pm 4\.1\}76\.9±4\.176\.9\_\{\\pm 4\.1\}2948±2502948\_\{\\pm 250\}0\.69\-\-\-\-GLM\-5\.273\.8±3\.673\.8\_\{\\pm 3\.6\}84\.3±3\.584\.3\_\{\\pm 3\.5\}5229±4865229\_\{\\pm 486\}1\.09\-\-\-\-DeepSeek\-V4\-Flash67\.6±3\.967\.6\_\{\\pm 3\.9\}79\.6±3\.979\.6\_\{\\pm 3\.9\}3648±3913648\_\{\\pm 391\}0\.29\-\-\-\-DeepSeek\-V4\-Pro70\.4±3\.870\.4\_\{\\pm 3\.8\}80\.6±3\.880\.6\_\{\\pm 3\.8\}2232±2012232\_\{\\pm 201\}0\.63\-\-\-\-GPT\-oss\-20b52\.3±4\.152\.3\_\{\\pm 4\.1\}67\.6±4\.567\.6\_\{\\pm 4\.5\}2611±3172611\_\{\\pm 317\}0\.02\-\-\-\-GPT\-oss\-120b60\.2±4\.160\.2\_\{\\pm 4\.1\}72\.2±4\.372\.2\_\{\\pm 4\.3\}1555±2041555\_\{\\pm 204\}0\.02\-\-\-\-Qwen3\-Next\-80B\-A3B59\.5±4\.259\.5\_\{\\pm 4\.2\}72\.2±4\.372\.2\_\{\\pm 4\.3\}6460±4446460\_\{\\pm 444\}0\.27\-\-\-\-VLMsKimi\-K2\.568\.3±3\.868\.3\_\{\\pm 3\.8\}80\.6±3\.880\.6\_\{\\pm 3\.8\}4163±3364163\_\{\\pm 336\}0\.8642\.4±6\.842\.4\_\{\\pm 6\.8\}53\.5±7\.753\.5\_\{\\pm 7\.7\}4994±6904994\_\{\\pm 690\}1\.07Kimi\-K2\.665±3\.965\_\{\\pm 3\.9\}78\.7±478\.7\_\{\\pm 4\}3797±2663797\_\{\\pm 266\}0\.741\.9±6\.841\.9\_\{\\pm 6\.8\}53\.5±7\.753\.5\_\{\\pm 7\.7\}4501±5274501\_\{\\pm 527\}0\.87Gemma\-4\-E2B\-it32\.4±432\.4\_\{\\pm 4\}42\.6±4\.842\.6\_\{\\pm 4\.8\}1533±1271533\_\{\\pm 127\}0\.0118±4\.718\_\{\\pm 4\.7\}30\.2±7\.130\.2\_\{\\pm 7\.1\}1049±1391049\_\{\\pm 139\}0\.01Gemma\-4\-E4B\-it41\.4±441\.4\_\{\\pm 4\}57\.4±4\.857\.4\_\{\\pm 4\.8\}1601±1471601\_\{\\pm 147\}0\.0122\.7±5\.822\.7\_\{\\pm 5\.8\}32\.6±7\.232\.6\_\{\\pm 7\.2\}1335±2871335\_\{\\pm 287\}0\.01Gemma\-4\-26B\-A4B\-it62\.5±4\.262\.5\_\{\\pm 4\.2\}72\.2±4\.372\.2\_\{\\pm 4\.3\}4541±3414541\_\{\\pm 341\}0\.0640\.7±6\.940\.7\_\{\\pm 6\.9\}53\.5±7\.753\.5\_\{\\pm 7\.7\}3587±5783587\_\{\\pm 578\}0\.05Qwen3\-VL\-30B\-A3B49\.5±4\.149\.5\_\{\\pm 4\.1\}66\.7±4\.666\.7\_\{\\pm 4\.6\}3673±2563673\_\{\\pm 256\}0\.0725±625\_\{\\pm 6\}37\.2±7\.537\.2\_\{\\pm 7\.5\}2796±5382796\_\{\\pm 538\}0\.06Qwen3\-VL\-235B\-A22B58\.3±4\.258\.3\_\{\\pm 4\.2\}71\.3±4\.471\.3\_\{\\pm 4\.4\}3435±2223435\_\{\\pm 222\}0\.3232±6\.832\_\{\\pm 6\.8\}39\.5±7\.539\.5\_\{\\pm 7\.5\}2604±3512604\_\{\\pm 351\}0\.26Qwen3\.5\-35B\-A3B63\.9±4\.263\.9\_\{\\pm 4\.2\}74\.1±4\.274\.1\_\{\\pm 4\.2\}7278±3747278\_\{\\pm 374\}0\.1147\.1±6\.547\.1\_\{\\pm 6\.5\}69\.8±7\.169\.8\_\{\\pm 7\.1\}6492±6436492\_\{\\pm 643\}0\.1Qwen3\.5\-122B\-A10B63\.9±4\.263\.9\_\{\\pm 4\.2\}75±4\.275\_\{\\pm 4\.2\}5637±2845637\_\{\\pm 284\}0\.2548\.8±6\.948\.8\_\{\\pm 6\.9\}62\.8±7\.562\.8\_\{\\pm 7\.5\}3350±3533350\_\{\\pm 353\}0\.16Qwen3\.5\-397B\-A17B71\.1±3\.971\.1\_\{\\pm 3\.9\}81\.5±3\.881\.5\_\{\\pm 3\.8\}5849±2885849\_\{\\pm 288\}0\.8948\.3±7\.248\.3\_\{\\pm 7\.2\}58\.1±7\.658\.1\_\{\\pm 7\.6\}5479±5025479\_\{\\pm 502\}0\.85Gemini\-2\.5\-flash55\.3±4\.155\.3\_\{\\pm 4\.1\}70\.4±4\.470\.4\_\{\\pm 4\.4\}3884±5133884\_\{\\pm 513\}0\.9732±6\.532\_\{\\pm 6\.5\}41\.9±7\.641\.9\_\{\\pm 7\.6\}3383±9973383\_\{\\pm 997\}0\.85Gemini\-2\.5\-pro60\.4±460\.4\_\{\\pm 4\}76\.9±4\.176\.9\_\{\\pm 4\.1\}2905±3112905\_\{\\pm 311\}2\.9236±6\.836\_\{\\pm 6\.8\}44\.2±7\.744\.2\_\{\\pm 7\.7\}3283±7483283\_\{\\pm 748\}3\.32Gemini\-3\-flash76\.2±3\.776\.2\_\{\\pm 3\.7\}83\.3±3\.683\.3\_\{\\pm 3\.6\}8130±6378130\_\{\\pm 637\}2\.4564\.5±6\.264\.5\_\{\\pm 6\.2\}83\.7±5\.783\.7\_\{\\pm 5\.7\}2942±6392942\_\{\\pm 639\}0\.94Gemini\-3\.1\-flash\-lite61\.8±3\.961\.8\_\{\\pm 3\.9\}78\.7±478\.7\_\{\\pm 4\}1537±1661537\_\{\\pm 166\}0\.2342\.4±7\.142\.4\_\{\\pm 7\.1\}51\.2±7\.751\.2\_\{\\pm 7\.7\}1335±1121335\_\{\\pm 112\}0\.23Gemini\-3\.1\-pro83\.3±3\.283\.3\_\{\\pm 3\.2\}90\.7±2\.890\.7\_\{\\pm 2\.8\}5550±6455550\_\{\\pm 645\}3\.3466\.9±6\.466\.9\_\{\\pm 6\.4\}76\.7±6\.576\.7\_\{\\pm 6\.5\}3959±8583959\_\{\\pm 858\}2\.49GPT\-5\-mini67\.6±467\.6\_\{\\pm 4\}77\.8±477\.8\_\{\\pm 4\}2032±2252032\_\{\\pm 225\}0\.4133\.1±6\.533\.1\_\{\\pm 6\.5\}44\.2±7\.744\.2\_\{\\pm 7\.7\}1437±2211437\_\{\\pm 221\}0\.3GPT\-570\.6±3\.870\.6\_\{\\pm 3\.8\}82\.4±3\.782\.4\_\{\\pm 3\.7\}3104±3293104\_\{\\pm 329\}3\.1237\.8±6\.737\.8\_\{\\pm 6\.7\}51\.2±7\.751\.2\_\{\\pm 7\.7\}2279±2822279\_\{\\pm 282\}2\.35GPT\-5\.270\.6±3\.770\.6\_\{\\pm 3\.7\}83\.3±3\.683\.3\_\{\\pm 3\.6\}1165±1691165\_\{\\pm 169\}1\.6650±750\_\{\\pm 7\}60\.5±7\.560\.5\_\{\\pm 7\.5\}1022±2211022\_\{\\pm 221\}1\.54GPT\-5\.4\-nano62±462\_\{\\pm 4\}75\.9±4\.175\.9\_\{\\pm 4\.1\}1734±2761734\_\{\\pm 276\}0\.2225±6\.225\_\{\\pm 6\.2\}32\.6±7\.232\.6\_\{\\pm 7\.2\}1369±3671369\_\{\\pm 367\}0\.18GPT\-5\.4\-mini63±463\_\{\\pm 4\}77\.8±477\.8\_\{\\pm 4\}2205±3892205\_\{\\pm 389\}145\.3±6\.745\.3\_\{\\pm 6\.7\}62\.8±7\.562\.8\_\{\\pm 7\.5\}2224±8472224\_\{\\pm 847\}1\.05GPT\-5\.478\.9±3\.278\.9\_\{\\pm 3\.2\}90\.7±2\.890\.7\_\{\\pm 2\.8\}1299±1811299\_\{\\pm 181\}1\.9858\.1±6\.758\.1\_\{\\pm 6\.7\}74\.4±6\.774\.4\_\{\\pm 6\.7\}1420±2861420\_\{\\pm 286\}2\.29GPT\-5\.584±3\.284\_\{\\pm 3\.2\}89\.8±2\.989\.8\_\{\\pm 2\.9\}945±110945\_\{\\pm 110\}2\.8958\.7±6\.558\.7\_\{\\pm 6\.5\}74\.4±6\.774\.4\_\{\\pm 6\.7\}1491±3361491\_\{\\pm 336\}4\.8
#### Overall Performance

Gemini\-3\.1\-Proachieves the strongest overall performance, with an accuracy of≈\\approx83\.3% on textual problems and≈\\approx66\.9% on multimodal problems[Table 1](https://arxiv.org/html/2607.08317#S4.T1)\. WhileGPT\-5\.5also attains very high accuracy on textual problems \(≈\\approx84%\), its performance does not transfer to visual questions \(≈\\approx58\.7%\)\. The strongest open\-weight models remain competitive mainly on text\-only problems, but lag behind in the multimodal setting\.GLM\-5\.2leads among open\-weights models with a text\-only accuracy of≈\\approx74%\.Qwen3\.5\-397Balso demonstrates strong performance \(≈\\approx70% on text,≈\\approx48% on multimodal\), surpassing much larger models such asKimi\-K2\.6andDeepSeek\-V4\-Pro\. This result might come as a consequence of longer reasoning, as suggested by the high average number of output tokens\. Overall we observe substantial improvements across successive iterations of the same model families in the different iterations of the same model families \(e\.g\., fromGemini\-2\.5toGemini\-3\), indicating that the latest generation of models is significantly more robust than models from late 2025\.

For image\-generation models[Subsec\. 4\.2](https://arxiv.org/html/2607.08317#S4.SS2.SSS0.Px4)\)Gemini\-3\-Pro\-Imageattains highest accuracy \(54\.8%\), but at a substantially higher inference cost than its OpenAI counterpart\.GPT\-Image\-2achieves slightly lower accuracy \(51\.2%\) while being approximately 4×\\timescheaper\. A detailed view of the failures distribution across the different problems is provided in[Appendix E](https://arxiv.org/html/2607.08317#A5)\.

![Refer to caption](https://arxiv.org/html/2607.08317v1/x7.png)Figure 4:Accuracy onblind\-spots\-benchvs\. average cost for 100 samples\. Colors distinguish model families; models of the same version but different sizes are connected\.
#### Cost\-Performance Trade\-off

We further analyze the relation between each model’s accuracy onblind\-spots\-benchand its average inference cost\. As shown in[Fig\. 4](https://arxiv.org/html/2607.08317#S4.F4), the rate of improvement in accuracy slowly decreases with higher cost\. In text\-only tasks in particular, we observe that a 10% accuracy gain, from 60% to 70%, requires at least an order\-of\-magnitude increase in inference cost \(approximately 10×\\times\)\. Few open models, in paricularDeepSeek\-V4,Qwen3\.5, andGLM\-5\.2demonstrate particularly cost\-efficient performance, outperforming the frontier alternatives at similar cost levels \(such asGemini\-3\.1\-Flash\-LiteandGPT\-5\.4\-mini\)\. Among frontier models, theGemini\-3family displays an overall better accuracy\-cost trade\-off, especially in the multimodal domain, strengthening the evidence of its greater performance when compared toGPT\-5\.4and5\.5\.

#### Alignment with General Intelligence Assessment

We further compareblind\-spots\-benchaccuracy with external general intelligence scores, focusing on the Artificial Analysis Intelligence Index \(AAII\)\[[4](https://arxiv.org/html/2607.08317#bib.bib69)\]\. This composite benchmark aggregates ten established text\-only evaluations that span diverse capabilities, including Humanity Last Exam\[[39](https://arxiv.org/html/2607.08317#bib.bib70)\], IFBench\[[41](https://arxiv.org/html/2607.08317#bib.bib72)\], and TerminalBench\[[33](https://arxiv.org/html/2607.08317#bib.bib73)\]\. As shown in[Fig\. 1](https://arxiv.org/html/2607.08317#S0.F1)a, we observe a positive relationship between Intelligence Index scores and performanceblind\-spots\-bench, with models that perform well on broad reasoning, coding, and mathematical evaluations also scoring highly onblind\-spots\-bench333For some models, an AAII score is not available at exactlymediumthinking effort\. In these cases, we use the best available approximation: the average oflowandhighwhen both are reported, or the single reasoning\-enabled score when nothinking\_effortoption is provided\.\. This supports the view thatblind\-spots\-benchmeasures general reasoning ability rather than an capturing an unrelated set of adversarial prompts\.

However, this relation differs between open\- and closed\-weight models\. Open models tend to perform worse onblind\-spots\-benchthan closed models with comparable AAII scores\. One possible explanation is that broad public benchmarks may overestimate robustness on underrepresented tasks, especially when models have been optimized for widely used evaluations that emphasize specific skills rather than global robustness\.

Table 2:Image\-generation model results\. Accuracy \(mean±stderr\\texttt\{mean\}\_\{\\pm\\texttt\{stderr\}\}\) is shown as a percentage, generation cost as USD\-per\-100\-samples\.Table 3:Delta in performance and tokens, when enabling code execution for text\-only problems\.

#### Tool Use Impact on Reasoning

Some of the most challenging tasks for LLMs often involve meeting hard constraints, counting and performing string operations\. Because of this, the use of external tools, particularly code execution environments, can substantially improve performance\. Therefore, we repeat the evaluation for a subset of models, equipping them with a Python execution environment and allowing up to five tool calls per question\. Results in[Table 3](https://arxiv.org/html/2607.08317#S4.T3)show mixed effects, with some models likeGemini\-3\.1\-Flashachieving notably higher accuracy with reduced token consumption, whereas others, such asGPT\-5\.4andQwen3\.5\-397B, lose performance\. A qualitative evaluation of these failures indicates that, despite generating correct code snippets, models often mishandle inputs, for example, when copy\-pasting an input string\.

![Refer to caption](https://arxiv.org/html/2607.08317v1/x8.png)Figure 5:Performance comparison of leading image generation and VLM models on the largest\-sample subtasks within each model type\.

### 4\.3Results Analysis by Task Taxonomy

We analyze model performance across the defined taxonomy to better understand their strengths and limitations, with the full results reported in[Table 8](https://arxiv.org/html/2607.08317#A6.T8)and[Table 9](https://arxiv.org/html/2607.08317#A6.T9)\. Scores are computed using mean@4 for all models except image\-generation models\. For the 15 questions annotated with multiple subtask labels, we further validate their fine\-grained task scores\. Since different question formats contain different numbers of subtask categories, we mainly conduct the analysis within each input\-output format\. Several interesting findings emerge\.

First,*fine\-grained visual perception remains a major bottleneck, with models consistently struggling on attribute and pattern recognition and perceptual counting\.*Even the strongest models obtain only 41\.67% and 57\.14% on these two subtasks, which contain 6 and 21 questions, respectively\. For perceptual counting, many models achieve only around 30–40% accuracy\.

Second,*different model families exhibit complementary strengths rather than a single dominant pattern across all subtask categories\.*Here, we compare among the best\-performing model from each family\. For image\-generation models, Gemini and GPT achieve comparable overall performance, but exhibit complementary strengths: Gemini performs better on abstract reasoning winning 3 more questions, whereas GPT is stronger in language and knowledge[Fig\. 5](https://arxiv.org/html/2607.08317#S4.F5)\. A similar pattern appears among text\-only models\. GPT\-OSS remains competitive on math\-related tasks, but lags behind others on all three language\-and\-knowledge subtasks\. DeepSeek\-V4 performs particularly well on character\-level manipulation, reaching 75% accuracy\. Among VLMs, the GPT and Gemini series form the leading group\. GPT trails Gemini on object\-centric tasks, but slightly outperforms it on the other two major categories\. Although Qwen and Kimi sometimes perform strongly on individual subtasks, such as Kimi’s 83% accuracy on arithmetic reasoning, their performance is less consistent, with noticeable gaps from the leading models in other cases, including a 10\-point gap on character\-level manipulation\.

Third,*model size scaling does not consistently improve performance across all subtask types\.*We compare different model sizes within the same model family and observe several cases where larger models underperform smaller variants\. For example, in the Gemma\-4 series, the 26B model improves on most subtasks but performs about 8% worse than the E2B model on irrelevant\-context robustness\. Similarly, in the Qwen3\.5 series, the 122B model shows a 10–14% drop over the 35B model on several abstract\-reasoning subtasks, including logical reasoning, geometric and graph reasoning, and arithmetic reasoning\. Although the 397B model recovers part of this drop, it still does not surpass the 35B model on these subtasks\.

## 5Conclusion

In this work, we presentblind\-spots\-bench, a dataset of 235 questions that remain challenging for modern models but straightforward for humans\. We introduce a fine\-grained taxonomy, curated reference solutions, and an automatic grading pipeline to evaluate diverse VLMs, LLMs, and image\-generation models\. Our analyses show that closed\-source frontier models outperform open\-weight models, while open\-weight models are often more cost\-effective\. We further find that scaling model size or enabling tool use does not consistently improve performance\. Finally, taxonomy\-level analysis reveals several distinctive patterns in model failure modes\.

Limitations\.Whileblind\-spots\-benchprovides useful insights into model failure modes, several limitations remain\. First, the dataset is relatively modest in size and imbalanced across sub\-tasks\. Although sufficient for our analyses, increasing the number of samples per sub\-task would strengthen the statistical reliability of the results\. Second, because the questions were primarily created by students to challenge two frontier models, they may be biased toward those models’ specific weaknesses and may miss other types of reasoning failures\. Finally, althoughblind\-spots\-benchquestions are intended to be easy for humans, adding a human baseline would more directly quantify the gap between model and human performance\.

## Acknowledgments and Disclosure of Funding

This project was first proposed in the EPFL course Reasoning in Artificial Intelligence \(MATH\-700\), taught by Emmanuel Abbé\. We thank all students in the course for each proposing five questions to challenge frontier models at the time\. We also thank Kevin Qiu, Amene Gafsi, Rafaila Galanopoulou, Szymon Sobczak, Julien David Laurendeau, and Zhipeng Xue for their early efforts to dataset cleaning\. Finally, we thank the RCP team at EPFL for GPU and model support, and the Laboratory of Mathematical Data Science, which ran the course, for covering the associated costs\.

## References

- \[1\]\(2019\)Tallyqa: answering complex counting questions\.33\(01\),pp\. 8076–8084\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/4815),[Document](https://dx.doi.org/10.1609/aaai.v33i01.33018076)Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p2.1)\.
- \[2\]S\. Agarwalet al\.\(2025\)gpt\-oss\-120b and gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.External Links:[Link](https://arxiv.org/abs/2508.10925)Cited by:[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[3\]Anonymous\(2025\)MATH\-perturb: benchmarking llms’ math reasoning abilities against hard perturbations\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2502.06453)Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px3.p1.1)\.
- \[4\]Artificial Analysis\(2026\)Artificial Analysis Intelligence Benchmarking Methodology: Artificial Analysis Intelligence Index v4\.0\.4\.Note:[https://artificialanalysis\.ai/methodology/intelligence\-benchmarking](https://artificialanalysis.ai/methodology/intelligence-benchmarking)Cited by:[§4\.2](https://arxiv.org/html/2607.08317#S4.SS2.SSS0.Px4.p1.1)\.
- \[5\]Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou,et al\.\(2024\)Longbench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 3119–3137\.Cited by:[§3\.1](https://arxiv.org/html/2607.08317#S3.SS1.SSS0.Px2.p1.1)\.
- \[6\]V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhan\(2025\)τ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.arXiv preprint arXiv:2506\.07982\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p1.1)\.
- \[7\]L\. Binyamin, Y\. Tewel, H\. Segev, E\. Hirsch, R\. Rassin, and G\. Chechik\(2025\)Make it count: text\-to\-image generation with an accurate number of objects\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 13242–13251\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p2.1),[§1](https://arxiv.org/html/2607.08317#S1.p2.1),[§3\.2](https://arxiv.org/html/2607.08317#S3.SS2.p2.1)\.
- \[8\]L\. Chen, W\. Xie, Y\. Liang, H\. He, H\. Zhao, Z\. Yang, Z\. Huang, H\. Wu, H\. Lu, Y\. Bao,et al\.\(2026\)BabyVision: visual reasoning beyond language\.arXiv preprint arXiv:2601\.06521\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p2.1)\.
- \[9\]G\. Comaniciet al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.External Links:[Link](https://arxiv.org/abs/2507.06261)Cited by:[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[10\]DeepSeek\-AI\(2026\)DeepSeek\-V4: technical report\.Note:[https://huggingface\.co/deepseek\-ai/DeepSeek\-V4\-Pro/blob/main/DeepSeek\_V4\.pdf](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf)Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[11\]J\. Dekoninck, N\. Jovanović, T\. Gehrunger, K\. Rögnvalddson, I\. Petrov, C\. Sun, and M\. Vechev\(2026\)Beyond benchmarks: matharena as an evaluation platform for mathematics with llms\.External Links:2605\.00674,[Link](https://arxiv.org/abs/2605.00674)Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p1.1)\.
- \[12\]C\. Fu, P\. Chen, Y\. Shen, Y\. Qin, M\. Zhang, X\. Lin, J\. Yang, X\. Zheng, K\. Li, X\. Sun, Y\. Wu, R\. Ji, C\. Shan, and R\. He\(2023\)MME: a comprehensive evaluation benchmark for multimodal large language models\.arXiv preprint arXiv: 2306\.13394\.Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px2.p1.1)\.
- \[13\]T\. Fu, R\. Ferrando, J\. Conde, C\. Arriaga, and P\. Reviriego\(2024\)Why do large language models \(llms\) struggle to count letters?\.arXiv preprint arXiv:2412\.18626\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p4.1),[§1](https://arxiv.org/html/2607.08317#S1.p2.1),[§3\.2](https://arxiv.org/html/2607.08317#S3.SS2.p2.1)\.
- \[14\]M\. Gardner, Y\. Artzi, V\. Basmov, J\. Berant, B\. Bogin, S\. Chen, P\. Dasigi, D\. Dua, Y\. Elazar, A\. Gottumukkala,et al\.\(2020\)Evaluating models’ local decision boundaries via contrast sets\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 1307–1323\.Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px2.p1.1)\.
- \[15\]Google DeepMind\(2025\)Gemini 3 Pro Model Card\.Note:[https://storage\.googleapis\.com/deepmind\-media/Model\-Cards/Gemini\-3\-Pro\-Model\-Card\.pdf](https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf)Cited by:[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[16\]Google DeepMind\(2026\)Gemini 3\.1 Pro Model Card\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-1\-pro/](https://deepmind.google/models/model-cards/gemini-3-1-pro/)Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[17\]Google\(2026\)Gemma 4 model card\.Note:[https://ai\.google\.dev/gemma/docs/core/model\_card\_4](https://ai.google.dev/gemma/docs/core/model_card_4)Cited by:[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[18\]M\. Guo, J\. Xu, Y\. Zhang, J\. Song, H\. Peng, Y\. Deng, X\. Dong, K\. Nakayama, Z\. Geng, C\. Wang,et al\.\(2025\)R\-bench: graduate\-level multi\-disciplinary benchmarks for llm & mllm complex reasoning evaluation\.arXiv preprint arXiv:2505\.02018\.Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px1.p1.1)\.
- \[19\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan\(2023\)Swe\-bench: can language models resolve real\-world github issues?\.InThe twelfth international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p1.1)\.
- \[20\]D\. Keysers, N\. Schärli, N\. Scales, H\. Buisman, D\. Furrer, S\. Kashubin, N\. Momchev, D\. Sinopalnikov, L\. Stafiniak, T\. Tihon,et al\.\(2019\)Measuring compositional generalization: a comprehensive method on realistic data\.arXiv preprint arXiv:1912\.09713\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p2.1)\.
- \[21\]D\. Kiela, M\. Bartolo, Y\. Nie, D\. Kaushik, A\. Geiger, Z\. Wu, B\. Vidgen, G\. Prasad, A\. Singh, P\. Ringshia,et al\.\(2021\)Dynabench: rethinking benchmarking in nlp\.InProceedings of the 2021 conference of the North American chapter of the Association for Computational Linguistics: human language technologies,pp\. 4110–4124\.Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px2.p1.1)\.
- \[22\]Kimi Team\(2026\)Kimi K2\.5: visual agentic intelligence\.arXiv preprint arXiv:2602\.02276\.External Links:[Link](https://arxiv.org/abs/2602.02276)Cited by:[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[23\]F\. Kong, W\. Zu, X\. Chen, Y\. Yang, S\. Zhu, and X\. Feng\(2025\)Siv\-bench: a video benchmark for social interaction understanding and reasoning\.arXiv preprint arXiv:2506\.05425\.Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px2.p1.1)\.
- \[24\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica\(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p2.1)\.
- \[25\]Q\. Li, L\. Cui, X\. Zhao, L\. Kong, and W\. Bi\(2024\-08\)GSM\-plus: a comprehensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),External Links:[Link](https://aclanthology.org/2024.acl-long.163)Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px3.p1.1)\.
- \[26\]H\. Liu, Y\. Ding, Z\. Fu, C\. Zhang, X\. Liu, and Y\. Zhang\(2025\)Evaluating the logical reasoning abilities of large reasoning models\.arXiv preprint arXiv:2505\.11854\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p2.1)\.
- \[27\]W\. Liu, Q\. Xue, H\. Wang, X\. Yin, B\. Yang, and W\. Gao\(2025\)Spatial reasoning in multimodal large language models: a survey of tasks, benchmarks and methods\.arXiv preprint arXiv:2511\.15722\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p2.1)\.
- \[28\]Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu,et al\.\(2024\)Mmbench: is your multi\-modal model an all\-around player?\.InEuropean conference on computer vision,pp\. 216–233\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p1.1)\.
- \[29\]P\. Lu, H\. Bansal, T\. Xia, J\. Liu, C\. Li, H\. Hajishirzi, H\. Cheng, K\. Chang, M\. Galley, and J\. Gao\(2023\)Mathvista: evaluating mathematical reasoning of foundation models in visual contexts\.arXiv preprint arXiv:2310\.02255\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p1.1)\.
- \[30\]A\. Maksai\(2026\)InkSlop: vibe\-coded benchmark for spatial reasoning with digital ink\.Note:Benchmark for evaluating VLMs on digital ink tasksExternal Links:[Link](https://inkslop.github.io/)Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p2.1)\.
- \[31\]A\. Masry, X\. L\. Do, J\. Q\. Tan, S\. Joty, and E\. Hoque\(2022\)Chartqa: a benchmark for question answering about charts with visual and logical reasoning\.InFindings of the association for computational linguistics: ACL 2022,pp\. 2263–2279\.Cited by:[§3\.1](https://arxiv.org/html/2607.08317#S3.SS1.SSS0.Px2.p1.1)\.
- \[32\]R\. T\. McCoy, E\. Pavlick, and T\. Linzen\(2019\)Right for the wrong reasons: diagnosing syntactic heuristics in natural language inference\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 3428–3448\.Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px2.p1.1)\.
- \[33\]M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan, J\. Shen, G\. Ye, H\. Lin, J\. Poulos, M\. Wang, M\. Nezhurina, J\. Jitsev, D\. Lu, O\. M\. Mastromichalakis, Z\. Xu,et al\.\(2026\)Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.arXiv preprint arXiv:2601\.11868\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.11868),[Link](https://arxiv.org/abs/2601.11868)Cited by:[§4\.2](https://arxiv.org/html/2607.08317#S4.SS2.SSS0.Px4.p1.1)\.
- \[34\]I\. Mirzadeh, K\. Alizadeh, H\. Shahrokhi, O\. Tuzel, S\. Bengio, and M\. Farajtabar\(2025\)GSM\-symbolic: understanding the limitations of mathematical reasoning in large language models\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2410.05229)Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px3.p1.1)\.
- \[35\]B\. Mitrevski, A\. Rak, J\. Schnitzler, C\. Li, A\. Maksai, J\. Berent, and C\. C\. Musat\(2025\)InkSight: offline\-to\-online handwriting conversion by teaching vision\-language models to read and write\.Transactions on Machine Learning Research\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p2.1)\.
- \[36\]OpenAI\(2025\)OpenAI GPT\-5 System Card\.arXiv preprint arXiv:2601\.03267\.External Links:[Link](https://arxiv.org/abs/2601.03267)Cited by:[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[37\]OpenAI\(2026\)Introducing GPT\-5\.4\.Note:[https://openai\.com/index/introducing\-gpt\-5\-4/](https://openai.com/index/introducing-gpt-5-4/)Cited by:[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[38\]OpenAI\(2026\)Introducing GPT\-5\.5\.Note:[https://openai\.com/index/introducing\-gpt\-5\-5/](https://openai.com/index/introducing-gpt-5-5/)Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[39\]L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi, M\. Choi, A\. Agrawal, A\. Chopra, A\. Khoja, R\. Kim, R\. Ren, J\. Hausenloy, O\. Zhang, M\. Mazeika,et al\.\(2025\)Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2501.14249),[Link](https://arxiv.org/abs/2501.14249)Cited by:[§4\.2](https://arxiv.org/html/2607.08317#S4.SS2.SSS0.Px4.p1.1)\.
- \[40\]V\. Pyatkin, S\. Malik, V\. Graf, H\. Ivison, S\. Huang, P\. Dasigi, N\. Lambert, and H\. Hajishirzi\(2025\)Generalizing verifiable instruction following\.arXiv preprint arXiv:2507\.02833\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p1.1)\.
- \[41\]V\. Pyatkin, S\. Malik, V\. Graf, H\. Ivison, S\. Huang, P\. Dasigi, N\. Lambert, and H\. Hajishirzi\(2025\)Generalizing verifiable instruction following\.Advances in Neural Information Processing Systems38\.External Links:[Link](https://arxiv.org/abs/2507.02833)Cited by:[§4\.2](https://arxiv.org/html/2607.08317#S4.SS2.SSS0.Px4.p1.1)\.
- \[42\]M\. F\. Qharabagh, M\. Ghofrani, and K\. Fountoulakis\(2024\)Lvlm\-count: enhancing the counting ability of large vision\-language models\.arXiv preprint arXiv:2412\.00686\.Cited by:[§3\.2](https://arxiv.org/html/2607.08317#S3.SS2.p2.1)\.
- \[43\]K\. Qian, S\. Wan, C\. Tang, Y\. Wang, X\. Zhang, M\. Chen, and Z\. Yu\(2024\)VarBench: robust language model benchmarking through dynamic variable perturbation\.InFindings of the Association for Computational Linguistics: EMNLP 2024,External Links:[Link](https://aclanthology.org/2024.findings-emnlp.950/)Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px3.p1.1)\.
- \[44\]Qwen Team\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[45\]Qwen Team\(2026\)Qwen3\.5\-omni technical report\.arXiv preprint arXiv:2604\.15804\.External Links:[Link](https://arxiv.org/abs/2604.15804)Cited by:[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[46\]F\. Redhardt, Y\. Akram, and S\. Schug\(2025\)Scaling can lead to compositional generalization\.arXiv preprint arXiv:2507\.07207\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p2.1)\.
- \[47\]D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman\(2023\)Gpqa: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p1.1),[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px1.p1.1)\.
- \[48\]M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. Singh\(2020\)Beyond accuracy: behavioral testing of nlp models with checklist\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 4902–4912\.Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px2.p1.1)\.
- \[49\]A\. Saparov and H\. He\(2022\)Language models are greedy reasoners: a systematic formal analysis of chain\-of\-thought\.arXiv preprint arXiv:2210\.01240\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p3.1),[§3\.2](https://arxiv.org/html/2607.08317#S3.SS2.p2.1)\.
- \[50\]R\. Saxena, A\. P\. Gema, and P\. Minervini\(2025\)Lost in time: clock and calendar understanding challenges in multimodal llms\.arXiv preprint arXiv:2502\.05092\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p2.1)\.
- \[51\]J\. Seely, Y\. Imajuku, T\. Zhao, E\. Cetin, and L\. Jones\(2025\)Sudoku\-bench: evaluating creative reasoning with sudoku variants\.arXiv preprint arXiv:2505\.16135\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p2.1)\.
- \[52\]P\. Song, P\. Han, and N\. Goodman\(2025\)A survey on large language model reasoning failures\.In2nd AI for Math Workshop@ ICML 2025,Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px2.p1.1)\.
- \[53\]A\. Srivastava, A\. Rastogi, A\. Rao, A\. A\. M\. Shoeb, A\. Abid, A\. Fisch, A\. R\. Brown, A\. Santoro, A\. Gupta, A\. Garriga\-Alonso,et al\.\(2023\)Beyond the imitation game: quantifying and extrapolating the capabilities of language models\.Transactions on machine learning research\.Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px1.p1.1)\.
- \[54\]I\. Stogiannidis, S\. McDonagh, and S\. A\. Tsaftaris\(2025\)Mind the gap: benchmarking spatial reasoning in vision\-language models\.arXiv preprint arXiv:2503\.19707\.Cited by:[§3\.2](https://arxiv.org/html/2607.08317#S3.SS2.p2.1)\.
- \[55\]S\. Tong, Z\. Liu, Y\. Zhai, Y\. Ma, Y\. LeCun, and S\. Xie\(2024\)Eyes wide shut? exploring the visual shortcomings of multimodal llms\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 9568–9578\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p2.1)\.
- \[56\]J\. Tso, P\. Schmittou, Q\. Huynh, and J\. Hutchins\(2026\)ConstraintBench: benchmarking llm constraint reasoning on direct optimization\.arXiv preprint arXiv:2602\.22465\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p2.1)\.
- \[57\]UK AI Security Institute\(2024\-05\)Inspect AI: Framework for Large Language Model Evaluations\.Note:MIT LicenseExternal Links:[Link](https://github.com/UKGovernmentBEIS/inspect_ai)Cited by:[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px1.p2.1)\.
- \[58\]K\. Valmeekam, M\. Marquez, A\. Olmo, S\. Sreedharan, and S\. Kambhampati\(2023\)Planbench: an extensible benchmark for evaluating large language models on planning and reasoning about change\.Advances in Neural Information Processing Systems36,pp\. 38975–38987\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p3.1)\.
- \[59\]V\. Veitch, A\. D’Amour, S\. Yadlowsky, and J\. Eisenstein\(2021\)Counterfactual invariance to spurious correlations: why and how to pass stress tests\.arXiv preprint arXiv:2106\.00545\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p4.1)\.
- \[60\]A\. Vo, K\. Nguyen, M\. R\. Taesiri, V\. T\. Dang, A\. T\. Nguyen, and D\. Kim\(2025\)Vision language models are biased\.arXiv preprint arXiv:2505\.23941\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p2.1)\.
- \[61\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. Bowman\(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP,pp\. 353–355\.Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px2.p1.1)\.
- \[62\]A\. T\. Wasi, W\. Faisal, A\. Rahman, M\. A\. Anik, M\. Shahriar, M\. M\. Topu, S\. T\. Meem, R\. N\. Priti, S\. A\. Mitu, M\. I\. Hoque,et al\.\(2026\)SpatiaLab: can vision\-language models perform spatial reasoning in the wild?\.arXiv preprint arXiv:2602\.03916\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p2.1),[§1](https://arxiv.org/html/2607.08317#S1.p2.1),[§3\.1](https://arxiv.org/html/2607.08317#S3.SS1.SSS0.Px2.p1.1)\.
- \[63\]T\. Wiedemer, P\. Mayilvahanan, M\. Bethge, and W\. Brendel\(2023\)Compositional generalization from first principles\.Advances in Neural Information Processing Systems36,pp\. 6941–6960\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p2.1),[§1](https://arxiv.org/html/2607.08317#S1.p2.1),[§3\.2](https://arxiv.org/html/2607.08317#S3.SS2.p2.1)\.
- \[64\]Z\. Xiong, Y\. Cai, B\. Hooi, N\. Peng, Z\. Li, and Y\. Wang\(2025\)Enhancing llm character\-level manipulation via divide and conquer\.arXiv preprint arXiv:2502\.08180\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p2.1)\.
- \[65\]W\. Yu, Z\. Yang, L\. Li, J\. Wang, K\. Lin, Z\. Liu, X\. Wang, and L\. Wang\(2023\)Mm\-vet: evaluating large multimodal models for integrated capabilities\.arXiv preprint arXiv:2308\.02490\.Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px2.p1.1)\.
- \[66\]J\. Yuan, T\. Peng, Y\. Jiang, Y\. Lu, R\. Zhang, K\. Feng, C\. Fu, T\. Chen, L\. Bai, B\. Zhang,et al\.\(2025\)MME\-reasoning: a comprehensive benchmark for logical reasoning in mllms\.arXiv preprint arXiv:2505\.21327\.Cited by:[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px2.p1.1)\.
- \[67\]X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun,et al\.\(2024\)Mmmu: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 9556–9567\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p1.1),[§2](https://arxiv.org/html/2607.08317#S2.SS0.SSS0.Px1.p1.1)\.
- \[68\]A\. Zenget al\.\(2026\)GLM\-5: from vibe coding to agentic engineering\.arXiv preprint arXiv:2602\.15763\.External Links:[Link](https://arxiv.org/abs/2602.15763)Cited by:[§4\.1](https://arxiv.org/html/2607.08317#S4.SS1.SSS0.Px3.p1.1)\.
- \[69\]X\. Zhang, J\. Cao, and C\. You\(2024\)Counting ability of large language models and impact of tokenization\.arXiv preprint arXiv:2410\.19730\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p4.1),[§3\.2](https://arxiv.org/html/2607.08317#S3.SS2.p2.1)\.
- \[70\]Y\. Zhang, J\. Hare, and A\. Prügel\-Bennett\(2018\)Learning to count objects in natural images for visual question answering\.arXiv preprint arXiv:1802\.05766\.Cited by:[§A\.1](https://arxiv.org/html/2607.08317#A1.SS1.p2.1)\.
- \[71\]Y\. Zhang, Y\. Li, and S\. Gao\(2026\)Do vision language models rotate in mind? evaluating spatial transformation reasoning\.External Links:[Link](https://openreview.net/forum?id=up2LD7vVdW)Cited by:[§3\.2](https://arxiv.org/html/2607.08317#S3.SS2.p2.1)\.
- \[72\]J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou\(2023\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§1](https://arxiv.org/html/2607.08317#S1.p2.1)\.

## Appendix AExamples of problems

To illustrate a practical example for the annotation and task taxonomy, we provide the following prompt from the dataset:

Example 1Prompt: Can you read the words marked in this “find a word” game\.![[Uncaptioned image]](https://arxiv.org/html/2607.08317v1/tables/dataset/img_q18.png)Solution: The words marked in colors are: WORD, SEARCH, JOURNEY, LOVE, RELAX, ESCAPE, FUN, PLAY, PUZZLE, ENJOY\. Any answer that does not contain one of these words, or contains an extra word not in this list, is considered wrong\.Question type:multi\-to\-text Task Type:object\-centricSub\-Task Type: Attribute and pattern recognition\.

Example 2Prompt: Given a directed weighted graph\. What is the shortest path from A to B?![[Uncaptioned image]](https://arxiv.org/html/2607.08317v1/tables/dataset/img_q22.png)Solution: The correct answer is a shortest directed path from A to B together with its total weight\. An output is correct if and only if \(1\) every step follows an existing directed edge in the graph \(respecting arrow direction\) and \(2\) the total weight equals the minimum over all valid directed A→B paths\.The correct shortest path is A → F → I → D → E → C → G → B with total weight 17\.This is correct because each edge in the path exists with the shown direction and weights \(A→F=2, F→I=3, I→D=5, D→E=1, E→C=2, C→G=2, G→B=2\), summing to 17, and no other valid directed path from A to B has a smaller total weight\.A common incorrect behavior is ignoring edge directions \(treating C→A as A→C or B→C as C→B\) or inventing non\-existent edges \(e\.g\., claiming C→D or D→B\), producing paths that are not actually reachable in the directed graph\.Question type:multi\-to\-text Task Type:Abstract reasoningSub\-Task Type: Geometric and graph reasoning\.

We note that some prompts require multi\-task categories, in this case we discuss further the failure mode depending on the model’s answer\. We provide an example below:

Multi category examplePrompt: Can you generate an image according to the problem description and solve the problem? Let ABCD be a square, and l be a line segment from B to a point on side AD\. A is 5 cm from l and C is 7 cm from l\. What is the area of ABCD? Question type:image\-gen Task Type:abstract reasoning,object\-centricSub\-Task Type: Geometric and Graph reasoning, Attribute binding\.Failure Mode: If the model generates an image with incorrect properties, such as placing point B off side AD or producing square ABDC instead of ABCD, then it fails in attribute binding\. If the model fails to compute the area of ABCD, then it fails in geometric and graph reasoning\.

Table 4:We propose a taxonomy of capability dimensions, including object\-centric, abstract reasoning, and language and knowledge, with detailed sub\-task categories\. For each question, we primarily identify its dominant task type for analysis; but still 15 questions have more than one requirement\.Task TypeSub\-Task typeIDDescriptionExampleObject\-centricAttribute and pattern recognition1\-1Perceive and ground objects/attributes/relations in visual inputs or required outputs \(e\.g\., detection, attribute recognition\)\.•Given a picture, how many red and blue balls are in the picture?Spatial reasoning1\-2The ability to perceive, represent, and manipulate spatial relations among objects\. Also includes spatial transformations such as mirroring and flipping to an image or generated object\.•Generate an image of a fully furnished living room\. No furniture can touch the carpet\.•Mirror the input image\.Generative counting1\-3Enforce discrete quantity constraints during generation \(e\.g\., generate exactlykkobjects/parts\)\.•Make a figure that repeats a circle, a triangle, and a square 4 times\.Perceptual counting1\-4Perceptual counting is for counting visible objects in an image, like trees, chess pieces, dice, clouds\.•Given a picture, how many chess pieces are in the picture?Attribute binding1\-5Correctly associate object attributes such as color, shape, size, or identity with the corresponding objects, avoiding swaps or mismatches between objects and their properties\.•Generate an image with the writing ”Google” where the two ”g”s are red, ”o”s are blue and green, ”l” is red and ”e” is yellow\.Abstract reasoningLogical reasoning2\-1Task that requires multiple sequential steps of inference where the solution is derived through a directed chain of implications or valid state\-changes\.•If every square were a circle, and every circle were a triangle, what shape would a square be?•Your task is to deduce the two steps of a security protocol and their correct order of application from the log file entries below\. Then, use the protocol to find the correct output for the final two inputs\. Log File Entries:GOLD→\\rightarrowKSPH,GLASS→\\rightarrowXXFQL,STEEL→\\rightarrowQJJYX,PASTA→\\rightarrowFYXFU,IMAGE→\\rightarrowNRFLJ\.Find the outputs for:QUARTZ→\\rightarrow?,SILVER→\\rightarrow?\.Arithmetic reasoning2\-2Task that requires numerical operations, quantitative inference, working with arithmetic expression and reasoning under uncertainty using probabilistic rules\.•Find me three integers a, b, and c such thatexp⁡\(9​log⁡\(a\)\)−exp⁡\(18​log⁡\(c\)/2\)\+\(b3\)3=0\.\\exp\(9\\log\(a\)\)\-\\exp\(18\\log\(c\)/2\)\+\(b^\{3\}\)^\{3\}=0\.Can you solve this with a, b and c being all non\-zero?•Give a smallest non\-negative number which can be written as sum of cubes of two different pairs where each pair has different integers•I draw a white marble from a bag with two marbles\. What is the probability the other marble is also white?Geometric and graph reasoning2\-3Tasks that require reasoning about geometric properties, spatial grid navigation, or graph structures, such as shape properties, distances, connectivity, and board\-state positioning\.•Please draw a diagonal in a triangle\.•Given a directed weighted graph, what is the shortest path from A to B?Constraint reasoning2\-4Tasks that require satisfying multiple hard constraints within a complex search space; often equivalent to constraint satisfaction, feasibility checking, finding optimal tactical sequences where most possible paths are invalidated by the rules\.•Solve the attached sudoku please\.Language and knowledgeIrrelevant\-context robustness3\-1Tasks where irrelevant or misleading context interferes with correct reasoning\.•Imagine you are in a dark room\. In a drawer there are 10 red socks, 10 white socks, and 10 green socks\. The light comes on\. How many socks do you need to take out to be sure you have at least one pair of the same color?Character\-level manipulation3\-2Represent, process, and manipulate text at the atomic granularity of character as opposed to higher\-level tokens\.•Repeat the number “12341234” exactly 31 times without spaces\.•Write a sentence that ends with the letter “m”\.World knowledge3\-3Requires factual knowledge \(often stable world facts\) and avoiding hallucinated details\. Or Requires everyday commonsense inferences beyond explicit text \(e\.g\., typical object uses, social/physical plausibility\)\.•Draw a calendar for February with 29 days\.•What has no holes and holds water?

### A\.1Task type analsysis

We provide detailed description and examples of task taxonomy in[Table 4](https://arxiv.org/html/2607.08317#A1.T4)\. Beyond the distribution of question format and task taxonomy, we examine the resulting distribution and observe several recurring patterns in the designed challenging tasks\. Since the prompts were intentionally created to probe the weaknesses of modern AI agents, these frequently appearing categories may offer useful clues about the capabilities that remain difficult for current models\. Below, we provide a brief dataset\-level analysis and link the possible reasons to several related studies for easier future study\.

Object\-centric tasks:We observe a failure for questions related to attribute and pattern recognition because models tend to extract and report frequently occurring patterns from their training dataset distribution rather than perceiving the actual content as reported in\[[60](https://arxiv.org/html/2607.08317#bib.bib33),[35](https://arxiv.org/html/2607.08317#bib.bib35),[30](https://arxiv.org/html/2607.08317#bib.bib34)\]\. In attribute binding or generative counting, they could fail in producing novel recombinations of familiar concepts that rarely occur in natural data\. This challenge has been emphasized in prior work\[[63](https://arxiv.org/html/2607.08317#bib.bib17),[20](https://arxiv.org/html/2607.08317#bib.bib15),[46](https://arxiv.org/html/2607.08317#bib.bib16),[7](https://arxiv.org/html/2607.08317#bib.bib20)\]\. Another example in perceptual counting\[[70](https://arxiv.org/html/2607.08317#bib.bib21),[1](https://arxiv.org/html/2607.08317#bib.bib22)\], models often misidentify, merge or overlook objects in cluttered scenes, especially when instances are small, visually similar, or partially occluded\. Spatial reasoning has also been widely recognized as a challenging capability\[[62](https://arxiv.org/html/2607.08317#bib.bib74),[27](https://arxiv.org/html/2607.08317#bib.bib96)\]for current models, as it requires understanding and manipulating spatial relations rather than merely recognizing objects\. While humans can usually solve these problems through intuitive geometric and physical reasoning, models may fail when the task requires precise relational understanding or multi\-step spatial manipulation\. In our taxonomy, we therefore include spatial reasoning as a key subcategory of object\-centric tasks, covering questions that test whether models can correctly interpret, transform, or reason about spatial configurations\.

Abstract Reasoning:In abstract reasoning, model often fails in problems that require several logical steps and reveals the reasoning gap where the accuracy of the model decreases with the increase of number of inference step\. This is justified in\[[49](https://arxiv.org/html/2607.08317#bib.bib49)\], where they show that due to the greedy nature of the model, it makes choices locally at each step without a global verification\. This leads to accumulation of the error and fails in giving a final correct solution\. Moreover, on the problems with constraint reasoning, where strict rules should be followed, or graph ones, such as the maze problems, where a certain path needs to be found, the model suffers from backtracking as mentioned in\[[58](https://arxiv.org/html/2607.08317#bib.bib50)\]\. In this case, it cannot effectively re\-evaluate earlier decision when a path leads to a dead\-end or violates a certain constraint and ends up in hallucinated answer\.

Language and Knowledge tasks:For Character\-level manipulation, we observe failure in several tasks , such as generating an exact number of characters or repeating a token sequence under strict constraints, especially in settings where external tools are disallowed\. We conjecture that such failures may often be attributed to limitations of subword tokenization in large language models, making exact character\-level counting and length control difficult\[[69](https://arxiv.org/html/2607.08317#bib.bib13),[13](https://arxiv.org/html/2607.08317#bib.bib14)\]\. Also, models are not robust to irrelevant changes in the input\. Prior work\[[59](https://arxiv.org/html/2607.08317#bib.bib68)\]has studied this phenomenon as spurious correlations, where model predictions depend on features that are not causally relevant to the task\.

## Appendix BFull Grader Prompts

Grader Prompt \(text generation tasks\)You are an expert grader\. Your task is to evaluate the correctness of a submitted answer based on the provided question and ground truth answer defining grading criteria\. \-\-\-\# \[Question\] \{question\} \-\-\-\# \[Submission\] \{answer\} \-\-\-\# \[Ground Truth\] \{criterion\} \-\-\-Important grading rules: \- The submission is correct only if it clearly states a final answer to the question\. \- A submission that only contains code, tool calls, search queries, reasoning, or instructions for how to derive the answer must be graded ’I’ if that answer is not clearly reported at the end, even if those steps would likely produce the correct answer\. \- If the submission is ambiguous about what the final answer is, grade it ’I’\.After assessing the submitted answer, reply with ’GRADE: $LETTER’ \(without quotes\) where LETTER is either of C or I\. Please choose ONE option for the grade: either "C" for correct answers, or "I" for incorrect answers\. No intermediate grades are allowed\. If the grading criterion is met only partially or the ground truth is ambiguous, use your best judgement to assign the most appropriate grade\. Start by briefly analyzing the submission and compare it against the ground truth\. Be as concise as possible in your analysis\. The ground truth will provide you with the necessary information to easily determine if the submission is correct or incorrect\. Only if needed, execute code snippets to verify the correctness of the submission\. Write a minimal explanation of your judgement and end with your answer formatted as ’GRADE: $LETTER’ \(without quotes\) where LETTER is either of C or I\.

Grader Prompt \(image generation tasks\)You are an expert grader evaluating whether a generated image correctly satisfies the given task\.\-\-\-\# \[Task/Prompt\] \{question\} \{optional\-input\-image\-section\} \-\-\-\# \[Generated Image \(Candidate Answer to Grade\)\] The image below is the generated output that needs to be evaluated\. \(The generated image is attached after this text\) \-\-\-\# \[Grading Criteria\] \{criterion\} \-\-\-\# \[Instructions\] Evaluate whether the generated image correctly satisfies the task requirements based on the grading criteria above\.Consider the following aspects when grading: 1\. \*\*Content Accuracy\*\*: Does the image contain the required elements/objects/scenes? 2\. \*\*Instruction Following\*\*: Does the image follow the specific instructions in the prompt? 3\. \*\*Quality\*\*: Is the generated image of reasonable quality \(not corrupted, incomplete, or nonsensical\)? 4\. \*\*Relevance\*\*: Is the generated image relevant to the task?If needed, you can execute Python code snippets to help verify aspects of the generated image \(e\.g\., counting objects, checking colors, analyzing dimensions, etc\.\)\. After your analysis, provide your final grade\. Reply with ’GRADE: $LETTER’ \(without quotes\) where LETTER is either: \- "C" for CORRECT: The generated image satisfies the task requirements \- "I" for INCORRECT: The generated image does NOT satisfy the task requirements Start with a brief analysis of the generated image compared to the requirements, then provide your final grade\.

## Appendix CGrader Validation

[Table 5](https://arxiv.org/html/2607.08317#A3.T5)and[table6](https://arxiv.org/html/2607.08317#A3.T6)collect the detailed metrics computed to verify the alignment between the automatic evaluation pipeline and human judgment\.

Table 5:Pipeline Validation Results: agreement scores between human\-annotated correctness scores and values obtained with the automatic evaluation pipeline\.Table 6:Disaggregated grader validation by solver model family\. We report agreement between the automatic grader and human labels, together with false positive rate \(FPR\) and recall\. FPR measures the rate at which the grader incorrectly marks a human\-incorrect response as correct, and is therefore the most relevant metric for detecting potential overestimation of a model family’s performance\.
## Appendix DModels Info List

[Table 7](https://arxiv.org/html/2607.08317#A4.T7)collects all the specifics of the evaluated models\.

Table 7:Model metadata, pricing, datatype, and serving hardware on the serving platform\. Prices are reported in USD per 1M tokens\. Size is reported in billions of parameters when available\. Serving GPU counts and data\-types are taken from the serving platform default model configuration\.VLMscan process both images and textual inputs but only produce text, whileLLMonly operate in text\.Image generationmodels can instead processed both modalities as input, but their output are only images\.ModelSize \(B\)ReleaseIn\. \($\)Out\. \($\)dtypeGPUsLanguage modelsGLM\-4\.73582025\-120\.56061\.6817fp84 x H200GLM\-57542026\-021\.07583\.2274fp88 x H200GLM\-5\.17542026\-040\.77162\.3149fp88 x H200GLM\-5\.27542026\-060\.692\.07fp88 x H200DeepSeek\-V4\-Flash1582026\-040\.26660\.7998fp84 x H100DeepSeek\-V4\-Pro8622026\-040\.93242\.7971fp88 x H200GPT\-oss\-20b212025\-080\.02040\.0611bf161 x A100GPT\-oss\-120b1172025\-080\.04860\.1457bf161 x A100Qwen3\-Next\-80B\-A3B\-Thinking802025\-090\.13650\.4097bf164 x A100Vision\-language modelsKimi\-K2\.510002026\-020\.68562\.0567int48 x H200Kimi\-K2\.610002026\-040\.61211\.8362int48 x H200Gemma\-4\-E2B\-it52026\-040\.01640\.0493bf161 x A100Gemma\-4\-E4B\-it82026\-040\.02490\.0746bf161 x A100Gemma\-4\-26B\-A4B\-it262026\-040\.04220\.1266bf161 x A100Qwen3\-VL\-30B\-A3B\-Thinking302025\-090\.06330\.1902bf161 x A100Qwen3\-VL\-235B\-A22B\-Thinking2352025\-090\.30950\.9285fp84 x A100Qwen3\.5\-35B\-A3B352026\-030\.04830\.1448fp81 x A100Qwen3\.5\-122B\-A10B1222026\-030\.14910\.4474fp82 x A100Qwen3\.5\-397B\-A17B3972026\-030\.50471\.5142fp88 x A100Gemini\-2\.5\-flash–2025\-060\.30002\.5000––Gemini\-2\.5\-pro–2025\-061\.250010\.0000––Gemini\-3\-flash\-preview–2025\-120\.50003\.0000––Gemini\-3\.1\-flash\-lite\-preview–2026\-030\.25001\.5000––Gemini\-3\.1\-pro\-preview–2026\-021\.00006\.0000––GPT\-5\-mini–2025\-080\.25002\.0000––GPT\-5–2025\-081\.250010\.0000––GPT\-5\.2–2025\-121\.750014\.0000––GPT\-5\.4\-nano–2026\-030\.20001\.2500––GPT\-5\.4\-mini–2026\-030\.75004\.5000––GPT\-5\.4–2026\-032\.500015\.0000––GPT\-5\.5–2026\-045\.000030\.0000––Generative image modelsGemini\-2\.5\-flash\-image–2025\-080\.300030\.0000––Gemini\-3\-pro\-image\-preview–2025\-112\.0000120\.0000––Gemini\-3\.1\-flash\-image\-preview–2026\-030\.500060\.0000––GPT\-image\-1\-mini–2025\-102\.50008\.0000––GPT\-image\-1\.5–2025\-128\.000032\.0000––GPT\-image\-2–2026\-048\.000030\.0000––
## Appendix EFull Models Performance

Hereafter are reported the scores of all models on each single question, allowing to identify shared failures among models\. Three heatmaps collect the models mean@k for each problem\. Problems are sorted based on the average accuracy across models\. Results are reported in[fig\.6](https://arxiv.org/html/2607.08317#A5.F6)for text\-only,[fig\.7](https://arxiv.org/html/2607.08317#A5.F7)for multi\-to\-text, and[fig\.8](https://arxiv.org/html/2607.08317#A5.F8)for image generation\.

![Refer to caption](https://arxiv.org/html/2607.08317v1/x9.png)Figure 6:Average accuracy for each model on each task, ontext\-onlyproblems\.![Refer to caption](https://arxiv.org/html/2607.08317v1/x10.png)Figure 7:Average accuracy for each model on each task, onmulti\-to\-textproblems\.![Refer to caption](https://arxiv.org/html/2607.08317v1/x11.png)Figure 8:Average accuracy for each model on each task, onimage\-generationproblems\.
## Appendix FAdditional analysis

Analyses relying on the proposed taxonomy are presented in[Table 8](https://arxiv.org/html/2607.08317#A6.T8)\.

Table 8:Detailed accuracy performance of each model on individual subtask categories\. The mapping between task IDs and task names can be found in[Table 4](https://arxiv.org/html/2607.08317#A1.T4)\.Model nameModel type1\-11\-21\-31\-41\-52\-12\-22\-32\-43\-13\-23\-3Total countALL653182130182218983219Total countGEN037180301072106gemini\-2\.5\-flash\-imageGEN\-29\.7338\.89\-40\.000\.00\-0\.000\.00100\.00\-50\.00gemini\-3\-pro\-image\-previewGEN\-54\.0561\.11\-73\.330\.00\-42\.8650\.00100\.00\-66\.67gemini\-3\.1\-flash\-image\-previewGEN\-64\.8655\.56\-60\.000\.00\-57\.140\.00100\.00\-83\.33gpt\-image\-1\-miniGEN\-21\.6233\.33\-46\.670\.00\-0\.000\.00100\.00\-0\.00gpt\-image\-1\.5GEN\-51\.3550\.00\-63\.330\.00\-14\.290\.00100\.00\-33\.33gpt\-image\-2GEN\-54\.0566\.67\-66\.670\.00\-14\.290\.00100\.00\-83\.33Total countLM11001017187673111zai\-org/GLM\-4\.7LM0\.0062\.50\-0\.00\-67\.6568\.0671\.4337\.5064\.2954\.8468\.18zai\-org/GLM\-5LM75\.0060\.00\-0\.00\-57\.3566\.6778\.5741\.6785\.7157\.2684\.09zai\-org/GLM\-5\.1LM50\.0062\.50\-0\.00\-54\.4163\.8985\.7145\.8385\.7167\.7486\.36zai\-org/GLM\-5\.2LM50\.0072\.50\-0\.00\-75\.0073\.6185\.7141\.6778\.5773\.3984\.09deepseek\-ai/DeepSeek\-V4\-FlashLM0\.0062\.50–0\.00–73\.5368\.0682\.1445\.8357\.1472\.5863\.64deepseek\-ai/DeepSeek\-V4\-ProLM0\.0065\.00–0\.00–64\.7173\.6182\.1433\.3371\.4375\.0084\.09openai/gpt\-oss\-20bLM0\.0055\.00–0\.00–51\.4759\.7271\.4333\.3353\.5752\.4240\.91openai/gpt\-oss\-120bLM0\.0055\.00–0\.00–66\.1866\.6785\.7150\.0057\.1459\.6843\.18Qwen/Qwen3\-Next\-80B\-A3B\-ThinkingLM0\.0062\.50\-0\.00\-66\.1866\.6782\.1433\.3350\.0056\.4554\.55Total countVLM6160210172211773213moonshotai/Kimi\-K2\.5VLM20\.8354\.69–33\.33–76\.4782\.9570\.4553\.5760\.7164\.0657\.69moonshotai/Kimi\-K2\.6VLM12\.5051\.56–34\.52–63\.2475\.0070\.4546\.4350\.0069\.5361\.54google/gemma\-4\-E2B\-itVLM4\.1739\.06–9\.52–27\.9446\.5927\.2721\.4350\.0025\.0025\.00google/gemma\-4\-E4B\-itVLM0\.0031\.25–16\.67–52\.9457\.9529\.5525\.0057\.1434\.3832\.69google/gemma\-4\-26B\-A4B\-itVLM4\.1756\.25–34\.52–69\.1275\.0061\.3639\.2942\.8666\.4150\.00Qwen/Qwen3\-VL\-30B\-A3B\-ThinkingVLM0\.0035\.94–17\.86–55\.8867\.0554\.5514\.2939\.2951\.5632\.69Qwen/Qwen3\-VL\-235B\-A22B\-ThinkingVLM8\.3343\.75–25\.00–64\.7168\.1854\.5528\.5753\.5760\.1653\.85Qwen/Qwen3\.5\-35B\-A3BVLM–51\.56–47\.62–75\.0079\.5572\.7332\.1475\.0055\.4753\.85Qwen/Qwen3\.5\-122B\-A10BVLM29\.1751\.56–53\.57–64\.7169\.3259\.0939\.2975\.0065\.6253\.85Qwen/Qwen3\.5\-397B\-A17BVLM25\.0060\.94–55\.95–72\.0671\.5965\.9139\.2982\.1467\.9769\.23gemini\-2\.5\-flashVLM4\.1745\.31–27\.38–70\.5972\.7363\.6421\.4360\.7141\.4148\.08gemini\-2\.5\-proVLM4\.1754\.69–29\.76–55\.8868\.1868\.1832\.1471\.4353\.9169\.23gemini\-3\-flash\-previewVLM41\.6775\.00–59\.52–76\.4772\.7388\.6453\.5771\.4377\.3482\.69gemini\-3\.1\-flash\-lite\-previewVLM4\.1760\.94\-40\.48\-57\.3568\.1861\.3628\.5760\.7164\.0663\.46gemini\-3\.1\-pro\-previewVLM41\.6787\.50\-57\.14\-92\.6575\.0088\.6464\.2996\.4384\.3876\.92gpt\-5\-miniVLM0\.0053\.12\-26\.19\-77\.9479\.5575\.0032\.1464\.2968\.7542\.31gpt\-5VLM12\.5059\.38\-38\.10\-77\.9471\.5968\.1857\.1460\.7169\.5355\.77gpt\-5\.2VLM41\.6762\.50\-39\.29\-70\.5976\.1475\.0071\.4364\.2971\.0959\.62gpt\-5\.4\-nanoVLM4\.1748\.44\-16\.67\-72\.0672\.7354\.5546\.4353\.5764\.0634\.62gpt\-5\.4\-miniVLM12\.5050\.00\-36\.90\-60\.2975\.0075\.0035\.7175\.0066\.4153\.85gpt\-5\.4VLM41\.6770\.31\-52\.38\-85\.2977\.2781\.8264\.2989\.2981\.2563\.46gpt\-5\.5VLM25\.0073\.44\-51\.19\-89\.7188\.6484\.0960\.71100\.0082\.0380\.77

Table 9:Detailed accuracy performance of each model on individual task categories\.ModelModel typeObject\-centricAbstract ReasoningLanguage and KnowledgeALL\-1286759Total countGEN85107gemini\-2\.5\-flash\-imageGEN35\.290\.0057\.14gemini\-3\-pro\-image\-previewGEN62\.3540\.0071\.43gemini\-3\.1\-flash\-image\-previewGEN61\.1840\.0085\.71gpt\-image\-1\-miniGEN32\.940\.0014\.29gpt\-image\-1\.5GEN55\.2910\.0042\.86gpt\-image\-2GEN61\.1810\.0085\.71Total countLM124849zai\-org/GLM\-4\.7LM52\.0864\.5859\.18zai\-org/GLM\-5LM56\.2561\.9867\.35zai\-org/GLM\-5\.1LM56\.2561\.4674\.49zai\-org/GLM\-5\.2LM64\.5871\.8876\.53deepseek\-ai/DeepSeek\-V4\-FlashLM52\.0869\.2768\.37deepseek\-ai/DeepSeek\-V4\-ProLM54\.1766\.6776\.53openai/gpt\-oss\-20bLM45\.8355\.2150\.00openai/gpt\-oss\-120bLM45\.8367\.1955\.61Qwen/Qwen3\-Next\-80B\-A3B\-ThinkingLM52\.0864\.5855\.10Total countVLM435752moonshotai/Kimi\-K2\.5VLM39\.5375\.0062\.02moonshotai/Kimi\-K2\.6VLM37\.7967\.1164\.90google/gemma\-4\-E2B\-itVLM19\.7734\.2128\.37google/gemma\-4\-E4B\-itVLM19\.7746\.9337\.02google/gemma\-4\-26B\-A4B\-itVLM38\.3766\.2359\.13Qwen/Qwen3\-VL\-30B\-A3B\-ThinkingVLM22\.0954\.8245\.19Qwen/Qwen3\-VL\-235B\-A22B\-ThinkingVLM29\.6559\.6557\.69Qwen/Qwen3\.5\-35B\-A3BVLM43\.6071\.0557\.69Qwen/Qwen3\.5\-122B\-A10BVLM49\.4262\.2863\.94Qwen/Qwen3\.5\-397B\-A17BVLM53\.4966\.6770\.19gemini\-2\.5\-flashVLM30\.8164\.0445\.67gemini\-2\.5\-proVLM35\.4760\.0960\.10gemini\-3\-flash\-previewVLM62\.7974\.5677\.88gemini\-3\.1\-flash\-lite\-previewVLM43\.0258\.7763\.46gemini\-3\.1\-pro\-previewVLM66\.2881\.5884\.13gpt\-5\-miniVLM32\.5672\.3761\.54gpt\-5VLM42\.4471\.0564\.90gpt\-5\.2VLM48\.2673\.6867\.31gpt\-5\.4\-nanoVLM26\.7465\.7955\.29gpt\-5\.4\-miniVLM38\.3765\.7964\.42gpt\-5\.4VLM57\.5678\.9577\.88gpt\-5\.5VLM55\.8184\.6584\.13

## Appendix GBroader Impact

Firstly, we provide a diagnostic benchmark to detect persistent gaps in modern models’ language understanding\. By selecting tasks that can be easily solved by humans but remain difficult for modern AI,blind\-spots\-benchcan help to better understand cases of failure which are otherwise overlooked due to aggregate success in other benchmarking tasks\. In certain use\-cases, where minor mistakes in number or spatial reasoning, attribute binding, or instruction following could lead to misinterpretations or even unsafe actions on the part of an AI system,blind\-spots\-benchmay become relevant for evaluating the applicability of the models in question, including applications to education, assistive technologies, document parsing, and human\-AI collaboration workflows\.

Theblind\-spots\-benchmay prove useful as a tool for more insightful model comparisons, showing how different approaches behave within individual categories\. At the same time, it would be unwise to consider the proposed benchmark as a means for thorough reliability assessment\. Being relatively small in terms of scale, biased towards particular types of problems and limited in question formulation and answer distribution, the dataset will inevitably show some signs of bias when it comes to the overall reliability of various model architectures\. Another problem associated with public benchmarks is that in the future, models can be specifically trained to overfit to the given data\. For all that, the proposed benchmark can only serve as a stress\-test and not a thorough safety evaluation of model robustness\.

## Appendix HLicense information

We distributeblind\-spots\-bench, containing the benchmark prompts, reference solutions, task and sub\-task annotations, failure\-mode annotations, and associated benchmark metadata for research purposes\. This dataset is available on HuggingFace under CC\-BY\-4\.0 license\. This data set contains questions suggested by students in an AI course at a graduate level and refined, filtered, and annotated by the authors\. The provided annotations consist of the structured reference solution, correctness conditions, and typical failure modes\. Some samples include or refer to images and external links to interact with shared models; the respective external service and third\-party contents accessible via such links are governed by their own license terms and are not being re\-licensed by the benchmark\.

The evaluation code and scripts are provided separately from the data set\. The evaluation pipeline uses Inspect AI, available under the MIT License\. All the third\-party software packages used here retain their original license agreements\. Outputs from models, evaluation logs, and generated images are not included with the distribution of the benchmark\.