On Locality and Length Generalization in Visual Reasoning

Hugging Face Daily Papers Papers
visual-reasoning length-generalization locality recurrence foveated-vision global-shortcuts out-of-distribution

Summary

This paper shows that state-of-the-art vision-language models fail at length generalization in visual reasoning due to 'global shortcuts', and demonstrates that combining local foveated perception with recurrence enables robust out-of-distribution generalization.

A striking feature of the human visual system is that it ingests visual information through a series of local foveated glimpses, rather than a single global computation. This makes human vision distinctly different from most popular computer vision models in use today, which input images globally and in a single shot. A natural question therefore is whether local, sequential vision models may provide any fundamental computational benefits in addition to being biologically more plausible than global models. In this work, we investigate this question from the perspective of visual state tracking and length generalization. Inspired by recent studies of length generalization in language models, we study the behavior of vision models trained on simple vision tasks that require the aggregation of local information across an image. Our experiments reveal that, similar to language models, vision models can learn to exploit global shortcuts and thereby fail to generalize over task length or complexity. We also show that recurrent vision policies based on strictly local perception can mitigate these failures, thereby allowing models to generalize on these tasks. Our results show that local attention may be an essential overlooked requirement for robust compositional generalization.
Original Article
View Cached Full Text

Cached at: 07/20/26, 09:41 AM

Paper page - On Locality and Length Generalization in Visual Reasoning

Source: https://huggingface.co/papers/2607.09061 Your eyes don’t see a whole scene at once — they dart around in a sequence of foveated glimpses. Modern vision models take in the whole image in one shot. That difference decides if a model can generalize to scenarios that are out-of-distribution.

Adobe Express - ACzVXh45Nj4LA_b8

We built simple visual puzzles (e.g. “read these switches, navigate, track the state”) and made them longer at test time than in training. SOTA VLMs — GPT-5.4, Claude Sonnet 4.6, Qwen — ace the short ones… then fall off a cliff the moment the puzzle gets longer. A tiny recurrent agent (green) extrapolates the performance at test time. Even with task-specific training, the global model (Qwen) still breaks out-of-distribution.

image

The failure has a name:global shortcuts. When a model sees the whole image at once, it can memorize a parallel “trick” that works for the lengths it saw — but doesn’t actually implement the step-by-step computation. So it breaks out-of-distribution. Same thing happens to LLMs on parity/state-tracking. We show vision has the same problem.

You might think making the model recurrent (giving it a memory that updates step by step) is the fix. It isn’t — on its own. Same recurrent LSTM, three ways ofseeingthe image:

  • Global— sees the whole image at once → collapses
  • Local + Global— adds small high-res crops but keeps the global view → still collapses
  • Foveated— only small local glimpses, no global view → generalized out-of-distribution. Hand a recurrent net the whole image and itstillmemorizes a shortcut.Locality is a necessary ingredient.

image

Now flip it: fix thelocalglimpses, swap only the backbone. Strict recurrent nets (LSTM/GRU/RNN) generalize. Transformers, Mamba, xLSTM — all degrade. So the recipe isboth: local perception + recurrence. Neither alone is enough. Together they’re sufficient. 🔑

image

There’s a real trade-off. Small/low-res glimpses → generalize but slow to explore. Big/high-res glimpses → easy to explore but invite shortcuts. With the right glimpse settings, we show that FoveAgentLSTM holds its accuracy at resolutions well beyond training — a global model only works in/near the resolution it was trained at.

image

But locality + recurrence isn’t a free win everywhere — and knowingwhenit helps is the point. On arecalltask the global VLMwinsand our local agent lags. State-tracking wants recurrence + locality. Pure retrieval doesn’t. The two split exactly like they do in language models.

image

The synthetic tasks isolate the mechanism. Does it carry over to a real task? We tested reasoning over math plots — finding a function’s roots. At thesame visual-compute budget, a foveated Qwen adds**+29 pts (~100%)**of accuracy over the global baseline. Uniformly cranking resolution 10× buys almost nothing (+3.8 pts). Same lesson as the synthetic tasks: locality + recurrence beats brute-force scaling. How you spend visual compute > how much visual compute you throw at it.

image

X post

Similar Articles

Reinforcing Dual-Path Reasoning in Spatial Vision Language Models

Hugging Face Daily Papers

This paper introduces SR-REAL, a unified framework for spatial vision-language models that combines linguistic deduction and 3D geometric reasoning via reinforcement learning, enabling robust multi-step spatial reasoning across diverse tasks.

More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models

Papers with Code Trending

This paper uncovers that prolonged reasoning in vision-language models can impair perceptual grounding, causing recognition failures on basic visual questions. It proposes Vision-Anchored Policy Optimization (VAPO) to steer reasoning toward visually grounded trajectories, achieving state-of-the-art performance with the VAPO-Thinker-7B model.

Why Far Looks Up: Probing Spatial Representation in Vision-Language Models

Hugging Face Daily Papers

Investigates spatial representation in vision-language models, revealing a consistent bias where models conflate vertical image position with distance, and introduces SpatialTunnel synthetic benchmark to expose this shortcut; finds that better disentangled spatial representations improve robustness.