JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles

Hugging Face Daily Papers Papers
benchmark vision-language-models geometric-reasoning jigsaw-puzzle evaluation fine-tuning

Summary

Introduces JigShape, a jigsaw puzzle benchmark with interlocking pieces to evaluate visual-geometric reasoning in vision-language models. It finds that frontier VLMs largely fail at geometric reasoning and all models collapse on larger puzzle sizes, revealing a 'scaling cliff' for constraint satisfaction.

Jigsaw puzzle solving requires jointly reasoning about visual content and geometric constraints, yet existing benchmarks use rectangular cuts that create ambiguous ground truth in texture-repeated regions. We introduce \ours{}, a benchmark with tab-and-blank interlocking pieces where geometric constraints provide strong local compatibility requirements that, combined with visual content, yield unambiguous ground truth. Across 95K instances at four grid densities (4times4 to 16times16), we find that zero-shot VLMs largely lack geometric reasoning: only one of five frontier models (GPT-5.5) exceeds random baseline on 4times4 puzzles, while all others perform at chance level. While supervised fine-tuning achieves >97\% on 4times4, all models collapse on larger grids: GPT-5.5 drops from 70\% to near-random on 8times8, and even fine-tuned models fall below 5\% on 12times12. This ``scaling cliff'' suggests current architectures cannot maintain consistent constraint satisfaction as the number of pieces increases. establishes scalable geometric reasoning as an open challenge for vision-language models.
Original Article
View Cached Full Text

Cached at: 08/12/26, 08:19 AM

Paper page - JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles

Source: https://huggingface.co/papers/2607.27670 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

A new jigsaw benchmark with interlocking pieces reveals that vision-language models fail at geometric reasoning and suffer a sharp performance drop as puzzle size increases.

Jigsaw puzzle solving requires jointly reasoning about visual content and geometric constraints, yet existing benchmarks use rectangular cuts that create ambiguous ground truth in texture-repeated regions. We introduce \ours{}, a benchmark withtab-and-blank interlocking pieceswhere geometric constraints provide strong local compatibility requirements that, combined with visual content, yield unambiguous ground truth. Across 95K instances at four grid densities (4times4 to 16times16), we find that zero-shot VLMs largely lackgeometric reasoning: only one of five frontier models (GPT-5.5) exceeds random baseline on 4times4 puzzles, while all others perform at chance level. Whilesupervised fine-tuningachieves >97\% on 4times4, all models collapse on larger grids: GPT-5.5 drops from 70\% to near-random on 8times8, and even fine-tuned models fall below 5\% on 12times12. This ``scaling cliff’’ suggests current architectures cannot maintain consistent constraint satisfaction as the number of pieces increases. establishes scalablegeometric reasoningas an open challenge forvision-language models.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2607\.27670

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.27670 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.27670 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.27670 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MentalThink: Shaping Thoughts in Mental SVG World

arXiv cs.AI

MentalThink introduces a visual-symbolic reasoning paradigm for multimodal LLMs that uses SVG code as intermediate visual representations for multi-turn reasoning. The two-stage training with SFT and RL achieves strong performance on spatial reasoning benchmarks.

VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark

arXiv cs.AI

VAMPS is a new benchmark of 1,168 multimodal bilingual math problems designed to evaluate whether LLMs can benefit from constructing and reasoning over graphs/visualizations. Key finding: direct analytical solving surprisingly outperforms tool-enabled visual solving even on problems where plotting is a natural strategy.