ShapeCodeBench: A Renewable Benchmark for Perception-to-Program Reconstruction of Synthetic Shape Scenes

Hugging Face Daily Papers Papers
benchmark synthetic perception-to-program reconstruction drawing-programs evaluation

Summary

ShapeCodeBench is a synthetic benchmark for perception-to-program reconstruction where models generate executable drawing programs from raster images, evaluated on metrics like exact match and pixel accuracy. The benchmark is designed to be renewable via seeded RNG, and current models still achieve low exact match rates, indicating room for improvement.

We introduce ShapeCodeBench, a synthetic benchmark for perception-to-program reconstruction: given a rendered raster image, a model must emit an executable drawing program that a deterministic evaluator re-renders and compares with the target. The v1 DSL has four primitives on a 512 x 512 black-on-white canvas, but every instance is generated from a seeded RNG, so fresh held-out sets can be created to reduce exact-instance contamination. We release a frozen eval_v1 split with 150 samples across easy, medium, and hard tiers, scored by exact match, pixel accuracy, foreground IoU, parse success, and execution success. We evaluate an empty-program floor, a classical computer-vision heuristic, Claude Opus 4.7 at high and max effort, and GPT-5.5 at medium and extra_high reasoning effort. The heuristic is competitive on easy scenes but collapses when overlaps fuse components; the strongest multimodal configuration preserves much of the foreground structure but still misses exact match because of small parameter errors. Best overall exact match remains low, so ShapeCodeBench is far from saturated. The benchmark code, frozen dataset, run artifacts, and paper sources are released to support independent replication and extension.
Original Article
View Cached Full Text

Cached at: 05/14/26, 04:16 AM

Paper page - ShapeCodeBench: A Renewable Benchmark for Perception-to-Program Reconstruction of Synthetic Shape Scenes

Source: https://huggingface.co/papers/2605.11680

Abstract

ShapeCodeBench presents a synthetic benchmark for perception-to-program reconstruction where models generate executable drawing programs from raster images, evaluated on multiple metrics including exact match and pixel accuracy.

We introduce ShapeCodeBench, asynthetic benchmarkforperception-to-program reconstruction: given a renderedraster image, a model must emit anexecutable drawing programthat adeterministic evaluatorre-renders and compares with the target. The v1DSLhas fourprimitiveson a 512 x 512 black-on-whitecanvas, but every instance is generated from aseeded RNG, so freshheld-out setscan be created to reduce exact-instance contamination. We release a frozen eval_v1 split with 150 samples across easy, medium, and hard tiers, scored byexact match,pixel accuracy,foreground IoU,parse success, andexecution success. We evaluate an empty-program floor, a classical computer-vision heuristic, Claude Opus 4.7 at high and max effort, and GPT-5.5 at medium and extra_high reasoning effort. The heuristic is competitive on easy scenes but collapses when overlaps fuse components; the strongestmultimodal configurationpreserves much of the foreground structure but still missesexact matchbecause of smallparameter errors. Best overallexact matchremains low, so ShapeCodeBench is far from saturated. The benchmark code, frozen dataset, run artifacts, and paper sources are released to support independent replication and extension.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2605\.11680

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.11680 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.11680 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.11680 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

ProgramBench (5 minute read)

TLDR AI

ProgramBench is a new benchmark that evaluates AI agents' ability to reconstruct complete software projects from compiled binaries and documentation without access to source code or decompilation tools.

MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

Hugging Face Daily Papers

MemoBench is a diagnostic benchmark for evaluating video generation models' memory consistency in dynamically changing environments, where objects disappear and reappear in updated states. It includes 360 ground-truth clips and an evaluation suite combining automated metrics with VQA-based assessment, revealing insights into memory consistency challenges.

scShapeBench: Discovering geometry from high dimensional scRNAseq data

arXiv cs.LG

Introduces scShapeBench, a benchmark dataset for shape detection in high-dimensional single-cell data, and scReebTower, a baseline method that uses diffusion geometry and Reeb graphs to classify data shapes into clusters, trajectories, multi-branches, and archetypes.