Paper page - Beyond Starry Night: Shortcut-Aware Control-State Planning for Artist-Grounded Text to Image Generation

Source: https://huggingface.co/papers/2608.06751 Beyond Starry Night: Shortcut-Aware Control-State Planning for Artist-Grounded Text-to-Image Generation

📄 arXiv:https://arxiv.org/abs/2608.06751

TL;DR:Naming an artist in a prompt isn’t style control. T2I models tend to fall back oncanonical shortcuts— the recurring motifs, stock palettes, and over-represented period signatures they associate with that name — and quietly overwrite the scene the user actually asked for. This paper introducesAtelier, which plans an explicit control statebeforegeneration instead of hoping the backend infers intent.

What Atelier does

• Decomposes an underspecified request (“a quiet subway platform in the style of Van Gogh”) into an explicit control state:scene anchors,preserve vs. transformdecisions,style-regime hypotheses,role-bound artist evidence, andanti-shortcut constraints.
• Grounds that state with artist-level knowledge pluslocal patchreferences — and unlike generic RAG prompting, binds each patch to a specific scene role rather than dumping it in as an undifferentiated style exemplar.
• Compilesbackend-awaregeneration plans, then iteratively refines candidates with global + local authenticity critics.

ArtIntentBench A companion benchmark that trades artist breadth for supervision depth — two deliberately contrastive artists (Van GoghandQi Baishi, different media and cultural traditions) across four tasks: artwork re-rendering, period-controlled generation (Paris / Arles / Saint-Rémy / Auvers), historically unseen subjects, and Qi Baishi re-rendering, plus shortcut auditing and human preference eval. The argument for going deep: a wide benchmark of many artist names with shallow labels would end up measuring generic style association, not artist-grounded control.

Results Across both open-weight and closed-source generators, Atelier improves artist-level style fidelity, preserves source structure better, and substantially cuts shortcut substitution relative to prompt-engineered, retrieval-augmented, and general-purpose agent baselines.

**Why it’s interesting:**the framing that artist-grounded generation is bottleneckedupstreamof the diffusion model — in inferring explicit, evidence-grounded controls — is a nice counterpoint to the “just scale the generator” default. The shortcut-auditing setup also seems reusable well beyond these two artists.

Curious how the control state holds up on artists with less catalogued, more diffuse oeuvres, and whether the period-regime hypotheses transfer to movements rather than individuals.