TILT: Improving Compositional Generation in Diffusion Models with a Model-Intrinsic Reward
Summary
TILT is a training-free framework that improves compositional generation in diffusion models by using a model-intrinsic reward to align sampling trajectories at test time, enhancing fidelity to complex prompts without external supervision.
View Cached Full Text
Cached at: 07/28/26, 06:26 PM
Paper page - TILT: Improving Compositional Generation in Diffusion Models with a Model-Intrinsic Reward
Source: https://huggingface.co/papers/2607.21606
Abstract
Recentadvancesinpowerfultext-to-imagegenerationmodelshavemadeitincreasinglyimportanttodeveloptest-timemethodsthatmodifythesamplingtrajectorytoproduceimagesmorefaithfultocomplexcompositionalprompts.WepresentTILT,atraining-freeframeworkforcompositionaltext-to-imagegenerationviatest-timerewardalignment.Weinterpretcompositionalfailuresasoverlapmodesbetweenjointandsingle-conceptdistributions,anddefinearewardthatfavorssampleswhereallconceptsarejointlypresent.Thisrewardisintrinsictothebasemodelanddoesnotrequireanyexternalsupervisionorrewardmodels.ThisyieldsaKL-constrainedobjectivewithaclosed-formtiltedtargetdistributionandprincipledguidingstepsfordiffusionsampling.Theinteractionofconceptdistributionstogetherwiththeaboverewardnaturallyleadstotwodifferentguidancestrategieswhileahybridapproachthatbalancestheirrespectivebenefitsproducesstrongerperformance.ExperimentsonpromptsfromT2ICompBenchshowthatourmethodimprovescompositionalalignmentwhilepreservingimagequalitycomparedtopreviousbaselines.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2607\.21606
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.21606 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.21606 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.21606 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Learnability-Informed Fine-Tuning of Diffusion Language Models
We propose LIFT, a learnability-informed fine-tuning algorithm for diffusion language models that aligns training with token difficulty and time step, achieving substantial gains on reasoning benchmarks.
Diffusion-Guided Search via Exponential Tilting (DiffTilt): An Application to Falsification of Safety-Critical Systems
This paper introduces DiffTilt, a distributional framework that exponentially tilts a diffusion model-induced joint distribution over environments and executions to efficiently discover rare safety-critical failures in autonomous and cyber-physical systems, outperforming conditional sampling strategies on ARCH-COMP benchmarks and a new tractor-trailer benchmark.
Are we really tilting? The mechanics of reward guidance in flow and diffusion models
This paper explains the root cause of reward hacking in reward-guided flow and diffusion models, attributing it to finite-particle plug-in estimation of the Doob h-function, and proposes a reward damping schedule to correct within-mode bias without additional computational cost.
Reinforcing Few-step Generators via Reward-Tilted Distribution Matching
RTDMD is a two-stage framework combining distribution matching distillation with reward-guided reinforcement learning to improve few-step image generation alignment with human preferences. It achieves state-of-the-art results on multiple models with only 4 inference steps.
DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimization
This paper proposes DRIFT, a framework that combines offline trajectories with importance-weighted supervised fine-tuning to efficiently achieve multi-turn interactive learning performance comparable to reinforcement learning.