Controllable Diverse Sampling through Pullback Directions

# Controllable Diverse Sampling through Pullback Directions
Source: [https://arxiv.org/html/2610.14294](https://arxiv.org/html/2610.14294)
\\optauthor

1École Polytechnique, Institut Polytechnique de Paris2New York Genome Center3Irving Institute for Cancer Dynamics, Columbia University4Department of Systems Biology, Columbia University5Department of Computer Science, Columbia University6Department of Biomedical Engineering, Columbia University7Data Science Institute, Columbia University
These authors contributed equally\.###### Abstract

Conditional diffusion models can generate high\-quality images that align closely with a given prompt\. However, samples generated from different random seeds may remain visually similar, limiting creative exploration\. Existing approaches can increase variation, but often require additional optimization or sampling overhead, while stronger perturbations may introduce distortions that degrade image quality or prompt alignment\. We propose a training\-free method that leverages the local pullback geometry induced by the mapping from the conditioning\-embedding space to the image space to identify directions of maximal output sensitivity\. By steering different sampling trajectories along these directions, we encourage distinct generations while maintaining consistency with the conditioning signal\. Experiments on MS\-COCO and BrushNet show the efficacy of the method in improving diversity on prompt\-conditioned image generation and inpainting\.

## 1Introduction

Recent advances in diffusion models\([Ho et al\., 2020](https://arxiv.org/html/2610.14294#bib.bib3);[Song and Ermon, 2019](https://arxiv.org/html/2610.14294#bib.bib5);[Song et al\., 2020a](https://arxiv.org/html/2610.14294#bib.bib4);[Song et al\., 2020b](https://arxiv.org/html/2610.14294#bib.bib6);[Rombach et al\., 2022](https://arxiv.org/html/2610.14294#bib.bib7)\), have enabled text\-to\-image systems to generate visually compelling images that closely follow natural\-language prompts\. However, image quality and conditional alignment do not fully characterize a generative model\. For a fixed condition, the model should also represent diverse valid outputs\.

In practice, trajectories may concentrate around the same dominant modes of the conditional distribution, resulting in semantically redundant outputs\. This limited coverage increases the cost of obtaining a diverse set of samples\. Inverse problems like image inpainting\([Ju et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib14)\)further constrain diversity, as only a small masked region is generated\.

Many methods consider postprocessing steps that require many more gradient calculations, typically fine\-tuning either the score model or the initial noise distribution to optimize a measure of diversity\. Alternatively, inference\-time methods based on sequential Monte Carlo can introduce diversity by drawing many samples and iteratively resampling to enforce a final distribution with high diversity\. Both these methods suffer from substantially increased time complexity, either during fine\-tuning or during inference\. In this work, we therefore seek to answer:

How can a pretrained diffusion model generate high\-quality, diverse samples without fine tuning or sampling prohibitively many particles?

We propose an efficient guidance method to encourage diverse samples\. Namely, for a given prompt and possible inpainting mask, we require only a constant number of Jacobian evaluations \(as few as one\) from which we can generate an arbitrary number of diverse samples that are conditioned by the text and/or mask\. We demonstrate improved performance on diversity metrics over comparable baselines operating in the regime of no fine\-tuning and cheap inference\.

## 2Preliminaries

#### Diffusion Models\.

We consider Denoising Diffusion Probabilistic Models \(DDPM\), which define a forward process that gradually corrupts samples from the data distribution with Gaussian noise\([Ho et al\., 2020](https://arxiv.org/html/2610.14294#bib.bib3)\)\. Given a clean imagex0∼pdatax\_\{0\}\\sim p\_\{\\mathrm\{data\}\}, the forward Markov chain is defined as

q\(x1:T∣x0\)=∏t=1Tq\(xt∣xt−1\),q\(xt∣xt−1\)=𝒩\(xt;αtxt−1,\(1−αt\)I\),q\(x\_\{1:T\}\\mid x\_\{0\}\)=\\prod\_\{t=1\}^\{T\}q\(x\_\{t\}\\mid x\_\{t\-1\}\),\\qquad q\(x\_\{t\}\\mid x\_\{t\-1\}\)=\\mathcal\{N\}\\left\(x\_\{t\};\\sqrt\{\\alpha\_\{t\}\}\\,x\_\{t\-1\},\(1\-\\alpha\_\{t\}\)I\\right\),\(1\)whereαt=1−βt\\alpha\_\{t\}=1\-\\beta\_\{t\}and\{βt\}t=1T\\\{\\beta\_\{t\}\\\}\_\{t=1\}^\{T\}denotes a predefined noise schedule\. We letα¯t=∏s=1tαs\\bar\{\\alpha\}\_\{t\}=\\prod\_\{s=1\}^\{t\}\\alpha\_\{s\}\. For a sufficiently long diffusion process,xTx\_\{T\}approaches an isotropic Gaussian distribution\. Given a networkϵθ\\epsilon\_\{\\theta\}trained to predict the noise used to generatextx\_\{t\}conditioned on the conditioncc, the reverse process is modeled by

pθ​\(xt−1∣xt,c\)=𝒩⁡\(xt−1,1αt​\(xt−βt1−α¯t​ϵθ​\(xt,t,c\)\),σt2​I\)\.p\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\},c\)=\\mathcal\{N\}\\left\(x\_\{t\-1\};\\frac\{1\}\{\\sqrt\{\\alpha\_\{t\}\}\}\\left\(x\_\{t\}\-\\frac\{\\beta\_\{t\}\}\{\\sqrt\{1\-\\bar\{\\alpha\}\_\{t\}\}\}\\epsilon\_\{\\theta\}\(x\_\{t\},t,c\)\\right\),\\sigma\_\{t\}^\{2\}I\\right\)\.\(2\)
The predicted noise can then be used to estimate the original clean image at any timestep\. Using Tweedie’s formula, the denoised estimate is

x^0\|t\(xt,t,c\):=𝔼\[x0∣xt,c\]=1α¯t\(xt−1−α¯tϵθ\(xt,t,c\)\)\.\\widehat\{x\}\_\{0\\mid t\}\(x\_\{t\},t,c\):=\\mathbb\{E\}\[x\_\{0\}\\mid x\_\{t\},c\]=\\frac\{1\}\{\\sqrt\{\\bar\{\\alpha\}\_\{t\}\}\}\\left\(x\_\{t\}\-\\sqrt\{1\-\\bar\{\\alpha\}\_\{t\}\}\\,\\epsilon\_\{\\theta\}\(x\_\{t\},t,c\)\\right\)\.\(3\)It therefore defines a differentiable mapping from the current noisy state to an estimate of the final clean image, which will later be used to construct the pullback geometry\.

#### Inpainting as an inverse problem\.

In image restoration problems, only a partial or corrupted measurementyyof imagex0x\_\{0\}is available\. For noiseless image inpainting, letM∈\{0,1\}dM\\in\\\{0,1\\\}^\{d\}denote the mask of the observed region, theny=M⊙x0y=M\\odot x\_\{0\}\. Text guidance controls the semantic content throughcc, whereas the mask constrains the generated sample to agree with the visible regionyy\.

## 3Conditional Pullback Metric

In this section, we introduce a differential geometric framework for identifying directions in the conditioning space of diffusion models\. In Euclidean spaces with a smooth mapf:ℝn→ℝmf:\\mathbb\{R\}^\{n\}\\rightarrow\\mathbb\{R\}^\{m\}, the pullback metric atx∈ℝnx\\in\\mathbb\{R\}^\{n\}evaluated atv∈ℝnv\\in\\mathbb\{R\}^\{n\}is defined as‖Jf⁡\(x\)​v‖\\\|J\_\{f\(x\)\}v\\\|\. We therefore seek directions in the input space that induce proportionally larger shifts in the output space via the top right singular vectors ofJf⁡\(x\)J\_\{f\(x\)\}\. We consider the Jacobian of the noise predictor with respect to condition:

Jc​\(xt,t,c\)=∂ϵθ​\(xt,t,c\)∂c,J\_\{c\}\(x\_\{t\},t,c\)=\\frac\{\\partial\\epsilon\_\{\\theta\}\(x\_\{t\},t,c\)\}\{\\partial c\},\(4\)
In the case of image inpainting, we can chain the linear masking process with Tweedie’s formula to define a Jacobian from the context to the masked portion of the image\. LetMe​d​i​t=1−MM\_\{edit\}=1\-Mdenote the bit\-flipped mask ofMM, then we can define the linear operatorWMW\_\{M\}acting on image space by

WM​v=Me​d​i​t⊙v,W\_\{M\}v=M\_\{edit\}\\odot v,The differential of the masked clean estimate is

∂\(WM​x^0\|t​\(xt,t,c\)\)∂c=−1−α¯tα¯t​WM​Jc​\(xt,t,c\)\.\\frac\{\\partial\\left\(W\_\{M\}\\widehat\{x\}\_\{0\\mid t\}\(x\_\{t\},t,c\)\\right\)\}\{\\partial c\}=\-\\sqrt\{\\frac\{1\-\\bar\{\\alpha\}\_\{t\}\}\{\\bar\{\\alpha\}\_\{t\}\}\}W\_\{M\}J\_\{c\}\(x\_\{t\},t,c\)\.\(5\)
Using the conditional pullback metric, we can infer a subspace that locally produces large variability in the latent space\. Explicitly, we consider span of the topKKright singular vectors of JacobianJ~c=WM​Jc\\widetilde\{J\}\_\{c\}=W\_\{M\}J\_\{c\}\. Since explicitly constructing the condition Jacobian is computationally intractable, we use a sketching method to obtain these directions using only Jacobian\-vector products dominant singular directions[Halko et al\. \(2011\)](https://arxiv.org/html/2610.14294#bib.bib26), for more details see Section[B\.1](https://arxiv.org/html/2610.14294#A2.SS1)\.

## 4Pullback\-informed prompt perturbation

With the top singular vectors, we can identify directions in the prompt embedding space along which small perturbations induce large changes in the denoising prediction and, consequently, in the predicted clean latent\. We use these directions to diversify the reverse trajectories while preserving the semantic content of the original condition\.

#### Initialization and trajectory diversification\.

We first sample a batch ofBBindependent initial latent variables,𝒵T=\{zT\(i\)\}i=1B\\mathcal\{Z\}\_\{T\}=\\\{z\_\{T\}^\{\(i\)\}\\\}\_\{i=1\}^\{B\}withzT\(i\)∼𝒩⁡\(0,I\),z\_\{T\}^\{\(i\)\}\\sim\\mathcal\{N\}\(0,I\),and use the same latent batch for all methods considered in our experiments\. Given the base promptc0c\_\{0\}, a pullback basis is then estimated from an anchor trajectory at the reference timesteptbt\_\{b\}\. The dominant eigenvectors of the condition\-space metricGtb=Jc0,tb⊤​Jc0,tbG\_\{t\_\{b\}\}=J\_\{c\_\{0\},t\_\{b\}\}^\{\\top\}J\_\{c\_\{0\},t\_\{b\}\}are partitioned into mutually disjoint subspaces, one for each particle\. Each particle is assigned a persistent direction obtained by projecting a fixed Gaussian field onto its corresponding subspace\. The perturbations are mutually orthogonal and encourage different local variations of the conditional model\. For particleii, the perturbed condition is therefore written as

ci​\(t\)=c0\+ρi⋆​γt​Δ​ci,c\_\{i\}\(t\)=c\_\{0\}\+\\rho\_\{i\}^\{\\star\}\\gamma\_\{t\}\\Delta c\_\{i\},whereΔ​ci\\Delta c\_\{i\}denotes the prompt\-relative pullback direction defined in Equation[7](https://arxiv.org/html/2610.14294#A2.E7),γt\\gamma\_\{t\}is the time\-dependent annealing coefficient, andρi⋆\\rho\_\{i\}^\{\\star\}controls the particle\-specific perturbation strength\.

#### Scheduled perturbation\.

LetS\>ES\>Edenote the perturbation window in reverse\-diffusion timestep coordinates\. We define the annealing schedule as a hard\-sigmoid raised to a powerp\>0p\>0, namely

γ⁡\(t,S,E\)=max⁡\(0,min⁡\(1,t−ES−E\)\)p\.\\gamma\(t;S,E\)=\\max\\left\(0,\\min\\left\(1,\\frac\{t\-E\}\{S\-E\}\\right\)\\right\)^\{p\}\.
The perturbation is fully active at the beginning of the reverse process, progressively annealed betweenSSandEE, and completely removed afterward\. For particleii, the condition used at timestepttisci,t=c0\+ρi⋆​γ​\(t,S,E,p\)​Δ​cic\_\{i,t\}=c\_\{0\}\+\\rho\_\{i\}^\{\\star\}\\gamma\(t;S,E,p\)\\,\\Delta c\_\{i\}whereΔ​ci\\Delta c\_\{i\}is its assigned pullback direction\.

#### Adaptive pullback refresh\.

The pullback basis computed at the reference timesteptbt\_\{b\}only captures the local geometry around the current state\. Since this geometry evolves along the reverse trajectory, we update the basis atPPintermediate timesteps:

trref≈S\+rP\+1\(E−S\),r=1,…,P\.t\_\{r\}^\{\\mathrm\{ref\}\}\\approx S\+\\frac\{r\}\{P\+1\}\(E\-S\),\\qquad r=1,\\ldots,P\.At each refresh timestep, we recompute the dominant eigenspace of the local pullback metric using the current latent of the anchor particle, and reproject each persistent Gaussian fieldηi\\eta\_\{i\}onto its refreshed particle\-specific subspace\. The previous and refreshed directions are interpolated over a short transition window to avoid abrupt changes in the conditioning trajectory[8](https://arxiv.org/html/2610.14294#A2.E8)\.

#### Particle\-specific scale selection\.

The sensitivity of the condition to response map varies along the diffusion landscape\. Consequently, a single perturbation scale may be too weak for some particles and too strong for others\. We therefore select a particle’s specific scaleρi∗\\rho\_\{i\}^\{\*\}from a finite candidate setℛ=\{ρ1,…,ρM\}\\mathcal\{R\}=\\\{\\rho\_\{1\},\\ldots,\\rho\_\{M\}\\\}by using the selection method in[B\.3](https://arxiv.org/html/2610.14294#A2.SS3)\.

## 5Experiments & Results

For text\-to\-image generation, we use captions from the MSCOCO 2017 validation\([Lin et al\., 2014](https://arxiv.org/html/2610.14294#bib.bib24)\)\. For image inpainting, we use the BrushBench validation set introduced with BrushNet\([Ju et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib14)\)\. We generate using the Stable Diffusion family of models, comparing our pullback method against baselines of DDIM, CADS\([Sadat et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib8)\), and TPSO\([Meng et al\., 2025](https://arxiv.org/html/2610.14294#bib.bib13)\)with standard metrics of prompt alignment and diversity\. Full details are given in Section[C](https://arxiv.org/html/2610.14294#A3)\. Tables[1](https://arxiv.org/html/2610.14294#S5.T1)and[2](https://arxiv.org/html/2610.14294#S5.T2)summarize the results for image generation and text\-guided inpainting, respectively\. Across both tasks, pullback\-guided sampling consistently improves diversity over clean DDIM while maintaining a high level of alignment with the text condition\. The results are particularly good for image inpainting, where the editable region constrains the possible variations and therefore makes the pullback metric a more significant initial signal\.

![Refer to caption](https://arxiv.org/html/2610.14294v1/figures/inpainting_comparison.png)Figure 1:Qualitative comparison on a BrushBench example with BrushNet\-SDXL\. Each row contains masked samples \(see Figure[2](https://arxiv.org/html/2610.14294#A3.F2.fig1)in the Appendix\)\. Clean DDIM, CADS, and TPSO introduce local changes, whereas adaptive pullback produces more pronounced variations\.Table 1:Generation results on MSCOCO 2017 validation captions with five generated images per caption\. Stable Diffusion 1\.5 and XL results use all 500 captions\. Entries report mean±\\pmstandard deviation across captions\. The best mean within each backbone is shown in bold\.Table 2:BrushBench inpainting results for Stable Diffusion 1\.5 and Stable Diffusion XL, with five images per example\. Stable Diffusion 1\.5 and XL results use all 500 examples\. Entries report mean±\\pmstandard deviation across examples\. The best result within each backbone is shown in bold\.
## 6Conclusion

We introduced a training\-free method for diversity conditional diffusion models through pullback\-informed prompt perturbations\. Our method identifies locally expressive directions in the conditioning space that induce the largest variation in the image space, by estimating the dominant directions of the pullback operator\. These results demonstrate that the local geometry of the denoiser with respect to condition provides useful directions for diversifying generation without fine\-tuning\. Future work will evaluate the effectiveness of pullback directions on other constrained generation tasks\.

## References

- Astolfiet al\.\(2024\)P\. Astolfi, M\. Careil, M\. Hall, O\. Mañas, M\. Muckley, J\. Verbeek, A\. R\. Soriano, and M\. DrozdzalConsistency\-diversity\-realism pareto fronts of conditional image generative models\.arXiv preprint arXiv:2406\.10429\.Cited by:[§A\.1](https://arxiv.org/html/2610.14294#A1.SS1.SSS0.Px3.p1.1)\.
- Banet al\.\(2025\)Y\. Ban, R\. Wang, T\. Zhou, B\. Gong, C\. Hsieh, and M\. ChengThe crystal ball hypothesis in diffusion models: anticipating object positions from initial noise\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 34384–34408\.Cited by:[§A\.1](https://arxiv.org/html/2610.14294#A1.SS1.SSS0.Px3.p1.1)\.
- Corsoet al\.\(2024\)G\. Corso, Y\. Xu, V\. De Bortoli, R\. Barzilay, and T\. JaakkolaParticle guidance: non\-iid diverse sampling with diffusion models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 22480–22507\.Cited by:[§A\.3](https://arxiv.org/html/2610.14294#A1.SS3.p1.1)\.
- De Santiet al\.\(2025\)R\. De Santi, M\. Vlastelica, Y\. Hsieh, Z\. Shen, N\. He, and A\. KrauseProvable maximum entropy manifold exploration via diffusion models\.arXiv preprint arXiv:2506\.15385\.Cited by:[§A\.2](https://arxiv.org/html/2610.14294#A1.SS2.p1.1)\.
- Domingo i Enrichet al\.\(2025\)C\. Domingo i Enrich, M\. Drozdzal, B\. Karrer, and R\. T\. ChenAdjoint matching: fine\-tuning flow and diffusion generative models with memoryless stochastic optimal control\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 53791–53846\.Cited by:[§A\.2](https://arxiv.org/html/2610.14294#A1.SS2.p1.1)\.
- Eyringet al\.\(2024\)L\. Eyring, S\. Karthik, K\. Roth, A\. Dosovitskiy, and Z\. AkataReno: enhancing one\-step text\-to\-image models through reward\-based noise optimization\.Advances in Neural Information Processing Systems37,pp\. 125487–125519\.Cited by:[§A\.3](https://arxiv.org/html/2610.14294#A1.SS3.p1.1)\.
- Friedman and Dieng \(2022\)D\. Friedman and A\. B\. DiengThe vendi score: a diversity evaluation metric for machine learning\.arXiv preprint arXiv:2210\.02410\.Cited by:[§C\.1](https://arxiv.org/html/2610.14294#A3.SS1.SSS0.Px3.p1.1)\.
- Halkoet al\.\(2011\)N\. Halko, P\. Martinsson, and J\. A\. TroppFinding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions\.SIAM review53\(2\),pp\. 217–288\.Cited by:[§B\.1](https://arxiv.org/html/2610.14294#A2.SS1.p5.1),[§3](https://arxiv.org/html/2610.14294#S3.p5.1)\.
- Harringtonet al\.\(2025\)A\. Harrington, A\. Koepke, S\. Karthik, T\. Darrell, and A\. A\. EfrosIt’s never too late: noise optimization for collapse recovery in trained diffusion models\.arXiv preprint arXiv:2601\.00090\.Cited by:[§A\.3](https://arxiv.org/html/2610.14294#A1.SS3.p1.1)\.
- Hesselet al\.\(2021\)J\. Hessel, A\. Holtzman, M\. Forbes, R\. Le Bras, and Y\. ChoiClipscore: a reference\-free evaluation metric for image captioning\.InProceedings of the 2021 conference on empirical methods in natural language processing,pp\. 7514–7528\.Cited by:[§C\.1](https://arxiv.org/html/2610.14294#A3.SS1.SSS0.Px1.p1.1)\.
- Hoet al\.\(2020\)J\. Ho, A\. Jain, and P\. AbbeelDenoising diffusion probabilistic models\.Advances in neural information processing systems33,pp\. 6840–6851\.Cited by:[§1](https://arxiv.org/html/2610.14294#S1.p1.1),[§2](https://arxiv.org/html/2610.14294#S2.SS0.SSS0.Px1.p1.1)\.
- Ho and Salimans \(2022\)J\. Ho and T\. SalimansClassifier\-free diffusion guidance\.arXiv preprint arXiv:2207\.12598\.Cited by:[§A\.1](https://arxiv.org/html/2610.14294#A1.SS1.SSS0.Px1.p1.1)\.
- Juet al\.\(2024\)X\. Ju, X\. Liu, X\. Wang, Y\. Bian, Y\. Shan, and Q\. XuBrushnet: a plug\-and\-play image inpainting model with decomposed dual\-branch diffusion\.InEuropean Conference on Computer Vision,pp\. 150–168\.Cited by:[§A\.1](https://arxiv.org/html/2610.14294#A1.SS1.SSS0.Px2.p1.1),[Appendix C](https://arxiv.org/html/2610.14294#A3.SS0.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2610.14294#A3.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.14294#S1.p2.1),[§5](https://arxiv.org/html/2610.14294#S5.p1.1)\.
- Linet al\.\(2014\)T\. Lin, M\. Maire, S\. Belongie, J\. Hays, P\. Perona, D\. Ramanan, P\. Dollár, and C\. L\. ZitnickMicrosoft coco: common objects in context\.InEuropean conference on computer vision,pp\. 740–755\.Cited by:[Appendix C](https://arxiv.org/html/2610.14294#A3.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.14294#S5.p1.1)\.
- Lugmayret al\.\(2022\)A\. Lugmayr, M\. Danelljan, A\. Romero, F\. Yu, R\. Timofte, and L\. Van GoolRepaint: inpainting using denoising diffusion probabilistic models\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 11461–11471\.Cited by:[§A\.1](https://arxiv.org/html/2610.14294#A1.SS1.SSS0.Px2.p1.1)\.
- Menget al\.\(2025\)D\. Meng, C\. Jin, Z\. Gao, Y\. Li, I\. Patras, and G\. TzimiropoulosTraining\-free generation of diverse and high\-fidelity images via prompt semantic space optimization\.arXiv preprint arXiv:2511\.19811\.Cited by:[§A\.3](https://arxiv.org/html/2610.14294#A1.SS3.p1.1),[3rd item](https://arxiv.org/html/2610.14294#A3.I1.i3.p1.1),[§5](https://arxiv.org/html/2610.14294#S5.p1.1)\.
- Nicholet al\.\(2021\)A\. Nichol, P\. Dhariwal, A\. Ramesh, P\. Shyam, P\. Mishkin, B\. McGrew, I\. Sutskever, and M\. ChenGlide: towards photorealistic image generation and editing with text\-guided diffusion models\.arXiv preprint arXiv:2112\.10741\.Cited by:[§A\.1](https://arxiv.org/html/2610.14294#A1.SS1.SSS0.Px2.p1.1)\.
- Oquabet al\.\(2023\)M\. Oquab, T\. Darcet, T\. Moutakanni, H\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby,et al\.Dinov2: learning robust visual features without supervision\.arXiv preprint arXiv:2304\.07193\.Cited by:[§C\.1](https://arxiv.org/html/2610.14294#A3.SS1.SSS0.Px2.p1.1)\.
- Parket al\.\(2023\)Y\. Park, M\. Kwon, J\. Choi, J\. Jo, and Y\. UhUnderstanding the latent space of diffusion models through the lens of riemannian geometry\.Advances in Neural Information Processing Systems36,pp\. 24129–24142\.Cited by:[§D\.1](https://arxiv.org/html/2610.14294#A4.SS1.p1.1),[§D\.1](https://arxiv.org/html/2610.14294#A4.SS1.p2.2)\.
- Rombachet al\.\(2022\)R\. Rombach, A\. Blattmann, D\. Lorenz, P\. Esser, and B\. OmmerHigh\-resolution image synthesis with latent diffusion models\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 10684–10695\.Cited by:[§A\.1](https://arxiv.org/html/2610.14294#A1.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.14294#S1.p1.1)\.
- Sadatet al\.\(2024\)S\. Sadat, J\. Buhmann, D\. Bradley, O\. Hilliges, and R\. WeberCADS: unleashing the diversity of diffusion models through condition\-annealed sampling\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 23723–23755\.Cited by:[§A\.3](https://arxiv.org/html/2610.14294#A1.SS3.p1.1),[2nd item](https://arxiv.org/html/2610.14294#A3.I1.i2.p1.1),[§C\.1](https://arxiv.org/html/2610.14294#A3.SS1.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2610.14294#S5.p1.1)\.
- Sahariaet al\.\(2021\)C\. Saharia, W\. Chan, H\. Chang, C\. A\. Lee, J\. Ho, T\. Salimans, D\. J\. Fleet, and M\. NorouziPalette: image\-to\-image diffusion models\.arXiv preprint arXiv:2111\.05826\.Cited by:[§A\.1](https://arxiv.org/html/2610.14294#A1.SS1.SSS0.Px1.p1.1)\.
- Songet al\.\(2020a\)J\. Song, C\. Meng, and S\. ErmonDenoising diffusion implicit models\.arXiv preprint arXiv:2010\.02502\.Cited by:[§1](https://arxiv.org/html/2610.14294#S1.p1.1)\.
- Song and Ermon \(2019\)Y\. Song and S\. ErmonGenerative modeling by estimating gradients of the data distribution\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2610.14294#S1.p1.1)\.
- Songet al\.\(2020b\)Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. PooleScore\-based generative modeling through stochastic differential equations\.arXiv preprint arXiv:2011\.13456\.Cited by:[§1](https://arxiv.org/html/2610.14294#S1.p1.1)\.
- Zhanget al\.\(2023\)L\. Zhang, A\. Rao, and M\. AgrawalaAdding conditional control to text\-to\-image diffusion models\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 3836–3847\.Cited by:[§A\.1](https://arxiv.org/html/2610.14294#A1.SS1.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2023\)G\. Zheng, X\. Zhou, X\. Li, Z\. Qi, Y\. Shan, and X\. LiLayoutdiffusion: controllable diffusion model for layout\-to\-image generation\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 22490–22499\.Cited by:[§A\.1](https://arxiv.org/html/2610.14294#A1.SS1.SSS0.Px1.p1.1)\.

## Appendix ARelated works

### A\.1Image Diffusion Models

#### Conditional image generation

Conditional image generation aims to synthesize realistic images that satisfy user\-specified controls\. These conditions may take the form of textual descriptions, reference images, class labels, segmentation maps, depth maps, or spatial layouts\. Diffusion models have become a dominant framework for this task because of their strong image quality and flexibility in incorporating heterogeneous conditions\. Text\-conditioned latent diffusion models introduce semantic information through cross\-attention, enabling high\-resolution text\-to\-image synthesis\([Rombach et al\., 2022](https://arxiv.org/html/2610.14294#bib.bib7)\), while classifier\-free guidance strengthens alignment by combining conditional and unconditional denoising predictions\([Ho and Salimans, 2022](https://arxiv.org/html/2610.14294#bib.bib15)\)\. Other approaches introduce image\-to\-image conditioning\([Saharia et al\., 2021](https://arxiv.org/html/2610.14294#bib.bib16)\)or additional spatial controls such as edges, poses, depth maps, and layouts\([Zheng et al\., 2023](https://arxiv.org/html/2610.14294#bib.bib17);[Zhang et al\., 2023](https://arxiv.org/html/2610.14294#bib.bib21)\)\. These developments have substantially improved the controllability and fidelity of conditional image synthesis\.

#### Image inpainting

Image inpainting consists in reconstructing a missing region of an image while preserving the visible content and maintaining semantic and visual coherence across the edited region\. Diffusion\-based approaches address inpainting either by training models that explicitly receive the masked image and mask as conditioning inputs or by adapting the reverse process of a pretrained generative model\([Ju et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib14);[Lugmayr et al\., 2022](https://arxiv.org/html/2610.14294#bib.bib22);[Nichol et al\., 2021](https://arxiv.org/html/2610.14294#bib.bib18)\)\. These approaches aim to generate plausible content within the masked region while preserving the observed pixels and maintaining semantic and visual coherence with the unmasked region\.

#### Diversity image generation

Although diffusion models generate samples from random initial noise, stochasticity alone does not guarantee meaningful diversity under a fixed condition\. Different initializations may still converge toward semantically similar outputs, particularly when strong guidance, textual constraints, or spatial observations restrict the reverse process\. Recent analyses therefore emphasize a trade\-off between realism, condition consistency, and distributional coverage\([Astolfi et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib20)\), while other work shows that the initial noise can already strongly determine the spatial organization of the final image\([Ban et al\., 2025](https://arxiv.org/html/2610.14294#bib.bib19)\)\. Consequently, producing multiple distinct yet equally valid completions remains challenging, especially for constrained tasks such as text\-guided inpainting\. Existing methods address this limitation through additional training\(finetuning diversity methods\), condition perturbations, or interactions between sampling trajectories\.

### A\.2Fine\-tuning diversity methods

Fine\-tuning methods enable further training towards a target reward or distributional loss\. For a scalar reward, Adjoint Matching\([Domingo i Enrich et al\., 2025](https://arxiv.org/html/2610.14294#bib.bib2)\)backpropagates through the entire dynamics to improve the reward while staying close to the original reverse process\. This has been generalized as a tool to promote diversity in the generated distribution in[De Santi et al\. \(2025\)](https://arxiv.org/html/2610.14294#bib.bib1)\. Notably, these methods require expensive sampling through the diffusion model and are therefore no longer simulation\-free\.

### A\.3Inference\-Time Diversity Methods

Inference\-time diversity methods aim to improve the coverage of the conditional distribution without retraining the underlying diffusion model\. Existing approaches differ primarily in the component of the sampling process on which they intervene\. Condition\-space methods modify the conditioning signal supplied to the denoising network\. CADS\([Sadat et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib8)\)perturbs the condition embedding with Gaussian noise whose magnitude is progressively annealed throughout sampling, thereby increasing variation during the early stages of generation\. TPSO\([Meng et al\., 2025](https://arxiv.org/html/2610.14294#bib.bib13)\)instead explores underutilized directions of the token\-embedding space through optimized prompt perturbations, while constraining the resulting samples to preserve semantic consistency\. Some family directly modifies the source of stochasticity by optimizing the initial diffusion noise\([Harrington et al\., 2025](https://arxiv.org/html/2610.14294#bib.bib25);[Eyring et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib27)\), an others instead construct a joint, non\-i\.i\.d\. distribution sample\([Corso et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib9)\)\. Particle Guidance introduces pairwise repulsive interactions during denoising to prevent particles from converging toward the same modes\([Corso et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib9)\)\.

## Appendix BPullback\-Informed Prompt Perturbation

LetC0\+∈ℝL×dcC\_\{0\}^\{\+\}\\in\\mathbb\{R\}^\{L\\times d\_\{c\}\}denote the positive token condition, wheredcd\_\{c\}is the token\-embedding dimension\. After vectorization, the condition belongs toℝD\\mathbb\{R\}^\{D\}, withD=L​dcD=Ld\_\{c\}\.

### B\.1Matrix\-free estimation of expressive directions

We first follow the clean CFG\-DDIM trajectory of one reference particle to a basis\-estimation timesteptbt\_\{b\}, obtainingztbrefz\_\{t\_\{b\}\}^\{\\mathrm\{ref\}\}\. At the fixed state\(ztbref,tb,C0\+\)\(z\_\{t\_\{b\}\}^\{\\mathrm\{ref\}\},t\_\{b\},C\_\{0\}^\{\+\}\), we seek directions in the conditioning space that induce the largest local variation in the denoiser output\. For a unit directionvv, this variation can be measured by

1ε2​‖fb​\(C0\+\+ε​v\)−fb​\(C0\+\)‖22,‖v‖2=1,\\frac\{1\}\{\\varepsilon^\{2\}\}\\left\\\|f\_\{b\}\(C\_\{0\}^\{\+\}\+\\varepsilon v\)\-f\_\{b\}\(C\_\{0\}^\{\+\}\)\\right\\\|\_\{2\}^\{2\},\\qquad\\\|v\\\|\_\{2\}=1,wherefb​\(C\)=ϵθ​\(ztbref,tb,C\)\.f\_\{b\}\(C\)=\\epsilon\_\{\\theta\}\(z\_\{t\_\{b\}\}^\{\\mathrm\{ref\}\},t\_\{b\},C\)\.Using the first\-order approximationfb​\(C0\+\+ε​v\)−fb​\(C0\+\)≃ε​Jb​vf\_\{b\}\(C\_\{0\}^\{\+\}\+\\varepsilon v\)\-f\_\{b\}\(C\_\{0\}^\{\+\}\)\\simeq\\varepsilon J\_\{b\}v, the problem of finding theKKmost expressive orthonormal directions becomes

V⋆=arg⁡maxV⊤​V=IK⁡‖Jb​V‖F2=arg⁡maxV⊤​V=IK​tr⁡\(V⊤​Jb⊤​Jb​V\)\.V^\{\\star\}=\\arg\\max\_\{V^\{\\top\}V=I\_\{K\}\}\\\|J\_\{b\}V\\\|\_\{F\}^\{2\}=\\arg\\max\_\{V^\{\\top\}V=I\_\{K\}\}\\operatorname\{tr\}\\\!\\left\(V^\{\\top\}J\_\{b\}^\{\\top\}J\_\{b\}V\\right\)\.\(6\)
Therefore, by the Rayleigh–Ritz theorem, the columns ofV⋆V^\{\\star\}are theKKdominant eigenvectors of

Gb=Jb⊤​Jb,Jb=∂fb∂C\|C=C0\+\.G\_\{b\}=J\_\{b\}^\{\\top\}J\_\{b\},\\qquad J\_\{b\}=\\left\.\\frac\{\\partial f\_\{b\}\}\{\\partial C\}\\right\|\_\{C=C\_\{0\}^\{\+\}\}\.Equivalently, they are the dominant right singular vectors ofJbJ\_\{b\}\. We estimate these directions without explicitly constructingJbJ\_\{b\}\.

Jb​v≈fb​\(C0\+\+ε​v\)−fb​\(C0\+−ε​v\)2​ε,Gb​v=Jb⊤​\(Jb​v\),J\_\{b\}v\\approx\\frac\{f\_\{b\}\(C\_\{0\}^\{\+\}\+\\varepsilon v\)\-f\_\{b\}\(C\_\{0\}^\{\+\}\-\\varepsilon v\)\}\{2\\varepsilon\},\\qquad G\_\{b\}v=J\_\{b\}^\{\\top\}\(J\_\{b\}v\),where the second operation is evaluated with a vector–Jacobian product\.

To approximate theKKdominant eigenvectors, we use randomized block subspace iteration followed by Rayleigh–Ritz extraction\([Halko et al\., 2011](https://arxiv.org/html/2610.14294#bib.bib26)\)\. We sampleΩ∈ℝD×K\\Omega\\in\\mathbb\{R\}^\{D\\times K\}with Gaussian entries and set

Q0=QR\(Ω\),Qr\+1=QR\(GbQr\),r=0,…,R−1\.Q\_\{0\}=\\operatorname\{QR\}\(\\Omega\),\\qquad Q\_\{r\+1\}=\\operatorname\{QR\}\(G\_\{b\}Q\_\{r\}\),\\quad r=0,\\ldots,R\-1\.We then construct

B=12​\(QR⊤​Gb​QR\+\(QR⊤​Gb​QR\)⊤\),B=\\frac\{1\}\{2\}\\left\(Q\_\{R\}^\{\\top\}G\_\{b\}Q\_\{R\}\+\(Q\_\{R\}^\{\\top\}G\_\{b\}Q\_\{R\}\)^\{\\top\}\\right\),diagonalizeB=U​Λ​U⊤B=U\\Lambda U^\{\\top\}, sort the eigenvalues in decreasing order, and lift the resulting vectors back to condition space:

V=QR​U=\[v1,…,vK\]\.V=Q\_\{R\}U=\[v\_\{1\},\\ldots,v\_\{K\}\]\.

### B\.2Fixed disjoint perturbation directions

The basis indices are divided into balanced, mutually disjoint setsℐ1,…,ℐN\\mathcal\{I\}\_\{1\},\\ldots,\\mathcal\{I\}\_\{N\}\. For each particle, a Gaussian fieldξi∼𝒩⁡\(0,ID\)\\xi\_\{i\}\\sim\\mathcal\{N\}\(0,I\_\{D\}\)is sampled once and kept fixed\. Its projection onto the assigned pullback subspace is

d¯i=∑k∈ℐi⟨ξi,vk⟩​vk,di=d¯i‖d¯i‖2\.\\bar\{d\}\_\{i\}=\\sum\_\{k\\in\\mathcal\{I\}\_\{i\}\}\\langle\\xi\_\{i\},v\_\{k\}\\rangle v\_\{k\},\\qquad d\_\{i\}=\\frac\{\\bar\{d\}\_\{i\}\}\{\\\|\\bar\{d\}\_\{i\}\\\|\_\{2\}\}\.
To express the perturbation relative to the magnitude of the original condition, each real token is scaled by its clean embedding norm:

Δ​ci=‖C0\+‖2​di,\\Delta c\_\{i\}=\|\|C\_\{0\}^\{\+\}\|\|\_\{2\}d\_\{i\},\(7\)where‖C0\+‖2\|\|C\_\{0\}^\{\+\}\|\|\_\{2\}denote the norms of the token embeddings\.

#### Adaptive pullback refresh

For the refresh, the pullback basis and the corresponding directions are recomputed at intermediate denoising steps\. Letdioldd\_\{i\}^\{\\mathrm\{old\}\}anddinewd\_\{i\}^\{\\mathrm\{new\}\}denote the directions before and after recomputation\. To avoid an abrupt change, we interpolate between them overL=2L=2steps:

di\(ℓ\)=\(1−βℓ\)​diold\+βℓ​dinew‖\(1−βℓ\)​diold\+βℓ​dinew‖2,βℓ=ℓ2,ℓ∈\{1,2\}\.d\_\{i\}^\{\(\\ell\)\}=\\frac\{\(1\-\\beta\_\{\\ell\}\)d\_\{i\}^\{\\mathrm\{old\}\}\+\\beta\_\{\\ell\}d\_\{i\}^\{\\mathrm\{new\}\}\}\{\\left\\\|\(1\-\\beta\_\{\\ell\}\)d\_\{i\}^\{\\mathrm\{old\}\}\+\\beta\_\{\\ell\}d\_\{i\}^\{\\mathrm\{new\}\}\\right\\\|\_\{2\}\},\\qquad\\beta\_\{\\ell\}=\\frac\{\\ell\}\{2\},\\quad\\ell\\in\\\{1,2\\\}\.\(8\)\.

### B\.3Particle\-specific magnitude selection

We select a particle’s specific magnitude from a finite candidate setℛ=\{ρ1,…,ρM\}\\mathcal\{R\}=\\\{\\rho\_\{1\},\\ldots,\\rho\_\{M\}\\\}at a specific timesteptpt\_\{p\}before the final sampling run\.

For each particleiiand candidate magnitudeρm\\rho\_\{m\}, we compute the predicted clean imagex^i,m\\widehat\{x\}\_\{i,m\}attpt\_\{p\}\. We additionally useρ0=0\\rho\_\{0\}=0to obtain the unperturbed referencex^i,0\\widehat\{x\}\_\{i,0\}\. Semantic diversity between two candidates is measured using the cosine distance between their DINO representations:

dDINO​\(x,x′\)=1−cos⁡\(ϕDINO​\(x\),ϕDINO​\(x′\)\),d\_\{\\mathrm\{DINO\}\}\(x,x^\{\\prime\}\)=1\-\\cos\\left\(\\phi\_\{\\mathrm\{DINO\}\}\(x\),\\phi\_\{\\mathrm\{DINO\}\}\(x^\{\\prime\}\)\\right\),\(9\)whereϕDINO\\phi\_\{\\mathrm\{DINO\}\}is a frozen DINO image encoder\. Prompt alignment is measured by

sCLIP​\(x,C\)=cos⁡\(ϕCLIPI​\(x\),ϕCLIPT​\(C\)\),s\_\{\\mathrm\{CLIP\}\}\(x,C\)=\\cos\\left\(\\phi\_\{\\mathrm\{CLIP\}\}^\{I\}\(x\),\\phi\_\{\\mathrm\{CLIP\}\}^\{T\}\(C\)\\right\),\(10\)whereϕCLIPI\\phi\_\{\\mathrm\{CLIP\}\}^\{I\}andϕCLIPT\\phi\_\{\\mathrm\{CLIP\}\}^\{T\}are the frozen CLIP image and text encoders, respectively\.

The particle\-specific magnitudes are jointly selected by maximizing the average pairwise DINO distance while limiting the loss of prompt alignment relative to the unperturbed prediction:

\(m1⋆,…,mB⋆\)∈arg⁡maxm1,…,mBmi∈\{1,…,M\}\\displaystyle\(m\_\{1\}^\{\\star\},\\ldots,m\_\{B\}^\{\\star\}\)\\in\\arg\\max\_\{\\begin\{subarray\}\{c\}m\_\{1\},\\ldots,m\_\{B\}\\\\ m\_\{i\}\\in\\\{1,\\ldots,M\\\}\\end\{subarray\}\}2B⁡\(B−1\)​∑i<jdDINO​\(x^i,mi,x^j,mj\)\\displaystyle\\frac\{2\}\{B\(B\-1\)\}\\sum\_\{i<j\}d\_\{\\mathrm\{DINO\}\}\\left\(\\widehat\{x\}\_\{i,m\_\{i\}\},\\widehat\{x\}\_\{j,m\_\{j\}\}\\right\)\(11\)subject to\\displaystyle\\text\{subject to\}sCLIP\(x^i,mi,C\)≥sCLIP\(x^i,0,C\)−δCLIP,i=1,…,B\.\\displaystyle s\_\{\\mathrm\{CLIP\}\}\\left\(\\widehat\{x\}\_\{i,m\_\{i\}\},C\\right\)\\geq s\_\{\\mathrm\{CLIP\}\}\\left\(\\widehat\{x\}\_\{i,0\},C\\right\)\-\\delta\_\{\\mathrm\{CLIP\}\},\\qquad i=1,\\ldots,B\.
The selected magnitudes areρi⋆=ρmi⋆\\rho\_\{i\}^\{\\star\}=\\rho\_\{m\_\{i\}^\{\\star\}\}\.

### B\.4Scheduled reverse sampling

After selectingρi⋆\\rho\_\{i\}^\{\\star\}, sampling is restarted from the original initial latents\. The perturbation envelope is

α⁡\(t\)=\{1,t≥TS,\(t−TETS−TE\)p,TE<t<TS,0,t≤TE\.\\alpha\(t\)=\\begin\{cases\}1,&t\\geq T\_\{S\},\\\\\[5\.69054pt\] \\left\(\\dfrac\{t\-T\_\{E\}\}\{T\_\{S\}\-T\_\{E\}\}\\right\)^\{p\},&T\_\{E\}<t<T\_\{S\},\\\\\[8\.53581pt\] 0,&t\\leq T\_\{E\}\.\\end\{cases\}At each reverse step, the particle\-specific positive condition is

Ci,t\+=C0\+\+α⁡\(t\)​ρi⋆​ΔC​i,tCi,t−=C0−\.C\_\{i,t\}^\{\+\}=C\_\{0\}^\{\+\}\+\\alpha\(t\)\\rho\_\{i\}^\{\\star\}\\Delta\_\{Ci,t\}\\qquad C\_\{i,t\}^\{\-\}=C\_\{0\}^\{\-\}\.The resulting pair is used in the standard CFG prediction, followed by one DDIM update\.

## Appendix CExperimental Details

#### Tasks and datasets

For complete text\-to\-image generation, we use captions from the MSCOCO 2017 validation\([Lin et al\., 2014](https://arxiv.org/html/2610.14294#bib.bib24)\)\. For image inpainting, we use the BrushBench validation set introduced with BrushNet\([Ju et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib14)\)\. Each inpainting example contains a source image, a binary edit mask, and a text condition describing the expected content\. We evaluate 500 conditions from each benchmark and generate five images per condition, resulting in 2,500 generated images for every method on each task\.

#### Configuration

Complete generation is performed with Stable Diffusion 1\.5\. The inpainting evaluation uses the corresponding Stable Diffusion 1\.5 and XL BrushNet backbone\([Ju et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib14)\)\. We use 50 DDIM sampling steps, classifier\-free guidance with scale7\.57\.5, andη=0\\eta=0\. We follow the publicly available default configurations of the corresponding models\.

For every condition, all compared methods receive exactly the same initial latent particles\. All experiments use fixed random seeds to ensure fair and model\-consistent comparisons\.

#### Compared methods

We compare the following sampling procedures:

- •Clean DDIM: standard DDIM sampling without condition perturbation;
- •CADS: condition annealing with isotropic Gaussian perturbations\([Sadat et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib8)\);
- •TPSO: token\-prompt embedding space optimization applied during sampling\([Meng et al\., 2025](https://arxiv.org/html/2610.14294#bib.bib13)\);
- •Pullback: our method, which perturbs the condition along dominant pullback directions and uses particle\-specific perturbation magnitudes\.

### C\.1Evaluation Metrics

The evaluation measures three complementary properties: semantic alignment, perceptual diversity, and set\-level diversity\.

#### Text–image alignment\.

Text–image consistency is evaluated using the CLIP Score\([Hessel et al\., 2021](https://arxiv.org/html/2610.14294#bib.bib10)\), defined as the cosine similarity between CLIP embeddings of the generated image and its corresponding prompt\. Higher values indicate stronger semantic alignment with the prompt\.

#### Semantic diversity\.

We extract DINOv2 image representations\([Oquab et al\., 2023](https://arxiv.org/html/2610.14294#bib.bib12)\)and compute the mean pairwise cosine similarity within each set of samples: Lower mean similarity indicates greater overall semantic diversity\.

#### Set\-level diversity\.

We report the Mean Similarity Score \(MSS\) and Vendi Score\([Friedman and Dieng, 2022](https://arxiv.org/html/2610.14294#bib.bib11)\)for per\-prompt \(intra\-set\) diversity, both computed using SSCD features following\([Sadat et al\., 2024](https://arxiv.org/html/2610.14294#bib.bib8)\)\. Lower MSS and higher Vendi indicates greater within\-condition diversity\.

![Refer to caption](https://arxiv.org/html/2610.14294v1/figures/inpainting_comparison_source.png)

Figure 2:Illustration of the text\-guided inpainting task\. Given a text prompt, the model regenerates the masked region while preserving the unmasked region\.The prompts are*“a small bird perched on top of a rock”*\(top\) and*“a young girl holding a husky dog on a black background”*\(bottom\)\.

![Refer to caption](https://arxiv.org/html/2610.14294v1/figures/condition_influence.png)

Prompt\. “a red car with tinted windows driving through the desert, photorealistic, 8k”

Figure 3:Token\-dependent cross\-attention at DDIM timestept=499t=499\. Each visualization averages 28 U\-Net cross\-attention maps over attention heads and layers\. Warmer regions receive greater relative attention for the corresponding token\.

## Appendix DFurther Experimental Details

### D\.1Feature\-space and conditioning\-space pullback directions

Park et al\.[Park et al\. \(2023\)](https://arxiv.org/html/2610.14294#bib.bib23)use the pullback metric to identify locally sensitive directions in the noisy latent space of a diffusion model\. Their construction relies on the observation that the intermediate representations of the denoising model encode semantic and spatial information about the generated image\. Let

ft​\(⋅,c\):𝒳t→ℋtf\_\{t\}\(\\cdot;c\):\\mathcal\{X\}\_\{t\}\\rightarrow\\mathcal\{H\}\_\{t\}\(12\)denote the mapping from the noisy latentxtx\_\{t\}to a selected intermediate U\-Net feature representationht=ft​\(xt,c\)h\_\{t\}=f\_\{t\}\(x\_\{t\};c\)under conditioncc\. Its Jacobian and induced pullback metric are

Jx=∂ft​\(xt,c\)∂xt,Gx=Jx⊤​Jx\.J\_\{x\}=\\frac\{\\partial f\_\{t\}\(x\_\{t\};c\)\}\{\\partial x\_\{t\}\},\\qquad G\_\{x\}=J\_\{x\}^\{\\top\}J\_\{x\}\.\(13\)The dominant eigenvectors\{vk\}\\\{v\_\{k\}\\\}ofGxG\_\{x\}, identify input directions that produce large local variations in the feature representation\.

For directionvkv\_\{k\}and strengthss, we construct the norm\-matched latent perturbation

δ​xt,k​\(s\)=s​∥xt∥2∥vk∥2​vk\.\\delta x\_\{t,k\}\(s\)=s\\frac\{\\lVert x\_\{t\}\\rVert\_\{2\}\}\{\\lVert v\_\{k\}\\rVert\_\{2\}\}v\_\{k\}\.\(14\)Following thexx\-space guidance formulation of Park et al\.[Park et al\. \(2023\)](https://arxiv.org/html/2610.14294#bib.bib23), the perturbed latent is

x~t,k​\(s\)=xt\+γ⁡\[ϵθ​\(xt\+δ​xt,k​\(s\),t,c\)−ϵθ​\(xt,t,c\)\],\\widetilde\{x\}\_\{t,k\}\(s\)=x\_\{t\}\+\\gamma\\left\[\\epsilon\_\{\\theta\}\\bigl\(x\_\{t\}\+\\delta x\_\{t,k\}\(s\),t,c\\bigr\)\-\\epsilon\_\{\\theta\}\(x\_\{t\},t,c\)\\right\],\(15\)whereγ\\gammacontrols the magnitude of the noise\-prediction response\.

We compare this construction with the conditioning\-space pullback\. For a fixed latentxtx\_\{t\}, we consider the condition\-response mapping

gt​\(C\)=ϵθ​\(xt,t,C\),g\_\{t\}\(C\)=\\epsilon\_\{\\theta\}\(x\_\{t\},t,C\),\(16\)with Jacobian and pullback metric

JC=∂ϵθ​\(xt,t,C\)∂C,GC=JC⊤​JC\.J\_\{C\}=\\frac\{\\partial\\epsilon\_\{\\theta\}\(x\_\{t\},t,C\)\}\{\\partial C\},\\qquad G\_\{C\}=J\_\{C\}^\{\\top\}J\_\{C\}\.\(17\)The dominant eigenvectors\{dk\}\\\{d\_\{k\}\\\}ofGCG\_\{C\}identify condition directions that induce large changes in the predicted noise\.

For both constructions, we compute a rank\-44approximation of the pullback metric at the same latent and retain the four dominant directions\. We keep others parameters same for both\. For the latent\-feature pullback, we evaluate

s∈\{0\.005,0\.015,0\.02,0\.025,0\.03,0\.04,0\.05\},s\\in\\\{0\.005,0\.015,0\.02,0\.025,0\.03,0\.04,0\.05\\\},\(18\)withγ=5\\gamma=5\. For the conditioning pullback, we evaluate

s∈\{0\.1,0\.25,0\.5,1\.0,1\.5,2\.0,2\.5\}\.s\\in\\\{0\.1,0\.25,0\.5,1\.0,1\.5,2\.0,2\.5\\\}\.\(19\)The numerical values ofssare not directly comparable across the two methods because the perturbations operate in different spaces with different sensitivity scales\. We therefore select a specific range of values that clearly reveals the progressive effect of the perturbation on the generated image\. The experiment compares how the output evolves within each method as the perturbation strength increases\.

#### Qualitative observations\.

The latent\-feature pullback primarily produces localized variations and does not induce substantial semantic diversity\. Increasing the perturbation strength does not necessarily amplify these variations in a meaningful way; instead, it progressively introduces high\-contrast contours, geometric distortions, and other off\-manifold artifacts\. In contrast, the conditioning\-space pullback produces broader semantic transformations that affect a larger portion of the image\. These transformations remain direction\-dependent across the strength sweep, with different pullback directions controlling distinct semantic attributes\.

![Refer to caption](https://arxiv.org/html/2610.14294v1/figures/feature_direction_strength_grid.png)Figure 4:Effect of the perturbation strength along latent\-feature pullback directions\. Rows correspond to the four dominant directionsv1,…,v4v\_\{1\},\\ldots,v\_\{4\}of the pullback metricGx=Jx⊤​JxG\_\{x\}=J\_\{x\}^\{\\top\}J\_\{x\}, whereJxJ\_\{x\}is the Jacobian of the selected U\-Net feature representation with respect to the noisy latentxtx\_\{t\}\. Columns show the clean sample followed by increasing strengths\. Moderate strengths produce predominantly localized, whereas larger strengths introduce visible distortions\.![Refer to caption](https://arxiv.org/html/2610.14294v1/figures/conditioning_direction_strength_grid.png)Figure 5:Effect of the perturbation strength along conditioning\-space pullback directions\. Rows correspond to the four dominant directions of the condition pullback metricGC=JC⊤​JCG\_\{C\}=J\_\{C\}^\{\\top\}J\_\{C\}, whereJC=∂ϵθ​\(xt,t,C\)/∂CJ\_\{C\}=\\partial\\epsilon\_\{\\theta\}\(x\_\{t\},t,C\)/\\partial C\. Columns show the clean sample followed by increasing strengths\. The different directions produce distinct global semantic transformations, while large strengths can reduce image quality and fidelity\.![Refer to caption](https://arxiv.org/html/2610.14294v1/figures/combined_feature_conditioning_directions.png)Figure 6:Qualitative comparison between latent\-feature and conditioning\-space pullback directions computed from the same anchor\. The top row shows the clean sample and the four dominant latent\-feature directions ats=0\.025s=0\.025, using anxx\-space guidance scale ofγ=5\\gamma=5\. The bottom row shows the clean sample and the four dominant conditioning directions ats=1s=1\. In this example, the latent\-feature directions primarily affect localized appearance, pose, and nearby image structure, whereas the conditioning\-space directions induce broader semantic changes\. All results use the same prompt, initial noise, and reverse\-process parameters\.