Deep Dreams Are Made of This: Visualizing Monosemantic Features in Diffusion Models
Summary
This paper introduces Latent Visualization by Optimization (LVO), a mechanistic interpretability technique that uses sparse autoencoders to visualize monosemantic features in diffusion models like Stable Diffusion 1.5.
View Cached Full Text
Cached at: 05/12/26, 07:07 AM
# Deep Dreams Are Made of This: Visualizing Monosemantic Features in Diffusion Models Source: [https://arxiv.org/abs/2605.08218](https://arxiv.org/abs/2605.08218) [View PDF](https://arxiv.org/pdf/2605.08218) > Abstract:This paper proposes latent visualization by optimization \(LVO\), a mechanistic interpretability technique that extends feature visualization by optimization \- originally developed for convolutional neural networks \- to latent diffusion models\. LVO employs sparse autoencoders \(SAEs\) to disentangle polysemantic layer representations into monosemantic features\. Key contributions include latent\-space optimization, time\-step activity analysis, schedule\-matched noise injection, prior initialization through feature steering, and suitable regularization strategies\. We demonstrate the method on Stable Diffusion 1\.5 fine\-tuned on the Style50 dataset, showing that SAE features produce clear visualizations of recognizable concepts \- including diagonal compositions, human figures, roses, cables, and waterfall foam \- that correlate with dataset examples, while the baseline without disentanglement produces less coherent results\. We further show that regularization techniques from pixel\-space feature visualization transfer to the latent domain, though they require different configurations for the raw\-layer and SAE variants\. Compared to dataset examples and steering, LVO provides complementary insights by directly revealing what activates a feature rather than its downstream effects\. ## Submission history From: Adam Szokalski \[[view email](https://arxiv.org/show-email/9097c59b/2605.08218)\] **\[v1\]**Wed, 6 May 2026 13:21:24 UTC \(26,164 KB\)
Similar Articles
Subliminal Clocks: Latent Time Modelling in Diffusion Language Models
This paper demonstrates that diffusion language models (DLMs) internally represent denoising progress as a latent timestep signal in their residual streams, which can be probed and steered to modulate model confidence and entropy.
On the Diffusibility of High-Dimensional Latents
This paper shows that fine-tuning autoencoders for reconstruction reduces effective dimensionality, making standard velocity prediction inefficient in diffusion models, and proposes using x0-prediction to focus on the signal manifold, consistently improving text-to-image generation.
@volokuleshov: New blog post: How to Build a Diffusion Language Model. Diffusion LLMs went from open problem to reality in 2 years (Me…
A comprehensive blog post by Volodymyr Kuleshov's Cornell group explains how to build diffusion language models, covering core techniques like masked diffusion, iterative refinement, variable-length generation, controllable generation, fast samplers, and RL post-training, using open-source models such as Mercury, Gemma Diffusion, and Nemotron Diffusion as examples.
Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design
This paper theoretically analyzes diffusion language models through a bias-variance lens, identifying trade-offs between masking and uniform diffusion kernels. It proposes SemDLM+, which adds a global transition and semantic-frequency penalty to overcome the semantic basin problem, achieving competitive generation quality on LM1B and OpenWebText benchmarks.
Concept-based Visual Counterfactual Explanations with Diffusion Models
Introduces C-VCE, a diffusion framework that builds an interpretable concept bottleneck layer into the generative model, enabling human-guided visual counterfactual explanations without relying on external noise-robust classifiers.