DEMON: Diffusion Engine for Musical Orchestrated Noise
Summary
DEMON presents a real-time diffusion engine that enables live musical performance by controlling the denoising process, achieving up to 12.3 decoder completions per second on a single RTX 5090. It introduces heterogeneous scheduling, shared mutable state, per-frame blending, and windowed VAE decode for responsive control.
View Cached Full Text
Cached at: 06/01/26, 07:21 PM
Paper page - DEMON: Diffusion Engine for Musical Orchestrated Noise
Source: https://huggingface.co/papers/2605.28657
Abstract
DEMON enables real-time diffusion model control as a musical instrument through specialized scheduling, shared state management, and optimized decoding techniques.
We present DEMON, a real-timediffusion enginethat makes thedenoising processplayable as a live musical instrument: a control surface both broad (many parameters shaped per-frame across the output) and responsive (each control taking effect as fast as its place in thedenoising loopallows). Built onACE-Step 1.5andStreamDiffusion’sring-buffer architecturewithTensorRT acceleration, it sustains up to 12.3 decoder completions per second for 60-second music on a single consumer GPU (RTX 5090), or 11.3 generations per second at our production ring-depth of 4. At these ratesdenoising parametersbecome viable as live performance controls, but the ring buffer propagates per-request changes only at its drain rate, a floor of S denoising steps. We contribute four mechanisms. (1) Per-slotheterogeneous denoise scheduling: each ring-buffer slot owns its timestep schedule, so a moving denoise slider is tracked without wiping the in-flight queue, where the upstream global-schedule design must rebuild and discard it. (2)Shared mutable per-step state, giving any parameter consulted at every solver step next-tick effect, bypassing ring-buffer drain. (3)Per-frame source blending: a sampling-time control on the standard SDE re-noise step, giving a framewise transformation-strength axis that complements scalar denoise scheduling. (4) WindowedVAE decodeexploitingreceptive-field analysisfor an 8.0x decode speedup. Together these separate streaming-diffusion parameters into four propagation classes, by onset and convergence latency.
View arXiv pageView PDFProject pageGitHub186Add to collection
Get this paper in your agent:
hf papers read 2605\.28657
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### daydreamlive/DEMON Audio-to-Audio• Updatedabout 4 hours ago
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.28657 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.28657 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators
This paper introduces Live Music Diffusion Models (LMDMs), which modify the diffusion process to enable efficient block-wise processing and novel training paradigms for real-time interactive music generation on consumer hardware, outperforming discrete autoregressive models in inference complexity and enabling stable post-training alignment.
Roll Out and Roll Back: Diffusion LLMs are Their Own Efficiency Teachers
This paper introduces WINO and WINO+, methods that enable revokable parallel decoding in diffusion LLMs and distill efficient denoising trajectories, significantly improving the quality-speed trade-off.
Nemotron-Labs-Diffusion from NVIDIA
NVIDIA released the Nemotron-Labs-Diffusion model family (3B to 14B) that supports both AR and diffusion decoding with novel self-speculation, achieving significant speedups (up to 4x) over standard AR and Eagle3 methods across hardware platforms.
IFM/K2-Horizon-7B-Uno · Hugging Face - 5200tps with no quality loss
The article presents K2-Horizon-7B-Uno, a diffusion-augmented LLM that combines autoregressive and diffusion pathways to achieve 5200 tokens per second throughput without quality loss, with benchmarks showing competitive performance across various tasks.
Stefano Ermon: Autoregressive inference is sequential and memory-bound. Diffusion is built to map to GPUs — that's why it wins.
Augment Code switched its coding-agent backend to Stefano Ermon's Mercury 2.5 diffusion model, achieving 82% latency reduction and 90% cost cut in production. The article highlights the performance advantages of diffusion models and the need for independent AI benchmarking tools.