Orthrus-Qwen3-8B : up to 7.8×tokens/forward on Qwen3-8B, frozen backbone, provably identical output distribution

Reddit r/LocalLLaMA Papers
inference-acceleration diffusion attention-module speculative-decoding frozen-backbone parallel-decoding qwen3

Summary

Introduces Orthrus, a method that injects a trainable diffusion attention module into a frozen autoregressive transformer to achieve up to 7.8× tokens per forward pass and ~6× wall-clock speedup on MATH-500, with provably identical output distribution to the base Qwen3-8B model. The approach requires minimal additional parameters and training, and avoids the TTFT penalty of external drafters.

* Code: [https://github.com/chiennv2000/orthrus](https://github.com/chiennv2000/orthrus) * Paper: [https://arxiv.org/abs/2605.12825](https://arxiv.org/abs/2605.12825) * HF: [https://huggingface.co/chiennv/Orthrus-Qwen3-1.7B](https://huggingface.co/chiennv/Orthrus-Qwen3-1.7B) ; [https://huggingface.co/chiennv/Orthrus-Qwen3-4B](https://huggingface.co/chiennv/Orthrus-Qwen3-4B) ; [https://huggingface.co/chiennv/Orthrus-Qwen3-8B](https://huggingface.co/chiennv/Orthrus-Qwen3-8B) * Disclosure: co-author. Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the base model. Results: * Up to 7.8× TPF, \~6× wall-clock on MATH-500. * 16% of params trained, <1B tokens, 24h on 8×H200. * vs. diffusion LMs (Dream, Fast-dLLM-v2, SDAR, Mercury, Gemini Diffusion): they modify base weights and lose accuracy (Fast-dLLM-v2: -11 pts on MATH-500). Orthrus freezes the backbone; accuracy matches Qwen3-8B exactly. * vs. Speculative Decoding (EAGLE-3, DFlash): No external drafter, no separate cache, and zero Time-To-First-Token (TTFT) penalty because we don't have to initialize and sync a separate drafter model. KV overhead is O(1) (\~4.5 MiB flat). Acceptance length on MATH-500: 11.7 vs. 7.9 (DFlash) vs. 3.5 (EAGLE-3). * Single-step denoising beats multi-step (6.35 vs. 3.53 TPF). KL distillation beats CE on acceptance rate. Limitations: strictly bounded by the frozen base model (inherits its biases, hallucinations, knowledge gaps); Qwen3-only evaluation; greedy + rejection sampling only.
Original Article

Similar Articles

qwen 3.6 27B AR-> Diffusion - local training on 5090

Reddit r/LocalLLaMA

The author details attempts to locally train a Qwen 3.6 27B autoregressive-to-diffusion model on an Nvidia 5090 GPU using qlora and modifications from open-dllm and d3LLM, facing VRAM constraints and hardware issues while exploring one-shot diffusion techniques.