Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

Hugging Face Daily Papers Papers
diffusion-models transformers scaling-laws chinchilla mixture-of-experts visual-generation long-context

Summary

This paper introduces Chimera, a hybrid visual diffusion backbone with a principled scaling recipe, combining Kimi Delta Attention, Multi-head Latent Attention, and sparse Mixture-of-Experts to efficiently handle long-context image and video generation. It also presents HeteroP, a module-wise hyperparameter transfer scheme, and Chinchilla-style scaling laws to train an 11B-parameter model with 2B activated parameters.

Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibitive. We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe. Chimera processes text, image, and video tokens in one raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while controlling activated compute. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and model depth. HeteroP yields a consistently tuned family used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments show three results. First, measured by pretraining diffusion loss, the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x. Second, without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds. Third, the fitted laws show that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets. These results establish a foundation for designing and scaling efficient long-context diffusion architectures.
Original Article
View Cached Full Text

Cached at: 07/31/26, 05:53 AM

Paper page - Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

Source: https://huggingface.co/papers/2607.28611 Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

Visualgenerationincreasinglyrequireshigh-resolutionimages,longvideos,andmultimodalcontext,makingthequadraticcostoffullattentionprohibitive.WeintroduceChimera,ahybridvisualdiffusionbackbonewithaprincipledscalingrecipe.Chimeraprocessestext,image,andvideotokensinoneraster-orderedstreamwithoutpositionalembeddings.ItcombinesKimiDeltaAttention(KDA)forlong-contextstatetrackingwithO(N)complexity,interleavedMulti-headLatentAttention(MLA)fordirectglobalinteraction,andmodality-awareshortconvolutionsforlocalspatiotemporalcontext.SparseMixture-of-Experts(MoE)layersexpandcapacitywhilecontrollingactivatedcompute.Toscalethisheterogeneousarchitecture,weintroduceHeteroP,amodule-wiseschemethattransfershyperparametersacrosswidthanddepthaccordingtoeachtensor’sfunctionalfan-inandmodeldepth.HeteroPyieldsaconsistentlytunedfamilyusedtofitChinchilla-stylecompute-optimallawsforactivatedmodelsize,training-tokencount,andimage-videodataratio.Guidedbytheselaws,wetrainan11B-parameterChimerawith2Bactivatedparameters.Experimentsshowthreeresults.First,measuredbypretrainingdiffusionloss,thedensebackboneis1.7xascompute-efficientasamatchedfull-attentionWan-2.12Bbaseline,whilethecompletesystemreaches7.3x.Second,withoutlength-specificfine-tuning,Chimeraextrapolateszero-shotfrom5-secondtrainingclipsto30-secondvideos,withonly6.5%FIDdegradationinthelastfiveseconds.Third,thefittedlawsshowthatcompute-optimalimagepretrainingdividescomputenearlyevenlybetweenactivatedmodelsizeandtraining-tokencount,whereasvideopretrainingmodestlyfavorsmodelsizeathigherbudgets.Theseresultsestablishafoundationfordesigningandscalingefficientlong-contextdiffusionarchitectures.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.28611

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.28611 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.28611 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.28611 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

Papers with Code Trending

VideoChat3 is a fully open, efficient, and generalist video-centric multimodal large language model that introduces Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame Resolution for streaming video perception, along with scalable video data synthesis pipelines, achieving superior performance with only 4B parameters.

DiffusionBench: On Holistic Evaluation of Diffusion Transformers

Hugging Face Daily Papers

Researchers introduce NanoGen, a unified framework for training and evaluating diffusion transformers, and propose DiffusionBench, a holistic benchmark combining ImageNet class-conditional and text-to-image generation to better assess progress in generative modeling.

@VukRosic99: Long-context Transformers hit two walls: quadratic attention compute and a KV cache that reaches hundreds of GB at 1M t…

X AI KOLs Timeline

MiniCPM-SALA is a 9B-parameter hybrid attention model that interleaves sparse and linear attention to overcome the quadratic compute and large KV cache bottlenecks of long-context Transformers. It achieves 3.5x faster inference than Qwen3-8B at 256K tokens and supports up to 1M tokens on consumer GPUs, with a cost-effective continual training approach that reduces training costs by ~75%.