Multi-Branch Transformer
Summary
The Multi-Branch Transformer (MBT) splits Transformer blocks into parallel parameterized branches with shared initial/final layers, and its Balanced variant with a shared Joint-KV cache outperforms sequential Transformers at matched compute while reducing KV-cache storage by 34% and boosting decoding throughput by up to 51.4% on an H200.
View Cached Full Text
Cached at: 11/19/26, 03:42 AM
Paper page - Multi-Branch Transformer
Source: https://huggingface.co/papers/2611.19800
Abstract
Branching Transformer blocks into parallel parameterized branches, combined with shared KV caching, improves language model quality and inference efficiency compared with standard sequential Transformers.
Transformer language models are commonly scaled by making their blocks wider or stacking them deeper. In this work, we explore a complementary path:branching, the parallel composition of full Transformer blocks. We introduce theMulti-Branch Transformer (MBT), which processes the input throughshared initial layers, executesindependently parameterized branchesofsequential blocksin parallel, and fuses their outputs beforeshared final layers. At comparable training compute, we study three MBT variants:Wide-Shallowpreserves block width,Narrow-Deeppreserves sequential depth, andBalancedinterpolates between them. In controlled pretraining experiments at fourmixture-of-experts scales(0.8B to 2.1B total parameters), 2-branchWide-ShallowMBTs consistently outperformsequential Transformersunder matched parameter and training-compute budgets.BalancedMBTs improve further and retain comparable quality as the number of branches increases. To improve inference efficiency at long contexts, we introduceJoint-KV, which constructs ashared KV cachefrombranch representationsat each depth. Our final recipe combinesBalancedMBTs withJoint-KV, supported by aninference enginethat exploitsbranch groupingandshared-cache reuse. Fittedscaling curvesindicate that standard Transformers require up to 1.26x as much training body compute to match our final models’validation loss. These models reduce full-modelKV-cache storageby up to 34% and deliver up to 51.4% higherdecoding throughputon a single H200 GPU. Ablation studies show thatbranchingyields gains beyond changes in width and depth, enables an effective KV-sharing strategy (Joint-KV), remains robust acrossfusion functions, and performs best in themiddle layers. These findings show thatbranchingoffers a practical path for scaling language models toward both better modeling quality and faster inference.
View arXiv pageView PDFProject pageGitHub9Add to collection
Get this paper in your agent:
hf papers read 2611\.19800
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2611.19800 in a model README.md to link it from this page.
Datasets citing this paper1
#### gaotang/MBT-Validation Viewer• Updated2 days ago • 12.2k • 29
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2611.19800 in a Space README.md to link it from this page.
Collections including this paper3
Similar Articles
BCMT: Blockwise Causal Memory Transformer
BCMT introduces a blockwise causal memory transformer architecture that decouples local and global dependencies in long-context language modeling, improving efficiency while maintaining performance comparable to dense transformers.
Block-Based Double Decoders
Proposes block-based double decoders, a novel transformer architecture using doubly-causal block-based attention masks to combine decoder-only training efficiency with encoder-decoder inference efficiency, achieving strong scaling performance and reduced KV-cache memory.
Variable-Width Transformers
Proposes a nonuniform width allocation transformer (hourglass shape) that outperforms uniform baselines in language modeling, reducing FLOPs and KV cache size.
kvcache-ai/ktransformers
KTransformers is a flexible research framework for cutting-edge LLM inference and fine-tuning, leveraging CPU-GPU heterogeneous computing and supporting many recent models.
Dual-Flow Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation
This paper introduces Dual-Flow Transformers, an architecture that decouples prefill and decode computation by adding an auxiliary flow for continuation prediction while sharing weights and the primary KV cache, enabling phase-specific compute allocation and improved efficiency.