Multi-Branch Transformer

Hugging Face Daily Papers Papers

Summary

The Multi-Branch Transformer (MBT) splits Transformer blocks into parallel parameterized branches with shared initial/final layers, and its Balanced variant with a shared Joint-KV cache outperforms sequential Transformers at matched compute while reducing KV-cache storage by 34% and boosting decoding throughput by up to 51.4% on an H200.

Transformer language models are commonly scaled by making their blocks wider or stacking them deeper. In this work, we explore a complementary path: branching, the parallel composition of full Transformer blocks. We introduce the Multi-Branch Transformer (MBT), which processes the input through shared initial layers, executes independently parameterized branches of sequential blocks in parallel, and fuses their outputs before shared final layers. At comparable training compute, we study three MBT variants: Wide-Shallow preserves block width, Narrow-Deep preserves sequential depth, and Balanced interpolates between them. In controlled pretraining experiments at four mixture-of-experts scales (0.8B to 2.1B total parameters), 2-branch Wide-Shallow MBTs consistently outperform sequential Transformers under matched parameter and training-compute budgets. Balanced MBTs improve further and retain comparable quality as the number of branches increases. To improve inference efficiency at long contexts, we introduce Joint-KV, which constructs a shared KV cache from branch representations at each depth. Our final recipe combines Balanced MBTs with Joint-KV, supported by an inference engine that exploits branch grouping and shared-cache reuse. Fitted scaling curves indicate that standard Transformers require up to 1.26x as much training body compute to match our final models' validation loss. These models reduce full-model KV-cache storage by up to 34% and deliver up to 51.4% higher decoding throughput on a single H200 GPU. Ablation studies show that branching yields gains beyond changes in width and depth, enables an effective KV-sharing strategy (Joint-KV), remains robust across fusion functions, and performs best in the middle layers. These findings show that branching offers a practical path for scaling language models toward both better modeling quality and faster inference.
Original Article
View Cached Full Text

Cached at: 11/19/26, 03:42 AM

Paper page - Multi-Branch Transformer

Source: https://huggingface.co/papers/2611.19800

Abstract

Branching Transformer blocks into parallel parameterized branches, combined with shared KV caching, improves language model quality and inference efficiency compared with standard sequential Transformers.

Transformer language models are commonly scaled by making their blocks wider or stacking them deeper. In this work, we explore a complementary path:branching, the parallel composition of full Transformer blocks. We introduce theMulti-Branch Transformer (MBT), which processes the input throughshared initial layers, executesindependently parameterized branchesofsequential blocksin parallel, and fuses their outputs beforeshared final layers. At comparable training compute, we study three MBT variants:Wide-Shallowpreserves block width,Narrow-Deeppreserves sequential depth, andBalancedinterpolates between them. In controlled pretraining experiments at fourmixture-of-experts scales(0.8B to 2.1B total parameters), 2-branchWide-ShallowMBTs consistently outperformsequential Transformersunder matched parameter and training-compute budgets.BalancedMBTs improve further and retain comparable quality as the number of branches increases. To improve inference efficiency at long contexts, we introduceJoint-KV, which constructs ashared KV cachefrombranch representationsat each depth. Our final recipe combinesBalancedMBTs withJoint-KV, supported by aninference enginethat exploitsbranch groupingandshared-cache reuse. Fittedscaling curvesindicate that standard Transformers require up to 1.26x as much training body compute to match our final models’validation loss. These models reduce full-modelKV-cache storageby up to 34% and deliver up to 51.4% higherdecoding throughputon a single H200 GPU. Ablation studies show thatbranchingyields gains beyond changes in width and depth, enables an effective KV-sharing strategy (Joint-KV), remains robust acrossfusion functions, and performs best in themiddle layers. These findings show thatbranchingoffers a practical path for scaling language models toward both better modeling quality and faster inference.

View arXiv pageView PDFProject pageGitHub9Add to collection

Get this paper in your agent:

hf papers read 2611\.19800

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2611.19800 in a model README.md to link it from this page.

Datasets citing this paper1

#### gaotang/MBT-Validation Viewer• Updated2 days ago • 12.2k • 29

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2611.19800 in a Space README.md to link it from this page.

Collections including this paper3

Similar Articles

BCMT: Blockwise Causal Memory Transformer

arXiv cs.CL

BCMT introduces a blockwise causal memory transformer architecture that decouples local and global dependencies in long-context language modeling, improving efficiency while maintaining performance comparable to dense transformers.

Block-Based Double Decoders

arXiv cs.LG

Proposes block-based double decoders, a novel transformer architecture using doubly-causal block-based attention masks to combine decoder-only training efficiency with encoder-decoder inference efficiency, achieving strong scaling performance and reduced KV-cache memory.

Variable-Width Transformers

Hugging Face Daily Papers

Proposes a nonuniform width allocation transformer (hourglass shape) that outperforms uniform baselines in language modeling, reducing FLOPs and KV cache size.

kvcache-ai/ktransformers

GitHub Trending (daily)

KTransformers is a flexible research framework for cutting-edge LLM inference and fine-tuning, leveraging CPU-GPU heterogeneous computing and supporting many recent models.