Dynamic Multi-Byte Prediction With Hierarchical Language Models
Summary
The paper introduces multi-byte prediction to speed up inference in byte-level language models by generating multiple bytes in parallel with minimal performance impact.
View Cached Full Text
Cached at: 08/19/26, 03:59 AM
Paper page - Dynamic Multi-Byte Prediction With Hierarchical Language Models
Source: https://huggingface.co/papers/2608.15454
Abstract
Multi-byte prediction accelerates byte-level hierarchical language models by generating parallel bytes via variable-length windows and causal attention masking, improving inference speed with minimal quality loss.
Byte-level hierarchical language models(LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introducemulti-byte prediction(MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popularmulti-token prediction(MTP) paradigm with two crucial innovations. First, we introduce avariable-length prediction windowthat aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novelattention-maskingscheme that enables parallel byte prediction without violating causality. We show thatmulti-byte predictionstrikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance andinference throughput.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2608\.15454
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.15454 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.15454 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.15454 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Hierarchical Latent Prediction for Language Models
This paper introduces HiLP, a hierarchical representation training method that adds multi-scale self-predictive learning to transformer pretraining, aiming to reduce compounding error and improve long-horizon reasoning and speculative decoding efficiency.
Fast Byte Latent Transformer
This paper introduces BLT Diffusion and speculative decoding techniques for byte-level language models to significantly reduce generation latency and memory bandwidth costs while maintaining quality.
Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors
This paper introduces HYMELL, a hybrid analytical-machine-learning framework for estimating LLM inference latency and energy across prefill and decode phases, validated on NVIDIA H100 with under 5% error for LLaMA 3 8B.
Training Hybrid Block Diffusion Language Models with Partial Bidirectionality
This paper proposes a hybrid Mamba-attention architecture for block diffusion language models that restricts reverse Mamba scans to the active denoising block, enabling exact caching across blocks and achieving high throughput for long-context generation.
Multi-Block Diffusion Language Models
This paper proposes Multi-Block Diffusion Language Models (MBD-LMs), extending single-block diffusion to concurrent multi-block decoding with improved training strategies like Multi-block Teacher Forcing and an optimized Block Buffer decoding algorithm. Experiments show increased tokens per forward pass and improved accuracy on benchmarks.