TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning

Hugging Face Daily Papers Papers
token-pruning long-context inference-efficiency transformers llm prefill attention

Summary

TokenGate 是一个轻量级可学习的 token 剪枝模块，通过 token-gated attention 机制在长上下文预填充阶段跳过不重要的 token，实现 1.4–11.3 倍的 TTFT 加速和 1.5–3.4 倍的峰值内存降低且无精度损失。

Expert human readers process large inputs (e.g., code repositories and long documents) by skimming to closely read only what matters. On the other hand, language models process all input tokens through each model layer, resulting in significant processing costs for long-context inputs. We introduce TokenGate, a small, drop-in component that extends existing trained models to enable end-to-end learnable token pruning over long-context inputs for fast and memory-efficient prefill. To enable end-to-end learning despite the non-differentiable top-K operator for token pruning, we introduce token-gated attention, a mechanism inspired by Mixture-of-Experts that re-weights attention logits using token importance scores. Across diverse tasks, TokenGate achieves 1.4-11.3times faster time-to-first token (TTFT) and 1.5-3.4times lower peak memory at no accuracy degradation for prefill in transformers. Compared to the best-performing prior methods for token-pruning, TokenGate gives up to 6.4times faster prefill and 2times lower peak memory. TokenGate natively extends both standard transformers (e.g., Qwen3) and hybrid-attention models (e.g., Qwen3.5) with simple lightweight adaptation comparable to standard parameter-efficient fine-tuning. We make our code available at: https://github.com/tokengate-models/TokenGate.
Original Article
View Cached Full Text

Cached at: 10/21/26, 07:04 AM

Paper page - TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning

Source: https://huggingface.co/papers/2610.26754 Published on Oct 19

·

Submitted byhttps://huggingface.co/liana313

LIanaon Oct 20

Abstract

TokenGate is a lightweight learnable token-pruning module that accelerates long-context prefill in transformers by selectively skipping unimportant tokens via token-gated attention.

Expert human readers process large inputs (e.g., code repositories and long documents) by skimming to closely read only what matters. On the other hand, language models process all input tokens through each model layer, resulting in significant processing costs for long-context inputs. We introduceTokenGate, a small, drop-in component that extends existing trained models to enable end-to-end learnabletoken pruningover long-context inputs for fast and memory-efficientprefill. To enable end-to-end learning despite the non-differentiable top-K operator fortoken pruning, we introducetoken-gated attention, a mechanism inspired byMixture-of-Expertsthat re-weights attention logits using token importance scores. Across diverse tasks,TokenGateachieves 1.4-11.3times faster time-to-first token (TTFT) and 1.5-3.4times lower peak memory at no accuracy degradation forprefillintransformers. Compared to the best-performing prior methods for token-pruning,TokenGategives up to 6.4times fasterprefilland 2times lower peak memory.TokenGatenatively extends both standardtransformers(e.g., Qwen3) andhybrid-attentionmodels (e.g., Qwen3.5) with simple lightweight adaptation comparable to standardparameter-efficient fine-tuning. We make our code available at: https://github.com/tokengate-models/TokenGate.

View arXiv pageView PDFAdd to collection

Community

Get this paper in your agent:

hf papers read 2610\.26754

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper6

#### liana313/tokengate-biodexr-qwen3-1.7b Text Generation• Updatedabout 7 hours ago • 2 #### liana313/tokengate-niah-qwen3-1.7b Text Generation• Updatedabout 7 hours ago #### liana313/tokengate-2wikimqa-qwen3-1.7b Text Generation• Updatedabout 7 hours ago #### liana313/tokengate-multinews-qwen3-1.7b Text Generation• Updatedabout 7 hours ago Browse 6 models citing this paper## Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.26754 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.26754 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Efficient Pre-Training with Token Superposition

Hugging Face Daily Papers

Token-Superposition Training (TST) improves LLM pre-training efficiency by combining contiguous tokens into bags during a superposition phase with a multi-hot cross-entropy objective, achieving up to 2.5x reduction in training time without architectural changes.

Optimizing Korean-Centric LLMs via Token Pruning

arXiv cs.CL

This paper presents a systematic benchmark of token pruning—a compression technique that removes tokens and embeddings for irrelevant languages—applied to Korean-centric LLM tasks. The study evaluates popular multilingual models (Qwen3, Gemma-3, Llama-3, Aya) across different vocabulary configurations and finds that token pruning significantly improves generation stability and reduces memory footprint for domain-specific deployments.

Stage-adaptive Token Selection for Efficient Omni-modal LLMs

Hugging Face Daily Papers

SEATS is a training-free, stage-adaptive token selection method that reduces computational overhead in omni-modal LLMs by progressively pruning redundant visual and audio tokens, achieving a 9.3x FLOPs reduction and 4.8x prefill speedup while preserving 96.3% performance.