Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention

Hugging Face Daily Papers Papers
mixture-of-experts grouped-query-attention transformer efficiency attention query-heads routing

Summary

Grouped Query Experts (GQE) improves Transformer efficiency by applying a mixture-of-experts layer on top of grouped-query attention, selectively activating query heads per token while keeping key-value cache benefits, matching baseline accuracy with half the query-head compute at 250M parameter scale.

Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.
Original Article
View Cached Full Text

Cached at: 06/23/26, 01:43 PM

Paper page - Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention

Source: https://huggingface.co/papers/2606.20945

Abstract

Grouped Query Experts (GQE) improves Transformer efficiency by selectively activating query heads based on token content while maintaining key-value cache benefits of grouped-query attention.

Self-attentionis central toTransformerperformance and is often the most expensive part of theTransformerat long context lengths because its pairwisetoken interactionsscale quadratically with sequence length. Standard dense attention also applies the same set ofattention headsto every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer andattention costincreases rapidly. We propose Grouped Query Experts (GQE), amixture-of-expertslayer on top ofgrouped-query attention(GQA). Within each GQA group, arouterselects kquery-head expertsper token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2606\.20945

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.20945 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.20945 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.20945 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

GQA-{\mu}P: The maximal parameterization update for grouped query attention

arXiv cs.LG

This paper extends the maximal update parameterization (μP) framework to grouped-query attention (GQA), deriving scaling laws for hyperparameter transfer across model architectures. It introduces spectral norm conditions for feature learning and addresses issues with low-rank weight matrices in GQA.

GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding

arXiv cs.LG

GQLA proposes a minimal modification to Multi-head Latent Attention (MLA) that exposes both an MQA-absorb path and a GQA path over the same trained weights, enabling hardware-adaptive decoding without retraining. The method compresses KV cache and supports tensor parallelism, demonstrated by converting LLaMA-3-8B from GQA to GQLA.

Mixture of Experts (MoEs) in Transformers

Hugging Face Blog

Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.

Hierarchical Global Attention (HGA)

arXiv cs.LG

Hierarchical Global Attention (HGA) is a drop-in replacement for dense causal attention in pretrained long-context transformers that uses hierarchical two-level routing to enable exact attention over a small routed working set, allowing models like Qwen3-30B to run at 64K context on a single RTX 5090 with minimal quality loss.