Confidence-Adaptive SwiGLU for Mixture-of-Experts

Hugging Face Daily Papers Papers
mixture-of-experts swiglu confidence-adaptive routing transformer gating-function

Summary

Proposes Confidence-Aware SwiGLU (κ-SwiGLU) that adjusts expert gate sharpness in Mixture-of-Experts models based on token-level routing confidence, improving performance with minimal computational overhead.

SwiGLU has become a standard gated activation in modern Transformer MLPs, yet its gate sharpness -- the smoothness and selectivity of the gating function -- is typically fixed throughout training. In this work, we propose Confidence-Aware SwiGLU (κ-SwiGLU), a variant of SwiGLU for Mixture-of-Experts (MoE) models that adjusts expert gate sharpness according to token-level routing confidence. Specifically, κ-SwiGLU parameterizes the SiLU gate sharpness coefficient as a learnable function of the router logit, enabling each expert gate unit to interpolate between smooth, broadly active gating and sharp, selective gating. We evaluate κ-SwiGLU on the FineWeb-Edu dataset across MoE Transformer models ranging from 8 to 28 layers. Across these settings, κ-SwiGLU improves mean CORE performance while adding negligible parameters and incurring only a small computational overhead, demonstrating that confidence-aware gate sharpness is a promising mechanism for improving MoE MLPs. The code is available at https://github.com/askerlee/kappa-swiglu.
Original Article
View Cached Full Text

Cached at: 06/02/26, 03:36 PM

Paper page - Confidence-Adaptive SwiGLU for Mixture-of-Experts

Source: https://huggingface.co/papers/2606.00761

Abstract

Confidence-Aware SwiGLU adjusts expert gate sharpness in Mixture-of-Experts models based on token-level routing confidence, improving performance with minimal computational overhead.

SwiGLUhas become a standard gated activation in modern Transformer MLPs, yet its gate sharpness -- the smoothness and selectivity of thegating function-- is typically fixed throughout training. In this work, we propose Confidence-AwareSwiGLU(κ-SwiGLU), a variant ofSwiGLUforMixture-of-Experts(MoE) models that adjusts expert gate sharpness according totoken-level routing confidence. Specifically, κ-SwiGLUparameterizes theSiLUgate sharpness coefficient as a learnable function of therouter logit, enabling each expert gate unit to interpolate between smooth, broadly active gating and sharp, selective gating. We evaluate κ-SwiGLUon the FineWeb-Edu dataset acrossMoETransformer models ranging from 8 to 28 layers. Across these settings, κ-SwiGLUimproves meanCORE performancewhile adding negligible parameters and incurring only a small computational overhead, demonstrating that confidence-aware gate sharpness is a promising mechanism for improvingMoEMLPs. The code is available at https://github.com/askerlee/kappa-swiglu.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2606\.00761

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.00761 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.00761 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.00761 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts

arXiv cs.CL

该论文提出 Elastic Expert Routing，通过从以 k 为中心的局部离散分布中随机采样激活专家数量，平滑传统 top-k 路由的硬边界，在保持计算成本不变的前提下提升稀疏 MoE 模型在微调与从头预训练场景下的下游表现。

Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

Hugging Face Daily Papers

This paper introduces the Expert Subspace Separation Index (ESSI) to disentangle route coherence, candidate quality, and candidate-by-context interaction in sparse Mixture-of-Experts language models, revealing a pattern of coherent overlap where routing selects token-relevant experts from a shared geometric neighborhood yet multi-expert computation remains useful.

Introducing Laguna XS 2.1 (5 minute read)

TLDR AI

Poolside releases Laguna XS 2.1, a 33B parameter Mixture-of-Experts model with 3B activated parameters per token, designed for agentic coding, with improvements on SWE-bench Multilingual and other benchmarks, now available under the permissive OpenMDW-1.1 license.