Tag
清华团队发表 TokenRouter 论文，提出一种按 token 级别在大小模型间路由的 LLM 服务系统，通过独立服务器协作与 KV cache 传递，在 5 种路由方法上实现相比现有最强框架 2.01 到 64.15 倍的吞吐提升。
LRCC proposes token-dependent compute allocation by training lightweight routers over nested low-rank paths in frozen pretrained LLMs, improving perplexity and downstream accuracy over static low-rank compression at matched active-parameter budgets and decoding latency on Llama and Qwen models.
MOPD-Router proposes a token-level routing framework for multi-teacher on-policy distillation, enabling the use of unlabeled data and cross-domain supervision without domain labels. Experiments demonstrate significant performance improvements over standard methods in both unlabeled and labeled scenarios.
The article explains the engineering aspects of Mixture-of-Experts (MoE) inference, detailing token routing, expert batching, GPU distribution, and performance trade-offs for efficient serving.
The author describes a token routing system to optimize LLM usage, reducing API costs by up to 70% through techniques like cost-based routing, prompt caching, and output discipline.
This paper introduces Role-Conditioned Sub-Token Routing (RoleSub), a method to efficiently compress vision-language-action models by routing sub-token groups, reducing computational costs while maintaining strong performance on robotic tasks.
This paper proposes SATS (Sensitivity-Aware Thresholding for Sparsity) and a token routing framework to improve inference efficiency in LLMs by dynamically sparsifying MLP activations. The methods achieve better quality-throughput trade-offs compared to percentile-based baselines.
This paper proposes DPVR-LF, a modality-asymmetric routing framework for MLLMs that routes vision tokens at their saturation point into a lightweight side branch and performs late fusion, reducing visual computation while maintaining competitive performance.
This paper presents Token-Selective Attention (TSA), a differentiable token routing mechanism that learns to skip unnecessary computations per token in transformer layers, reducing token-layer operations by 14–23% with minimal quality loss on language modeling tasks.