Tag
MedMix is a semantic-alignment framework for federated multimodal sparse Mixture-of-Experts that addresses modality heterogeneity by coordinating routing and expert specialization, achieving improved performance in medical AI datasets.
Moonshot's Kimi K3, a 2.8 trillion parameter open weights model with 896 experts (16 active per token), exemplifies the trend of scaling total parameters while holding active compute constant, and uses attention compression to reduce KV cache size, making frontier inference more accessible but with high storage costs.
SILX AI releases Quasar-Preview, an 18B parameter MoE foundation model with 2B active parameters and experimental 5M-token context, built on a hybrid recurrent/attention architecture and designed for decentralized training via Bittensor SN24.
HodgeCover uses higher-order topological coverage to compress sparse Mixture-of-Experts layers by addressing irreducible mergeability barriers that pairwise signals miss, matching state-of-the-art baselines on expert reduction and leading on aggressive compression.