moe

Moonshot AI announced Kimi K3.1 Pro, its most capable 3T-class open-weight MoE model with 1M-token context, and Modal will offer day 0 support for it this week.

tandem.cpp is a patch set for llama.cpp that offloads MoE models overflowing VRAM by keeping hot experts on GPU while the CPU handles the rest, achieving 97.3 tok/s on a single RTX 4090 (3.3× faster than llama.cpp) for Qwen3.8-Flash-Next, with gains for other MoE models too.

A Hugging Face repo called burner22/numbat claims to contain an Anthropic internal training checkpoint, reportedly 14T parameters, 114 layers, and 1,224 MoE experts — suspected to be one of the largest leaks of closed-source model weights ever.

A user shares their custom MoE-centric AI workstation build featuring an AMD EPYC 7452, 256GB DDR4, and a Radeon AI PRO R9700 32GB GPU, costing roughly half as much as an NVIDIA DGX Spark and ready for experimenting with Qwen4.

Modal announced first-class support for post-training Reflection's Beam model and released Modal Dojo, an open-source library and skill bundle for production-grade LLM training (including RL) on Modal, with an example training Qwen3.5-4B on GSM8K.

@NVIDIARTXSpark shares three techniques to accelerate Mixture-of-Experts (MoE) inference on the NVIDIA DGX Spark, covering optimized kernels, FP8 quantization, and multi-token prediction to boost tokens per second.

Prime Intellect made DeepSeek's MegaMoE kernels trainable, fusing computation and communication in the MoE layer to achieve up to 55% faster MoE training than TorchTitan, with support for BF16, MXFP8, and FP8 on Blackwell and Hopper GPUs.

A zero-retraining REAP-pruned derivative of NVIDIA's Nemotron-3.5-Transcribe-30B-A4B speech LLM shrinks it from 67.5GB to 10GB (4-bit) while keeping WER at 5.16% vs the teacher's 4.12%, and it runs 27% faster in single-stream throughput on an H200 with vLLM.

一篇技术博客深入解析 Meta 于 10 月 15 日发布的开源模型 Muse Spark Open（816B 总参数、42B 激活参数的 MoE 架构），并介绍 Daniel Gross 发起的 10 万美元 Open Kernel Challenge，邀请开发者在 H200 上优化其推理内核与 SGLang 服务性能。

A community post laying out feature requests for Google's upcoming Gemma 5, including looped transformers for better reasoning, DeepSeek's Engram memory, cheaper KV cache via Compressed Sparse Attention, a potential ~120B MoE variant, drafter improvements, and keeping Gemma well-rounded beyond coding benchmarks.

A community member released the Groot Series, a family of four MoE models ranging from 19B to 280B parameters, touting strong benchmark results including a 19B model running 128K context on a 16GB GPU and a 280B model scoring 63.47 on SWE-bench Pro at low cost.

论文提出开源训练配方 ALTO，用于在 MXFP4 微缩放格式下训练 20B 参数 MoE 模型 GPT-OSS-20B，通过混合块量化、随机 Hadamard 变换、随机舍入与权重去振荡方案，将 FP4 与 BF16 的验证损失差距缩小至 0.007，收敛开销从 +20% 降至 +5%，并报告了若干无效尝试的负面结果。

OrcaRouter 发布了 DeepSeek-V4.1-Flash-1017 的 abliterated（移除拒绝）GGUF 版本，可在单机 236GB 内存上通过 llama.cpp 本地运行这个 763B MoE 模型，其中 197B 参数的 Engram n-gram 记忆表按需从 SSD 流式读取。

Ant Group and the vLLM community are open-sourcing Humming, a general-purpose, high-performance kernel library for quantized inference that supports broad GPU coverage, diverse quantization formats, and outperforms existing implementations like Marlin, FlashInfer, and DeepGEMM.

llama.cpp PR #29887 adds an LRU MoE expert cache that exploits locality of reference, boosting Qwen3.8-Flash-Next (94GB) runs to 67.8 tok/s on an RTX 5090 (2.2x), with SGLang merging a similar 'paged experts' feature days later.

Ant Group’s inclusionAI has open-sourced the Ming-3.0-tts speech synthesis model, which ranked first overall in pronunciation robustness with a score of 93.2% on Artificial Analysis’s October leaderboard and first among open-source models for voice controllability. It supports voice cloning, control through natural-language instructions, 14 paralinguistic tags, and speech synthesis across multiple dialects and languages, including Sichuanese and Cantonese.

DeepSeek open-sources DeepInfer, the high-performance inference framework powering its large-scale production services, featuring optimized kernels, MoE expert parallelism, speculative decoding, batch-invariant inference, and support for NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

A Hugging Face release of DeepSeek-V4.1-Flash-1017 with refusal behavior removed via abliteration, shipped in native FP8 (e4m3) weights with FP4 MoE experts while preserving vision, video, tool calling, reasoning tiers, and 1M-token context. The model card emphasizes it is not a quantized derivative, not a chat-template jailbreak, and not LoRA/RLHF-based — the edited weights are the model.

A developer built Red Curtain to run Fish Audio's Drama 3 text-to-speech model (35B MoE, ~3B active) on a 16 GB Radeon RX 9070 despite its official 48 GB NVIDIA requirement, offloading FP8 experts to system RAM and caching active ones on the GPU. The tool ships with a web UI supporting voice cloning and multi-speaker scripts, and the author shares tuning tips and a known first-line silence issue.

腾讯混元团队成员公开了 Hy4 Preview（770B 参数 MoE）的极致量化方案，通过 STQ1_0 编码与混合精度分配将约 1.5TB 权重压缩至 214 GiB，并在 llama.cpp 中实现接近 IQ1_M 的推理速度。