moe

Qwen3.8 Max has been upgraded and now leads China's AI leaderboard with a score of 45 on the Artificial Analysis Intelligence Index, surpassing GLM-5.3 and Kimi K3 after a 30-day improvement to the 2.4T MoE model.

An implementation of expert lookahead achieves over 10% performance improvement for MoE models running on low-memory devices using slotstream, with additional gains from a correction model.

A permanently modified version of DeepSeek-V4.1-Flash with surgically removed safety guardrails, retaining full capabilities including vision, reasoning, and tools, and demonstrating 100% compliance on HarmBench evaluations.

A llama.cpp implementation that expands MoE expert routing beyond native top-K during inference using adaptive thresholds and layer-specific linear decay, without requiring model retraining or fine-tuning.

A developer built a custom branch of llama.cpp that implements expert expansion for Mixture-of-Experts models, tested it on Metal, and is seeking cross-platform feedback.

IFM AI's K2-Horizon-400B-MoE continues to outperform Thinking Machine's Inkling on Artificial Analysis benchmarks, according to a team member highlighting the year-old lab's progress.

This paper diagnoses intra-adapter contention in MoE+LoRA fine-tuning and introduces SpawnLoRA to dynamically add sub-adapters, reducing negative transfer across domains.

Marin has grown from one to ten full-time employees since joining Open Athena a year ago. The team is currently in the middle of a large-scale 535B/A23B Mixture of Experts model training run.

Ant's Ling team has released Ling-3.0-flash-Fin, a finance-enhanced AI model designed for financial workflows, with free access on OpenRouter for one month and plans to open-source the weights.

ExFold is a unified training-free framework that accelerates MoE model inference by folding excluded expert contributions into retained experts, achieving up to 1.41× speedup while maintaining high quality.

Alibaba will open-source the Qwen3.8-Flash-Next model on ModelScope at 23:00 tonight, featuring Qwen4's GDN hybrid layers and sparse attention architecture, but lacking benchmark data.

The author shares findings from testing quantization impacts on local LLMs for agentic use, revealing that many quants are statistically indistinguishable, MoEs are less affected than dense models, and significant degradation occurs below Q4 quantization.

PyTorch provides a native fine-tuning library with Day-0 Hugging Face checkpoint support for Alibaba's open-weight Qwen3.8-2.4T-A95B model, enabling efficient training and deployment on NVIDIA systems with configurable reasoning.

Ornith-1.5, a family of open-source LLMs, is introduced with variants up to 397B MoE, achieving state-of-the-art performance among comparable models and rivaling Claude Opus in benchmarks.

This article compares the performance of NVIDIA DGX Spark and a modified RTX 4090 in locally deploying the Qwen3.8-27B and Ling-3.0-flash models, providing benchmark data and purchase recommendations.

NVIDIA released their expert models for MOPD, making AI research more accessible with the NVIDIA-Nemotron-Labs-Teacher-Competition-Coding model designed for competitive programming and code reasoning.

Speculation about the upcoming Qwen 3.8 release, questioning whether it will be a dense 27B model or a MoE variant like the previous 35B-A3B, with discussion of performance implications for local hardware.

This paper introduces Dual-Flow Transformers, an architecture that decouples prefill and decode computation by adding an auxiliary flow for continuation prediction while sharing weights and the primary KV cache, enabling phase-specific compute allocation and improved efficiency.

Discussion about running LLMs on 12GB VRAM, noting current focus on dense models like Muse Glimmer 30B and Qwen 3.8 27B, and questioning whether upgrading to 24GB VRAM is needed.

The author trained a 40M-parameter connector on 100K examples to give DeepSeek V4 Flash basic vision, freezing both the language model and MoonViT image encoder, demonstrating a low-cost approach to turning a text-only MoE into a basic VLM.