@techNmak: The smartest way to run a giant MoE model is not to add more GPUs. It is to stop treating every expert as GPU-worthy. L…

X AI KOLs Timeline Tools
moe inference cpu-gpu heterogeneous ktransformers open-source fine-tuning

Summary

KTransformers is a framework that optimizes inference and fine-tuning of large Mixture-of-Experts models by dynamically placing only active experts on the GPU while keeping the rest in CPU memory, enabling large models like DeepSeek-V3 to run on limited consumer GPU memory.

The smartest way to run a giant MoE model is not to add more GPUs. It is to stop treating every expert as GPU-worthy. Large Mixture-of-Experts models contain hundreds of specialized feed-forward networks called experts. But the model does not use all of them for every token. In the architecture shown by KTransformers, the router activates only 8 out of 256 experts at a time. So keeping the entire expert pool in expensive GPU memory makes little sense. KTransformers splits the work based on what the hardware is actually good at: → Attention and shared components stay on the GPU → The enormous routed-expert pool sits in CPU memory → Frequently used experts can be placed on the GPU → Colder experts continue running on the CPU → AMX and AVX kernels accelerate INT4/INT8 CPU inference → NUMA-aware scheduling reduces memory-access overhead The CPU is not treated as a slow overflow bin. It becomes an active part of the inference engine. The GPU handles operations with high arithmetic intensity and the experts that benefit most from fast memory. The CPU provides the capacity needed to hold the model’s long tail of experts. KTransformers is specifically designed for heterogeneous CPU–GPU inference and supports dynamic expert placement for large MoE models. That distinction matters. A dense model needs most of its weights for every token. A Mixture-of-Experts model needs only a small working set. So the real optimization problem is no longer: How do we fit the entire model onto the GPU? It becomes: Which parts of the model deserve GPU memory right now? KTransformers also integrates with SGLang for serving and with LLaMA-Factory for fine-tuning large MoE models using limited GPU memory. Its documented setup includes DeepSeek-V3/R1 fine-tuning at roughly 80GB total GPU memory across four RTX 4090s, and Qwen3-30B-A3B on one RTX 4090. The project now has 17K+ GitHub stars and supports models including GLM-5.2, MiniMax-M3, DeepSeek-V4-Flash and Kimi-K2.5. The larger idea: GPU memory should not be where the whole model lives. It should be where the model’s current working set lives. GitHub: https://github.com/kvcache-ai/ktransformers…
Original Article
View Cached Full Text

Cached at: 07/12/26, 11:01 PM

The smartest way to run a giant MoE model is not to add more GPUs. It is to stop treating every expert as GPU-worthy.

Large Mixture-of-Experts models contain hundreds of specialized feed-forward networks called experts. But the model does not use all of them for every token.

In the architecture shown by KTransformers, the router activates only 8 out of 256 experts at a time.

So keeping the entire expert pool in expensive GPU memory makes little sense.

KTransformers splits the work based on what the hardware is actually good at: → Attention and shared components stay on the GPU → The enormous routed-expert pool sits in CPU memory → Frequently used experts can be placed on the GPU → Colder experts continue running on the CPU → AMX and AVX kernels accelerate INT4/INT8 CPU inference → NUMA-aware scheduling reduces memory-access overhead

The CPU is not treated as a slow overflow bin. It becomes an active part of the inference engine.

The GPU handles operations with high arithmetic intensity and the experts that benefit most from fast memory.

The CPU provides the capacity needed to hold the model’s long tail of experts. KTransformers is specifically designed for heterogeneous CPU–GPU inference and supports dynamic expert placement for large MoE models.

That distinction matters.

A dense model needs most of its weights for every token. A Mixture-of-Experts model needs only a small working set.

So the real optimization problem is no longer: How do we fit the entire model onto the GPU?

It becomes: Which parts of the model deserve GPU memory right now?

KTransformers also integrates with SGLang for serving and with LLaMA-Factory for fine-tuning large MoE models using limited GPU memory. Its documented setup includes DeepSeek-V3/R1 fine-tuning at roughly 80GB total GPU memory across four RTX 4090s, and Qwen3-30B-A3B on one RTX 4090.

The project now has 17K+ GitHub stars and supports models including GLM-5.2, MiniMax-M3, DeepSeek-V4-Flash and Kimi-K2.5.

The larger idea: GPU memory should not be where the whole model lives. It should be where the model’s current working set lives.

GitHub: https://github.com/kvcache-ai/ktransformers…

kvcache-ai/ktransformers

Source: https://github.com/kvcache-ai/ktransformers

KTransformers

A Flexible Framework for Experiencing Cutting-edge LLM Inference/Fine-tune Optimizations

🎯 Overview | 🚀 Inference | 🎓 SFT | 🔥 Citation | 🚀 Roadmap(2026Q2)

🎯 Overview

KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project now exposes two user-facing capabilities from the kt-kernel source tree: Inference and SFT.

🔥 Updates

  • June 21, 2026: MiniMax-M3 Day0 Support! (Tutorial)
  • June 17, 2026: GLM-5.2 Day0 Support! (Tutorial)
  • May 6, 2026: KTransformers at GOSIM Paris 2026 — “Agentic AI on Edge” track. We’ll present KT’s inference performance on consumer hardware.
  • May 02, 2026: DeepSeek-V4-Flash Support! (Tutorial)
  • Apr 30, 2026: KTransformers v0.6.1 refreshes kt-kernel inference and SFT docs with separate Inference and SFT Quick Start entry points.
  • Mar 26, 2026: Support AVX2-only CPU backend for KT-Kernel inference. (Tutorial)
  • Feb 13, 2026: MiniMax-M2.5 Day0 Support! (Tutorial)
  • Feb 12, 2026: GLM-5 Day0 Support! (Tutorial)
  • Jan 27, 2026: Kimi-K2.5 Day0 Support! (Tutorial) (SFT Tutorial)
  • Jan 22, 2026: Support CPU-GPU Expert Scheduling, Native BF16 and FP8 per channel Precision and AutoDL unified fine-tuning and inference
  • Dec 24, 2025: Support Native MiniMax-M2.1 inference. (Tutorial)
  • Dec 22, 2025: Support RL-DPO fine-tuning with LLaMA-Factory. (Tutorial)
  • Dec 5, 2025: Support Native Kimi-K2-Thinking inference (Tutorial)
  • Nov 6, 2025: Support Kimi-K2-Thinking inference (Tutorial) and fine-tune (Tutorial)
  • Nov 4, 2025: KTransformers Fine-Tuning × LLaMA-Factory Integration. (Tutorial)
  • Oct 27, 2025: Support Ascend NPU. (Tutorial)
  • Oct 10, 2025: Integrating into SGLang. (Roadmap, Blog)
  • Sept 11, 2025: Support Qwen3-Next. (Tutorial)
  • Sept 05, 2025: Support Kimi-K2-0905. (Tutorial)
  • July 26, 2025: Support SmallThinker and GLM4-MoE. (Tutorial)
  • July 11, 2025: Support Kimi-K2. (Tutorial)
  • June 30, 2025: Support 3-layer (GPU-CPU-Disk) prefix cache reuse.
  • May 14, 2025: Support Intel Arc GPU (Tutorial).
  • Apr 29, 2025: Support AMX-Int8、 AMX-BF16 and Qwen3MoE (Tutorial)
  • Apr 9, 2025: Experimental support for LLaMA 4 models (Tutorial).
  • Apr 2, 2025: Support Multi-concurrency. (Tutorial).
  • Mar 15, 2025: Support ROCm on AMD GPU (Tutorial).
  • Mar 5, 2025: Support unsloth 1.58/2.51 bits weights and IQ1_S/FP8 hybrid weights. Support 139K Longer Context for DeepSeek-V3 and R1 in 24GB VRAM.
  • Feb 25, 2025: Support FP8 GPU kernel for DeepSeek-V3 and R1; Longer Context.
  • Feb 15, 2025: Longer Context (from 4K to 8K for 24GB VRAM) & Slightly Faster Speed （+15%, up to 16 Tokens/s), update docs and online books.
  • Feb 10, 2025: Support Deepseek-R1 and V3 on single (24GB VRAM)/multi gpu and 382G DRAM, up to 3~28x speedup. For detailed show case and reproduction tutorial, see here.
  • Aug 28, 2024: Decrease DeepseekV2’s required VRAM from 21G to 11G.
  • Aug 15, 2024: Update detailed tutorial for injection and multi-GPU.
  • Aug 14, 2024: Support llamfile as linear backend.
  • Aug 12, 2024: Support multiple GPU; Support new model: mixtral 8*7B and 8*22B; Support q2k, q3k, q5k dequant on gpu.
  • Aug 9, 2024: Support windows native.

📦 Capabilities

🚀 Inference - High-Performance kt-kernel Serving

CPU-optimized kernel operations for heterogeneous LLM inference.

image

Key Features:

  • AMX/AVX Acceleration: Intel AMX and AVX512/AVX2 optimized kernels for INT4/INT8 quantized inference
  • MoE Optimization: Efficient Mixture-of-Experts inference with NUMA-aware memory management
  • Quantization Support: CPU-side INT4/INT8 quantized weights, GPU-side GPTQ support
  • Easy Integration: Clean Python API for SGLang and other frameworks

Quick Start:

cd kt-kernel
pip install .

Use Cases:

  • CPU-GPU hybrid inference for large MoE models
  • Integration with SGLang for production serving
  • Heterogeneous expert placement (hot experts on GPU, cold experts on CPU)

Performance Examples:

ModelHardware ConfigurationTotal ThroughputOutput Throughput
DeepSeek-R1-0528 (FP8)8×L20 GPU + Xeon Gold 6454S227.85 tokens/s87.58 tokens/s (8-way concurrency)

👉 Full Documentation →

🎓 SFT - Fine-Tuning with LLaMA-Factory

KTransformers × LLaMA-Factory integration for ultra-large MoE model fine-tuning.

KTransformers SFT

Key Features:

  • Multi-Backend Support: CPU/GPU hybrid fine-tuning with INT8/INT4 quantization
  • Ultra-Large MoE Support: Fine-tune models like DeepSeek-V3/R1 on limited GPU memory
  • Faster than ZeRO-Offload: 6-12x training speedup in benchmarked MoE SFT workloads
  • Lower CPU Memory: About half the CPU memory of the previous KT SFT path in the benchmarked setup
  • LLaMA-Factory Integration: Seamless integration with popular fine-tuning framework
ModelGPU MemoryTraining SpeedHardware
DeepSeek-V3~80GB total3.7 it/s4x RTX 4090
DeepSeek-R1~80GB total3.7 it/s4x RTX 4090
Qwen3-30B-A3B~24GB total8+ it/s1x RTX 4090

Quick Start:

cd /path/to/LLaMA-Factory
pip install -e .
pip install -r requirements/ktransformers.txt
CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch \
  --config_file examples/ktransformers/accelerate/fsdp2_kt_int8.yaml \
  src/train.py \
  examples/ktransformers/train_lora/qwen3_5moe_lora_sft_kt.yaml

👉 Quick Start → 👉 Full Documentation →

🔥 Citation

If you use KTransformers in your research, please cite our paper:

@inproceedings{10.1145/3731569.3764843,
  title = {KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models},
  author = {Chen, Hongtao and Xie, Weiyu and Zhang, Boxin and Tang, Jingqi and Wang, Jiahao and Dong, Jianwei and Chen, Shaoyuan and Yuan, Ziwei and Lin, Chen and Qiu, Chengyu and Zhu, Yuening and Ou, Qingliang and Liao, Jiaqi and Chen, Xianglin and Ai, Zhiyuan and Wu, Yongwei and Zhang, Mingxing},
  booktitle = {Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles},
  year = {2025}
}

👥 Contributors & Team

Developed and maintained by:

We welcome contributions! Please feel free to submit issues and pull requests.

💬 Community & Support

📦 KT original Code

The original integrated KTransformers framework has been archived to the archive/ directory for reference. The project now organizes the two capabilities above from the kt-kernel source tree for clearer documentation and maintenance.

For the original documentation with full quick-start guides and examples, see:

Similar Articles

Mixture of Experts (MoEs) in Transformers

Hugging Face Blog

Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.

Multi Tier MoE Caching

Reddit r/LocalLLaMA

Discusses multi-tier caching strategies for MoE models to improve inference speed by keeping frequently activated experts on GPU, referencing existing implementations like PowerInfer and llama.cpp branches.