ExLlamaV3 is underrated

Reddit r/LocalLLaMA Tools
exllamav3 nvidia cpu-offload quantization local-ai llama-cpp

Summary

The author advocates for ExLlamaV3, an underrated local AI inference tool that outperforms llama.cpp on NVIDIA hardware, with recent updates like CPU MoE offload enhancing its performance.

I moght get shit on for posting this but, I feel like i don't see this being talked enough and it feels like such a waste of a good piece of software. Exl3 is incredible, albeit only if you have NVIDIA cards I think? Exl3 quants are higher quality for its size, much lower KLD metrics, faster, all compared to llama.cpp just from a few personal sets of tests I like to give my local models (these are not benchmarks). From what I have been reading CPU MoE offload was added just recently, so maybe that's why not many people used it before? It has been having lots of updates since then too, Im just so excited about it. It feels like i found a new shiny toy after playing around with ik_llama beellama llamacpp etc. I have been using tabbyAPI exl3 backend + qwen 3.8 27b sc 6bpw H6 and qwen 3.8 flash next 4bpw as my daily drivers and it's incredible what it can do. I hope this software gets known to more people too. I'm not affiliated with them or anything. i judt wanted to share it. It's just so cool, please give it a try!!
Original Article

Similar Articles

ExLlamaV3 Major Updates!

Reddit r/LocalLLaMA

ExLlamaV3 has released a series of major updates including Gemma 4 support, improved caching efficiency, and the new DFlash technology for significantly faster inference speeds across various model categories.