Don't Sleep on EXL3 Quants
Summary
The author shares their experience running a 30B parameter model with EXL3 quantization on a 12GB VRAM GPU, achieving efficient performance and speed for coding and agent tasks.
Similar Articles
yeasah/Qwen3.8-27B-exl3 quants - lowest KLD of 0.0002
A new EXL3 quant of Qwen3.8-27B by yeasah achieves the lowest KLD (0.0002) among competitors while requiring only 18.8 GiB of VRAM for weights, outperforming quants from Unsloth and turboderp.
ExLlamaV3 is underrated
The author advocates for ExLlamaV3, an underrated local AI inference tool that outperforms llama.cpp on NVIDIA hardware, with recent updates like CPU MoE offload enhancing its performance.
@QuixiAI: LESSON LEARNED: Always use BF16 kv cache. I was using turboquant. Yeah the VRAM consumption sucks - so another trick is…
The author shares a technical lesson on using BF16 KV cache instead of turboquant for AI model optimization and implements CPU offloading in SlimServe for Qwen models to manage VRAM consumption.
Consider running a bigger quant if possible
A user reports that switching from a highly-compressed IQ4_XS quant to the larger IQ4_NL_XL quant of Qwen 3.6 dramatically improves agentic-coding accuracy, despite lower tok/s, urging others to favor bigger quants when VRAM allows.
@Tech2Wild: I’m seeing a lot of NFP4 vs EXL3 and need a verdict. Is EXL3 wave moving forward on quants ?
A user asks for a verdict on whether EXL3 quantization is advancing compared to NFP4 in AI models.