exllama

社区开发者发布 EXL3-QLoRA：一个基于 ExLlamaV3 分支的原生 PyTorch 训练实现，支持在 2-8bpw 量化模型上进行 QLoRA 训练，取代传统的 BitsAndBytes 4bit 方案，并在损失和显存占用上超越 bnb NF4。

A hobbyist reports stabilizing a dual CMP 170HX (64GB each) rig running GLM-5.3-Flash via EXL3 3.05bpw at 384K context and ~90 tok/s, and compares it against Qwen3.8-Flash-Next on AWQ INT4 + FP8 PLE under vLLM, including agent-task latency. The post also analyzes trellis-based EXL3 quantization fidelity versus common UD-Q4/Q5 GGUF variants.

Running Swift 1.5 (UkisAI's reasoning-efficient fine-tune of Qwen3.8-Flash-Next) on sf-stav's veloGB10 Rust/CUDA engine across 2× DGX Spark yields ~110 tok/s decode, letting xhigh reasoning effort finish faster than base Flash-Next at medium on an older vLLM NVFP4 setup — including a header-only safetensors fix so 12 of 14 EXL3 community packs load, since current exllamav3 writes the n-gram table as 128 shards that velo can't read.

Yamz Labs released Kyojin, a ROCm inference engine built on ExLlamaV3 for AMD Strix Halo, enabling two ~300B-class MoE models (GLM-5.3-Flash and MiMo-V2.6-Flash) to each fit and run on a single 128 GB mini PC, with EXL3 quantized weights hitting up to 580 tok/s prefill and 44 tok/s speculative decode.

ExLlamav3 has released major updates including CPU offload for MoE experts, support for new AI models like GLM-5.3-Flash and Qwen-3.8-Flash, and various performance optimizations.

ExLlamaV3 v1.0.0 brings major performance upgrades for running large language models locally.