LoRA over GGUF: Train Qwen3.8-Flash-Next in 40G VRAM
Summary
An update on the LoRA over GGUF series showing that Qwen3.8-Flash-Next (125B-A6B + 51B engram) can now be LoRA-trained in 40 GiB VRAM without CPU offloading, achieving 200 token/s on Strix Halo, with initial GGUF support merged in transformers 5.18 and further kernel optimizations (GGTensile) on the horizon.
Similar Articles
Train LoRA directly on Quantized GGUFs, no PyTorch, no Parent(FP16) needed, just your GGUF and the data
介绍了 Runner 工具，可直接在量化 GGUF 权重上训练 LoRA 适配器，无需 PyTorch、transformers 或 FP16 母模型，并支持合并为独立 GGUF 文件用于推理。
Qwen3.6 27B Pure Quant: 40 tok/s on 16 GB VRAM
A quantized version of Qwen3.6 27B using a pure Q4_K_M method fits entirely in 16 GB VRAM, achieving up to 40 tok/s token generation speed with MTP, and significantly reducing model size compared to other GGUF variants.
gufo-Qwen3.6-35B-A3B-Q6dense - 3095tok/s prefill; 190 tok/s decode on Strix Halo
Gufo is an open-source local inference engine optimized for AMD Strix Halo (Ryzen AI MAX+ 395, 128 GiB unified memory), achieving 3,095 tok/s prefill and up to 190 tok/s decode on a Q6dense quantized Qwen3.6-35B-A3B model, with benchmarks for Qwen3.8 and DeepSeek V4 Flash.
I reverse-engineered an NPU vendor's engine format (int8 weights stored as two nibble planes) to run GGUFs with no model conversion — now 1.5× faster than the vendor's own runtime
Reverse-engineered the Axera AX8850 NPU's int8 weight format to enable direct GGUF inference in llama.cpp, achieving up to 24.5 t/s decode and 716 t/s prefill on a Raspberry Pi 5, outperforming the vendor's runtime by 1.5×.
Strata with Qwen3.8 Flash Next UD-Q4_K_XL
A user reports strong single-request decode speeds running Qwen3.8-Flash-Next in Unsloth's UD-Q4_K_XL quant on Strata with an RTX PRO 6000, showing roughly 2x faster throughput than vLLM on the same GPU across prose, counting, coding, and reasoning tasks.