unsloth

A user shares their positive experience with low quantizations of Qwen 27B 3.8 on a Mac mini M4, using Unsloth's Q3 XXS quant, and asks for others' experiences with sub-Q3 quants.

This article provides a local deployment guide for the Qwen3.8-27B model, recommends using Q4 quantization and the Unsloth GGUF tool, and shares performance test results compared to the FP8 benchmark.

A user tested the unsloth 1-bit quantized version of the Qwen 3.8 27B AI model on an 8GB VRAM system and found the results amusing.

Unsloth has released Dynamic v3.0 GGUFs for Qwen3.8 models, offering >10% better accuracy at the same size through improved quantization techniques and calibration methods.

Unsloth has released an NVFP4 quantized version of the Qwen3.8-27B AI model, which offers enhanced capabilities in coding, professional work, agentic tasks, and native vision-language understanding.

Unsloth announces that Qwen3.8 can now run locally, shrinking the 2.4T-parameter model from 4.9TB to 397GB via Dynamic 1-bit quantization, with a guide and GGUF release.

Unsloth Desktop is a new open-source desktop app for running and training models locally on Mac, Windows, and Linux, with support for connecting Claude Code and Codex to local LLMs.

A user shares local testing of Muse Glimmer (Q4 quant via Unsloth) on llama.cpp with OpenCode, noting it performs below Qwen3.6 27B but had reliable tool calls.

Unsloth releases a GGUF-quantized version of Meta's Muse Glimmer 30B model, designed for local agentic tasks with multimodal input, tool use, and multi-step reasoning.

Unsloth releases new GGUF quantizations of Kimi K3, ranging from 466GB to 649GB, enabling efficient deployment of the large model.

Unsloth releases GGUF quantizations of MiniMax-H3, an omni-modal generative system for video with native stereo audio, enabling local execution via sd-cli and other platforms.

Unsloth AI announces DSpark, enabling DeepSeek-V4-Flash GGUF models to run ~1.4–2× faster locally, reaching 120 tokens/s with no accuracy change.

Daniel Han of Unsloth validates that Qwen3.8-27B will run in only 17GB VRAM, making it accessible for local inference.

Alibaba announces Qwen3.8-27B open-weights release, capable of running locally on 17GB RAM/VRAM, alongside the larger Qwen3.8-Max.

Unsloth releases an IQ3 GGUF quantization of DeepSeek-V4-Flash-0731, enabling local inference via llama.cpp, Ollama, LM Studio, and other tools.

DeepSeek-V4-Flash-0731 is shown running as an unsloth GGUF quant on a single 40GB A100, with 17.7 tok/s and 6 experts loaded into VRAM, enabling a full agentic coding loop.

A reply to Unsloth AI expressing excitement about the rumored DeepSeek-V4-Flash model and the possibility of running it locally via quantized versions.

Unsloth teases the upcoming release of DeepSeek V4 Flash GGUF quantized model on Hugging Face.

Unsloth released GGUF quantizations of the Laguna S 2.1 Mixture-of-Experts model, a 118B parameter coding model with 8B active parameters, 1M context window, and agentic capabilities. The quantized versions enable efficient local deployment.

A detailed benchmark of Unsloth's Qwen3.6-27B NVFP4 model on RTX 5090 GPUs, showing MTP (multi-token prediction) gives large speedups for single requests at short context but becomes detrimental under batch concurrency or long contexts.