unsloth

A reply to Unsloth AI expressing excitement about the rumored DeepSeek-V4-Flash model and the possibility of running it locally via quantized versions.

Unsloth teases the upcoming release of DeepSeek V4 Flash GGUF quantized model on Hugging Face.

Unsloth released GGUF quantizations of the Laguna S 2.1 Mixture-of-Experts model, a 118B parameter coding model with 8B active parameters, 1M context window, and agentic capabilities. The quantized versions enable efficient local deployment.

A detailed benchmark of Unsloth's Qwen3.6-27B NVFP4 model on RTX 5090 GPUs, showing MTP (multi-token prediction) gives large speedups for single requests at short context but becomes detrimental under batch concurrency or long contexts.

Unsloth, the efficient LLM fine-tuning library, now supports AMD GPUs, expanding hardware accessibility.

The author argues that the commonly recommended learning rate of 2e-4 for QLoRA fine-tuning is too high for datasets under 10k samples, leading to overfitting and poor evaluation, and suggests using a lower learning rate like 1e-4.

The unsloth/inkling-GGUF page provides the quantized GGUF version of Inkling, a 975B-parameter multimodal MoE model (41B active) from Thinking Machines, designed for text, image, and audio inputs with open weights and support for local deployment via libraries like Unsloth, SGLang, and vLLM.

Unsloth collaborated with AWS on a comprehensive guide to dynamic quantization and deployment of LLMs on Amazon SageMaker, covering model formats, tools, and best practices.

Unsloth improved Qwen3.6-27B Q6_K context length from 23K to 64K on a single 32GB card by accurately counting the draft model's KV cache in bytes instead of using a flat VRAM cushion.

Voodoo Quant, a quantization method, outperforms Unsloth Dynamic 2.0 KLD by 95% on Qwen3.5 0.8B and 2B models.

Benchmark results of the unsloth/Qwen3.6-27B-NVFP4 model running on 4x RTX 5060 Ti GPUs with peer-to-peer and pipeline parallelism at various concurrency levels.

Unsloth releases a new version of Qwen3.6 27B NVFP4 quantization scheme, introducing FP8_E4M3 intermediate precision layer and refined weight protection, achieving 2.5x speed improvement on 24GB VRAM, while improving accuracy and tool-calling capabilities.

Unsloth AI releases quantized Qwen3.6 models that run 2.5× faster on consumer GPUs, with the 27B model fitting in 24GB VRAM and the 35B-A3B achieving high throughput.

Unsloth releases quantized Qwen3.6 models using NVFP4 format, achieving 2.5x faster inference speeds.

A guide on running Z.ai's open model GLM-5.2 locally using Unsloth Dynamic GGUFs. The model features 744B total parameters (40B active) and a 1M context window, with quantized versions reducing memory to 239GB for 2-bit, enabling local inference on 256GB Macs.

A developer fine-tunes a small Qwen 3 0.6B model using the Unsloth framework to categorize household questions, achieving good results with only 850 training examples.

A tweet promoting the Qwen 3.6 27b model and recommending UnslothAI for running it on any GPU.

Unsloth 成功将 GLM-5.2 模型以 2-bit 量化压缩至 238GB，可在 256GB Mac 上本地运行，保留约 82% 的准确率。

UnslothAI announces GLM-5.2, Z.ai's strongest open model with 744B parameters, now runnable locally via dynamic GGUF quantization reducing size by ~84% to 239GB while retaining ~82% accuracy. It fits on 256GB Macs and supports long-context, reasoning, and agentic tasks.

GLM 5.2 GGUF quantized model is released, with 8-bit version half the size of the full model; smaller versions are coming soon.