model-quantization

The team quantized Qwen 3.8 27B into various GGUF formats and benchmarked them on an RTX 6000, finding similar performance across quants with AD-Q6_K recommended for safety.

Introducing colibri, an open-source inference engine written in pure C that unifies VRAM, RAM, and disk as a hierarchical structure to stream large model weights, supporting various cutting-edge MoE models to run locally on consumer hardware, lowering the barrier to using large models.

The article critiques the lack of systematic benchmarking for AI models in different quantization formats, highlighting discrepancies between benchmark results and real-world usage, and calls for more thorough evaluation.

A detailed guide on running the quantized NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B model on two RTX 3090s using vLLM with full 262K context, achieving high inference speeds without CPU offloading.

Poolside releases GGUF quantizations of the Laguna S 2.1 AI model, including a DFlash speculative decoding draft model, enabling efficient local inference with llama.cpp.

A user tested the unsloth quantized GLM-5.2 model on a high-end consumer-like system with dual RTX 5090, achieving 12 tokens per second.

This tutorial from NVIDIA walks through the end-to-end workflow of converting an FP8-quantized PyTorch model into a TensorRT inference engine for production deployment, covering ONNX export and performance profiling.

Ported the EXL3 LLM codec to run on Apple Silicon via Metal, achieving high prefill and generation speeds on M5 Max (e.g., ~600 tok/s prefill, 17-80 tok/s gen on various models).

Unsloth is uploading a GGUF quantized version of the MiniMax M3 model to Hugging Face.

This article is the middle part of the AI Engineering Landscape series, detailing core techniques such as inference optimization, model slimming (quantization, distillation, pruning, MoE), and speculative decoding, while reviewing the latest advances from hardware to the engineering stack.

A user shares their GGUF quantized version of Qwen 3.6 27B using AutoRound, claiming it performs better than other quants, and invites feedback.

The poster states that the MoQ GGUFs of the LFM2.5 8B A1B model offer the best accuracy-to-size ratio, advising against using versions with less than 95% accuracy recovery.

This post from NVIDIA explains how to use the NVIDIA Model Optimizer library to quantize a CLIP model to FP8 using post-training quantization, reducing VRAM usage and improving inference performance on consumer GPUs.

MagicQuant v2.0 is a pipeline for creating hybrid mixed GGUF quant models, learning from Unsloth and other methods to find optimal quant configurations based on KLD benchmarks, with a focus on nonlinear wins and anomaly detection.

Comparison of hardware requirements for running multi-agent AI workflows locally, highlighting VRAM and KV Cache constraints.

The article highlights a significant surge in the creation of local AI GGUF models on Hugging Face, with monthly additions nearly doubling to over 9,000 in recent months, driven by improved tooling and new open-weight releases.

0xSero has released new FP8 and NVFP4 quantized versions of the Tencent Hy3-preview model, enabling it to run on 256GB VRAM with full context.

Solo researcher released Spiral, a tool that compresses LLMs to INT3 and KV-cache to INT2 with custom fused Metal kernels for Apple Silicon, currently shipping Qwen-7B preview.

Developer @0xSero achieved high-performance inference on an optimized GLM-5.1-505B variant using NVFP4 quantization and 32% pruning, reaching 45 tokens/s decode and 1350 tokens/s prefill speeds.