Tag
The poster states that the MoQ GGUFs of the LFM2.5 8B A1B model offer the best accuracy-to-size ratio, advising against using versions with less than 95% accuracy recovery.
This post from NVIDIA explains how to use the NVIDIA Model Optimizer library to quantize a CLIP model to FP8 using post-training quantization, reducing VRAM usage and improving inference performance on consumer GPUs.
MagicQuant v2.0 is a pipeline for creating hybrid mixed GGUF quant models, learning from Unsloth and other methods to find optimal quant configurations based on KLD benchmarks, with a focus on nonlinear wins and anomaly detection.
Comparison of hardware requirements for running multi-agent AI workflows locally, highlighting VRAM and KV Cache constraints.
The article highlights a significant surge in the creation of local AI GGUF models on Hugging Face, with monthly additions nearly doubling to over 9,000 in recent months, driven by improved tooling and new open-weight releases.
0xSero has released new FP8 and NVFP4 quantized versions of the Tencent Hy3-preview model, enabling it to run on 256GB VRAM with full context.
Solo researcher released Spiral, a tool that compresses LLMs to INT3 and KV-cache to INT2 with custom fused Metal kernels for Apple Silicon, currently shipping Qwen-7B preview.
Developer @0xSero achieved high-performance inference on an optimized GLM-5.1-505B variant using NVFP4 quantization and 32% pruning, reaching 45 tokens/s decode and 1350 tokens/s prefill speeds.