quantization

用户分享使用 Strata 替换 llama.cpp 运行 Qwen 3.8 Flash Next 后，解码速度从 70-80 tk/s 提升至 150 tk/s，并顺带测试了 DeepSeek v4.1flash 2.25bit 量化在多 GPU 上的表现。

作者对比了通过 Z.ai 云订阅与本地部署的 GLM 5.3 Flash 在相同任务上的表现，本地推理在输出几乎相同的情况下快了 4.2 倍，指出本地 AI 除省钱外的最大价值是节省时间、提升代码产出效率。

Strata's latest update significantly boosts inference performance for AMD R9700 users running Flash Next, roughly doubling token output from ~70 tok/s to 140-160 tok/s while still trailing Radiance. The user notes quantization limits (Q5 falling just short of VRAM for split-engine mode) and hopes future updates close the gap further.

A quantization report for LiquidAI's LFM3-1.7B shows it runs losslessly in 3.6 GB VRAM (2.2 GB with modest degradation) and outperforms MiniCPM5-2B on LiquidAI benchmarks while offering 2x context in half the memory.

An analysis points out that all four Unsloth quantizations of DeepSeek-V4.1-Flash-1017 plus the Q8 baseline simultaneously improved by 2–3 points on LiveCodeBench v6 in an updated chart, a shift statistically improbable under an independent-sampling model, while GSQ-RCO results behaved inconsistently.

A community post discusses the rivalry between ISTA's GSQ-RCO quantization method and Unsloth, noting that both teams' competition benefits users with better quants, while questioning why ISTA doesn't publish KLD metrics for its quants.

Manning Books is offering a free ebook, 'Local AI and Small Language Models', featuring chapters on small language models, CPU inference with GGUF and llama.cpp, quantization, LoRA/DoRA specialization, and building private local AI apps with Ollama's Python library.

FLAMIA 提出了一族面向 NVIDIA Blackwell GPU 的混合精度注意力内核，利用 MXFP8 与 NVFP4/FP8 路径，在避免因果信息泄漏的同时加速 Transformer 训练，在 B200/B300 上实现约 1.06–1.42 倍的前向与反向注意力加速。

Unsloth published benchmark results for its DeepSeek Dynamic V3.1 GGUF quants, claiming >10–20% gains over GSQ-RCO on agentic benchmarks like Terminal Bench and DeepSWE at 512K context, and statistically disputing GSQ-RCO's quantization recovery claims.

AtomicChat released GSQ-RCO GGUF quantizations of DeepSeek-V4.1-Flash-1017 with refusal directions projected out of the dense residual writers, letting a 763B MoE model run on a single 12 GB GPU via Strata with minimal MMLU degradation.

MiniMax thanked community builders who shrank M3.1 Flash open weights from 252.4 GB to as low as 93 GB in 10 days, with quantized variants ranging from GGUF to EXL3 and MXFP4 delivering strong accuracy and throughput on consumer and datacenter hardware.

ISTA DASLab released a Coder build of DeepSeek-V4.1-Flash-1017, a 763B sparse MoE model compressed to roughly 1 bit per transformer parameter via GSQ quantization and RCO-based expert pruning (192 of 384 experts kept), retaining 98.7% of LiveCodeBench v6 and 94.8% of SWE-bench Verified performance in a 72.1 GB GGUF.

A hobbyist benchmark pits locally run quantized models (Qwen3.8-Flash-Next, Qwen3.8-27B, Gemma 4) against Claude models on 9 real C++/CUDA bug-fixing tasks, finding 3-bit quantization costs little in score but makes models think longer, and that local Flash-Next roughly matches Claude Haiku 5.5 while Opus 5.5 leads overall.

A community proposal in r/LocalLLaMA urging members to stop fixating on tok/s speed metrics and instead create private quality benchmarks, especially to evaluate model quality across quantisation levels, and to share results with each other.

The author tested three local open-source models — DeepSeek-V4.1-Flash-1017, Qwen-3.8-Flash-Next, and Qwen-3.8-27b — on an RTX 5090, an RTX 4090, and dual V100s, comparing the real-world experience gap against cloud models like GPT-5.5 and Opus-4.8 through a Core-300 blind test and 200K+ long-context stress tests.

The paper investigates three calibration approaches—HiGPTQ, HiAutoRound, and HiFlatQuant—for quantizing LLMs with the HiFloat4 block floating-point format, reducing the accuracy gap to BF16 below 1% and outperforming NVFP4 across models and tasks.

该论文研究随机舍入（stochastic rounding）如何影响低精度训练中的预测风险，针对两层线性网络推导了匹配的上下界，并给出了随计算量、模型规模、训练时长与精度变化的 scaling laws，揭示前向与后向舍入对最优计算缩放的不同影响。

An academic paper proposing correlated private quantization for communication-efficient, locally differentially private federated reinforcement learning in vehicular networks, coupled with post-quantum authentication; it reports 92.4% smaller authenticated uplink sizes and 14.3% lower mean travel time versus independent quantization.

An updated 2.3bpw quantized checkpoint of a Qwen 27B model for 12 GB Ampere cards ties Qwen3.6 BF16 on LiveCodeBench (84.3% vs 83.9%) and cuts KLD error by ~40% versus the previous release, using the new AmperInteger+ quant format in llamAmpere.

A user shares a detailed setup guide for running DeepSeek-V4.1-Flash-1017 (~35 tok/s) and Qwen3.8-Flash-Next (~130-140 tok/s) on older consumer hardware (RTX 3090 Ti, 128GB RAM, Threadripper 3990X), including an NVIDIA firmware-throttle unlock via an open-source kernel module and an Xolotl inference engine build.