gguf

Unsloth published benchmark results for its DeepSeek Dynamic V3.1 GGUF quants, claiming >10–20% gains over GSQ-RCO on agentic benchmarks like Terminal Bench and DeepSWE at 512K context, and statistically disputing GSQ-RCO's quantization recovery claims.

AtomicChat released GSQ-RCO GGUF quantizations of DeepSeek-V4.1-Flash-1017 with refusal directions projected out of the dense residual writers, letting a 763B MoE model run on a single 12 GB GPU via Strata with minimal MMLU degradation.

ISTA DASLab released a Coder build of DeepSeek-V4.1-Flash-1017, a 763B sparse MoE model compressed to roughly 1 bit per transformer parameter via GSQ quantization and RCO-based expert pruning (192 of 384 experts kept), retaining 98.7% of LiveCodeBench v6 and 94.8% of SWE-bench Verified performance in a 72.1 GB GGUF.

A humorously named Gemma 4 E2B fine-tune called CRAP (Compiler Replacement for Assembling Programs) that emits working x86_64 ELF binaries directly from text prompts, skipping compilers and assemblers, now at v1.1 with a browser-based WebGPU demo.

The author tested Liquid AI's LFM3-1.7B model (Q8_0 GGUF via llama.cpp/ROCm) for compression and summarization, reporting strong prefill/decode throughput on Strix Halo and tensor-split 7900 XTX+XT setups, plus unexpectedly coherent performance at 32k-64k context lengths for a 1.7B model.

A discussion on whether an LLM's tendency to second-guess already-reasoned conclusions is an inherent model trait or an artifact of aggressive quantization (e.g., IQ2-XXS vs Q4_K_M), and whether it would disappear on BF16 GGUFs.

ik_llama.cpp now fully supports PrismML's Ternary Bonsai 2 with native CUDA offload, lossless weight repacking, and fast CPU prompt ingestion via -rtr 1, eliminating prior sign tensor bottlenecks. The post provides a setup guide covering PQ2_0 conversion and recommended command-line flags.

Announcement recommending users upgrade from the MTP head to the DFlash 2 draft model for Qwen3.8-Flash-Next speculative decoding in llama.cpp, claiming ~17% average speedup, with Q4_0 quantization recommended as it matches Q8_0 acceptance/speed while using 287 MiB less memory. The card also details runtime memory trade-offs between the MTP head and the DFlash 2 drafter, noting the smaller on-disk drafter costs about the same at runtime due to its larger compute buffer.

Unsloth has released a Dynamic V3.1 quantized GGUF version of DeepSeek-V4.1-Flash-1017, enabling the model to run locally on devices with just 64GB of RAM. It claims up to 3× faster inference, support for a 1-million-token context window, and across-the-board leadership in benchmarks against comparable quantized models.

Unsloth 发布 DeepSeek-V4.1-Flash-1017 Dynamic V3.1 (preview) GGUF 量化模型，宣称在 7 项基准上精度保持优于 ISTA 的 GSQ-RCO GGUFs，并公开了无法复现对方 AIME25 100% 精度恢复声明的对比结果。

Release of Swift 1.5 Flash Next - Heretic v2, a custom GGUF quantization of Qwen 3.8 Flash optimized for reverse engineering and coding, featuring new heretic weights and a layer-aware bit allocation strategy based on activation-energy auditing.

OpenBonsai 是一个 Apache-2.0 的后训练三值量化框架，可将 BF16 模型和微调压缩为约 2 GB 的三值 GGUF 文件，兼容 Prism 的 PTQ1_0 格式，并用 Qwen3.5-9B（18 GB → 2 GB）验证了效果。

Liquid AI 发布 LFM3-Omni-2B 的 GGUF 版本，支持在 CPU/GPU 上本地推理。该模型可理解文本、图像、视频和音频，并以文本和语音输出，专为设备端使用打造。

kaitchup released statistically-lossless SLQ-quantized GGUF versions of Qwen3.8-27B in three accuracy-target variants (EAR98, EAR98.5, EAR99), built by mixing donor quantized weights across 498 matrices.

AtomicChat 发布了基于 LiquidAI/LFM3-1.7B 的安全消融版 GGUF 模型，通过投影秩-1 LoRA 移除有害提示的拒答（61.7%降至0%），同时在 MMLU 上无精度损失，并提供 AD-Q6_K 和 AD-Q4_K_M 量化版本。

CCORE 发布了基于 Qwen3.5-4B 的独立微调模型 Qwen3.5-4B-3D-Coder，专注于通用代码编写与高质量 Three.js 交互式 3D 场景生成，目前已提供 Q4_K_M 和 Q8_0 量化版本，完整 FP16 权重和训练数据集将在训练完成后公开。

研究审计了 Ollama 与 Hugging Face 上的量化 GGUF 文件，发现部分低比特量化模型因未校准的量化配方而功能失效，发布了 quantcheck 工具并建议在发布前进行筛查与校准。

Liquid AI 发布 LFM3-1.7B 的 GGUF 版本，支持在 CPU/GPU 上本地推理，兼容 llama.cpp、Ollama、LM Studio 等工具，并提供经过量化感知蒸馏（QAD）的 Q4_0 检查点以恢复量化损失的精度。

Unsloth 发布 NVIDIA Nemotron 3.5 Bolt E4B 的 GGUF 量化版本（Q4_K_M），这是一个基于 Gemma-4-E4B-it 进一步后训练的小语言模型（SLM），专为低内存、低首字延迟的本地语音助手与嵌入式/边缘设备场景优化。

ISTA-DASLab released GSQ-RCO non-uniform GGUF quantizations of DeepSeek-V4.1-Flash-1017 (763.2B MoE) at 1.75-2.25 bits per weight, assigning per-tensor quantization types via gradient-based search so the model can run on a single 12GB GPU with llama.cpp.