llama.cpp: Hy3 PR + GGUFs
Summary
The Hy3 model is now supported in llama.cpp via a pull request, with GGUF quantizations available. Early testing shows coherent output from Q2_K at 10-11 t/s on high-end hardware.
Similar Articles
model: add Hy3 (hy_v3) support with MTP speculative decoding by satindergrewal · Pull Request #25395 · ggml-org/llama.cpp
This pull request adds support for the Hy3 (hy_v3) model with MTP speculative decoding to llama.cpp, enabling efficient inference for this architecture.
Here is my llama.cpp NVFP4/MXFP6 GGUF quantizer tool
The author introduces an open-source GGUF quantizer tool for llama.cpp that creates NVFP4 and MXFP6 quantized models with advanced techniques like RSF, tensor promotion, and dynamic quantization, achieving better quality than existing methods like ModelOpt.
AngelSlim/Hy3-GGUF
AngelSlim/Hy3-GGUF is a toolkit for quantizing and deploying Hy3 large language models on llama.cpp, featuring MTP self-speculative decoding and a thinking/tool-call parser for efficient inference.
GnLOLot/MiniCPM5-1B-Claude-Opus-Fable5-Thinking-GGUF
A GGUF quantized version of MiniCPM5-1B-Claude-Opus-Fable5-Thinking model is released on Hugging Face, with usage instructions for llama.cpp, vLLM, and Ollama.
[llama.cpp] Asymmetric KV q8/q4 cache: current caveats and discussion in GGML repo
Discusses caveats of using asymmetric KV cache quantization in llama.cpp, where mismatched q8/q4 types cause prompt processing on CPU instead of GPU, and a proposed fix via compilation flags.