AngelSlim/Hy3-GGUF · Hugging Face

Source: https://huggingface.co/AngelSlim/Hy3-GGUF

Dedicated to building a more intuitive, comprehensive, and efficient LLMs compression toolkit.

📖Documentation| 🤗Hugging Face| 🤖ModelScope| 💬WeChat

https://huggingface.co/AngelSlim/Hy3-GGUF#hy3-llamacpp-quantizationHy3 llama.cpp Quantization

Quantize and run**Hy3 (hy_v3)**on llama.cpp with MTP self-speculative decoding and a thinking/tool-call parser. One script patches and builds llama.cpp; recipes and a chat template are included for quantizing your own low-bit GGUF from a calibration set.

Two parts below:Deploy(build & run) andQuantization.

https://huggingface.co/AngelSlim/Hy3-GGUF#quick-startQuick Start

https://huggingface.co/AngelSlim/Hy3-GGUF#buildBuild

bash setup_hyv3_llama.sh                    # clone into ./llama.cpp-hyv3, auto CUDA/CPU
bash setup_hyv3_llama.sh /path/to/target    # choose the clone directory
CUDA=0 bash setup_hyv3_llama.sh             # force a CPU-only build

The script pins a verified llama.cpp base commit, appliespatches/01(base arch) thenpatches/02(MTP + parser), and builds it. Binaries land in<target\>/build/bin/.

https://huggingface.co/AngelSlim/Hy3-GGUF#runRun

# plain serve (no speculative decoding)
./llama.cpp-hyv3/build/bin/llama-server -m /path/to/Hy3.gguf -ctk q8_0 -ctv q8_0 -fa on -c 65536

# serve + MTP self-speculative decoding (needs an MTP gguf, i.e. converted WITHOUT --no-mtp)
./llama.cpp-hyv3/build/bin/llama-server -m /path/to/Hy3-mtp.gguf --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 1 -ctk q8_0 -ctv q8_0 -ctkd q8_0 -ctvd q8_0 -fa on -c 65536

https://huggingface.co/AngelSlim/Hy3-GGUF#recommended-setupsRecommended setups

GPUsbuildMTP\-c(context)KV cache1× H20 (96 GB)IQ1_Mno\-c 65536``\-ctk q8\_0 \-ctv q8\_02× H20 (192 GB)IQ1_Myes/ (default)/ (f16)2× H20 (192 GB)Q4_K_Myes\-c 65536``\-ctk q8\_0 \-ctv q8\_0 \-ctkd q8\_0 \-ctvd q8\_04× H20 (384 GB)Q4_K_Myes/ (default)/ (f16) Weights: IQ1_M ~83 GiB, Q4_K_M ~166 GiB (MTP adds ~2 GiB plus a draft KV cache). 1 card fits only IQ1_M, and tightly — shrink context, quantize KV, no MTP. 2 cards run IQ1_M+MTP with room to spare (drop\-c/KV flags), but Q4_K_M+MTP is tight, so keep\-c 65536and q8_0 KV (main + draft). 4 cards run Q4_K_M+MTP comfortably.

https://huggingface.co/AngelSlim/Hy3-GGUF#troubleshootingTroubleshooting

• Error: no such instruction: vdpbf16ps(inggml\-cpu’svec\.cpp/sgemm\.cpp): the CPU has AVX512-BF16 but the system assembler (old binutils) can’t encode it. The script auto-detects this and adds\-DGGML\_NATIVE=OFF; force it withGGML\_NATIVE=0 bash setup\_hyv3\_llama\.shif needed. Unrelated to the patches; CUDA inference is unaffected.
• undefined reference to SSL\_get1\_peer\_certificate: the system OpenSSL is too old. OpenSSL is off by default here (\-DLLAMA\_OPENSSL=OFF); it only affects the server’s HTTPS model download, not local GGUF serving. SetOPENSSL=1if you have OpenSSL 3.0+ and need it.

https://huggingface.co/AngelSlim/Hy3-GGUF#quantize-with-your-own-dataQuantize with Your Own Data

If you have a calibration set, you can compute your own importance matrix and quantize the model yourself. All steps use stock llama.cpp tools; the mixed-precision recipes and a minja-compatible chat template are shipped in this folder.

https://huggingface.co/AngelSlim/Hy3-GGUF#the-mixed-precision-recipesThe mixed-precision recipes

The recipes inrecipes/spend bits where they matter:

• Attention(attn\_q/k/v) and thetoken embedding / output headstay high (q8_0 / q4_K / q6_K) — cheap in size, and where low bits hurt most.
• Shared experts(ffn\_\*\_shexp, active on every token) stay at q5_K–q6_K.
• Routed experts(ffn\_\*\_exps) carry the aggressive low bits and dominate the file size. In the IQ1_M recipe most layers are iq1_m, withffn\_downa bit higher (iq3_xxs) and sensitive layers upgraded (iq2_xxs) layer-by-layer.

recipetargetMTP headrecipes/hyv3\_q4km\_recipe\.txt~Q4_K_M mixednorecipes/hyv3\_q4km\_mtp\_recipe\.txt~Q4_K_M mixedyesrecipes/hyv3\_iq1m\_recipe\.txt~IQ1_M mixed (extreme)norecipes/hyv3\_iq1m\_mtp\_recipe\.txt~IQ1_M mixed (extreme)yes The\*\_mtpvariants add the MTP block (blk\.<n\>\.nextn\.\*and layer-<n\>experts). Use them only for a gguf that still has the MTP head; use the plain ones for a gguf converted with\-\-no\-mtp. (The MTP block’s experts stay at K-quant, not IQ*, because very-low-bit IQ types need an imatrix and the imatrix only covers the trunk layers.)

https://huggingface.co/AngelSlim/Hy3-GGUF#1-convert-hf-%E2%86%92-bf16-gguf1. Convert HF → BF16 GGUF

PYTHONPATH=./llama.cpp-hyv3/gguf-py python ./llama.cpp-hyv3/convert_hf_to_gguf.py /path/to/HYV3-hf --outfile Hy3-BF16.gguf --outtype bf16
#   add --no-mtp to drop the MTP head (then use a non-mtp recipe below)

https://huggingface.co/AngelSlim/Hy3-GGUF#2-compute-the-importance-matrix-imatrix2. Compute the importance matrix (imatrix)

The calibration file is plain text containing samples with the model’s special tokens already applied (one file, fed via\-f). The imatrix is independent of the target quant type — compute it once on BF16 and reuse it for any recipe.

./llama.cpp-hyv3/build/bin/llama-imatrix -m Hy3-BF16.gguf -f calib.txt -o imatrix.gguf --output-format gguf --parse-special
#   single GPU can't hold BF16 -> add --cpu-moe (keep experts on CPU) or -ngl 0

https://huggingface.co/AngelSlim/Hy3-GGUF#3-quantize-with-a-recipe3. Quantize with a recipe

Pick the recipe matching your target and whether the gguf has an MTP head (table above).\-\-token\-embedding\-typediffers: keep it at q8_0 for Q4_K_M (embedding is cheap, no reason to crush it), drop it to q4_K for the extreme IQ1_M build. The examples below are for MTP ggufs — use thenon\-mtprecipe for a\-\-no\-mtpgguf.

# ~Q4_K_M mixed
./llama.cpp-hyv3/build/bin/llama-quantize --imatrix imatrix.gguf --tensor-type-file recipes/hyv3_q4km_mtp_recipe.txt --token-embedding-type q8_0 --output-tensor-type q6_K Hy3-BF16-mtp.gguf Hy3-Q4_K_M-mtp.gguf Q4_K_M

# ~IQ1_M mixed (extreme)
./llama.cpp-hyv3/build/bin/llama-quantize --imatrix imatrix.gguf --tensor-type-file recipes/hyv3_iq1m_mtp_recipe.txt --token-embedding-type q4_K --output-tensor-type q6_K Hy3-BF16-mtp.gguf Hy3-IQ1_M-mtp.gguf IQ1_M

https://huggingface.co/AngelSlim/Hy3-GGUF#4-optional-embed-a-minja-compatible-chat-template4. (Optional) Embed a minja-compatible chat template

Hy3’s built-inchat\_templateuses Pythonstr\.format, which llama.cpp’s minja engine can’t evaluate → chat/\-\-jinjamode crashes.hyv3\_opensource\_chat\_template\.jinjais a static version (suffix hard-coded to:opensource,\.formatexpanded). Bake it into the gguf so\-\-jinjaworks without\-\-chat\-template\-fileat runtime:

PYTHONPATH=./llama.cpp-hyv3/gguf-py python -m gguf.scripts.gguf_new_metadata --chat-template "$(cat hyv3_opensource_chat_template.jinja)" --force Hy3-IQ1_M-mtp.gguf Hy3-IQ1_M-mtp-tmpl.gguf