Great to see labs shipping their own GGUFs 🔥 https://huggingface.co/AngelSlim/Hy4-preview-GGUF…

AngelSlim/Hy4-preview-GGUF · Hugging Face

Source: https://huggingface.co/AngelSlim/Hy4-preview-GGUF Three GGUF builds of Hy4-Preview:https://huggingface.co/tencent/Hy4-preview

Language / 语言:English·中文

filesizebpwnotesHy4\-preview\-Q4\_K\_M\.gguf435.20 GiB4.86standard 4-bit, safe defaultHy4\-preview\-UD\-IQ1\_M\.gguf219.83 GiB2.44Using the UD-IQ1_M quantization strategy, half the sizeHy4\-preview\-STQ1\_0\.gguf213.66 GiB2.38Using the MIX_STQ1_0 quantization strategy, half the size **Neither file runs on stock llama.cpp.**Thehyv4architecture is not upstream. Apply the patches inhy4\-preview\-patch/

https://huggingface.co/AngelSlim/Hy4-preview-GGUF#englishEnglish

https://huggingface.co/AngelSlim/Hy4-preview-GGUF#1-what-these-are1. What these are

Hy4\-preview\-Q4\_K\_M\.gguf— a conventional Q4_K_M. Most tensors are Q4_K;ffn\_down\_expsgets Q6_K on 37 layers via llama.cpp’s own logic. Use this unless you are memory-constrained.

Hy4\-preview\-UD\-IQ1\_M\.gguf- mixed precision with UD-IQ1_M strategy at ~2.44 bpw, roughlyhalf the sizefor the same model. The routed-expertgate/upprojections run at 1.75 bpw (IQ1_M) and 2.0625 bpw (IQ2_XXS).

Hy4\-preview\-STQ1\_0\.gguf— mixed precision with MIX-STQ1_0 strategy at ~2.38 bpw, roughlyhalf the sizefor the same model. The routed-expertgate/upprojections run at 1.3125 bpw (STQ1_0) on 29 layers and 2.0625 bpw (IQ2_XXS) on the other 48. See section 3.

https://huggingface.co/AngelSlim/Hy4-preview-GGUF#2-running-them2. Running them

Build a patched llama.cpp

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 0cea36222

git apply hy4-preview-patch/0001-hyv4-architecture.patch
git apply hy4-preview-patch/0002-stq1_0-quant-and-cuda.patch   # skip if only using Q4_K_M

export PATH=/usr/local/cuda-13.0/bin:$PATH CUDACXX=/usr/local/cuda-13.0/bin/nvcc
cmake -B build-cuda -DGGML_CUDA=ON -DLLAMA_CURL=OFF -DGGML_NATIVE=OFF \
      -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=90 \
      -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build-cuda --target llama-cli llama-bench llama-quantize -j 48

Set\-DCMAKE\_CUDA\_ARCHITECTURESfor your GPU (90 = H20/H100). Both\-DLLAMA\_BUILD\_UI=OFFand\-DLLAMA\_USE\_PREBUILT\_UI=OFFare needed for an offline build; the first alone still downloads prebuilt assets.

Then

# single prompt
build-cuda/bin/llama-cli -m Hy4-preview-Q4_K_M.gguf -ngl 99 -c 8192 \
    --temp 0 -n 512 --no-warmup --jinja -st -f prompt.txt

# throughput
build-cuda/bin/llama-bench -m Hy4-preview-STQ1_0.gguf -ngl 99 -p 512 -n 128 -r 3

• **\-\-jinjais required for chat.**The HY4 chat template matches no llama.cpp built-in family.
• **Keep the GGUF on local disk.**llama.cpp mmaps weights; over NFS random page faults run at ~12 MB/s, turning a 1-minute load into hours.
• Use\-st \-f prompt\.txtfor a single prompt.\-no\-cnvis ignored in this build and it will spin printing\>on EOF.
• VRAM for full residency: ~435 GiB (Q4_K_M) or ~214 GiB (STQ1_0). With less, lower\-ngl.

Measured on 8x H20:

prefill (pp512)decode (tg128)STQ1_0204.56 ± 1.42 t/s20.47 ± 0.02 t/s Python tools reading these files must use the patchedgguf\-pywith anabsolutepath:sys\.path\.insert\(0, '/path/to/llama\.cpp/gguf\-py'\).

https://huggingface.co/AngelSlim/Hy4-preview-GGUF#3-stq1_0-and-the-mixed-precision-strategy3. STQ1_0 and the mixed-precision strategy

The format.STQ1_0 comes from llama.cpp PR #22836. Weights are ternary\{\-d, 0, \+d\}, withexactly one of every four lanes forced to zero(3:4 sparsity). Each group of 4 weights is a 4-bit code plus a 1-bit table-select, indexing a 32-entry codebook; one fp16 scale covers 256 weights. That is2 \+ 32 \+ 8 = 42bytes per 256 weights =1.3125 bpw.

**Our encoder.**Upstream’s quantizer targets QAT inputs already on the ternary grid: it ignores the imatrix, setsd = amax, and zeroesargmin \|x\|. That is weak for post-training quantization. We keep the format byte-identical and change only two decisions:

1. Weighted least-squares scale,d = sum\(w\*sel\*x\) / sum\(w\*sel^2\)instead ofd = amax.
2. Imatrix-aware zero placement— zero the lane minimisingw\[j\]\*\(x\[j\]^2 \- \(\|x\[j\]\|\-d\)^2\), theincrementalcost rather than the smallest magnitude.

alternating for 3 rounds. Measured on 1200 real expert rows: the LS scale alone gives**-89.7%weighted SSD, and the imatrix terms a further-4.1%**of the remainder. The headline win is the scale —amaxpinsdto the single largest outlier among 256 weights.

**Where the bits go.**The three routed-expert families are 97.7% of all parameters, so the recipe spends freely on everything else:

familySTQ1_0 buildwhyffn\_gate\_exps/ffn\_up\_expsSTQ1_0 (29 layers) / IQ2_XXS (48 layers)the bulk; layer choice is imatrix-derivedffn\_down\_expsIQ3_XXS, IQ4_XS on last 3writes straight into the residual stream, so its error is not attenuated by a later gate — deliberately 2 levels higherattention out / gate / q_aQ5_Kllama.cpp only auto-bumps these whenn\_expert == 8; HY4 has 256MLAq\_b/k\_b/v\_b/kv\_a\_mqaQ8_0HY4’ssplitnames miss llama.cpp’s substring match, so they get no automatic bumpDSA indexerQ8_0 / F32105 tensors, 0.21 GiB total, gates which 2048 tokens each query seesiHC\*\_fn, router, norms, sinkF32mirrors the reference’s\_keep\_in\_fp32\_modules``output(lm_head)F32via\-\-leave\-output\-tensor

https://huggingface.co/AngelSlim/Hy4-preview-GGUF#4-building-a-runtime4. Building a runtime

https://huggingface.co/AngelSlim/Hy4-preview-GGUF#re-quantizing-from-bf16Re-quantizing from bf16

The recipe files are included.An imatrix is mandatory for STQ1_0— its encoder uses it for the scale solve and zero placement.

build-cuda/bin/llama-quantize --dry-run --imatrix imatrix.gguf \
    --tensor-type-file Hy4-preview-STQ1_0.tensortypes --leave-output-tensor \
    HY4.bf16.gguf out.gguf IQ1_M          # Q4_K_M build: use Q4_K_M as the base ftype

https://huggingface.co/AngelSlim/Hy4-preview-GGUF#%E4%B8%AD%E6%96%87中文

filesizebpwnotesHy4\-preview\-Q4\_K\_M\.gguf435.20 GiB4.86标准4bit ggufHy4\-preview\-UD\-IQ1\_M\.gguf219.83 GiB2.44使用 UD-IQ1_M 混合精度量化压缩Hy4\-preview\-STQ1\_0\.gguf213.66 GiB2.38使用 MIX-STQ1_0 混合精度量化压缩

https://huggingface.co/AngelSlim/Hy4-preview-GGUF#1-%E8%BF%99%E6%98%AF%E4%BB%80%E4%B9%881. 这是什么

Hy4\-preview\-Q4\_K\_M\.gguf—— 常规 Q4_K_M。多数张量为 Q4_K，ffn\_down\_exps由 llama.cpp 自身逻辑提到 Q6_K（37 层）。没有显存压力就用这个。****Hy4\-preview\-STQ1\_0\.gguf—— 约 2.44 bpw 的混合精度，使用 UD-IQ1_M 混合精度量化压缩, 同一个模型体积减半。Hy4\-preview\-STQ1\_0\.gguf—— 约 2.38 bpw 的混合精度，使用 MIX-STQ1_0 混合精度量化压缩, 同一个模型体积减半。路由专家的gate/up在 29 层用 1.3125 bpw（STQ1_0），另 48 层用 2.0625 bpw（IQ2_XXS）。见第 3 节。

https://huggingface.co/AngelSlim/Hy4-preview-GGUF#2-%E5%A6%82%E4%BD%95%E4%BD%BF%E7%94%A82. 如何使用

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 0cea36222

git apply hy4-preview-patch/0001-hyv4-architecture.patch
git apply hy4-preview-patch/0002-stq1_0-quant-and-cuda.patch   # 只用 Q4_K_M 可跳过

export PATH=/usr/local/cuda-13.0/bin:$PATH CUDACXX=/usr/local/cuda-13.0/bin/nvcc
cmake -B build-cuda -DGGML_CUDA=ON -DLLAMA_CURL=OFF -DGGML_NATIVE=OFF \
      -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=90 \
      -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build-cuda --target llama-cli llama-bench llama-quantize -j 48

\-DCMAKE\_CUDA\_ARCHITECTURES按自己的 GPU 设置（90 = H20/H100）。离线构建同时需要\-DLLAMA\_BUILD\_UI=OFF与\-DLLAMA\_USE\_PREBUILT\_UI=OFF，只给前者仍会去下载预构建资源。

# 单条 prompt
build-cuda/bin/llama-cli -m Hy4-preview-Q4_K_M.gguf -ngl 99 -c 8192 \
    --temp 0 -n 512 --no-warmup --jinja -st -f prompt.txt

# 测速
build-cuda/bin/llama-bench -m Hy4-preview-STQ1_0.gguf -ngl 99 -p 512 -n 128 -r 3

• **chat 必须加\-\-jinja。**HY4 的 chat template 不匹配 llama.cpp 任何内置模板家族。
• **GGUF 必须放本地盘。**llama.cpp 用 mmap，NFS 随机页错误约 12 MB/s，本来 1 分钟的加载会变 成几小时。
• **单条 prompt 用\-st \-f prompt\.txt。**本 build 忽略\-no\-cnv，遇 EOF 会一直打印\>。
• 全量驻留显存需求：约 435 GiB（Q4_K_M）或约 214 GiB（STQ1_0）。不够就降低\-ngl。

在 8 x H20 上实测（已确认 GPU 空闲、权重全驻显存）：

预填充 (pp512)解码 (tg128)STQ1_0204.56 ± 1.42 t/s19.52 ± 0.01 t/s 读这些文件的 Python 工具必须用打过补丁的gguf\-py，且用绝对路径：sys\.path\.insert\(0, '/path/to/llama\.cpp/gguf\-py'\)。

https://huggingface.co/AngelSlim/Hy4-preview-GGUF#3-stq1_0-%E4%B8%8E%E6%B7%B7%E5%90%88%E7%B2%BE%E5%BA%A6%E7%AD%96%E7%95%A53. STQ1_0 与混合精度策略

格式。STQ1_0 来自 llama.cpp PR #22836。权重为三值\{\-d, 0, \+d\}，且每 4 个 lane 强制 一个为零（3:4 稀疏）。每 4 个权重存成 4-bit code 加 1-bit 选表位，索引一张 32 项码本；每 256 个权重共用一个 fp16 scale。即每 256 权重2 \+ 32 \+ 8 = 42字节 =1.3125 bpw。

我们的编码器。上游的量化器面向已落在三值网格上的 QAT 输入：直接忽略 imatrix，取d = amax，并把零放在argmin \|x\|。这对训练后量化（PTQ）很弱。我们保持格式逐字节一致， 只改两个决策：

1. 加权最小二乘 scale：d = sum\(w\*sel\*x\) / sum\(w\*sel^2\)，取代d = amax。
2. imatrix-aware 零位置：零掉使w\[j\]\*\(x\[j\]^2 \- \(\|x\[j\]\|\-d\)^2\)最小的 lane，即比较增量代价，而非单纯的最小幅值。

两者交替 3 轮。在 1200 行真实专家权重上实测：仅最小二乘 scale 就带来**-89.7%加权 SSD， imatrix 项在残差上再补-4.1%**。主要收益来自 scale——amax会把d钉在 256 个权重里 的单个最大离群值上。

**bit 花在哪。**三个路由专家族占全部参数的 97.7%，所以配方在其余张量上舍得花：

张量族STQ1_0 产物原因ffn\_gate\_exps/ffn\_up\_expsSTQ1_0（29 层）/ IQ2_XXS（48 层）体积主体；选层由 imatrix 推导ffn\_down\_expsIQ3_XXS，最后 3 层 IQ4_XS直接写回残差流，误差不会被后续 gate 衰减，故刻意高两档attention out / gate / q_aQ5_Kllama.cpp 只在n\_expert == 8时自动提档，而 HY4 有 256 个专家MLAq\_b/k\_b/v\_b/kv\_a\_mqaQ8_0HY4 的拆分命名匹配不上 llama.cpp 的子串规则，完全拿不到自动提档DSA indexerQ8_0 / F32105 个张量共 0.21 GiB，却是决定每个 query 能看到哪 2048 个 token 的闸门iHC\*\_fn、router、norms、sinkF32对齐参考实现的\_keep\_in\_fp32\_modules``output（lm_head）F32通过\-\-leave\-output\-tensor

https://huggingface.co/AngelSlim/Hy4-preview-GGUF#%E4%BB%8E-bf16-%E9%87%8D%E6%96%B0%E9%87%8F%E5%8C%96从 bf16 重新量化

配方文件已随附。STQ1_0 强制需要 imatrix——它的编码器要用 imatrix 做 scale 求解与零位置选择。

build-cuda/bin/llama-quantize --dry-run --imatrix imatrix.gguf \
    --tensor-type-file Hy4-preview-STQ1_0.tensortypes --leave-output-tensor \
    HY4.bf16.gguf out.gguf IQ1_M          # Q4_K_M 产物：基础 ftype 用 Q4_K_M

https://huggingface.co/AngelSlim/Hy4-preview-GGUF#filesFiles

hy4-preview-patch/
  0001-hyv4-architecture.patch        18 files, +1632/-3   both GGUFs need this
  0002-stq1_0-quant-and-cuda.patch    25 files, +683/-4    STQ1_0 only
  Hy4-preview-STQ1_0.tensortypes      the STQ1_0 recipe
  Hy4-preview-Q4_K_M.tensortypes      the Q4_K_M recipe

Tencent Hy (@TencentHunyuan): We compressed Hy4-preview from 1.5TB to ～200GiB GGUF and it still works well !

Meet MIX-STQ1_0.The trick isn’t just going low, it’s deciding where: calibration data picks each layer’s bit-width, some down to 1.31-bit STQ1_0, some up to 2.06-bit IQ2_XXS. Same budget, lower