empero-ai/Qwythos-9B-v2-GGUF · Hugging Face

Source: https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF

Empero AI

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#qwythos-9b-v2-ggufQwythos-9B-v2-GGUF

GGUF quantizations of**empero-ai/Qwythos-9B-v2**forllama.cpp, Ollama, LM Studio, jan, KoboldCpp, and other GGUF runtimes.

Qwythos-9B-v2 is the new and improved Qwythos— all the deep chain-of-thought reasoning of the base Qwythos, with thelooping behavior fixed. The looping/degeneration that showed up under greedy or low-temperature decoding is trained out (6.7% → 0%), the nativeMTP head is restored, and the identity prompt is cleaned up — while knowledge and reasoning are held at (or above) the base Qwythos level.

The fix usesFTPO (Final-Token Preference Optimization): the exact token thatstartsa repetition loop is identified and the model is gently trained to prefer coherent alternatives at that one position, leaving the rest of the distribution — and therefore its knowledge and reasoning — untouched.

For full training details, evaluation numbers, and sample generations, see the**base model card**.

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#whats-new-vs-the-base-qwythosWhat’s new vs. the base Qwythos

• 🔁Looping behavior eliminated— repetition under greedy / low-temp decoding dropped6.7% → 0%.Greedy decoding is now safe— you no longer needrepeat\-penaltyas a band-aid.
• 🧩MTP head restored— the native multi-token-prediction module is back in the\-MTP\-files, so speculative-decoding (\-\-spec\-type draft\-mtp) works.
• 🧠Reasoning preserved— MMLU / GSM8K / GPQA / ARC held at or above the base Qwythos level (see the model card).
• 🪪Cleaner identity— states who it is once, only when asked.
• 🔓 Still intentionallyuncensored, still1M-token context(YaRN), stillmultimodal-capable(Qwen3.5 vision tower).

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#filesFiles

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#normal-text-weights–trunk-only-32-blocksNormal text weights — trunk only (32 blocks)

FileQuantSizeNotesQwythos\-9B\-v2\-Q4\_K\_M\.ggufQ4_K_M5.34 GiB / 5.74 GBrecommended default— smallest practical, good qualityQwythos\-9B\-v2\-Q5\_K\_M\.ggufQ5_K_M6.08 GiB / 6.52 GBbalanced quality / sizeQwythos\-9B\-v2\-Q6\_K\.ggufQ6_K6.95 GiB / 7.46 GBhigh qualityQwythos\-9B\-v2\-Q8\_0\.ggufQ8_08.87 GiB / 9.53 GBnear-losslessQwythos\-9B\-v2\-BF16\.ggufBF1616.69 GiB / 17.92 GBfull precision (conversion base) If you don’t know which to pick,Q4_K_M is the right starting point.

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#mtp-enabled-text-weights-33-blocks-nextn_predict_layers–1MTP-enabled text weights (33 blocks,nextn\_predict\_layers = 1)

These embed the restored Qwen3.5-compatible MTP head. Use them with llama.cpp builds that support MTP draft speculation (\-\-spec\-type draft\-mtp). The MTP matrices are retained atQ8_0in every quantized variant.

FileQuantSizeNotesQwythos\-9B\-v2\-MTP\-Q4\_K\_M\.ggufQ4_K_M + MTP5.50 GiB / 5.90 GBrecommended MTP defaultQwythos\-9B\-v2\-MTP\-Q5\_K\_M\.ggufQ5_K_M + MTP6.25 GiB / 6.71 GBbalanced quality / sizeQwythos\-9B\-v2\-MTP\-Q6\_K\.ggufQ6_K + MTP7.14 GiB / 7.67 GBhigh qualityQwythos\-9B\-v2\-MTP\-Q8\_0\.ggufQ8_0 + MTP9.11 GiB / 9.79 GBnear-losslessQwythos\-9B\-v2\-MTP\-BF16\.ggufBF16 + MTP17.14 GiB / 18.41 GBfull precision (conversion base)

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#vision-projector–for-image-inputVision projector — for image input

FileSizeNotesmmproj\-Qwythos\-9B\-v2\-BF16\.gguf0.86 GiB / 0.92 GBCLIP-style vision encoder + projector atBF16native precision;required for images, pairs with any text quant above The vision tower is inheritedunchanged from Qwen3.5-9B— it was frozen through both the base Qwythos SFT and the v2 FTPO fine-tune, so image behavior matches base Qwen3.5-9B. This mmproj is interchangeable with any Qwen3.5-9Bmmproj\-\*\.gguf.

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#hybrid-precision-quantization-gated-deltanet–ssm-tensorsHybrid-precision quantization (Gated-DeltaNet / SSM tensors)

Qwythos is ahybridmodel — a 3:1 mix of Gated-DeltaNet linear-attention (SSM) blocks and full-attention blocks. The linear-attention state tensors are disproportionately sensitive to low-bit quantization, so the K-quants here keep them at higher precision than the surrounding weights:

Quantssm\_alpha``ssm\_beta``ssm\_outQ6_KQ8_0Q8_0Q8_0Q5_K_MQ8_0Q8_0Q6_KQ4_K_MQ8_0Q8_0Q6_K The remaining SSM state tensors (ssm\_a,ssm\_conv1d,ssm\_dt,ssm\_norm) are kept atF32by the converter. This preserves the hybrid/SSM blocks for a small (~2–4%) increase in file size over a flat K-quant.Q8\_0andBF16are uniform and need no overrides.

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#quick-startQuick start

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#llamacppllama.cpp

llama-cli \
  -m Qwythos-9B-v2-Q4_K_M.gguf \
  -p "Walk through the biochemistry of how organophosphate nerve agents inhibit acetylcholinesterase." \
  -n 8192 \
  --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
  -c 16384

Because v2’s looping is trained out,\-\-repeat\-penaltyis now optional andgreedy decoding (\-\-temp 0) stays coherent.

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#ollamaOllama

ollama run hf.co/empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#lm-studio–jan–koboldcppLM Studio / jan / KoboldCpp

Drop any\.ggufinto your runtime’s model directory. Qwythos uses the standard Qwen3.5 chat template; modern GGUF runtimes load it automatically from the file.

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#mtp-draft-speculationMTP draft speculation

llama-server \
  -m Qwythos-9B-v2-MTP-Q4_K_M.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 6 \
  -c 16384 --port 8080

MTP support requires a recent llama.cpp build. If your runtime doesn’t support MTP yet, use the normal files.

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#vision-image-inputVision (image input)

Download a text quantplusthemmproj\-\*\.gguf, then run llama.cpp’s multimodal CLI/server:

llama-mtmd-cli \
  -m Qwythos-9B-v2-Q4_K_M.gguf \
  --mmproj mmproj-Qwythos-9B-v2-BF16.gguf \
  --image ./photo.jpg \
  -p "Describe this image in detail." \
  --temp 0.6 --top-p 0.95 --top-k 20 -c 16384

Honest note:all Qwythos training (base SFT and v2 FTPO) wastext-only— the vision tower was never fine-tuned, so image-grounded reasoning inherits base Qwen3.5-9B behavior and has not been independently evaluated for this release.

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#sampling-recommendationsSampling recommendations

Qwythos is a reasoning model — every response opens with a<think\>\.\.\.</think\>block before the answer.

ParameterValuetemperature0.6top\_p0.95top\_k20repeat\_penalty1.05 (optional in v2)max\_new\_tokens16384 Unlike the base Qwythos,v2 does not loop under greedy / low-temperature decoding— you can use\-\-temp 0for deterministic runs without repetition. The 0.6-temperature settings above still match Qwen3.5’s official thinking-mode recommendations for best quality.

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#long-context-1m-tokensLong context (1M tokens)

The GGUFs ship with YaRN rope-scaling baked in for a1,048,576-token context window(4× the 262,144 native). Set\-cup to1048576; lower it to reduce KV-cache memory for shorter prompts. A single H100/H200-class GPU comfortably handles 256k–512k; the full 1M typically needs multi-GPU or aggressive KV-cache offload.

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#conversion–verificationConversion & verification

• Converted and quantized withllama.cpp(convert\_hf\_to\_gguf\.py,llama\-quantize), architectureqwen35, GGUF v3.
• MTP variants: default conversion (33-block,nextn\_predict\_layers = 1, 15 MTP tensors, MTP matrices pinned Q8_0). Normal variants:\-\-no\-mtp(32-block trunk-only). mmproj:\-\-mmproj \-\-outtype bf16.
• Hybrid-precision overrides applied per the table above.
• Structurally verified (arch / block count /nextnkey / per-tensor types) and smoke-tested for load + coherent generation.
• shasum \-a 256 \-c SHA256SUMScovers all 11 artifacts.

https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF#license–acknowledgementsLicense & acknowledgements

Apache-2.0, inherited from Qwen3.5-9B. Shared for research and experimentation, as-is.

• Developed and released byEmpero AI
• Base model:Qwen3.5-9B(Alibaba Qwen team)
• Looping fixed withFTPO (Final-Token Preference Optimization)
• Quantization:llama.cpp(ggml-org)
• HF model:empero-ai/Qwythos-9B-v2