linuxid10t/G9v3-39A5B-GGUF · Hugging Face

Source: https://huggingface.co/linuxid10t/G9v3-39A5B-GGUF

https://huggingface.co/linuxid10t/G9v3-39A5B-GGUF#g9v3-39a5b-ggufG9v3-39A5B GGUF

This repository containsunofficial BF16 and Q4_K_M GGUF conversionsofai9stars/G9v3\-39A5B, a preview Mixture-of-Experts language model from AI9Stars.

The conversion was created and tested with a llama.cpp implementation of the G9v3 architecture available on thefeature/g9v3\-supportbranch. Until G9v3 support is merged upstream, standard llama.cpp builds will not load this model.

https://huggingface.co/linuxid10t/G9v3-39A5B-GGUF#available-filesAvailable files

FileFormatSizeSHA-256G9v3\-39A5B\-BF16\.ggufBF1672.63 GiB (77,988,959,200 bytes)016c0c8a708fc5d4e2378df1eb9b5f089d7e91b581914e2565fb819bcebe9196``G9v3\-39A5B\-Q4\_K\_M\.ggufQ4_K_M21.94 GiB (23,559,222,240 bytes)0c28e98eef634ca2304a492c7b2c92d8599189ae4409177f7edf50da5b082eb0 Choose the file according to your hardware and quality requirements:

FormatDescriptionRecommended system RAMBF16Highest-fidelity GGUF. Source BF16 tensors are preserved, along with required F32 tensors.96 GiB or moreQ4_K_MMixed K-quant that keeps selected tensors at higher precision; practical for local CPU inference.32 GiB or more The memory recommendations include headroom for the runtime, context, and operating system. Actual requirements vary with context size and llama.cpp settings.

https://huggingface.co/linuxid10t/G9v3-39A5B-GGUF#model-detailsModel details

PropertyValueArchitectureG9v3ForCausalLM, sparse MoEParametersApproximately 39B total / 5B active per tokenLayers38 (1 dense layer followed by 37 MoE layers)Hidden size2,048Attention32 query heads, 2 key/value heads, head dimension 128Experts320 routed experts, 32 selected per token, plus 1 shared expertExpert intermediate size512Vocabulary130,560 tokensMaximum context131,072 tokensLanguagesEnglish and ChineseFeaturesThink/No-Think modes, tool calling, long context The base model is a preview release. Refer to theai9stars/G9v3\-39A5Bmodel cardfor the original description and usage guidance.

https://huggingface.co/linuxid10t/G9v3-39A5B-GGUF#llamacpp-setupllama.cpp setup

Build the G9v3-enabled branch:

git clone --branch feature/g9v3-support --single-branch \
  https://github.com/linuxid10t/llama.cpp.git
cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp/build --target llama-cli llama-server -j

https://huggingface.co/linuxid10t/G9v3-39A5B-GGUF#command-lineCommand line

Both GGUF files contain the model’s chat template. Start an interactive conversation with the Q4_K_M file using:

./llama.cpp/build/bin/llama-cli \
  -m G9v3-39A5B-Q4_K_M.gguf \
  -c 4096 -t 32 -cnv

For BF16, use the same command with the other filename:

./llama.cpp/build/bin/llama-cli \
  -m G9v3-39A5B-BF16.gguf \
  -c 4096 -t 32 -cnv

Start with a modest context such as 4,096 tokens, then increase it according to your available memory and workload.

https://huggingface.co/linuxid10t/G9v3-39A5B-GGUF#openai-compatible-serverOpenAI-compatible server

./llama.cpp/build/bin/llama-server \
  -m G9v3-39A5B-Q4_K_M.gguf \
  -c 4096 -t 32 -tb 32 \
  --host 127.0.0.1 --port 8080

Replace the model filename withG9v3\-39A5B\-BF16\.ggufto serve the BF16 version on a machine with sufficient memory.

Example request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "G9v3-39A5B-Q4_K_M.gguf",
    "messages": [{"role": "user", "content": "Who are you?"}],
    "temperature": 0.7,
    "top_p": 0.95,
    "max_tokens": 256
  }'

Bind the server to a non-loopback address only after configuring appropriate authentication and network controls.

https://huggingface.co/linuxid10t/G9v3-39A5B-GGUF#conversion-provenanceConversion provenance

The source BF16 checkpoint was converted to GGUF and then quantized with:

python convert_hf_to_gguf.py G9v3-39A5B \
  --outfile G9v3-39A5B-BF16.gguf \
  --outtype bf16

./build/bin/llama-quantize \
  G9v3-39A5B-BF16.gguf \
  G9v3-39A5B-Q4_K_M.gguf \
  Q4_K_M 32

Conversion environment:

• llama.cpp base commit:9d57ce456(b10436)
• G9v3 support commit:77b13da1a
• GGUF version: V3
• Tensor count: 568 in each file
• BF16 weight size: 74,371.15 MiB, 16.01 bits per weight
• Q4_K_M weight size: 22,462.90 MiB, 4.84 bits per weight

https://huggingface.co/linuxid10t/G9v3-39A5B-GGUF#validation-and-performanceValidation and performance

The Q4_K_M model was tested end-to-end with bothllama\-cliandllama\-serveron a CPU-only system. The performance figures below apply only to Q4_K_M.

HardwareContextThreadsPrompt processingGenerationAMD Ryzen 9 9950X, 64 GiB RAM, CPU only4,0963250.18–57.9 tokens/s10.0–10.03 tokens/s The exact rate will vary with prompt length, thread count, memory speed, build options, and operating system.

Validation performed:

• Successful load and text generation from the Q4_K_M file
• Successful/v1/chat/completionsrequest with separate reasoning content
• BF16 GGUF metadata and all 568 tensor shapes loaded successfully
• BF16 file successfully used as the source for Q4_K_M quantization
• Converter dry run passed against all 15 source checkpoint shards after rebasing onto llama.cppb10436
• Synthetic G9v3 CPU architecture test: zero numerical error
• GGUF serialization round-trip test: passed

The BF16 file was not generation-benchmarked on the 64 GiB validation machine; it requires a system with more memory for representative testing.

https://huggingface.co/linuxid10t/G9v3-39A5B-GGUF#samplingSampling

The base model authors recommend the following settings:

ModeTemperatureTop-pThink1.00.95No Think0.70.95

https://huggingface.co/linuxid10t/G9v3-39A5B-GGUF#limitationsLimitations

• Q4_K_M is a lossy quantization and may perform differently from the BF16 file and original checkpoint.
• The BF16 GGUF preserves source precision but has much higher memory and storage requirements.
• G9v3 is a preview model and may produce inaccurate, biased, or unsafe output.
• Outputs should be independently reviewed before use in medical, legal, financial, safety-critical, or other high-stakes settings.
• This GGUF currently depends on the linked G9v3-enabled llama.cpp branch.
• Long-context capability was not benchmarked as part of this conversion test.

https://huggingface.co/linuxid10t/G9v3-39A5B-GGUF#license-and-attributionLicense and attribution

The original model is released by AI9Stars under the Apache License 2.0. This conversion retains that license. See the base model repository for its complete license terms and attribution information.

This conversion is independently produced and is not an official AI9Stars release.