empero-ai/Qwen3.8-9B-Distill · Hugging Face

Source: https://huggingface.co/empero-ai/Qwen3.8-9B-Distill

https://huggingface.co/empero-ai/Qwen3.8-9B-Distill#qwen38-9bQwen3.8-9B

Developed byEmpero

This repository contains model weights and configuration files in the Hugging Face Transformers format. These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, and other standard runtimes with Qwen3.5 architecture support.

Qwen3.8-9Bis a full-parameter distillation ofQwen3.8 2.4T A95Binto the Qwen3.5-9B architecture. The student was trained on**~70,000 curated teacher traces**from our internal Qwen3.8 distillation datasets — dense chain-of-thought spanning mathematics, code, general reasoning, instruction following, and tool use, quality-filtered before training.

The objective: bring the reasoning behavior of a frontier-scale teacher into a dense 9B that deploys on a single GPU.

https://huggingface.co/empero-ai/Qwen3.8-9B-Distill#highlightsHighlights

• Distilled chain-of-thought— every answer opens with a<think\>block learned directly from Qwen3.8 2.4T A95B traces rather than synthetic self-generated reasoning.
• Mathematics and code emphasis— the trace mix is deliberately weighted toward hard math and competitive programming, the domains where distillation moves the needle most at this scale.
• Native function callingper Qwen3.5’s specification — no wrapper or tool-specific fine-tune required.
• 262,144-token native context, inherited from the Qwen3.5 base.
• Full fine-tune— every parameter updated; not an adapter.

https://huggingface.co/empero-ai/Qwen3.8-9B-Distill#model-overviewModel Overview

• Type: Causal Language Model (text path of a vision-language base)
• Base:Qwen/Qwen3.5-9B
• Number of Parameters: 9B
• Training: SFT (off-policy distillation) on ~70,000 teacher traces
• Teacher: Qwen3.8 2.4T A95B (internal distillation datasets)
• Context Length: 262,144 natively

https://huggingface.co/empero-ai/Qwen3.8-9B-Distill#benchmark-resultsBenchmark Results

Measured withlm\-evaluation\-harness, HF backend, identical settings for base and student. Both models are reasoning models and are evaluated with the CoT protocols (gsm8k\_cot,mmlu\_flan\_cot\_zeroshot); MMLU covers all 57 subjects (~1,700 questions). Flexible-extract is the primary metric; strict-match requires exact answer formatting.

TaskMetricQwen3.5-9B (base)Qwen3.8-9BΔgsm8k_cotexact_match (flexible)0.8850.870−0.015gsm8k_cotexact_match (strict)0.8750.850−0.025mmlu (CoT, 57 subjects)acc (flexible-extract)0.5460.751****+0.205mmlu (CoT, 57 subjects)acc (strict-match)0.2510.511****+0.260 Sampling for generation:temperature=0\.6, top\_p=0\.95, top\_k=20(Qwen3.5 recommended settings).

https://huggingface.co/empero-ai/Qwen3.8-9B-Distill#quickstartQuickstart

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "empero-ai/Qwen3.8-9B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "A snail is at the bottom of a 10-meter well. Each day it climbs 3 meters, each night it slips back 2. How many days until it escapes?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)

out = model.generate(inputs, max_new_tokens=16384,
                     temperature=0.6, top_p=0.95, top_k=20, do_sample=True)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

A recenttransformersrelease with Qwen3.5 support is required, along with the Gated DeltaNet kernels (flash\-linear\-attentionand a CUDA-matchedcausal\_conv1dbuild) — without them the linear-attention layers fall back to slow, memory-hungry PyTorch ops.

https://huggingface.co/empero-ai/Qwen3.8-9B-Distill#best-practicesBest Practices

• Sampling:temperature=0\.6, top\_p=0\.95, top\_k=20. Greedy decoding on long generations is a known repetition-loop failure mode for reasoning models in this class.
• Output length: allow generousmax\_new\_tokens(16,384 recommended); every answer opens with a<think\>block. Parse and strip the<think\>\.\.\.</think\>span for end users.
• Scope: the model learned from teacher traces, not from its own rollouts — it inherits the teacher’s reasoning style, including occasional over-long deliberation on easy questions. The fine-tune is text-only; vision behavior is inherited from the base and was not evaluated here.

https://huggingface.co/empero-ai/Qwen3.8-9B-Distill#stay-in-the-loopStay in the loop

Sign up for the Empero newsletter at**empero.org**for releases, evals, and research notes.

https://huggingface.co/empero-ai/Qwen3.8-9B-Distill#support–donateSupport / Donate

If this model helped you, consider supporting the project:

• BTC:bc1qx6zepu6sfkvshgdmc4ewu6pk6rpadvpgffpp7v
• LTC:ltc1qv2mefzps2vtjcpwfx8xxdrpplrcvltswm68r7x

https://huggingface.co/empero-ai/Qwen3.8-9B-Distill#provenance–licensingProvenance & licensing

Weights are released underApache-2.0, inherited from the Qwen3.5-9B base. Shared for research and experimentation, as-is.

https://huggingface.co/empero-ai/Qwen3.8-9B-Distill#acknowledgementsAcknowledgements

• Developed and released byEmpero
• Base model:Qwen3.5-9B(Alibaba Qwen team)
• Training:TRL+Transformers
• Linear-attention kernels:flash-linear-attention,causal_conv1d
• Evaluation:lm-evaluation-harness(EleutherAI)