Ling 3.0 tiny a 7.9B/1.3B hybrid reasoning MoE https://huggingface.co/inclusionAI/Ling-3.0-tiny…

inclusionAI/Ling-3.0-tiny · Hugging Face

Source: https://huggingface.co/inclusionAI/Ling-3.0-tiny

🤗Hugging Face| 🤖ModelScope| 🐙OpenRouter

https://huggingface.co/inclusionAI/Ling-3.0-tiny#introductionIntroduction

We are introducingLing-3.0-tiny, a lightweight hybrid reasoning MoE model with7.9Btotal parameters and only1.3Bactivated parameters per token. It is designed to deliver strong reasoning and agentic capabilities at low inference cost, making advanced model capabilities more accessible for local and resource-constrained deployment. BF16, FP8, and INT4 weights are provided for a wide range of hardware and deployment settings.

Key highlights of the model are summarized below:

• **Efficient Hybrid-Linear Architecture:**Ling-3.0-tiny integrates a 3:1 alternating stacking of KDA and MLA (3 Kimi Delta Attention layers followed by 1 Multi-Head Latent Attention layer per 4-layer block) with a sparse MoE FFN comprising 128 routed experts. Each token activates only 8 routed experts and 1 shared expert, allowing the model to balance long-context modeling capability, parameter efficiency, and computational cost.
• **Native Hybrid Reasoning and Agentic Capabilities:**Ling-3.0-tiny supports both fast responses and multi-step reasoning, with thinking mode configurable per request throughenable\_thinking. It delivers balanced performance across general agent tasks, coding, mathematical and scientific reasoning, and instruction following.
• Local and Edge Deployment:Designed for efficient local deployment, Ling-3.0-tiny has been validated onNVIDIA DGX Spark, Apple Silicon MacBook, and Mac mini, enabling capable reasoning and agentic workloads without datacenter-class GPUs. With FP8, Ling-3.0-tiny reaches around100-105 tokens/s on DGX Sparkand86-90 tokens/s on an M4 Pro MacBook, with approximately8.34 GiB peak memory usageat an 8K context length.

https://huggingface.co/inclusionAI/Ling-3.0-tiny#model-overviewModel Overview

Ling-3.0-tiny inherits the hybrid linear attation architecture of Ling-3.0 series, while being specifically optimized for lightweight and accessible deployment. The model has 7.9B total parameters, with only 1.3B parameters activated per token.

The architecture of Ling-3.0-tiny is designed to make computational efficiency serve real-world agentic performance.

• A 3:1 KDA–MLA architecture (3 KDA layers and 1 MLA layer per 4-layer block) provides more efficient long-context processing;
• A sparse MoE FFN with 128 experts activates 8 routed experts and 1 shared expert per token, enabling broad model capabilities with only 1.3B activated parameters per token.
• Native hybrid reasoning enables fast responses for routine tasks and multi-step reasoning for complex tasks within a single model.

Overall, these designs deliver the inference efficiency needed to deploy lightweight models in real-world agentic workflows.

https://huggingface.co/inclusionAI/Ling-3.0-tiny#evaluationEvaluation

We evaluated Ling-3.0-tiny across agentic tasks, coding, long-context understanding, knowledge reliability, mathematical and scientific reasoning, and instruction following. Ling-3.0-tiny achieves a score of25on the Artificial Analysis Intelligence Index v4.1.1 and16on the Artificial Analysis Agentic Index. In Artificial Analysis testing, Ling-3.0-tiny reaches an output speed of over160 tokens/s, with approximately18 secondsof end-to-end latency for a 500-token response, including reasoning time. These results highlight the model’s efficiency relative to its 1.3B activated parameter footprint.

The following table presents representative benchmarks for Ling-3.0-tiny:

• Thinking mode is enabled by default. The recommended sampling parameters for Ling-3.0-tiny aretemperature=1\.0,top\_p=0\.95, andtop\_k=20. - Terminal-Bench 2.1: Evaluated under the Artificial Analysis (AA) protocol using the default Terminus 2 harness, a unified 2-hour timeout, the provided JSON parser in preserve-thinking mode, and 3 runs per task (mean). Decoding uses temperature=1.0, max_new_tokens=32K, with a 256K context window.

https://huggingface.co/inclusionAI/Ling-3.0-tiny#quickstartQuickstart

https://huggingface.co/inclusionAI/Ling-3.0-tiny#sglangSGLang

The hardware- and recipe-specific launch matrix (BF16/FP8 × Low-Latency / High-Throughput / HiCache + Mooncake), with a live command generator and verified configurations, lives in the SGLang cookbook:

Cookbook:https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-tiny

https://huggingface.co/inclusionAI/Ling-3.0-tiny#install-sglangInstall SGLang

Use the pre-built image that tracks the Ling-3.0 runtime:

docker pull lmsysorg/sglang:dev-Ling-3.0-tiny

https://huggingface.co/inclusionAI/Ling-3.0-tiny#run-inferenceRun Inference

Recommended low-latency recipe (built-in MTP / NEXTN, 256K YaRN context) on 1× 141GB-class GPU (H20-3e) or a 1-GPU Blackwell node:

Server

docker run --rm --gpus all --ipc=host --shm-size 32g \
  -p 30000:30000 \
  -e HF_TOKEN=<your-hf-token> \
  lmsysorg/sglang:dev-Ling-3.0-tiny \
  env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 \
  python3 -m sglang.launch_server \
    --model-path inclusionAI/Ling-3.0-tiny \
    --tp 1 \
    --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":2.0,"rope_theta":6000000,"partial_rotary_factor":0.5,"original_max_position_embeddings":131072}}' \
    --context-length 262144 \
    --speculative-algorithm NEXTN \
    --mem-fraction-static 0.8 \
    --host 0.0.0.0 \
    --port 30000

Client

Thinking is enabled by default by both the chat template and theling3reasoning parser. Disable it per request with"chat\_template\_kwargs": \{"enable\_thinking": false\}. We recommend the sampling parameterstemperature=1\.0,top\_p=0\.95, andtop\_k=20.

curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "auto",
       "messages": [{"role": "user", "content": "What is the capital of France?"}],
       "stream": true,
       "temperature": 1.0,
       "top_k": 20,
       "top_p": 0.95
     }'

For\-\-reasoning\-parser ling3/\-\-tool\-call\-parser ling3, the HiCache + Mooncake L3 setup, and GSM8K / bench_serving reproduction commands, see the cookbook page linked above.

https://huggingface.co/inclusionAI/Ling-3.0-tiny#vllmvLLM

https://huggingface.co/inclusionAI/Ling-3.0-tiny#install-vllm-with-ling-30-supportInstall vLLM with Ling-3.0 Support

pip install uv

uv venv ~/my_ling_env

source ~/my_ling_env/bin/activate

git clone -b ling_3_0 https://github.com/inclusionAI/vllm-ling-v3.git

cd vllm-ling-v3

VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto

https://huggingface.co/inclusionAI/Ling-3.0-tiny#run-inference-1Run Inference

Here is the example to run Ling-3.0-tiny with a single GPU, where the server port is$\{PORT\}:

Server

vllm serve "$MODEL_PATH" \
    --port "$PORT" \
    --trust-remote-code \
    --served-model-name auto \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.85 \
    --enable-prefix-caching \
    --mamba-cache-mode align \
    --enable-auto-tool-choice \
    --tool-call-parser ling3 \
    --reasoning-parser ling3

Client

For better performance, We recommend settingenable\_thinking=truewithtemperature=1\.0,top\_p=0\.95, andtop\_k=20.

curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "auto",
       "messages": [{"role": "user", "content": "What is the capital of France?"}],
       "chat_template_kwargs": {"enable_thinking": true},
       "stream": true,
       "temperature": 1.0, 
       "top_k": 20,
       "top_p": 0.95
     }'

https://huggingface.co/inclusionAI/Ling-3.0-tiny#ollamaOllama

• This configuration has been verified on an M4 Pro Mac with 48 GB of unified memory.

https://huggingface.co/inclusionAI/Ling-3.0-tiny#preparation-and-buildPreparation and Build

git clone https://github.com/ollama/ollama.git
cd ollama
git fetch origin refs/pull/17643/head:bailing-moe-v3
git switch bailing-moe-v3

cmake -B build .
cmake --build build --parallel 8

• Support is currently provided byollama/ollama#17643and is limited to running via MLX on Apple Silicon. - Use the local\./ollamaexecutable built from source in this section. This functionality is not yet included in the official Ollama release.

https://huggingface.co/inclusionAI/Ling-3.0-tiny#import-modelImport Model

Replace/absolute/path/to/bf16\_weightswith the absolute path to the BF16 model weights directory. The imported model will be namedling\-tiny\-bf16:

printf 'FROM /absolute/path/to/bf16_weights\n' > /tmp/Modelfile.ling
./ollama create ling-tiny-bf16 --experimental -f /tmp/Modelfile.ling

https://huggingface.co/inclusionAI/Ling-3.0-tiny#start-serviceStart Service

Set the default context length to 8192, and then start the Ollama service:

# The service listens on http://127.0.0.1:11434 by default
OLLAMA_CONTEXT_LENGTH=8192 ./ollama serve

https://huggingface.co/inclusionAI/Ling-3.0-tiny#call-apiCall API

curl -sS http://127.0.0.1:11434/api/generate -d '{
  "model": "ling-tiny-bf16",
  "prompt": "<role>SYSTEM</role>detailed thinking on<|role_end|><role>HUMAN</role>Calculate 17 × 23 and output only the number.<|role_end|><role>ASSISTANT</role>\n<think>",
  "raw": true,
  "think": true,
  "stream": false,
  "options": {
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20,
    "num_predict": 2048
  }
}' | jq -r .response

Ant Ling (@AntLingAGI): Ling-3.0-tiny is now available as an open-weight model in BF16, FP8 and INT4. On Artificial Analysis, it scores 25 on the Intelligence Index and 16 on the Agentic Index, with 772 Elo on GDPval-AA v2 and 20.80 on τ³-Banking—built for real task execution. 🧵