Ling-3.0-tiny is a very interesting model. Run on NVIDIA Orin Nano Super 8GB at 128K context with IQ4_NL quant.

I have been searching for suitable model to run on my 8GB RAM toy, NVIDIA Orin Nano Super 8GB. This little toy was priced at $249 earlier this year (not any more), and pulls very little power when idle. It was an interesting device that suitable for an agent to host on. It is likely won't do any coding job in meaningful way, but likely can handle simple things just fine. When the llama.cpp support showed up on github, I asked my Hermes to setup it up and run some basic test for me. The result is quite encouraging. Below is the summery from my Hermes run. ===============AI content below=================== # Ling-3.0-tiny at full 128K context on a $249 8GB Orin Nano Super — IQ4_NL, 33 tok/s, on llama.cpp master **Tl;dr:** I got inclusionAI's Ling-3.0-tiny (7.9B total, ~1.4B active MoE) running on an NVIDIA Jetson Orin Nano Super 8GB with the **IQ4_NL quant at its full native 131,072-token context** . Decode runs ~33 tok/s short-context (faster than you can read), prompt eval 220-760 tok/s, and the whole thing fits in 7.4 GiB unified RAM with only ~200 MB of swap. Model + KV + CUDA offload, all on one $249 board. ## The hardware - **Jetson Orin Nano Super Developer Kit** — [official NVIDIA product page](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/nano-super-developer-kit) — was **$249 at launch** (price has risen since; SparkFun lists it at $399 now). 8 GB unified LPDDR5, 67 TOPS, JetPack R39.2 / CUDA 13.2. ## The model + quant - **Original:** [inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny) — hybrid KDA+MLA reasoning MoE, 128 experts, 8 routed + 1 shared active per token - **GGUF repo:** [bartowski/Ling-3.0-tiny-GGUF](https://huggingface.co/bartowski/Ling-3.0-tiny-GGUF) - **File used:** [`Ling-3.0-tiny-IQ4_NL.gguf`](https://huggingface.co/bartowski/Ling-3.0-tiny-GGUF/blob/main/Ling-3.0-tiny-IQ4_NL.gguf) — 4.40 GB, IQ4_NL (4.5 bpw), imatrix-calibrated - **Why IQ4_NL:** at 4.30 GiB it's the largest ≥4-bit quant that still leaves room for the full 131K KV window on 8 GB. Q5/Q6 fit only to ~64K; IQ4_NL goes the distance. ## The engine — this matters The stock quantizer release **will not load this model** . Ling-3.0-tiny is a **BailingMoE V3** architecture, which only landed in llama.cpp via: - **PR [#26608 — BailingMoE3 Support]( https://github.com/ggml-org/llama.cpp/pull/26608 )** (merged into master Aug 17, 2026, adds arch + speculative MTP support) - **Branch: `ggml-org/llama.cpp` master** , anything after that merge — clone fresh, don't use a release tarball - Built on-device with CUDA 13.2 for sm_87 (`-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=87`), all 24 layers offloaded **Working command:** ``` llama-server -m Ling-3.0-tiny-IQ4_NL.gguf -c 131072 \ --cache-type-k q8_0 --cache-type-v q8_0 \ -ngl 99 -fa on -b 512 -ub 256 --jinja -t 6 \ --temp 1.0 --top-p 0.95 --top-k 20 ``` ## Speed (measured on-device) | Test | Result | |---|---| | Prompt eval (2-8K) | 450-760 t/s | | Prompt eval (full 125K prefill) | 220-264 t/s | | Decode (short ctx) | 33 t/s | | Decode @ 96-128K depth | 15-17 t/s | | TTFT (512 tok) | ~1 s | Memory at 128K: **7.2/7.4 GiB used** , ~200 MB swap. The hybrid design keeps the KV footprint small — only the 6 MLA layers carry context-proportional cache while the 18 KDA layers keep a fixed-size recurrent state — which is what makes the full window fit. ## Is it coherent at 128K? Mostly. I ran a proper needle-in-haystack: **128 unique passkeys** embedded across the context, each queried individually at temp 0: - **96K context: 92% retrieval** (118/128) — 100% in the first 32K - **128K context: 88% retrieval** (113/128) — with a visible cliff in the final 32K (67%) - Strict full-string match is lower (~30-40%) because the model often emits just the tail chunk of the passkey — a generation artifact, not a retrieval miss - Failure modes in the deep zone: digit-mangling (needle 105 → answers 015's key) and a couple of confabulations So: **majority passed, not a flawless 128K** — but at 96K the cliff disappears entirely, and for a $249 board that's genuinely impressive capability-per-dollar. ## Verdict Worth spending more time on. The hybrid KDA/MLA architecture + IQ4_NL + llama.cpp master is a compelling recipe for edge deployment: agentic reasoning MoE, native 128K window, stable 15-33 tok/s, all in 8 GB unified memory. I'd like to see more folks poke at the 96-128K attention degradation — and whether the MTP spec-decode support from #26608 helps on-device. Anyone else running Ling-3.0-tiny on constrained hardware? What quant/config are you using?