Ling 3.0 Tiny vs. Gemma 26B-A4B MoE

Introduction My audiobook pipeline has to decide who speaks each line of dialogue in a novel, so the TTS can cast a voice per character. It runs on Gemma 4 26B-A4B (QAT Q4) with the experts parked in system RAM. Ling 3.0 tiny looked like the opposite bet: 7.9B total, 1.3B active, hybrid KDA/MLA attention, 4.8 GB at Q4_K_M, so the whole model sits in VRAM on an 8 GB card and the prompt gets processed on the GPU instead of the CPU. I gave it the same three chapters as the incumbent, plus a smoke chapter. Here's what happened. Setup Hardware: NVIDIA T1000 8GB — Turing TU117, sm_75, no tensor cores, ~6.6 GB free with a browser and a mail client open. Ryzen 5 7600, 93 GB DDR5. Incumbent: unsloth/gemma-4-26B-A4B-it-qat-GGUF UD-Q4_K_XL, 14 GB. --cpu-moe with attention + KV on the GPU, 241 MB MTP draft head (--spec-type draft-mtp, n-max 2), temp 0.3 / top-p 0.95 / top-k 64, thinking off. About 3.2 GB resident. Challenger: inclusionAI/Ling-3.0-tiny-GGUF Q4_K_M, 4.82 GB (the official quant). 24 layers, 3 KDA (Kimi Delta Attention, recurrent) to 1 MLA per block, 128 experts with 8 routed + 1 shared active. Stock llama.cpp master: the bailingmoe3 arch was merged in August, no fork needed. Sampling per the model card: temp 1.0 / top-p 0.95 / top-k 20, thinking off through the chat template's enable_thinking flag. No draft: this checkpoint has num_nextn_predict_layers = 0, so the NEXTN head the model card advertises for sglang isn't in the weights and there's nothing for draft-mtp to load. Task: batched JSON extraction. Each request carries 32 quotations with ~700 chars of preceding and ~400 chars of following prose, the character roster, an intonation vocabulary (549 tags) and 6 turns of prior attributions — about 5.4K prompt tokens. What it took to get running, which was less than last time: It fits. llama.cpp's auto-fit projected 4801 MiB against 6346 MiB free and left 1544 MiB spare, so all 25/25 layers landed on the GPU with no forcing. Model buffer 4465 MiB plus 130 MiB of embeddings on the host, compute buffer 264 MiB. The KV cache is tiny because only 6 of the 24 layers are MLA (54 MiB at 8K, K only); the other 18 are recurrent state, 19 MiB in total. Resident: 4.9 GB at 8K, 5.0 GB at 16K. Context. Ling writes 50–70% more reasoning per quotation than Gemma, so 32-quotation batches overflowed 8192 (n_tokens = 8191, truncated = 1) on two of the first chapter's four batches; the pipeline's halve-and-retry recovered, at the cost of a wasted request each time. Gemma overflowed once on the same chapter. Raising ctx to 16384 costs 54 MiB of KV on Ling and 160 MiB on Gemma (only its 5 global-attention layers scale with context; the sliding-window layers don't), and nothing truncates after that. Peak use was 9855 tokens. One 32-quotation request through a hand-launched server: 245 tok/s prefill, 54 tok/s decode. Methodology Ground truth is three chapters (485 roster-attributed lines) of previously-attributed, hand-corrected output, plus an 11-line smoke chapter. I score exact speaker match on (start_offset, end_offset) keys, and separately: wrong roster name vs. roster character sent to Unknown vs. line dropped, attributions to a character flagged silent in the roster, narrator identification, off-roster speakers routed to the unknowns file instead of hallucinated onto the roster, and intonation agreement. The incumbent was re-run on the same chapters first, making for an A/B test under an identical prompt and pipeline. Single sample per chapter at each model's recommended sampling. Findings chapter lines Gemma Ling A (smoke) 11 not re-run 11/11 B 109 108/109 (99.1%) 79/109 (72.5%) C 152 148/152 (97.4%) 99/152 (65.1%) D 224 203/224 (90.6%) 98/224 (43.8%) total 485 459/485 (94.6%) 276/485 (56.9%) Wall-clock per chapter, engine time: B 697 s vs 265 s, C 517 s vs 325 s, D 857 s vs 551 s. Ling: 185–214 tok/s prefill, 53 tok/s decode, no draft. Gemma: 68–74 tok/s prefill, 21–24 tok/s decode with MTP acceptance ~0.9. Ling is 1.8× faster end to end. Narrator swap. B is a two-hander told in the first person. Ling exchanged the two leads on 30 of its 36 wrong lines (at 8K; 28 wrong at 16K), each with a confident rationale in which the narrator is being addressed by their own line. Narrator refusal. In C and D it labelled the narrator's own lines Unknown with the reason "an unnamed narrator character whose identity the prose does not reveal" — 16 lines in C, and 61 lines in D with no guess at all — while the prompt names the established narrator in so many words. Gemma sent 19 roster lines to Unknown across the three chapters, nearly all role-named characters it relabelled in lowercase ("engineer", "dispatch"). Third-party capture. Narrator lines handed to whichever other roster character is present: 8 to the household AI in C, 13 to another lead in D. Intonation regression. 8–13% agreement vs. Gemma's 54–73%. Same pattern as last time: it returns the reporting verb near the quote (asked, asserted, commanded, accused) where the reference carries a delivery tone (curt, deadpan, inquisitive). No mute character violations, no dropped lines, narrator field correct on every chapter for both. Ablations on chapter B. Gemma's sampling (temp 0.3, top-k 64): 76.1%, within noise of 72.5%. Thinking on at the pipeline's 4096-token reply budget: zero usable answers in ten minutes; it spends the whole budget reasoning before any JSON appears, and halve-and-retry went all the way down to 2 quotations per request. Thinking on with a 16000-token budget on one 32-quotation batch: 21/31 vs. 22/31 with thinking off, at 1.7× the tokens and 1.6× the time, with the same swaps. Conclusion Not adopting it. It's the fastest thing I've run on this card that still returns well-formed JSON for a 5K-token prompt, it fits with 1.5 GB spare where Gemma needs 14 GB of experts in system RAM, and it got the smoke chapter perfect. But 57% against 95% on the same chapters is unusable. Ling cannot hold on to who "I" is in a first-person scene, and no sampling or thinking setting moved that needle. It reads as a capability floor at 1.3B active, not a configuration problem. Follow-up Any other models to try? Must fit in ~6.6 GB or run with experts in RAM. Improving on Gemma's 90% for chapter D would be immensely helpful.