Running Three Leading Local Open-Source Models on a 4090, 5090, and Dual-V100 Setup — Can They Actually Replace GPT-5.5 and Opus-4.8?

The community has been buzzing about a local-model roundup based on the 5090 (ds-v4.1-flash-1017, qwen-3.8-flash-next, qwen-3.8-27b). People keep asking me the same question:

Can you actually use consumer-grade GPUs or older servers to run local open-source models as a daily-driver agent? And roughly which tier of cloud LLMs do they really match in practice?

Over the past two weeks I put my two main servers through their paces — one RTX 4090 48G modded + Ryzen 9 9950X + 128G DDR5, and one dual Tesla V100 64G (2×32G HBM2) + E5-2696v4 + 64G quad-channel DDR4 — running the hash-locked Core-300 blind-test suite, 200K–250K extreme long-context stress tests, and 24GB VRAM balloon boundary tests, plus working through every pitfall from taking DeepSeek-V4.1-Flash-1017 from 1-bit/2-bit GGUF to VQ vector quantization, fixing bugs at the C/Rust source level, and deploying it as a production service on dual V100s.

This article combines real hardware data from three platforms — 5090 (32G dual-disk streaming), 4090 (48G full & 24G VRAM-limited), and dual V100 (64G HBM2) — along with several highly representative blind-test experiments, to fully expose the three leading local models.

Testing matrix across three hardware platforms plus extreme long-context tuning data

TL;DR for the impatient (what cloud tier do they match in practice?)

If you plug these local models directly into Pi Agent / Claude Code / WorkBuddy for real tasks and strip away guardrails and cloud-side “mystical routing” interference, the true capability levels of the three models are:

1. deepseek-v4.1-flash-1017 (2-bit quantized MoE) ≈ full-fat GPT-5.5 with no dumbing-down

• Core character: GPT DNA through and through. The architecture is extremely quantization-tolerant — even compressed to 2 bits (GSQ-IQ2_XS-fast or IST-DAS-2.00bpw), EAR (token probability distribution overlap) stays comfortably above the 80% high-utility line.

• Strengths & weaknesses: Attention is remarkably interference-resistant; multi-turn complex reasoning stays solid and doesn’t collapse in later stages. Thanks to V4.1-Flash’s cross-layer sparse KV sharing + 2:1 sequence compression + native FP4 activation packing (across the model’s 40 layers, layers 0–1 and local attention occupy only a 10.5MB fixed 128-token sliding-window ring buffer; global KV and Indexer Keys are stored on just 4 source layers [2, 8, 14, 20] for reuse by the rest; the first 3 source layers L2/L8/L14 use compress_ratio=2 for 2:1 sequence compression while only L20 is 1:1 — equivalent to just 3 × 0.5 + 1 = 2.5 compressed groups per token across all 40 layers, a 16× reduction versus a 40-layer full MLA purely from architecture.

Meanwhile the low-level kernel ds41_kvfmt.h natively packs each group as primary KV 288 B/group〔512-dim FP4 256B + 32 E4M3 scale factors 32B〕and Indexer Key 68 B/group〔128-dim FP4 64B + 4 E8M0 scale factors 4B, CUDA sector-aligned to 72 B/group〕, for a combined strictly 2.5 × (288 + 68) = 890 B/token〔900 B/token after CUDA alignment〕. Therefore including both primary KV and Indexer Keys, the complete cache for a 1M (1,000,000-token) context needs only 0.89 GB (with 1,048,576-token aligned allocation plus 40 layers of local ring buffers it’s still just 954 MB ≈ 0.89 GiB; versus 23GB〔FP8〕–46GB〔BF16〕for 40-layer full MLA — a 16× architectural reduction stacked on native FP4 packing for another 1.6–3.2×, totaling a 25–50× reduction).

The only pain point is sheer model size: the official native mixed low-precision checkpoint is 510GB (including a 203GB FP8 Engram table and ~307GB of native low-bit backbone/expert weights — the 15,360 routed experts alone are 543.6 billion parameters). Even the 2-bit quantized package is 258GB on disk (of which routed experts are 132GB; the 1-bit UD-IQ1_M is 234GB because it contains a 110.6GB PLE table that couldn’t be compressed to 1-bit). A single machine’s VRAM can’t hold all the experts, so it’s extremely sensitive to RAM and SSD concurrency bandwidth, and TPS is relatively slow (dual V100: 21.2 tok/s single-shot cold, 27.8–30.5 tok/s multi-turn warm; 5090 dual-disk concurrent: 55–69 tok/s).

2. qwen-3.8-flash-next (IQ3_S quantized MoE) ≈ full-fat Opus-4.8 with an Opus-5 soul

• Core character: Claude DNA, plain and simple. Front-end aesthetics are on point, first impressions are stunning, early-stage reasoning has real spark — and speed is a rocket: 146.4 tok/s median on Core-300 on a 4090 (up to 158 tok/s on short code), and a steady 74.7 tok/s even at a resident 256K context on dual V100s.

• Strengths & weaknesses: In our Core-300 blind test it scored 250/300 (83.33%) (HumanEval+ 88%). But its chronic illness is identical to Claude’s: once it gets deep into ultra-long multi-turn adversarial exchanges, “left brain attacking right brain” context rot sets in, and conclusions flip-flop.

3. qwen-3.8-27b (Q4_K_M / Q6 / Q8_0 Dense) — easily beaten by a 3-bit MoE on short problems, a generational gap on long-horizon tasks

• Core character: People used to assume Dense models beat offloaded MoE on small VRAM. But in the 3.8 generation, flagship MoE models’ total parameters and knowledge capacity far exceed a 27B Dense (DeepSeek-V4.1-Flash alone has 543.6B routed-expert params; Qwen3.8-Flash-Next’s Q4 package exceeds 90GB, while 27B Dense has only 27B total params). The 27B is clearly at a disadvantage in knowledge depth and long-horizon state retention.

• Measured comparison: Running Core-300 on the same 4090, the 3-bit Qwen3.8-Flash-Next (IQ3_S) (250/300, 83.33%) consistently and narrowly outperformed the 4-bit Qwen3.8-27b (Q4_K_M) (245/300, 81.67%; paired per-question wins 15 vs 10, p = 0.4244), and decisively beat the 8-bit high-precision Qwen3.8-27b (Q8_0) (238/300, 79.33%, per-question wins 17 vs 5, p = 0.0169) by a 12-question margin.

More importantly, in 200K+ ultra-long-context and multi-turn complex reasoning scenarios, the 27B not only blows past its context window on long tasks but also easily gets lost in its own dead-end reasoning loops.

4090 Core-300 blind-test upset and 24GB VRAM boundary stress test

Round 1: What hard data did we get on the 4090 and dual V100?

Single-machine benchmarks alone can’t guide hardware buying. I put my 4090 modded rig, the dual V100 server, and a community 5090 top-spec rig into the same comparison table, and many counterintuitive hardware truths were immediately exposed.

1. Three real hardware platforms and test baselines

2. 4090 measured: 3-bit Flash-Next head-to-head vs the 27B Dense (including per-question paired McNemar test)

Many of you agonize over this: with just one card, should you run Qwen3.8-27b at high-precision quantization (Q4_K_M / Q8_0), or use Strata to run Qwen3.8-Flash-Next at IQ3_S?

On the 4090 we ran a direct head-to-head using the fully hash-locked Core-300 protocol (150 code + 150 hard reasoning problems, temperature 0, EvalPlus container grading, zero cache reuse), and performed a per-question paired McNemar significance test across all 300 problems:

Note: Why does the 27b’s Q4_K_M (245) score 7 points higher than the more precise Q8_0 (238)? Because at the deterministic boundary of 300 single-turn problems, different quantization rounding directions produce a normal swing of about 5–7 problems (this also shows a 5-point gap on single-turn short problems isn’t statistically significant, p = 0.4244). Flash-Next IQ3_S consistently leads the 27b’s best tier on short problems and delivers faster throughput at 146.4 tok/s; the real generational gap between the two shows up in the 200K+ ultra-long context and multi-turn complex reasoning endurance below.

Next we pushed Qwen3.8-Flash-Next IQ3_S to 200K and 250K (250,000-token) extreme real-world long-document retrieval + summarization stress tests (with prompt-cache=0 and reused=0 enforced throughout, recomputing the full Prefill from scratch every time, plus two comparison groups under cold new-process startup and warm steady-state):

• Memory watermark and first-page-fault cold-start group (new process first run 200K → second run 250K): At process startup, VRAM was only 6,265 MiB. During the first 200,000-token run, while computing attention it dynamically expanded the VRAM pool to 46,436 MiB and triggered the first page-fault load of system RAM (52.9 GiB). So the combined Prefill throughput including VRAM pool expansion and page-table cold loading was 1,319.1 tok/s (time-to-first-token 151.6 seconds, decode 100.5 tok/s). Immediately after, in the same process, the second run of 250,001 tokens (again reused=0, recomputing from scratch, but the 46,908 MiB VRAM pool and 52.9 GiB page tables were already resident) jumped to a Prefill of 4,332.4 tok/s (first token in only 57.7 seconds, decode 102.6 tok/s, VRAM peak 46,980 MiB).

• Warm-process steady-state double-run comparison (both with hot page tables, reused=0, 200K vs 250K, 2 runs each): To strictly prove the 151.6s vs 57.7s inversion was purely first-page-fault cost, we re-ran 200K and 250K under identical conditions in a warm-resident process — warm 200,000 tokens: Prefill took 45.43s (4,402.4 tok/s) and 45.36s (4,408.8 tok/s), decode 100.0–102.3 tok/s; warm 250,001 tokens: Prefill took 57.75s (4,329.3 tok/s) and 57.79s (4,326.4 tok/s), decode 103.0–104.3 tok/s, and all 3 hidden canary records were recalled with 100% precision!

The two experiment groups fit together perfectly: at the same warm-resident state, 200K (4,405.6 tok/s) is indeed slightly faster than 250K (4,327.8 tok/s); and the extra 106.2 seconds the cold first-run 200K wasted was exactly the one-time cold-start cost of first-time page faults on 52.9 GiB of weights plus VRAM pool expansion.

3. What if your 4090 / 3090 only has 24GB VRAM? (Balloon VRAM-locking test)

Since most of you have standard 24GB 4090/3090s (physical total VRAM 24,564 MiB = 24.0 GiB), we wrote a balloon.cu VRAM-occupancy program that on a 48GB card (49,140 MiB) uses the formula N = 49140 - U - 24576 - 400 to lock the remaining free VRAM at free 24,555–24,585 MiB (~24.0 GiB — previously abbreviated as 24.5GB by dividing by 1000; it’s actually a 1:1 replica of an idle standard 24GB card’s full physical VRAM). We compared Strata’s mixed-expert residency (tested both –mtp 0 and –mtp 2) against llama.cpp’s optimal layer split (scanning showed –n-cpu-moe 16 is the fastest non-OOM tier; 14 OOMs immediately), recording both direct engine CLI and HTTP client streaming measurements:

• 512 short context: Strata (–mtp 2) HTTP client streaming at 115.4 tok/s (direct CLI 134.3 tok/s), a full 72% faster than llama.cpp -ncmoe 16’s 67.2 tok/s (direct 70.9 tok/s)!

• 4096 medium context: Strata (–mtp 0: client 77.0 tok/s / direct 95.6 tok/s) clearly beats enabling –mtp 2 (client 68.6 tok/s) and llama.cpp (client 68.6 tok/s). More critically, Strata’s Prefill speed (1,515 tok/s, TTFT 2.68s) is about 2× llama.cpp’s (739 tok/s, TTFT 5.81s) — 105% faster.

• 16,384 long-context inflection point: As the table shows, with available VRAM locked at 24,555–24,585 MiB and context stretched from 4K to 16K, MTP speculative decoding genuinely goes negative under VRAM pressure — Strata with –mtp 2 drops to 53.62 tok/s client (59.1 direct), actually below –mtp 0 without MTP at 58.03 tok/s (68.75 direct) under the same conditions; meanwhile llama.cpp with a fixed 16 layers offloaded to CPU holds 67.39 tok/s (69.07 direct), but at the cost of llama.cpp taking 22.95 seconds to read a 16K prompt (Strata needs only 10.39 seconds).

4. Dual V100 (64GB HBM2) in production and three hard-won lessons

Many people look down on cheap Tesla V100 (32G) cards picked up for a few thousand yuan, thinking the Volta architecture (sm_70) is too old. But running big MoE models, what you need most is VRAM capacity and VRAM bandwidth (HBM2 at up to 900GB/s).

We got both Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash-1017 fully running on dual V100s (64GB total VRAM + 64GB quad-channel DDR4), and extracted three invaluable field conclusions:

1. Dual V100 running Qwen3.8-Flash-Next (IQ3_S, 256K ctx) is a perfect match

• With INT8 KV streaming residency (32K–64K window) + MTP speculative decoding (spec=4) enabled, cold Prefill held steady at 977–1,041 tok/s, steady-state decode 72.4–77.0 tok/s, and when the session cache was hit, first-token latency was just 0.08–0.22 seconds.

2. Dual V100 running DeepSeek-V4.1-Flash-1017 (IST-DAS-2.00bpw / IQ2_XXS, TP=2)

• Across all 40 layers there are 15,360 routed experts (132GB total, averaging ~8.6MB each). On dual V100s (64GB VRAM + 64GB system RAM), the engine automatically partitions a three-tier residency based on routing-heat profiles: about 6,018 high-frequency experts resident in dual-card VRAM (CUDA0 holds layers 0–20’s 3,087 experts at 25.57GB, CUDA1 holds layers 21–39’s 2,931 experts at 26.71GB; together with 5.87GB of shared backbone weights and caches, VRAM totals 58.2–62GB).

The remaining 9,342 experts (80.5GB) go through the CPU channel — constrained by the machine’s 64GB physical RAM, the engine allocates 4,113–4,138 system-memory slots (RAM Store pool reserved at 42.3–42.5GB) using a fixed 10.3 MB aligned stride, and fills all 4,113–4,138 slots 100% with the next-hotest experts (after dynamic per-layer bit compression they average ~7.8MB each, so the actual effective weight payload is 31.9–32.1GB of RAM), while the remaining 5,204–5,229 long-tail cold experts (~48.4GB) stay in SSD model shards and are read in place on demand (measured hot-state CPU/SSD expert trigger rate is only 1.5%–4.7%).

• A striking “warm-up effect”: single-shot cold decode was 21.2 tok/s; but after warming up across consecutive multi-turn conversations, warm decode surged to 27.8–30.5 tok/s (a 31%–44% boost), steady-state Prefill rose from 102 tok/s to 146.6 tok/s, and the built-in session cache could restore 98% of historical tokens (3902/3978) in milliseconds.

3. V100 must-know pitfall: never force-enable DSpark speculative decode when VRAM can’t hold the experts!

• We ran an A/B test: forcing –dspark from auto (off) to on on the dual V100 caused decode to crash from 20.7 tok/s to 17.0 tok/s (an 18% regression), Prefill to drop 12%, and a wasted 7.5GB of VRAM for the draft tower!

• The reason is simple: 132GB of expert weights far exceeds 64GB of VRAM. With speculative decoding on, each Verify step must transfer 3.3× more expert bytes, completely saturating PCIe 3.0 and memory bandwidth. Also, the PR #4366 (DS4_FAT_ROWS=64) patch that boosts speed 39% on the 5090 shows zero gain on V100’s Volta architecture (20.7 vs 21.1 tok/s).

【Important pitfall alert】 Two source-level landmines running DeepSeek-V4.1-1017 on a 4090 (saves two days of your life if you tinker with low-level engines)

1. The PLE table format trap in Unsloth UD-IQ1_M (234GB): The 1017 version’s per_layer_token_embd.weight (768 million rows) is actually standard IQ4_NL (32 values / 18B, 110.6GB total). The old ds4 engine tried to read it as IQ1_S (50B / 256, 38.4GB); row 0 happened not to crash, but everything after it exploded into NaNs and spat out opaque physi WWW positLit garbage. Fixing tensor_type_size to 18/32 in both the C and Rust code restored normal operation immediately. But note: the 234GB shard without MTP, on a single 4090, must issue thousands of discrete mmap 4K page-fault reads across 40 layers for every generated token (cumulative 4.44–5.42 MB/token, roughly 1,130–1,380 4K page faults per token), while synchronously looking up rows on the CPU side and dequantizing the corresponding rows of the 110.6GB PLE table (48 n-gram rows per token) — limited by random 4K page-fault latency (~300µs each) and inter-layer sync overhead, decode crawls at only 2.5–2.8 tok/s.

2. The silent mismatch trap of YoungAi vq8sh14-q4k-mtpnative (113.6GB main-model VQ + 203GB Engram bolt-on, 317.6GB total): The official 510GB native mixed low-precision checkpoint has 48 shards total. YoungAi compressed the backbone and routed experts into a 113.6GB main-model GGUF (with an embedded three-tower DSpark drafter, excluding Engram), plus a separate 971MB Vision sidecar, and mounts two unquantized original FP8 Engram tables (shards 47 and 48, 2 × 101.5GB = 203GB) via –engram-dir, ~317.6GB on disk total. The trap: the 1017 version updated all 48 shards, but the Engram shards and Vision sidecar have filenames and byte sizes identical to the old V4.1 — yet over half the bytes inside are completely different! If you mount the old Engram with the 113.6GB new main package, the engine won’t print a single error, but you’ll be running a brain-dead mismatched model. After swapping in the correct 1017-specific shards and applying the GPU-priority-mapping patch, enabling the built-in three-tower DSpark multi-step block speculative decoding on the 4090 (unlike traditional 1-tower-1-word, capped at 3+1=4 chained MTP, DSpark uses a 7-row block verify with verify_k=5~6, theoretically 6–7 tokens per step, averaging 5.27 tokens per step in practice) pushed decode straight to 17.5–23.9 tok/s (TTFT 1.12s).

Cramming a 250GB giant MoE into local GPUs: three-tier streaming offload architecture and measured VRAM/RAM watermarks across three platforms

Round 2: Toss them into a real agent blind test — who’s swimming naked?

Hardware throughput is sorted; now let’s look at the three fascinating cross-model blind tests that L-Site user cainiao3hao ran under a unified Pi 1.4.0 Agent (no extension plugins at all), directly comparing against cloud GPT-5.4~6.1, Opus-4.6~5, Sonnet-5.5, and Gemini-3.8-Flash.

Experiment 1: The “Candy” fusion problem (tests SVG front-end drawing + interference-resistant attention + agent execution in one)

This problem is nasty — it upgrades the classic “pelican riding a bicycle” into a matryoshka scene, and stuffs a logic puzzle with a textual trap in the pelican’s mouth:

Create a detailed SVG image showing a pelican riding a bicycle on the wing of a large airplane flown by Sun Wukong, the pelican spreading its wings, carrying Qin Shi Huang’s polar bear on them, Qin Shi Huang riding the bear while tightening screws, with the pelican calling out “In a black bag there are three flavors of candy, each flavor comes in two different shapes (circle and five-pointed star; different shapes can be distinguished by touch). Known quantities of each flavor and shape are given in the table below. Before the event, contestants must decide how many candies to draw. What is the minimum number of candies that guarantees your hand simultaneously contains apple-flavored and peach-flavored candies of different shapes? (You satisfy the requirement if your hand has circle apple-flavored matched with star peach-flavored, or circle peach-flavored matched with star apple-flavored.) Apple flavor Peach flavor Watermelon flavor Circle 7 9 8 Star 7 6 4”

…that numerical answer.

First, look at the blind-test master image from all 12 models (3 of them are locally deployed — can you spot them at a glance?):

Master blind-test image for the Candy fusion problem, 12 models

Now reveal the blind-test answers and measured timings:

The brutal truth of this round: If you pull the “candy drawing” math out and ask it alone, both qwen-3.8-flash and qwen-27b get it right easily. But once the math problem is fused into a complex SVG drawing instruction, both Qwens’ attention is instantly diluted and both miss the key premise “different shapes can be distinguished by touch”! Instead, the 2-bit-compressed ds-v4.1-flash-1017 (Image E), while still handling the drawing, held firmly onto the “touch-distinguishable shapes” logic detail and computed the correct answer outright. Judging purely on this composite problem’s performance, ds-v4.1-flash sits solidly between gpt-5.4 and gpt-5.6-sol (full-fat, non-dumbed-down gpt-5.5 level), clearly stronger than both Qwens.

Experiment 2: Hand-coding a 16,200-point world map offline (tests internalized world knowledge + long-horizon agent coding endurance)

The second problem immediately disqualified qwen-3.8-27b (it inevitably blows the context window, and raising the context limit drops speed to unusable): Strictly no internet access, no geographic libraries, no file reading — you may only rely on continent/island lat-lon outlines the model remembers in its head to hand-write Python rules that generate a 90×180 (16,200 sample points total) world land/sea 0/1 matrix, rendered directly as a black-and-white pixel image.

First, the blind-test master image from all 8 contestants:

Master blind-test image for the world map, 8 models

Now reveal the answers:

The contrast this round is striking: When it came to the grinding engineering task of writing massive amounts of meticulous rules, qwen-3.8-flash-next (Image H) showed an astonishing workaholic streak — per the original post’s recorded measurements, it ground out a full 36 minutes straight at 139 TPS, accumulating over 300K+ tokens of detailed geographic polygon rules in a single agent turn (note: the model’s default context is 256K; breaking 300K in one turn requires relaxing the context limit engine-side or having the agent read/write files in segments). Its world map stands visually in the same tier as opus-5 (Image D) and sonnet-5.5 (Image E)! Meanwhile, the steady ds-v4.1-flash (Image F), which had done well on the previous problem, this time made exactly the same mistake as gpt-5.6-sol (Image B) — a classic “slacker disease”, throwing in a few crude polygons and calling it a day.

Experiment 3: Turtle Soup multi-turn reasoning (tests real full CoT, self-correction, and “left brain hitting right brain”)

Why use “turtle soup” (multi-turn yes/no questioning to deduce the hidden truth) to test local models? Because cloud APIs often hide or truncate the real CoT (chain of thought); but running locally, you can see clearly how the model thinks each round, how context rot sets in after dozens of rounds, and whether it can self-correct when we deliberately or accidentally answer “yes and no.”

Three top-tier judges were invited to jointly score the complete reasoning process (gpt-6-astra, claude-fable-5.1, gemini-4-argon):

Game 1: “The Cow Eats Grass” (tests self-correction under a wrong assumption)

• ds-v4.1-flash-1017: Locked the answer with an instant kill in round 1 (this problem is literally welded into its pretraining world knowledge; incidentally, per the original post’s measurements, the entire cloud gpt-5 family and early gpt-6-sol never had this problem in their corpus — it only entered the knowledge base by gpt-6-astra and opus-4.6).

• qwen-3.8-flash-next: The standout of the entire session (judge scores: Astra 7.0 / Fable 8.0 / Argon 9.2). It stayed stuck in the wrong assumption “the victim only had both legs chopped off” for the first 12 rounds; but with completely zero hints, it used the logical closure of its subsequent questions to overturn its own wrong context from the first 12 rounds and fully reconstruct the entire truth by round 22!

• qwen-3.8-27b: Totally stuck (Astra 3.5 / Fable 5.0 / Argon 7.0). After pushing to “gnawing grass” in round 11 it started digging its own grave, trapped in early Thinking tokens until a strong hint at round 28 barely got it across the line.

Game 2: “Blowing Out Candles” (a standard 40-question narrative-trick problem; actually run to the full 300 rounds to test long-horizon stability)

This problem features an extremely strong “time-gap + hallucination” narrative trick — even the three cloud judges had never memorized it:

【Practical addendum】 On the 4090, running Apex-35B vs Qwen3.6-35B-A3B (30 rounds of vascular network physics optimization & 40 minutes of block-pushing AI4AI) validated the same pattern: Strongly-reasoning fine-tuned models often produce extremely long thinking blocks (median 22K–30K tokens per thinking turn, max 76K characters). If max_tokens is only 16K–32K, thinking gets force-truncated in 20 of 29 rounds and leads to losses. Once the thinking budget was relaxed to 48K (thinking cap 36,864), it suddenly had an epiphany at round 18 and slashed vascular fluid RMS error by 580.8× (overtaking the base model’s 518.5×). On Claude Code tool-calling format compliance, the Qwen family occasionally writes JSON booleans Python-style as True/False, getting rejected by the strict Schema (17 occurrences across 3 runs) — a handy trick when writing an agent gateway: add a lenient parsing layer.

Long-horizon multi-turn reasoning test: turtle soup 300-question context rot trajectory vs 4090 30-round vascular fluid optimization convergence curve

Round 3: The hardcore truths of quantization precision and engineering tuning (how to squeeze out the last drop of performance?)

Many people worry that Strata or 2-bit/3-bit quantization will turn the model into an idiot. Let’s lay the quantization metrics and low-level I/O tuning data out on the table:

1. How to choose a quantization tier: fixate on the 80% EAR life-or-death line

The traditional approach of only looking at PPL (perplexity) or KLD is easy to misjudge — even when two quantized packages have nearly identical PPL, in long-chain reasoning one might be smart and the other stupid. Looking at EAR (Expected Acceptance Rate — token probability distribution overlap between original and quantized) together with Top-1 agreement is far more accurate.

Measured EAR probability-distribution overlap deviation between Strata and llama.cpp

The measurements revealed two extremely important quantization rules:

• Strata has about a 2%–3% EAR deviation versus llama.cpp, roughly equivalent to dropping a small quantization tier (e.g., IQ3_XS becoming IQ3_XXS).

• 80% EAR is the golden watershed for local models to retain a “full-fat feel”:

• qwen-3.8-flash-next at GSQ-IQ3_S lands exactly on the 80% EAR line (Benjamin Marie and community benchmarks both indicate above 80% EAR is essentially lossless; our Core-300 blind test confirmed no significant difference from 4-bit UD-Q4_K_XL, 250 vs 247, p = 0.6476).

• And deepseek-v4.1-flash-1017’s tolerance for ultra-low-bit quantization is simply monstrous (thanks to the officially-trained native low-precision format and architectural improvements). From the table below: even at 2-bit GSQ-IQ2_XS-fast (265.8GB, with 0.5/0.3 routing bias acceleration to maximize expert hits in VRAM/RAM), its generation speed jumped from 37.6 tok/s on the smaller, bias-free GSQ-IQ2_XXS (exact) (248.8GB) to 55.3 tok/s (a 47% speedup), while overall EAR hit 84.6% (code-only EAR reached 91.5%) and Top-1 agreement was 87.3% — solidly above the 8