A Complete Guide to Local Deployment of Qwen3.8 27B From Scratch · Routes for Mac and NVIDIA Cards · Includes 4090 Benchmark Data Let me start with a real-world test result I didn’t expect myself: A compressed 9.83GB model matched the performance of the 30GB version on my set of 72 short tasks. The conclusion upfront: a standard 12GB gaming card (like the 3060 12G or 4070) can run it, and it handles short tasks just fine. It works offline, costs zero tokens, and you don’t have to upload your data to anyone. (First, let’s clarify two things: Equal scores don’t mean identical performance—the two versions differ in quantization, VRAM usage, and long-context performance. Chapter 4 will compare them in detail. Tie on short tasks doesn’t mean a tie on long tasks—community testing on 41 agent long tasks shows 2-bit versions lag noticeably (Q2_K_XL 27/41 vs Q4_K_M 35/41), and in my third round of testing, Q2 required 2.14 times more thinking time.) Let’s take half a minute to explain what this is. Qwen3.8 27B is a model just open-sourced by Alibaba on August 14 under the Apache 2.0 license, free for commercial use. It has 27 billion dense parameters, natively supports images and video, but its main strength is productivity: writing code, calling tools, reading long texts—its official positioning is an on-device Agent model with 256K context, expandable to 1M. Within two days of its release, downloads on Hugging Face surpassed a million, with immediate support for NVIDIA, Mac, and AMD. Many are calling it the “most cost-effective on-device Agent model available today.” Official benchmarks show SWE-bench Pro at 61.7, claiming to surpass Claude Opus 4.6 Max—that’s the official claim; third-party replications are still in progress. I’ll share my own tests in Chapter 4. It has two design features directly related to your budget, which we’ll reference repeatedly in later chapters: - Hybrid Attention: Out of 64 layers, only 16 use full attention, saving about 75% on KV Cache compared to traditional models—this is why even 24GB gaming cards can handle texts over 100,000 characters (Chapter 5 will break down the math). - Built-in MTP (Multi-Token Prediction): The speculative decoding module is integrated directly into the model file, no separate download needed. Just enable a parameter to double generation speed (Chapter 7 will toggle this switch). This article covers one thing: how to install it on your machine and unlock its full potential. I’ll start with “Can your computer run it, and will it be smooth?” and go all the way through installation, tuning, and understanding its speed and bottlenecks. Every technical term will be explained in plain language the first time it appears. No prior local deployment experience is required. Who this is for: - Those wanting to install a large model on their own computer but don’t know where to start. - Those with a decent Mac or NVIDIA card but unsure if the specs are sufficient. - Those who have installed it but don’t know how to optimize for speed (jump to Chapter 5 and Chapter 7). My direct reason for writing this was a misconception I had: I always thought “opening a larger window would slow down the model,” so I avoided increasing the window size. After testing, I realized window size, content length, and VRAM usage are three entirely different metrics—confusing these leads to all subsequent configuration being guesswork. Chapter 5 is dedicated to this. This article is long. Pick what you want to read. The full text is about 20,000 words, and you likely don’t need to read it all. It’s organized around four questions. Click the link to jump to the relevant section (if the link doesn’t work, search for the chapter number): 1. What is it, and is it worth installing? → Introduction + Chapter 2 2. Can your machine run it, and which version should you choose? - Can it run, and what’s the experience? → Chapter 1, 5 minutes. - Too many versions, don’t know which to download? → The selection table below is sufficient; see Chapter 4 for the rationale. - Using a Mac? → Section 6.9, written separately. 3. How to install - Already decided, just need the commands → Chapter 6.1, copy and paste (halfway through the text; search for “6.1” to find it). - Encountering errors during installation → Chapter 8 “Five Pitfalls” + Section 6.8. 4. How to optimize for speed - Installed but it’s slow → Chapter 5 + Chapter 7. Want the full process and data? Read from the beginning, about 40 minutes. Four rounds of quality testing, DFlash 2 comparison, 5090 benchmarks, and other details are covered in a separate article: “I Tried Four Times to Prove Quantized Versions Are Worse, but Couldn’t.” ## If You Don’t Want to Read It All, This Section is Enough Here are all the conclusions upfront; read on if you want to know how they were derived. Which version should your graphics card download? | Version | Size | Recommended Card | | — | — | — | | Original BF16 | 55.56GB | 80GB professional card or 128GB+ Mac | | Q8_0 / Q6 | 23–30GB | 32GB+ | | Q8attn (4-bit main body) | 19.1GB | 32GB+ | | Q4_K_M | 16.5GB | 20GB+, recommended | | Q2_K_XL | 9.83GB | 12–16GB card | | IQ2_S / IQ1_M | 6.7–8.4GB | Don’t use—significant quality loss | The naming seems messy, but the pattern is simple: The smaller the number, the smaller the file, the less VRAM used—but over-compression makes it dumb. The original is most accurate but requires 80GB-class hardware; since this guide is about running on a consumer GPU, we’ll start from compressed versions. Q4_K_M is the balance point I found through testing. > A note on the 12–16GB tier: I only tested Q2_K_XL in this range. Unsloth also offers Q3_K_XL (12.24 GiB) and IQ4_XS (13.27 GiB), which fit exactly in this VRAM range, but I haven’t tested them. Community data on 41 agent long tasks shows: Q2_K_XL 27/41, Q3_K_XL 33/41, Q4_K_M 35/41—2-bit versions complete about 20% fewer tasks. So: for short tasks only, Q2_K_XL is fine; for agent long tasks on a 16GB card, try Q3_K_XL first instead of copying my Q2 configuration. This table only answers “which file to download and what card is needed.” For each tier’s maximum window size, speed, and VRAM usage, refer to the summary table in Chapter 1—that’s the only complete reference in the entire article; other sections are based on it. Mac users should see the Mac-specific table in Chapter 1: 16GB can only make it work with strict conditions, while 24GB is the minimum. Key conclusions to remember: - Opening a large window only increases VRAM usage, not slowdown—it only becomes slow when you actually fill it with content. This is the most important point in the entire article. - Installation takes about 14 minutes (downloading 12.5 minutes is the bulk); use hf-mirror for domestic connections. - Don’t guess bit depth from the filename—Q8attn is 4-bit main body, not 8-bit. - The cost of quantization: it gets the answer right but thinks longer—smaller quantized versions have higher tok/s but may take 2–3 times longer thinking, so total time doesn’t necessarily save. - For speed configuration, MTP3 is sufficient—the newer DFlash 2 is actually slower in daily scenarios (this is for NVIDIA cards; on Mac it’s the opposite, see 6.9). - The model loading doesn’t mean it’s usable—always test with a real request; logs saying model loaded can be misleading. If you want to know how each conclusion was derived and what pitfalls I encountered, read on. For quality testing alone, I ran four rounds, and the first two were wasted—the main text only includes results that affect your selection, with the complete testing process detailed in a separate article. (If unsure where to start, go back to the introduction’s roadmap: “This article is long. Pick what you want to read.”) Chapter 0 below is a glossary—you can skip it for now. Each term will be explained when it first appears in the main text; just come back to check if you’re unfamiliar. If you already know terms like KV Cache, MTP, and quantization, no need to pause. (Common terms like token and context won’t be explained; only frequently used and potentially confusing terms are covered.) tok/s (tokens per second) How many tokens the model outputs per second; all speed metrics use this. Reference: Humans read about 5–10 characters per second, so 47 tok/s means “you can’t keep up.” VRAM / Unified Memory NVIDIA users look at VRAM (dedicated graphics card memory); Mac users look at Unified Memory (memory and VRAM are the same block, shared). The model must fit entirely into this to run fast, the most important hard metric in this article. Quantization (Q4, Q6, Q8, etc.) “Compressing” the model for storage—more compression saves more VRAM but may lose some precision. Smaller numbers mean more compression: Q2 saves the most, Q8 preserves the most. Chapter 3 explains the principle. GB and GiB (they’re not the same thing) GB uses 1000-based units; GiB uses 1024-based units. The same file has different numbers in two units—for example, Q2_K_XL is 9.83GB and 9.15GiB; don’t mistake them for two versions. This article uses GB; only near download commands is GiB noted, because HuggingFace pages and wget progress bars show GiB—match the numbers carefully when verifying file size. Window The maximum content the model can remember at once, like the size of a desk. You can set it to 32K (small desk) or 256K (large conference table) at startup. Remember: a big desk ≠ slow work, explained in detail in Chapter 5. Prefill When you send content, the model doesn’t respond immediately—it must first process everything you sent (build the KV Cache) before answering. This wait is prefill. Sending two sentences takes 0.2 seconds; sending 250,000 tokens takes about three and a half minutes. KV Cache Intermediate results cached during input processing. It allows generating each new token without recomputing all previous content. Longer context = larger KV Cache = more VRAM usage—this is one of the main VRAM consumers. MTP (Multi-Token Prediction) An acceleration technique: predicts upcoming tokens in advance, verifies them, and outputs multiple at once, saving time from sequential generation. The hit rate of predictions is called the “acceptance rate”—high acceptance means faster speed; low rates can be slower (wasted computation). Quantized KV Cache (q8 / q4) The KV Cache itself can also be compressed. q8 has higher precision and uses more VRAM; q4 saves half the VRAM with slightly lower precision. KV Cache is split into K and V parts, which can be set separately (e.g., K with q8, V with q4), a common practice. Idle / Loaded When comparing speeds, I’ll use these terms: Idle means the window is open but only a sentence or two has been sent; Loaded means the window is full of content. Speeds vary greatly between these two, explained in Chapter 5. Offload Large models are split into many layers. Layers on the GPU run fast but use VRAM; moving them to memory saves VRAM but runs slower. Used as a fallback when VRAM is insufficient. ubatch How many tokens the model processes at once when reading your content. Like flipping multiple pages at once—more pages might be read faster, but not always better, with real-world tests in Chapter 7. Cold Start / Hot Start Cold start is the first load after booting, when the model isn’t in VRAM, requiring extra loading time; Hot start is after the model has run once, making subsequent starts much faster. ## Chapter 1: Two Minutes to Check If Your Computer Can Run It and How Fast Models range from 7GB to 30GB after compression; downloading the wrong one wastes an hour. So spend two minutes checking. Step 1: Check your machine’s available memory Mac (M series): Top-left → About This Mac → Look at “Memory.” This number is your entire budget, shared between model and system. Windows / Linux with NVIDIA card: - Windows: Ctrl+Shift+Esc → Performance → GPU → Look at “Dedicated GPU Memory” - Or command line: nvidia-smi, check the total in the top-right. - Also note your system RAM (recommended 32GB minimum), which can supplement VRAM if needed. A summary table: find your row first Later chapters will explain each number’s origin. If you just want “which to download and maximum window size,” this table is enough. Remember one hard line: The most VRAM-efficient Q4_K_M configuration (q4 KV, 32K) needs 16,338 MiB. Cards with less VRAM have no alternative; cards with more have the same model quality, only a trade-off between long windows and speed. 1. Cards that can’t fit Q4_K_M (≤16GB): No choice | Card Capacity | Model | Max Window | VRAM Used | Notes | | — | — | — | — | — | | ≤8GB | No full GPU solution | — | — | Smallest IQ1_M needs 8.17GB. Some use offload (see 6.7), speed drops significantly. | | 12GB | Q2_K_XL | 32K | 10.92GB | 72-question score: 41/72 (Q4: 47). 2.14x more thinking per question. VRAM too tight for MTP. | | 16GB | Q2_K_XL | 128K | 14.66GB | Same as above. | 2. Cards that can fit Q4_K_M (18GB+): Quality fixed, choose window and speed All below use the same Q4_K_M model—answer quality is the same (47/72, highest in the full test), no “trading quality for window size.” Each cell’s speed is marked for “long text” or “short task.” | Card Capacity | Config 1: Prioritize Long Window | Config 2: Prioritize Speed | | — | — | — | | 18GB | q4 KV, 64K, 17,090 MiB, long text 40.36 tok/s | Same (MTP borderline: 18,148 MiB leaves only 284 MiB, not recommended) | | 20GB | q4 KV, 128K, 18,548 MiB, long text 33.56 tok/s | q4 KV + MTP3, 32K, 18,148 MiB, short task 105.70 tok/s (fastest overall) | | 24GB | q4 KV (no MTP), 256K, 21,492 MiB, long text 25.25 tok/s, 250K tokens 17/17 correct | q4 KV + MTP3, 192K, 22,637 MiB, long text 59.52 tok/s | | 32GB+ | q4 KV + MTP3, 256K, 24,422 MiB, long text 53.15 tok/s | Same—already optimal for this card. | Supplementary references (other routes, different sources and methods): - RTX 5090 32GB (⚠️ external report): LOW + MTP2, 256K only claimed borderline, short test 105.5 tok/s, ~30.2GB—only reference for short context. - AMD 7900 XTX 24GB (⚠️ external report): ~62 tok/s (third-party client output, quantization and window unspecified)—AMD has Day-0 support for this model; I don’t have an A-card, so only order-of-magnitude reference. - Mac 16GB (on-device test): Q2_K_XL, max 32K, 4–6 tok/s (strict conditions, see 6.9), 9.83GB + swap. - Mac 24GB+ (⚠️ external report/estimate): 24GB MLX 4-bit + MTP ~10 tok/s; 32GB+ ~10–40, with speculative decoding up to double—see “Mac users” table below. > Speeds in the table aren’t directly comparable. “Short task” is 32K small window; “long text” is filled with 250K tokens—same card, slower with more content, not because the card is slow. Step 2: Check compatibility—can it run, and what’s the experience? First, clarify what speed means. Reference: human reading speed, 5–10 characters per second: - Over 30 tok/s: Blazing fast, very smooth, no wait. - 15–30: Keeps up, comfortable, sufficient for daily work. - 8–15: Usable but you can see it generating character by character. - Under 5: Waiting, better use cloud. > ⚠️ Three easily confused “sizes”—distinguish first: - Card capacity: How much VRAM your card has. - File size: How much disk space the model file occupies (e.g., Q2_K_XL is 9.83GB). - Actual usage: VRAM consumed when running—always larger than file size because KV Cache and runtime overhead are added. For example, Q4_K_M is ~16GB, but the most efficient 32K window configuration (q4 KV) uses 16,338 MiB, so it won’t fit on a 16GB card (only 46 MiB left), requiring 18GB minimum. Buying a card based on file size will definitely fail. Mac users see this table: | Chip + Memory | Can it run? | Version | Without Speculative Decoding | With Speculative Decoding | Data Source | | — | — | — | — | — | — | | Base 16GB | Barely under strict conditions | Q2_K_XL | 4–6 tok/s (disable thinking, 8K window, clear background) | Not helpful (see 6.9) | On-device test, see 6.9 | | M4 mini 24GB | Yes | MLX 4-bit + MTP3 | 4.12 tok/s | 9.94 tok/s | External report | | M4 base 32GB | Yes | Q4_K_M + MTP | ~10–15 | ~23–35 | Estimated at 2.3× | | M2/M3 Pro 32–36GB | Yes | Q4_K_M + MTP | ~10–14 | ~23–32 | Estimate | | M2 Max | Comfortable | 4-bit + DFlash2 | — | 22 tok/s | External report | | M4 Pro 48–64GB | Comfortable | UD-Q4_K_XL + MTP | ~15–22 | ~35–50 | Estimate | | M3/M4 Max 64–128GB | Very comfortable | Q5/Q6 or even Q8 | ~25–40 | 72.1 (code) / 53.3 (prose) | External report (M4 Max) | | M2/M3 Ultra 128GB+ | Run anything | Q8 or even original | 34 tok/s | MTP 79 / DFlash2 88 | External report (M3 Ultra) | | M5 Max 128GB | Run anything | 4-bit | 26 tok/s | 70–87.9 tok/s | External report | > On Mac, speculative decoding is essential. M3 Ultra with vs without is 88 vs 34, a 2.6× difference; how to use MTP or DFlash2, see 6.9. Only the 16GB row in the table is on-device test; others are external reports or bandwidth-based estimates—tasks, windows, and quantization vary (same machine can differ 30% with different tasks), so only for order-of-magnitude reference. Speed depends on memory bandwidth, not CPU core count: bandwidth doubles, speed roughly doubles. When buying used—note M3 Pro bandwidth (150GB/s) is lower than previous M2 Pro (200GB/s); don’t buy it for model running. ⚠️ All numbers above are “model only” VRAM usage, not including KV Cache. Window size adds several GB for KV Cache. Example with my 48GB card: same model, 32K window uses 20.4GB, 256K window uses 30.3GB—just changing window size adds 10GB. So leave 20% margin, don’t choose based on the limit. A 12GB card running a 9.83GB model shouldn’t aim for a large window. Step 3: What if it can’t run? Three fallbacks, don’t force it: - Use a smaller model—8B-level models run on 8GB VRAM, sufficient for daily Q&A. - Accept more aggressive compression—but note the table above, 1-bit is just a demo, don’t get your hopes up. - Use cloud API directly—if you only use it occasionally, don’t bother. Buying a graphics card for rare use doesn’t add up. ## Chapter 2: Before Starting, Spend 10 Minutes Checking If It’s Worth Installing > This chapter explains how to avoid wasted effort. If already decided, skip. This step is often skipped in tutorials but saves the most time. Model downloads are 10–30GB, usually taking over an hour on average. If you download and find “this isn’t what I wanted,” that hour is wasted. So test the web version first. Ask a few questions you’d actually use—not “hello, who are you?” but real tasks: read a code snippet, summarize a document, generate something per your requirements. Key points to evaluate: - Instruction following—you gave three requirements, how many did it fulfill? - Performance on your most common tasks—code for coders, copywriting for writers. If satisfied, proceed. If not, stop here—save an hour of download and half a day of tinkering. Note: Web version and local version aren’t fully equivalent. Web usually runs uncompressed full version; local is compressed, with quality differences (Chapter 4 has tests). So the web experience represents the model’s upper limit—if unsatisfied with the limit, local will be worse. ## Chapter 3: Why Must You “Compress” This Model > This chapter explains the principle: why compression is necessary and why it still works. For hands-on, skip to “6.1” for installation commands. VRAM Reality Qwen3.8 27B’s official original—uncompressed full version—18 weight files totaling 55.56GB. Look at current card VRAM: official 4090 is 24GB, 5090 is 32GB, most people have 8–24GB. No consumer card fits 55.56GB. Even my special case fails: 4090 modded to 48GB, doubling official VRAM, usable ~49GB—55.56 > 49, still doesn’t fit. And this excludes runtime KV Cache and temporary overhead, adding over ten GB more. Even modded cards can’t fit it; regular cards have no chance. So on consumer cards, compression is mandatory—no choice—it’s not a cost-saving trade-off but a universal necessity. Many first worry “will quantization make it dumb”—if it can’t even load, this worry is meaningless. Why does it still work after compression? Use photos as an analogy. A DSLR raw image is 50MB, compressed to 2MB on WeChat, still looks the same. Photos contain lots of redundant information—large blue sky doesn’t need exact pixel values. Models are similar. Models store billions of numbers (weights); original uses 16-bit precision. But many numbers don’t need that precision—storing 3.14159265 as 3.14 hardly changes the model’s judgment. Quantization does this: stores each number with fewer bits. - Original (BF16): 16-bit, most precise, largest (55.56GB) - Q8: Compressed to 8-bit, half the size, minimal precision loss (~30GB) - Q4: Compressed to 4-bit, another half (~16GB) - Q2: Compressed to 2-bit, extreme compression (~10GB)—should collapse theoretically, but tests show surprise, see Chapter 4. Why same 4-bit, different formats matter Compressed to 4-bit, different formats vary greatly: some use fixed-block coarse cuts; others (like NVIDIA’s NVFP4) have finer blocks with per-block scaling. NVIDIA claims this reduces precision loss risk—I believed this, used it as primary for months, tested four rounds, couldn’t verify it. What happened? Chapter 4 has data. ## Chapter 4: Which Version to Choose > This chapter is about selection. If already decided on Q4_K_M (my recommendation), skip to “6.1” for installation—this is rationale only. Previous chapter explained why compression is needed. This chapter answers: Among compressed versions, which to download? I tested four rounds for this chapter and found two errors I made—both from inferring from filenames and docs without testing. First, acknowledge the errors, then see data. Error 1: I mistook 4-bit for 8-bit. I always thought Q8attn was “8-bit version.” Its full name Qwen3.8-27B-NVFP4-MTP-Q8attn actually clarifies: main body is NVFP4, i.e., 4-bit, with attention part in higher precision (hence Q8attn). Traditional 8-bit is Q8_0, a different file. Guessing bit depth from filename can be wrong. Error 2: I assumed NVFP4 was smarter. NVFP4 is NVIDIA’s own 4-bit format, indeed finer compression, officially claiming lower precision loss risk. I believed this, used it for months. Then tested four rounds, not one showed it smarter than Q4_K_M: | Test | Q8attn | Q4_K_M | Result | | — | — | — | — | | Fixed 72 questions | 41/72 | 47/72 | Q4 higher by 6 | | 25-question core accuracy | 25/25 | 25/25 | Tie | | Same 24 questions·thinking tokens | 1.20× | 0.97× | Q4 thinks 19% less | | Same 24 questions·time per question | 18.52s | 13.81s | Q4 25% faster | | 15 strict same questions (q8 KV) | 12/15 | 13/15 | Q4 higher by 1, p=1.0000 | | 15 strict same questions (q4 KV) | 13/15 | 12/15 | Q8attn higher by 1, p=1.0000 | | 250K long text exact recall | 3/3 | 3/3 | Tie | The largest difference is Q4 higher by 6—but that group wasn’t a controlled pair test; only one strict pair test was tied, p=1.0000. So accurate statement: I didn’t find NVFP4 smarter, nor proved it worse. NVIDIA likely wasn’t wrong; they tested their benchmarks, differing from my code, math, and 250K recall tests. How these four rounds were designed, why first two were wasted, which two papers woke me up, raw data per round—complete process in separate article: “I Tried Four Times to Prove Quantized Versions Are Worse, but Couldn’t.” This chapter only keeps data directly affecting “which to download.” So is NVFP4 still worth using? Yes. It lost the “smarter” claim, but has position in speed and VRAM—table below lists by window and KV settings; each cell’s speed corresponds only to that configuration. > Note on all speed numbers below: From my second round testing (stricter tasks, uniform conditions). Same config varies ~10% across tasks—I’ve seen two mismatched numbers, reason in test record article. Main tier real test (48GB card, full 250K tokens actual content) | Version | Time to read 250K tokens | Generation speed | VRAM | | — | — | — | — | | Old Q8 | 196.8s | 19.88 tok/s | 36.9GB | | Q6 | 205.2s | 22.75 tok/s | 31.0GB | | Q8attn q8 (my choice) | 214.0s | 47.67 tok/s | 30.3GB | | Q8attn q8K/q4V | 212.9s | 49.09 tok/s | 28.2GB | | Q8attn q4/q4 | 212.3s | 59.01 tok/s | 26.3GB | | LOW q8 | 219.8s | 39.07 tok/s | 26.8GB | Core conclusion: In this real 250K structured task, new version (Q8attn) generates 2.4× faster than old Q8 (47.67 vs 19.88 tok/s) while using 6.6GB less VRAM. > Comparing Q4_K_M to table’s Q8attn: at same 256K full load, Q8attn q4/q4 is 59.01 tok/s, 26.3GB; Q4_K_M q4/q4 with MTP3 is 53.15 tok/s, 24.4GB—Q8attn 11% faster, Q4_K_M saves 1.9GB VRAM. I recommend Q4_K_M because with no measurable quality difference, it uses less VRAM and is easier to run on 24GB cards. If you have 32GB+ and want max 256K speed, Q8attn q4/q4 is faster. A detail: new version “reads” slightly slower (214 vs 196.8s) but “writes” over twice as fast. Reading happens once; writing is continuous, so trade-off worthwhile. Extreme compression tiers: table summary I pushed compression to the limit to see if small VRAM can work: | Version | Size | Total Score | vs Old Q8 | Verdict | | — | — | — | — | — | | IQ1_M | 6.73GB | 24/72 | -45.5% | Like thumbnail; thought loops, HTTP 500. Tech demo, not production. | | IQ2_S | 8.37GB | 35/72 | -20.5% | Short tasks okay; 250K long text returns empty (0/17), avoid long tasks. | | Q2_K_XL | 9.83GB | 41/72 | -6.8% | Tie with my 30GB version; 250K long text 17/17 correct, 31.34 tok/s | So 9.83GB Q2_K_XL is the usable minimum; below this, real costs are paid—Unsloth docs warn: from Q2_K_XL to IQ2_S, 32-token prediction accuracy drops from 25% to 8–10%, with loops, empty responses, tool call failures, matching my tests. Q2_K_XL’s “tie” has boundaries: full 250K requires 19.65GB. So 12GB card using it for short tasks (32K uses 10.92GB) is fine; 250K long text needs 24GB. A counterintuitive finding: more compression, faster speed, but dumber. IQ1_M fastest generation (89.53 tok/s) but dumbest (24 score); Q2_K_XL slowest (72.03) but smartest (41 score). Don’t pick version based on tok/s alone. The cost of quantization: correct but thinks longer Cost found in four rounds: only comparing “24 same questions where traditional 8-bit Q8_0 also answered correctly”: | Version | Thinking tokens relative to Q8_0 | Total time per question | | — | — | — | | Q8_0 | 481.58 | 1.00× | 22.16s | | Q8attn | 576.62 | 1.20× | 18.52s | | Q4_K_M | 467.21 | 0.97× | 13.81s | | Q2_K_XL | 1,032.25 | 2.14× | 17.98s | | IQ2_S | 1,495.58 | 3.1