The Ultimate Offense and Defense: What to Buy for Local Deployment

When deploying local models, should you choose the NVIDIA DGX Spark or two modified RTX 4090s with 48GB VRAM each?

These two machines are extreme specialists. One has ridiculously large memory but limited bandwidth, while the other has massive bandwidth but relies on modifications for high VRAM.

The primary principle in choosing models is to maximize the utilization of these machines’ VRAM and computational power. Therefore, both selected models are too large to fit on a single card. Qwen3.8-27B in full precision requires 55.56 GB, and even after quantizing Ling-3.0-flash down to INT4, it still needs 77.01 GB—unfit for even a modified 4090 with 48GB VRAM, let alone considering space for context after loading the weights. Models of this scale require either multi-card VRAM pooling or a unified memory architecture. Conveniently, I have one of each setup at hand.

The second reason is that they represent two extremes: One is the dense Qwen3.8-27B, which processes all 27.78 billion parameters for every token generated. The other is the extremely sparse Mixture-of-Experts (MoE) model, Ling-3.0-flash-int4, with a total of 127 billion parameters but activating only about 5 billion per inference.

Two specialist machines, two specialist models, and four combinations—I tested them all.

This article provides real benchmark data, explains how to interpret VRAM configurations, and offers purchasing advice.

All technical terms are explained upon their first appearance, so no prior knowledge is required.

Let’s look at the conclusions first.

Guide

• For basic concepts like “dense,” “moe,” and “bandwidth,” see Chapter 1.
• To understand what determines “how many people can be served simultaneously,” see Chapter 3 (the most useful chapter).
• For complete data on all four combinations and real-world benchmarks, see Chapters 2-5.
• If you have two machines and want to assign them different roles, see Section 5.4.
• If you only want to know what to buy, see Chapter 6.

1. Four Specialists

1.1 Dense vs. MoE: What’s the Difference?

Dense refers to a dense model. A 27B-parameter model performs calculations across all 27.78B parameters for each generated token. Total parameters: 27B, activated parameters: 27B.

MoE stands for Mixture-of-Experts. It may have a total of hundreds of billions or even trillions of parameters but is divided internally into many “experts,” only a few of which are activated per inference. Ling has 512 experts, and for each token, it only selects 8 to use, leaving the rest idle.

The advantage of MoE is that its massive total parameter count pushes the boundaries of capability. The trade-off is that all these experts must reside in VRAM.

The two models have similar file sizes, but this is only because one uses 16-bit and the other uses 4-bit storage. In terms of parameter count, Ling is 4.6 times larger than Qwen. However, for each operation, Qwen activates 5.4 times more parameters than Ling.

1.2 The Two Machines

Spark’s biggest feature is its size: 121 GiB of unified memory, making it effortless to load any model. The dual 4090s represent a different path: Official cards have only 24GB, but the Huaqiangbei modified version has 48GB, offering ample bandwidth and compute power. However, my setup lacks NVLink, so cross-card communication relies on PCIe.

1.3 Compute Power vs. Bandwidth: Why Only Bandwidth Matters

When buying a GPU, two parameters in the spec sheet are easily confused:

• Compute Power (TFLOPS): How many operations per second, measuring computational speed.
• VRAM Bandwidth (GB/s): How much data can be moved from VRAM to compute units per second, measuring data transfer speed.

For text generation in large models, the bottleneck is VRAM bandwidth, which is counterintuitive. However, for training, computational power is much more critical.

When generating a token, the GPU must read all participating weight parameters from VRAM, compute, and output a character. Matrix multiplication is trivial for modern GPUs, taking just milliseconds; but moving tens of gigabytes of weights takes tens of milliseconds. Time is almost entirely spent on data transfer, making high compute power less useful.

Generation speed limit ≈ VRAM Bandwidth / Weight volume read per token

Thus, the bandwidth difference between the two machines directly determines the speed difference.

When a model doesn’t fit on a single card, it’s split across two cards (tensor parallelism). The advantage is both cards read simultaneously, effectively doubling bandwidth; the disadvantage is that after each layer, cross-card synchronization is required. My setup lacks NVLink and must use PCIe.

The dual 4090 have 7.4 times the bandwidth of Spark.

Now consider the model side: Qwen reads 27.8 GB per token, while Ling reads only about 1.3 GB. So at the same bandwidth, Ling should be an order of magnitude faster than Qwen.

These two factors—bandwidth and Ling’s faster speed—will be demonstrated in the actual tests in Chapter 2.

1.4 Drafting and Speculative Decoding

Everyone knows that during inference, models generate one token at a time, with each subsequent token inferred based on previous ones.

Popping out one at a time is slow—can we output more?

This involves speculative decoding, also known as drafting. Before the main model acts, cheaper methods guess the next token, or even the next three.

If all guesses are correct, speed increases by 1x or 3x.

Here’s how it works: The main model performs a forward pass as usual, which would normally output one token. However, it also verifies the draft’s correctness simultaneously. Thus:

• Correct guess: This step outputs two tokens—one it computed itself, plus the confirmed guess.
• Incorrect guess: Outputs only one token, discarding the guess.

The key is that verifying a guess and generating a token read the same weights and incur the same data transfer cost. So a correct guess is a free gain, and an incorrect guess only wastes some idle compute power, with no loss.

However, the statement about “the same weights” applies fully only to dense models. For MoE, it must be examined separately: Attention layers and shared experts are indeed shared across all positions, so multiple positions can be verified with a single read. But routed experts are selected individually for each token from 512 options, with minimal overlap between positions. Thus, the number of experts to read increases from 8 to about 16. Section 2.3 will calculate this in detail, and the result will actually favor Ling.

Moreover, this process is lossless: the output is identical to not using speculative decoding—it doesn’t trade quality for speed.

MTP (Multi-Token Prediction) is one way to implement this guessing. Traditional methods require an additional small model as a draftsman, while MTP adds an extra prediction head within the model itself to predict the next-next token directly from the model, reducing costs. Both models in this article support MTP: Ling-3.0-flash uses bailing_hybrid_v3_mtp, and Qwen3.8 uses Qwen3_5MTP, both configured to guess one token per step.

The hit rate determines the gain. The average number of tokens output per step is called the acceptance length, which can be read directly from server logs.

If guessing one token per step, the theoretical maximum acceptance length is 2.00. Acceptance means all inferred tokens are accepted: the first token is unconditionally accepted, while the second has a probability of being rejected. Thus, the acceptance length falls between 1 and 2. Both models achieve 1.8–1.9, indicating the technique is highly effective on real coding prompts—nearly doubling generation speed.

An important point: The hit rate depends on how easy the next token is to guess. Code, being highly patterned, is easy to guess, while random characters are not. The same model on the same machine can show over a 1x speed difference just by changing test data. Therefore, all benchmarks in this article use the same real coding prompt. Chapter 4 covers a real conversation, and it’s advisable to always ask what input was used when reviewing any benchmark.

1.4.1 The Number of Guesses is Adjustable

The number of guesses is a parameter. The previous section used the most conservative setting of 1. Increasing it means guessing several tokens ahead, with the main model verifying them all in one forward pass. Guessing k tokens sets the maximum acceptance length to k+1, enabling speed boosts of 1x or 3x.

However, gains diminish quickly due to compound probability: The second token is only counted if the first is guessed correctly. Assuming a per-position hit rate of p, the average tokens per step is 1 + p + p² + … + pᵏ. Using the measured acceptance length to infer hit rates (Ling: 0.90, Qwen: 0.80):

Increasing from 1 to 3 guesses triples the draftsman’s workload. Ling’s output increases only by 1.81x, and Qwen by 1.64x. This is an optimistic estimate—in reality, hit rates decay with depth, not remaining constant at 0.90.

For single-stream inference with dense models, this is nearly free, as it reads the same weights and compute power is idle. For sparse MoE, the benefit is reduced: k+1 positions require reading nearly k+1 sets of routed experts (guessing 3 means nearly 4 sets). When multiple users are concurrent, batch processing already utilizes compute power, making additional guesses a real cost. Thus, there’s no universal optimal setting—it depends on whether it’s for personal use or service.

Other approaches in this field include: guessing wider (preparing multiple candidates per position in a tree, verified in one pass—Medusa and EAGLE take this path) and drafting sources (using a separate small model sharing the vocabulary, finding repeated context snippets for free—almost zero-cost for code—or using MTP to train the drafting head into the model itself, as done here).

This article uses one guess per step throughout. Later, it’s worth investigating what actual gains come from increasing k, especially for Ling. Section 2.3 will show it only achieves 18% of the theoretical hardware limit, with bandwidth and compute still underutilized. Note that for sparse MoE, extra guesses incur not only compute costs but also proportional increases in expert memory access.

1.5 Three Questions This Article Aims to Answer

• For the same model, how much does performance vary across different machines?
• On the same machine, what’s the difference between a dense model and an MoE that only activates 4% of its parameters?
• If you need to spend money today, what should you buy?

The article conducts four rounds of testing.

2. Rounds One and Two: Single-Session Speed for Four Combinations

I set a uniform user experience threshold for all tests: single-user generation speed ≥ 20 tok/s.

This is the lower limit I consider comfortable for an interactive AI assistant.

This line determines where testing stops. Beyond it, server throughput can continue to increase, but at the expense of user experience for better-looking reports. So I didn’t push concurrency to extremes—not because it’s impossible, but because beyond that point, you’re no longer testing a usable system.

2.1 DGX Spark Tests

Ling is backed by a 127B-scale model, but only a small portion is activated during generation, so its memory bandwidth pressure is nothing like that of a 127B dense model.

Qwen3.8-27B on Spark achieves only 6.7 tok/s—essentially unusable. Moreover, this test run included speculative decoding: all possible accelerations were applied, and the result was still this low. The likely reason is that the model is relatively new, and Spark optimization hasn’t caught up yet: On the same machine, Qwen3.6 can easily exceed 30 tok/s, pending community optimization.

Thus, Test 1 ends here.

2.1.1 But Changing Quantization Format Changes Everything on This Machine

The two figures above are from their default deployments. Later, I tested all viable quantization formats on the same Spark using the same bench.py and prompt, with shocking results:

Spark struggles with full-precision dense models but excels at running MoE models. For full-precision dense models, speed improvements can only come from inference engines and speculative drafting, or by quantizing to reduce model size.

For Ling (MoE), with MTP it achieves an astonishing 39.2 and 54.9 tok/s. The same Ling model runs at 39 tok/s with INT4 on vLLM and 54.9 tok/s with MXFP4 on SGLang—a 41% difference, showing the importance of both inference engine and quantization compatibility.

At this point, the conclusion is clear: Spark cannot run full-precision dense models; you must switch to MoE or reduce precision.

2.2 RTX 4090 48G × 2

Looking directly at the results:

Both are much faster than Spark, though differing significantly: 145 vs. 50.4, a 2.9x gap.

I cross-verified both numbers. Ling was tested twice with entirely different clients, yielding 146.0 and 145.0 (less than 1% error). A third test with a concurrent script showed 144.9, consistent. Qwen was measured with the same script, ensuring comparable metrics.

On the same two cards, the 127B Ling outperforms the 27B Qwen by nearly 3x. This is due to two figures: parameter count and activation volume. Parameter count is the model’s size, and activation volume is the bandwidth/compute required per token. For dense models, these numbers align. For MoE, Ling here activates only 5B parameters.

2.3 Which Hardware Maxed Out Its Potential?

I compared each model to its physical limits. The theoretical limit is based on the formula from Section 1.3: weight volume read per step divided by VRAM bandwidth gives the minimum time per token, then inverted.

Both models used speculative decoding, so the theoretical limit is calculated as: Invert the time to read weights per step, then multiply by the actual tokens per step (acceptance length).

Qwen (full precision, 55.56 GB, split across two cards, each reading 27.8 GB):

27.8 GB ÷ 1008 GB/s = 27.6 ms/step 1.80 tokens ÷ 27.6 ms → Limit ≈ 65 tok/s

Ling (activates ~5B parameters per token in INT4, each card reads ~2.5 GB per token (1.3 GB × 2)). With MTP, it verifies two positions per step: shared parts are read once, but routed experts are read nearly twice. Since most of the 2.5 GB consists of experts, per-step reading is about 2.5 GB:

~2.5 GB ÷ 1008 GB/s ≈ 2.5 ms/step 1.90 tokens ÷ 2.5 ms → Limit ≈ 800 tok/s

Together:

Qwen’s 27.8 GB is exact (all BF16, fully activated); Ling’s 2.5 GB (1.3GB × 2) is an estimate (attention layers and shared experts remain BF16, requiring layer-by-layer calculation for precision). Both theoretical limits exclude draftsman overhead, making them optimistic—especially for Ling, where only the order of magnitude matters.

Qwen achieved about 77% of its limit, which is normal considering cross-card communication overhead.

Ling theoretically can reach 800 tok/s but only achieves 145. The missing 80% relates to my cards using PCIe and MoE’s sparsity: selecting 8 out of 512 experts requires routing calculations and scattered memory access.

Another factor favoring it: Expert memory access scales with token count, which only holds for small batches. With concurrency, the 512 experts quickly saturate, and a single data transfer is amortized across the batch, eliminating the MTP tax—it only exists in single-stream benchmarks, which aren’t the recommended scenario for Ling. This is essentially a space-for-time trade-off.

2.3.1 Introducing Two Metrics: Decode Speed and End-to-End Speed

The distinction between “decode speed” and “end-to-end speed” may seem redundant, but an example on Spark illustrates its necessity.

An SGLang deployment on that machine was tested once, yielding:

Decode speed (1000/TPOT) = 254 tok/s End-to-end (total tokens ÷ wall clock) = 50.3 tok/s

A 5x difference. The reason is its first-token latency of 8.2 seconds: The request waits silently for 8 seconds, then dumps 512 tokens in two seconds. TPOT only measures intervals after the first token, making it appear extremely fast, while the user actually waits 10 seconds.

This isn’t measurement error—both numbers are correct, and “first-token latency + decode time = end-to-end” holds precisely. They simply answer different questions.

Therefore, when reviewing any benchmark, besides asking about the input (Section 1.4), you must also ask which metric is reported. Decode speed without first-token latency can be inflated several times. All cross-machine comparisons in this article use “aggregate throughput ÷ concurrency,” as it avoids this distortion.

3. The Most Useful Chapter: What Determines Model Throughput

This chapter is the biggest takeaway from my experiments and goes beyond my initial understanding. It still uses experimental data for stress testing. All figures come from a fixed coding prompt: a Q&A format ending at 500 tokens, to reflect ideal conditions.

3.1 First, the Results

On dual 4090s, increasing concurrency:

Using the 20 tok/s threshold: Qwen supports up to 36 users, Ling up to 72.

Interestingly, Qwen hits the threshold gradually: Performance at 32 and 36 users is nearly identical (20.9 and 20.8), showing it hugs the red line before dropping, not plummeting.

At their respective limits, server total throughput: Qwen 642.8 tok/s, Ling 1553.7 tok/s with first-token latency of 392 ms.

Thus, on the same machine, Ling serves twice as many users (2.0x) and achieves 2.42x the total throughput.

3.2 A Trend: As Concurrency Increases, Ling’s Advantage Diminishes

Extracting the ratio per level:

At single-user, Ling is 2.88x faster; at 20 concurrent users, it drops to about 1.8x and stabilizes.

This shape makes sense. Low-activation MoE saves on “how many weights need to be moved per token”—a benefit most valuable at single-stream. As concurrency increases, both models enter batch processing, where a single data transfer is amortized across many users, diluting MoE’s relative advantage.

3.3 The Formula

Why can Ling support so many more users? The answer isn’t in architecture but in VRAM allocation.

Each active conversation occupies VRAM for context storage, called KV cache. More users mean more usage, but total allocatable KV space is fixed: it’s whatever remains after loading weights.

Concurrently servable users = KV cache capacity / Fixed cost per conversation

This isn’t an empirical rule; it’s simple division.

Note: This space must be explicitly requested. Inference engines often default to values much lower than the hardware’s actual capacity. On my first run, I only got half, missing out on dozens of potential users. It’s worth verifying this parameter during deployment.

So when someone asks, “How many people can this machine serve?” the answer lies not in parameter count or compute power, but in how much VRAM remains after loading weights and whether you’ve actually claimed it.

When buying a card, you look at compute power; when using it, concurrency is determined by the remaining VRAM.

3.4 What About Spark?

At 4 users on Spark, performance per user drops to 19.6, just below the threshold. Thus, its usable capacity is 3–4 users.

But I must be honest. Spark’s data was measured under conservative configurations, and the KV space issue likely exists there too. With 121 GiB unified memory, after loading 77 GB of Ling, it has more headroom than the 4090s. If KV is maximized, its service capacity could be far beyond 3–4 users.

Thus, the Spark row in subsequent tables should be understood as current configuration performance, not the hardware’s upper limit.

4. How Do These Multipliers Hold Up in Real Tasks?

All previous figures came from a fixed, short coding prompt: Q&A ending at 500 tokens. Real coding sessions aren’t like this: dozens of back-and-forth rounds, context growing continuously, with file reading, command execution, and rework.

How much of the benchmark multiplier translates here? I used the same task on the same dual 4090s with both models.

The task involved web animations: four versions of a cycling animation, three fans, a showcase page, plus a “it should be a pelican riding a bicycle” rework. Both were complete agent sessions with file read/write and command execution, with speculative decoding enabled.

Of course, Ling is a text-only model, so this tests its less-favorable scenario.

4.1 Same Task, One Takes 133 Minutes, the Other 69

Looking at the first row: Both models produced nearly identical code volume—96,151 vs. 95,889 tokens, a 0.3% difference. For the same task via two completely independent paths, matching output means the time comparison is meaningful: not one slacking and one verbose, but the same job taking 133.5 minutes vs. 68.7 minutes.

1.94x.

On the short-prompt benchmark, the gap was 145.0 vs. 50.4, 2.88x.

Including tool execution and wait times, the average throughput over the entire session reduces the gap to 1.45x. Real-world experience falls between these numbers.

4.2 Where Did Half the Multiplier Go?

Breaking down the time into two parts: reading context (prefill) and generating tokens (decode). Section 1.3 stated “generation is a data transfer task,” referring to the latter; prefill is a computational task.

Converting by measured rates:

Both align with actual measurements, validating this breakdown.

Thus, the gap’s source is clear: Of the 65 minutes Ling saved, 45 minutes came from faster context reading—a computational task where 5B activation beats 27.8B activation; sparsity truly pays off here. The actual token generation was only 1.43x faster.

Placing benchmark numbers side-by-side for clarity:

Both models slow down in real conversations, but Ling drops more sharply: Qwen retains 42% of original speed, Ling only 21%.

The reason matches Section 3.2: That chapter showed Ling’s advantage diminishes with more users; here, it diminishes with longer context. The mechanism is identical: Sparse activation saves on weight transfer, but context processing overhead is unrelated to sparsity. The longest input in this session was 131,233 tokens; at that scale, weight transfer is no longer the major cost, so saving there yields little benefit.

Multipliers from benchmarks using hundreds of tokens represent the model’s best-case performance. Your session length determines the discount.

4.3 Why Caching Matters

In this experiment, both models ingested a cumulative 3.59–4.86 million input tokens.

This isn’t due to long prompts, but because every round re-reads the entire conversation: Round 2 re-reads Round 1, Round 3 re-reads Rounds 1-2, and so on. The initial 13 minutes of prefill were mostly spent re-reading already processed content.

Theoretically, this has a solution: input caching—storing read contexts in VRAM for reuse in subsequent rounds. But for local deployments, caching mechanisms are absent.

This explains why storage is expensive, cache is valuable, and cache hits are cheap.

5. Experiment Summary

Writing this, I realize the table can be read differently.

Initially, I described the two machines as extreme specialists. After testing, the models proved equally specialized:

Thus, the four combinations represent four pairings:

• Spark + Qwen: Narrow bandwidth meets large data transfer—two weaknesses stacked, 6.7 tok/s, effectively unusable.
• Spark + Ling: Spark’s capacity accommodates Ling’s size; Ling’s low activation circumvents Spark’s bandwidth—two weaknesses offset, 38.7 to 54.9 tok/s, usable.
• 4090 + Qwen: High bandwidth feeds large data transfer, 50.4 tok/s, performs normally.
• 4090 + Ling: Two strengths combined, 145.0 tok/s, fastest overall—cost is realizing only 18% of theoretical hardware limit (Section 2.3).

Specialization isn’t a flaw; it’s a property requiring proper pairing. The only problematic scenario is combining two weaknesses.

5.1 Translating to Real Tasks

tok/s is engineering jargon; product managers only ask: How long do I wait? Is this machine enough for our team?

Based on generating 500 tokens per Q&A (roughly a complete function with test cases):

Converting to purchases: For a team of 70 needing simultaneous access, one dual-4090 + Ling suffices. Qwen requires two machines, Spark requires 18–23. User-perceived speed is identical across these three configurations.

Note: This estimate assumes the extremely short prompts used throughout. In real multi-turn coding, each round re-reads all previous context, worsening all numbers—Chapter 4’s real session reflects this discount. These are relative relationships, not absolute production values.

5.2 Individual Use or Small Teams, Primarily Coding: I’d Choose Qwen

First, let me clarify: This choice isn’t due to Ling being slow—quite the opposite.

Speed is where Ling excels most: 145 vs. 50.4, 2.88x faster.

So why choose Qwen? Because it means intentionally trading speed for three other things:

Quality certainty. Dense models process all 27.78B parameters at full BF16 precision, without expert routing. Asking the same question twice yields predictable performance. Ling uses 4-bit quantized MoE, where routing results and quantization errors are variables. In coding, a single erroneous output wastes far more time than a few seconds of waiting.

Ecosystem maturity. The dense path has more users, with the most complete toolchains, quantization solutions, and inference framework support.

It also has significant untapped potential. This round only added speculative decoding to Qwen; quantization wasn’t applied. Quantizing to INT4 would reduce weights from 55.56 GB to about 14 GB, fitting on a single card and reducing read volume by three-quarters. Per the formula in Section 1.3, there’s substantial speed headroom.

Translating to experience: A 500-token response takes Ling 3.4 seconds and Qwen 9.9 seconds—both within the “ask and wait briefly” range, not crossing a perceptual quality threshold. If those 6 seconds trade for more stable output and easier deployment, I consider it worthwhile.

In short: For local small setups, small teams, primarily coding—choose Qwen. But understand you’re spending speed for certainty.

5.3 Multi-User Services, Like a Doubao-like Frontend: I’d Choose Ling-3.0-flash

With multiple users, the rules change.

This is where the formula from Section 3.3 dominates: Your cost isn’t an engineer waiting three extra seconds, but how many accounts a single server can support.

On the same dual 4090, Qwen hits the threshold at 36 users; Ling can reach 72—exactly double. Though Ling’s relative advantage shrinks from 2.88x at single-user to 1.78x, its manifestation changes: Not “each person is faster,” but “can support twice as many people.”

Per the table in Section 5.1, for a team of 70, this means buying one machine instead of two.

MoE models like Ling are naturally suited for this scenario: Activating only 4% of parameters per token means more concurrent conversations within the same bandwidth budget. The trade-off is hardware utilization (only 18%), but it gains absolute speed and concurrency density—a worthwhile trade for service providers.

In short: For products serving many users, choose sparse MoE models like Ling flash.

5.4 The Third Coding Path: Choose Both, Assign Different Roles

The previous two sections presented a binary choice. But if you have both machines, there’s a third option: Let them handle different segments—one responsible for thinking, the other for execution.

This approach has recent evidence from a paper.

AI4AI at Test-Time demonstrates routing logic, prompt templates, deterministic solving code, format validation, and verification steps. Without modifying the weaker model’s parameters, performance improved from 0.49 to 0.91.

This aligns with my experimental findings.

In benchmarks, Ling is 2.88x faster. As long as each call has short context, Ling can achieve this speed.

And “breaking large tasks into clearly bounded small tasks” converts long contexts into short ones. This approach benefits sparse MoE doubly: It reduces errors and allows operation in its fastest range. The 3.59 million repetitive prefill tokens in Section 4.3 stem from the same issue: Long multi-turn conversations re-read all previous context each round. Switching to independent small tasks eliminates this overhead entirely.

The two models sum to 132 GB, too large for a single machine, so they must be deployed separately. Qwen as the planner, Ling as the executor.

6. What to Buy

Spark vs. 4090 48G ×2

The 121 GiB unified memory’s greatest value isn’t benchmark performance, but effortless model loading: larger models, higher-precision weights, single-machine development, model research, and avoiding multi-card topology hassles. Though dual 4090s offer 96 GB, after loading weights, VRAM for KV is limited, comfortably running models up to about 80G; Spark can handle up to 100G. Thus, models exceeding 80G must use Spark.

However, once the model fits on GPUs, I’d choose dual 4090s for the significant speed advantage.

But Spark’s weakness can be partially mitigated. Switching to MXFP4 + SGLang on the same machine improves single-session speed from 39 to 54.9—reaching 38% of dual 4090 performance, no longer “an order of magnitude slower.” The cost is manually handling quantization format and inference engine adaptation, which is more cumbersome on Spark than 4090.

Price-wise, Spark is about 30,000 cheaper than dual 4090s.

One scenario not previously mentioned: Buy both.

Both machines are specialists, and both models are too. With clear use cases, both can be purchased.

For local inference hardware, ignore TFLOPS and instead ask these four questions in order:

First, will it fit? Second, how many parameters need to be activated per token? Third, how large is the VRAM bandwidth? Fourth, is it for one person or dozens?

After these four questions, the purchasing choice becomes clear.

7. Final Notes

For completeness, I won’t finalize conclusions without addressing these points:

• Guessing multiple tokens per step. The entire article used the most conservative setting of one guess per step. The table in Section 1.4.1 is based on geometric series estimates, not measurements, and real hit rates decay with depth—actual gains require testing.

Writing this, I suddenly thought: If you took these two models and two machines to open a coffee shop, it would also make sense.

Second cup half-price, unlimited tokens.

Perfect loop.

If you’re also experimenting with local model deployment, follow @MinLiBuilds.