Six months on four RTX 3090s: everything I measured

Why this box exists

During the day, I manage the AI infrastructure at the bank. The cluster there consists of more than 200 H200 cards, and I’m able to get my hands on 10–12 of them. In the evenings, I go home to four used RTX 3090 cards installed in an open case held together with a grille from my kitchen oven.

The entire setup cost me about $6,000. Four graphics cards at about $900 each, plus about $2,000 for the platform: an AMD EPYC processor on a Chinese Huananzhi H12D-8D motherboard with a BMC, 128 GB of DDR4, riser cards, drives, and a power supply. That’s the same as renting a single high-end cloud GPU for several months, but my system sits right under my desk and doesn’t have to answer to anyone.

EPYC FURY

EPYC FURY

I didn’t build it based on a shopping list. It gradually grew. First, a single 3090 in my old home lab on a Xeon - just to see what local setups were capable of. Then a second one. For a long time, I made do with two cards until it became clear that the old Xeon couldn’t provide enough PCIe lanes for four cards, so I moved the GPUs over to an EPYC. Then two more cards. At one point, my wife and I spent an evening trying to figure out how to fit four cards with risers into a case that was completely unsuited for the task. She came back from the kitchen with an oven grill. It worked. It’s still there to this day.

yeah the egg carton is still here

yeah the egg carton is still here

After six months of experimenting, this article summarizes everything I’ve learned from this configuration. Nothing but real numbers. Every figure listed below is based on measurements taken on this system, and most of them surprised me at least once.

96 GB is the “magic” number in this case. It’s enough to run the 122B model with a context size of nearly 200K, which redefines the very concept of “local.” And you can achieve this for the price of a single high-end gaming PC.

The biggest lesson: how you split 4 cards matters more than anything

For several months, I ran every model with tensor parallelism across all four cards, because that’s what everyone does. TP4, done, next question. For any model that could fit on fewer cards, this habit resulted in me losing most of my throughput.

Here are all the load distribution options across four 3090 cards for the same model (Gemma 4 12B AWQ, vLLM) with a combined tokens-per-second metric at 220 W:

Layouttok/s4 separate copies, 1 card each, behind a router3,425Data parallel, native –data-parallel-size 43,308Pipeline parallel 2 x 22,619TP2 + DP22,433Two TP2 copies2,364Pipeline parallel = 41,428Tensor parallel = 41,007

Same hardware, same model, yet a 3.4x difference between the best and the most popular option.

The reason becomes obvious as soon as you see it. The TP4 synchronizes all four cards on every single layer, while in consumer-grade cards this is done via the PCIe interface. The 12B models don’t need this. The four independent copies do not interact with each other at all.

The practical answer is even simpler than my experiment with the router: native data parallel (–data-parallel-size 4) provides throughput that lags behind the router by less than 3.4%, while the time to first token is 2.3 times better (178 ms vs. 411 ms) - and all of this in a single process without the need to manage a load balancer. So, here’s the rule I now follow:

Replicate the model over the fewest cards it fits on. Split wider only when a single user’s latency is the whole job.

To be fair, it’s worth noting that a single-user scenario is entirely realistic. Gemma 4 31B for a single user at 300 W: TP4 delivers 78 tokens per second, while TP2 delivers 57. If you’re only interested in a single thread, the wide TP configuration pays off. But as soon as multiple users start sharing the system, replication becomes significantly more advantageous: with 64 concurrent threads on a Qwen3.6-27B, the TP4 configuration delivers 606 tokens per second, whereas two TP2 replicas deliver 1,273. With the same four cards, that’s a 2.1-fold increase.

Another finding that surprised people when I published it: additional cards pay off in terms of pure computing power, even if you don’t need the memory. Gemma 4 31B fits perfectly on two cards, but switching from TP2 to TP4 increased single-thread performance from 46 to 73 tokens per second at 220 W. The two extra cards added VRAM I didn’t need, and 58% more speed.

Unlocking P2P, or clawing back part of the TP tax

One reason for the low TP speed on consumer graphics cards is that NVIDIA disables peer-to-peer DMA on the 3090 model, so every synchronization between the cards goes through system memory. The GPUs are located on the same board and could exchange data directly, but the driver does not allow this.

There is a community-developed patch designed specifically for this purpose: open-source kernel modules with P2P support enabled, available in the aikitoria/open-gpu-kernel-modules repository. I installed them on top of driver 595.71.05, enabled Resizable BAR, and got 32 GB of BAR1 per GPU. The nvidia-smi topo -p2p r output changed.from “not supported” to “OK” for all four cards.

First, the synthetic results. Before applying the patch, the bandwidth between GPUs was 5–6 GB/s with a latency of 14–18 microseconds. Afterward: 13.2 GB/s in one direction, 25.5 GB/s in both directions, and 1.5 microseconds. The latency between the cards decreased tenfold thanks to kernel module swap.

Under real-world load, the gain is smaller, as is always the case. Qwen3.5-122B AWQ, TP4, run with a long context (4,096 inputs, 256 outputs) at a concurrency of 2: 65.5 total tokens/s before, 74.3 after. +13.5%. And a fair warning from my own notes: we’re talking about aggregate throughput at a concurrency of 2, not that “each request became 13.5% faster,” and this is far from the twofold increase that is sometimes promised in discussions about P2P.

That is precisely why this section is more of a note on the “chapter” rule than an exception to it. P2P reduces the overhead associated with tensor parallelism, but does not eliminate it entirely. The table above shows that the difference between TP4 and replication is a factor of 3.4, even though replication does not use P2P at all. If your model still requires you to use TP, patch the driver and take the extra 13%. But first, choose the right placement layout.

The eighth way to split: prefill/decode disaggregation

There is another configuration that does not fit into the table above: dedicating separate GPUs for prefill and decoding. Prefill requires intensive computation, while decoding requires intensive memory usage, and on a single server, a single heavy incoming prompt periodically stalls everyone else’s stream. You’ve likely encountered this phenomenon in the form of latency spikes observed during data streaming with some large hosted models.

I tested this separation on two 3090 cards (SGLang + Mooncake with standard TCP transport, without RDMA on a home-built system) compared to the same two cards operating using a round-robin algorithm. The throughput turned out to be the same: 913 tokens/s in the unified configuration versus 903 in the distributed configuration, with the same query rate. The point is something else:

Metric, 2 cardsUnifiedP/D splitInter-token latency, p9959 ms12 msWorst single stall132 ms15 msTime to first token~86 ms~216 ms

The decode gpu never sees a prefill, so the token stream is five to nine times more stable. The trade-off is TTFT: the key-value (KV) cache must be transferred between cards, and when using a TCP loopback, this takes about 130 ms per request. On data center hardware with NVLink or RDMA, this overhead virtually disappears, which is precisely why large inference service providers use a disaggregated architecture.

The conclusion for local AI systems: if your users are monitoring the token flow and cannot tolerate delays, P/D is a truly effective tool even on two consumer-grade cards. If, however, you’re simply processing tasks in batches, replication remains the best solution.

Free speed you are probably not using

Three things allowed me to achieve a real speed boost without any new hardware.

Speculative decoding with a built-in draft head. Qwen3.6-27B comes with its own MTP module, and vLLM uses it for speculative decoding. On two cards in single-thread mode, the code generation speed increased from 56 to 101 tokens/s. Same weights, same cards - a 79% increase thanks to a single flag. How it works: a draft head predicts a few tokens ahead, the full model checks the entire prediction in a single pass, and every accepted token is a token obtained practically for free. The code accepts many tokens because it is predictable. Prose accepts fewer - from 56 to 83 - but they’re still free. If your model comes with a draft head, enable it before you even think about buying anything.

Concurrency - that’s where this system really shines. A single system with four 3090 cards, with parallel thread settings ranging from 1 to 32: the Gemma 4 31B performance on all four cards increased from 78 to 522 tokens/s. In the Qwen3.6-27B test, using just two cards, the performance increased from 65 to 656. The speed per thread decreases as the number of users increases, so choose a parallelism level based on your actual workload, not on the most impressive test numbers. But if your use case involves agents or a small team, the consumer system will process hundreds of tokens per second throughout the day.

**Power consumption - **or, more precisely, power savings. I swept the per-card power cap from 220W to 300W and repeated the test with the same model. The decoding results were 73, 77, and 78 tokens per second. In other words, power consumption increased by 37% while speed increased by 7%. The number of tokens per watt decreased from 8.4 to 6.5 per 100 W. The 3090’s performance curve flattens out immediately after 220 W, so these systems run at a 220 W limit around the clock, and I’m keeping this difference. If you take only one piece of advice from this article, limit the power consumption of your graphics cards.

Quants on Ampere: what actually works

The 3090 model is an Ampere chip, sm_86. It lacks built-in FP8 support. This fact alone determines most bit-width choices in this generation, and half of the recommendations you read on X tacitly assume the use of Hopper.

What has proven effective on this platform:

• 4-bit AWQ or GPTQ-style int4 is the standard choice for everyday work. The Marlin kernels provide high int4 performance on Ampere; the quality of the 4-bit format is perfectly adequate for serving, and everything else in this article runs on it.

• FP8 weights are a pitfall here. Under real-world load, FP8 on the 3090 lost out to INT8-W8A8 in terms of throughput in my Qwen3.6-27B test, and the “just feed FP8 online” approach doesn’t work at all on Ampere. If someone recommends FP8 without asking which chip you’re running on, close the tab.

• The FP8 KV cache, on the other hand, is a free bonus. More on this in section 122B: it doubled my usable context and didn’t affect decoding speed. One fair point: I measured the speed performance of KV, not the accuracy (quality with fp16, fp8, and q8 KV on the same model). So consider the advice on fp8-KV verified in terms of speed; if your workload is sensitive to accuracy, conduct your own quality checks before deploying this in a production environment.

NVFP4 on a 3090: it runs, and that is the news

In early July, support for NVFP4 on the Ampere architecture - including sm_86 - was quietly added to vLLM 0.24. You can enable it using the –quantization modelopt parameter. This is not native support: the 3090 lacks FP4 units, so the weights pass through the Marlin kernels and are executed in W4A16 mode. But the model loads and runs, which I didn’t expect at all.

Why is this important if AWQ already exists? More and more models now come with an official NVFP4 build right out of the box. Running these models directly eliminates the need to search the community for a suitable quantization, and one of the models we’ll discuss in the next chapter runs on this computer precisely because of this.

Does NVFP4 replace AWQ? I analyzed three int4 quantization options for the same Qwen3.6-27B model, GSM8K with 200 samples, running 5 passes for each:

AWQNVFP4AutoRoundGSM8K0.950.890.52Zero-shot0.720.710.385

AWQ beats Nvidia’s own mixed-precision checkpoint by six points on GSM8K. Note AutoRound at the bottom of this table: its calibration does not yet support this model’s hybrid attention mechanism (gated DeltaNet layers), so the fastest quantization of the three (85.7 tok/s) comes with a quality gap you can’t justify. In the “122B” section below, AutoRound wins the same comparison on a different model. Conflicting results from the same tool mean that quantization rankings are not transferable across architectures. Be sure to test the exact model you plan to use in a production environment.

So, here’s a rule of thumb for the Ampere architecture as of mid-2026: AWQ remains the default standard for everyday use, NVFP4 is an option for ensuring compatibility with models that ship in this format, and each new architecture requires its own comparative testing.

The centerpiece: a 122B model as a daily driver

Here’s what I really want you to remember. A consumer device costing $6,000 is running the Qwen3.5-122B-A10B model - an MoE model with 122 billion parameters - at reading speed, with almost 200K context. And this isn’t just a demo. It’s a tool that runs continuously.

First, the numbers - all measured on vLLM, TP4, 220 W:

WhatNumberSingle stream, code92.7 tok/sSingle stream, prose90.2 tok/sTime to first token, short prompt~120ms16 concurrent streams468 tok/s aggregateMax context that fits (default KV)130K tokensMax context with fp8 KV198.6K tokensPrefill, 60K token prompt35 secondsPrefill, 191K token prompt163 seconds

Some of these deserve special mention.

Over 90 tokens per second from a 122-billion-parameter model - that’s faster than you can read. It’s the MoE architecture that makes this possible on consumer-grade graphics cards: 122B parameters in total, with about 10 billion active per token, so the computational load per token is low, while the knowledge capacity remains high. Dense models with 70 billion parameters perform worse on this computer than this 122-billion-parameter model.

Context isn’t quite that simple. With the default settings, you get 130K KV cache tokens and no more - the weights take up the remaining 96GB. Switch the KV cache to fp8, and the upper limit increases to 198.6K tokens; in my tests, this had no effect on performance: 92.7 tokens/s before, 93.2 after. A half-precision KV cache is the cheapest context improvement available on this hardware.

And here’s the real number that nobody mentions in performance tests: processing a 191K-token request requires 2 minutes and 44 seconds of prefill before the first output token appears. You load the codebase once per session - and after that, everything runs like a charm. With a “cold” context of 190K tokens, you can’t just chat away like that.

What do I use it for? Testing and experiments, of course, that is half the reason this account exists. My work code is written by Claude. The rig under my desk covers everything else. It answers the questions cloud models refuse to answer, at two in the morning, without asking anyone’s permission. It never says “no” to me.

AWQ vs. Intel AutoRound: Same Model, Two int4 Quants

Everyone runs this model in 4-bit mode, so I compared two popular int4 builds against each other: the AWQ quant from the community, which I’ve been using for several months, and the Intel AutoRound build (the one that’s been downloaded a million times). The same engine, the same configuration, the same seeds, one day of speed tests, and one night of quality tests.

AWQ-4bitIntel AutoRound int4Single stream92.7 tok/s110.4 tok/s16 streams468 tok/s495 tok/sKV cache at the same config130K tokens347K tokensGSM8K (200 tasks)97.5%97.5%MMLU-Pro (210 tasks)82.9%81.4%Disk size75GB91GB

I ran the same 122B model twice, and the “same” model in one of the quants turned out to be 19% faster, while having 2.7 times more context space at the same quality. According to the GSM8K results, they were literally tied - both scored 195 out of 200. In MMLU-Pro, the difference was three questions out of 210, which can be considered statistical noise.

The size paradox is my favorite part. The AutoRound build takes up 16 GB more on disk but significantly less space in VRAM, because disk layout and runtime packing are two different things. Never judge a quant by its download size.

So the recommendation is obvious: if you’re running this model, use the AutoRound build. I myself switched my daily configuration after looking at the table.

An unexpected twist: just as I was finishing this article, Nvidia dethroned it

For several months, every open-source release was either too large for 96 GB or so small that 122B simply outperformed it. Three days before I finished writing this article, that pattern was broken. Nvidia’s Nemotron-Puzzle-75B model, presented in the NVFP4 format from the quant chapter, took exactly the spot this box was waiting for: the same performance of ~103 tokens per second in a single thread, but with a 256K context, and the VRAM size doesn’t increase at all as the context grows.

The secret lies in the architecture. It’s a hybrid mamba2 + MoE model with a total capacity of 75 billion, of which ~9 billion are active. The mamba component maintains a fixed-size state rather than a growing “key-value” cache, which is why the system occupies 23 GB per GPU for ANY context length. Decode is virtually insensitive to depth: 106.6 tokens/s with a 1K context, 101.0 with a 200K context. It experiences a 5% drop on the curve where a conventional model with an attention mechanism drops off sharply. It passes the “needle” test even at 200K.

The 122B model still knows more - it’s simply a larger model, and a direct quality comparison test is next on my list. But for working with long contexts, the 75B model has become my primary working tool overnight. This is the subtle lesson I’ve learned from six months of working on fixed hardware: the system itself didn’t change, but its capabilities evolved every month solely through software updates and new releases.

The crazy experiments

Two things I ran mainly to see if they were actually feasible.

The frontier 284B model on this computer, running at a reasonable speed. It started out as a joke, but it’s not one anymore. DeepSeek V4 Flash is a model with 284 billion parameters. The full model, with 256 experts and quantized to 2-bit precision, takes up 87 GB and barely fits into 96 GB of video memory. “Out of the box,” it performed prefill at a rate of 15 tokens per second; it took 25 minutes to respond to a 23K-long prompt.

After four days of work on llama.cpp (I led the effort, while Fable 5 studied the kernel’s internal structure), the prefill speed reached 443 tokens per second with a 97K-long context. That’s a 29-fold speedup, and none of it is a new kernel. Attention was silently falling back to the CPU over a tensor shape check; the MoE matrix multiplication added padding to its blocks, so that 87% of the multiplications were for zeros, and the tool call parser discarded valid calls it deemed out of order. Now, prefill for a 253K prompt takes 16.5 minutes, the “needle” test passes at 200K, and the model’s own MTP head, run in speculative decoding mode, increases the generation rate from 15–18 to 37 tokens per second.

The fork is publicly available (github.com/alesha-pro/llama.cpp, ds4-longctx branch), and people I’ve never met have re-built it and reproduced the results on their own systems with multiple 3090s. That’s what I’m really proud of. Is a 284B 2-bit model your everyday choice? No. The quality tax is no joke, and my everyday models are the 75B and 122B I mentioned earlier. But the phrase “a big DeepSeek runs on GPUs from a classifieds site” is no longer a joke. It has a performance of 443 tokens per second on hardware that hasn’t been touched.

A million context tokens on consumer cards. Thanks to YaRN RoPE scaling and quantized KV, this server processes a 1 million token request with a 9B model. Prefill time: 128,000 in 21 seconds, 256,000 in 60 seconds, 512,000 in 197 seconds, and 1 million in 699 seconds. 11.5 minutes to the first token at 1M context. So yes, there is a context with millions of tokens on your home computer, but no, you won’t be using it interactively. Functionality and usefulness are different things, and it’s helpful to understand the difference.

What went wrong

An honest guide shouldn’t hide failures from you. Here’s a brief list of what took my evenings:

• Flavor one: silent CPU fallbacks. The main villain of this article. The CUDA kernel, which does not like the shape of the tensor, does not issue an error. It simply transfers the work to the central processor, and you wonder why four GPUs are loaded by only 7%. GGML_SCHED_DEBUG=2 in llama.cpp and one nsys profile will show you the truth in a few minutes. Don’t guess where the time goes. Measure.

• Flavor two: endless incompatibilities between everything and everything else. CUDA versions, nvcc versions, and FlashInfer JIT: cu13 kernels were installed in the venv, nvcc 12.6 was in the PATH, and half a day lost; the fix was two environment variables. Attention kernels and tensor shapes. Quantization calibration and new attention architectures. Parallel pipelines and models created without SupportsPP do not run. “Fits in video memory” vs “fits with only 960 tokens of context left” is what Gemma 4 31B AWQ really means on a single graphics card. The tool-call parser vs a model that writes parameters in its own order: everyone blames the 2-bit quantizer, but the log proxy shows that the parser is to blame. Even the system itself is against itself: the riser, under constant load from four graphics cards, gave a barrage of PCI PERR errors, and only the BMC was able to return the system to working order. All of this is not some exotic war story. It’s a typical week in local LLM serving, and the sooner you get used to it, the easier it will be.

If you boil six months down to one phrase, it would be: Local serving is 20% model and 80% stack.

What the money buys, and what I would buy today

The math that convinced me is in one paragraph. The box cost about $6,000, once. At 220W load per card, four cards and the platform consume less than 1.1kW during operation. It processes 122B at 93-110 tokens/s in a single thread, 75B at 256K contexts, a total of half a thousand tokens/s at load, all without cost per token. In the six months that I’ve spent benchmarking and experimenting, the equivalent API spend would have paid for the box several times over. If you only use a few chats per day, an API key is the way to go. However, if you’re pushing the system to its limits, as I do, the server comes out on top. Additionally, it serves as a testbed for a fork that strangers have already rebuilt and verified on their own hardware. This is not possible with an API key.

What would I buy today based on my budget:

• One 3090 GPU (~$900). This is the best option to start with. With 24 GB, you can quickly run 12B and 27B dense models at an acceptable speed. Everything mentioned in this article started here.

• Two 3090 GPU (~$1800). This is the best value for money. You can run 27B to 31B dense models with real-world context, MTP speculative decoding, and the ability to experiment.

• Four 3090 GPU (~$6000 total). 96GB club. 122B MoE at reading speed, 198K context, hundreds of tokens per second. This is where running local stops feeling like a compromise.

• What I’m dreaming of after the renovation: an RTX 6000 Pro, which at ~$10,000 for a single card is hard to justify, or maybe a DGX Spark (or two) if I can convince my wife. To be honest, this box has not hit a wall yet.

After six months, I keep coming back to the same idea. The fact is that all the numbers in this article were obtained at home, on old cards, by a single person using an oven grill. All the interesting findings that I share in this article came from the gap between ‘I read the benchmark’ and ‘I ran the benchmark’. Let’s measure together.

I’m testing everything that fits in 96 GB and sharing the results here. If you run local models on your hardware, you know where the follow button is.