vram

A US-based GPU repair technician claims that NVIDIA RTX 5090 variants with 64GB and 96GB of VRAM exist, suggesting potential pro-grade or workstation versions of the Blackwell consumer card.

A user benchmarked LLM inference on rented vast.ai servers with dual AMD EPYC 7642 (ROME) boards, large DRAM, and RTX 3090s, finding that RAM-heavy server builds deliver near-linear scaling up to 8 concurrent streams — hitting 94 tok/s at c=1 and 456 tok/s at c=8 on Qwen 3.8 Flash Next Unsloth Q8.

A user asks whether upgrading to 128GB DDR4 RAM alongside dual RTX 3090s (48GB VRAM) would let them run models like DeepSeek V4.1 and GLM 5.3 Flash at low quantizations with ~20 tokens/s, or whether a DDR5 system would be needed.

A local LLM user reports that sacrificing context length (96K→75K) to run higher-quality quantizations (IQ4_XS→UD-Q4_K_XS with KV-Q8) on a Qwen 27B model actually improved overall task completion time, cutting an 8-hour coding task to under 6 hours due to fewer error-prone drafts.

A community benchmark evaluated 21 Qwen3.8 27B model/KV quantization combinations (189 benchmarks) that fit in 16GB VRAM, finding q5_1/q5_1 is usually optimal and that upping model quant at q4_0/q4_0 beats lower quants at full f16 KV cache.

RTX 3090 Founders Edition owners running local LLMs may be cooking their VRAM: benchmarks show stock fan/power curves let GDDR6X hit 112°C within seconds of token generation, far above the 96°C continuous rating. The poster recommends capping the card at 270W with fans at 100% (or 240W at 70% fan) at roughly a 30% throughput cost, and notes undervolting doesn't help because Nvidia only applies clock offsets above 300W.

A community author ranks every LLM that can run on 10-16GB VRAM, evaluating models like Gemma 4, Qwen 3.6, GLM-4.7 Flash, and Mistral on speed, coherency, multilingual handling, slop patterns, instruction-following, and censorship for local roleplay use.

A buyer purchased an abandoned Russian crypto mining farm on Avito for about $35 and discovered it contained nine working P106-006 (GTX 1060 6GB) GPUs totaling 54GB of VRAM, booted fine from USB, and was only missing an SSD.

A creator shares benchmark experiments with the Strata inference framework, systematically limiting VRAM budgets for the expert cache on an RTX 4090 to measure how speed degrades across different simulated GPU memory tiers, using Qwen3.8-Flash-Next Q2_0 at 8192 context and temperature 0.

The author believes that after the Strata revolution, prices for memory above 32G and mid-range graphics cards above 16G are very likely to rise, and suggests that a configuration of 64G RAM + 16G VRAM can run Strata for just over a thousand yuan. The quoted post shows that DDR5 prices (especially 16G and 32G) have already deteriorated significantly.

A new strata architecture lets GPUs with 12G VRAM pair with 128G system memory to run 100B+ models, triggering a market shift where old cards (3090, V100) and cheap DDR3 memory are in renewed demand, and even iPhones/Macs are being used as memory expansion for inference.

LocalMaxxing is a community benchmarking platform for local LLM inference, where users submit real-world test results to compare tokens/sec, first-token latency, and VRAM usage across models, GPUs, inference engines, and quantization formats. You can also run benchmarks yourself via the CLI and contribute to the public leaderboards when picking hardware or tuning local setups.

A hands-on benchmark on a single RTX 3090 shows llama.cpp is 2-3x faster and much quicker to first token when the MoE model fits in VRAM, while FreeToken wins 7x on time-to-first-token with 32 concurrent users on gpt-oss-120b (63 GB) that overflows VRAM.

The user built a low-cost setup using five ex-mining BC-250 boards to run the Qwen3-Coder-Next AI model, achieving around 40 tokens per second at 30k context with plans to expand.

Unsloth has released GGUF quantized versions of Qwen-Image-2.1, enabling it to run locally on 12GB VRAM with performance comparable to Nano Banana 2.0.

The article discusses how 16GB of VRAM is the realistic high-end limit for most users due to financial constraints, but recent AI model improvements like Qwen 27B quants are enabling more capabilities on such hardware, with hopes for future architectural innovations.

The author compared the performance of Qwen3.8 27B IQ3_XXS and Bonsai Ternary PQ2 on limited VRAM, finding that Qwen is faster and uses fewer tokens, while Bonsai has a smaller file size but longer generation times.

A user built a 768GB VRAM system using 12x64GB CMP170HX cards for less than the cost of one RTX 6000 Pro, enabling local inference of various large AI models with strong performance.

The article benchmarks three AI models on 16GB VRAM, showing that Qwen3.8-27B performs best, followed by Ornith-1.5-9B, with IFM/K2-Horizon-7B lagging behind in speed and task completion.

The author shares their experience building a high-VRAM AI PC with multiple GPUs to run local models for software engineering, detailing solutions to hardware challenges like PSU limitations and GPU mounting.