A Reddit user asks for advice on cost-effective hardware setups to run a local AI model like Claude Opus or Qwen 3.8 Next, discussing GPU options such as Tesla V100s, AMD Strix, and Intel Arc Pro within a $4000 budget.
Hey Reddit. This is a post asking for advice / user experience. The goal is simple: deploy a small private server for a developer to run a harness that rivals/beats Claude Opus (in perf/intelligence, not necessarily speed). I believe the model to target is a Q3 or so version of the Qwen 3.8 Next model, and to build a dual GPU, 64GB setup. Streaming n-gram off RAM/SSD (~85GB for total model + KV, minus 27 n-gram = 63GB) would mean the 64GB might be enough. Budget: as low as possible, up to $4000. I am willing to get extremely scrappy though. I am wondering: if I want to run this model with a full 256k context, and want about ~500 tok/sec PP, and +20 tok/sec decode, what kind of cards do I need? I see my options as follows: Budget and weird Something super silly with 2 or 3 Tesla V100s, the 32GB version, which I assume is possible but have yet to see anyone actually deploy. This ofc would have the pro of being cheap, the con of being ancient and losing community support as even better models come out. It would also mean relying on a more serious mobo with proper PCIe lanes. Mobo and parts would be maybe $600, 2 cards on eBay ~$2000. Doable. Budget and interesting AMD Strix 64GB edition (way cheaper, around $2000) and a 5060 Ti 16GB, then run the model on Vulkan. This would mean we could utilize the iGPU and the fast RAM to handle the experts and have the model attention running on the 5060 Ti — this sounds like the best of both worlds. Costs $2600 + $750 + OcuLink riser = $3500. Doable, and would probably be very fast. Cons: running on Vulkan means slower PP, weird deployment. Budget and Intel 2x Arc Pro B70s. I have seen vLLM has a fork that plays nice on Intel. Going with vLLM over llama.cpp means that offloading the n-gram would be more finicky — so I would probably need 64GB of RAM. That's $400 today, so system price would go up significantly. And they aren't that fast — from what I can see anyways, the VRAM is about 2x the Strix Halo in Gbps. $1200 x2 + $800 for system = $3200. Con: probably slower. Not So Budget and Modern ROCm 2x the RX 7900 XTX or r9700. Better support than Intel, probably could offload to SSD properly and use this more modern + newer models down the line. Fastest, most future proof. Costs $1700 x2 + $500 ish = $4100 overall. I'm looking for any benchmarks of these systems, advice and/or anecdotes from anyone that wants to provide their two cents. Note that this model seems like a good target, but ofc this system would be meant to continually provide a server for a single dev with modern, frontier-matching AI without breaking the bank on a 5090 or something. Going forward, I expect the ~mid-tier, Opus-class of models to be what people actually can afford to run, and with optimizations like n-gram I assume they will be runnable on even less RAM as time goes on. Obviously, prices are getting crazy, but they seem to just keep rising — so let me know all/any thoughts. also - this isnt written by AI - so pls read.
A reminder that two RTX 3090s and open-source models like Qwen 3.6 27B or Gemma 4 31B can run powerful local AI agents, comparable to Opus 4.5, using tools like Claude Code and self-hosted SearXNG.
Discusses the cheapest hardware options for running Qwen 3.6 models, comparing RTX 3090 and Tesla V100 GPUs, and provides a detailed cost breakdown for a system at around $2000.
The tweet discusses the feasibility of running the Qwen3.8-27B AI model on $1500 hardware, highlighting its usability and cost-effectiveness for most people.
Andrew Chen shares his experience of buying multiple GPUs for local AI experimentation, running Qwen3.6 27B dense at 100 tok/s on a 5090 eGPU, and compares it to Sonnet 4.6.
A community thread asking how enthusiasts plan to run the large Qwen3.8-2.4T-A95B model locally, sharing the struggle of running huge AI models on personal hardware.