Tag
A builder shares three self-hosted 4x Radeon AI PRO R9700 inference servers (~$10.7k each, 384GB VRAM total) running Qwen3.8-27B via vLLM-Radiance/ROCm, with detailed throughput, latency, and long-context benchmarks after hosted models began refusing firearm-industry research tasks.
A user shares a 4x AMD Radeon R9700 AI Pro workstation build (128GB total VRAM, Threadripper 9970x) serving Qwen 3.8 Next Flash/27B and DSV4 Flash for a 10-person startup, reporting strong aggregate prefill/decode throughput across 16 concurrent sessions.
In July 2023, the user built a high-end computer with 8x RTX 3090s and an Epyc 7004 CPU to run Llama 2 70B locally, expressing confidence in the future of open-source AI.
Jeff Geerling built a portable homelab in a carry-on suitcase using Ubiquiti UniFi networking gear for a demo at VCF Midwest, featuring high-speed networking, 5G backup, and travel-ready design.
A showcase of a personal local AI inference build featuring Intel Arc B140 GPUs and custom hardware, running llama.cpp with SYCL back-end on Ubuntu.
The author shares their storage server build optimized for local AI inference, anticipating a rumored 26T-a3b model called "Le Chaton FAT" and using high-capacity NVMe drives with ZFS for model storage.
A grad student shares their experience building a multi-GPU workstation with 4x3090 Ti running on a single US wall outlet, detailing constraints, power-limiting challenges, and component choices.
A community member details a custom PC build using discontinued Intel Optane Persistent Memory to successfully run the 1-trillion parameter Kimi K2.5 model locally at roughly 4 tokens per second via llama.cpp.