gpu-setup

This article introduces local large model hardware configurations from $2,000 to $40,000, including detailed setups from dual RTX 3090 to quad RTX PRO 6000, covering PCIe switches, GPU communication, Docker configuration, and speech-to-text.

A comprehensive guide to building a local setup for running state-of-the-art LLMs, including hardware recommendations (from $2k to $40k), PCIe switching, and Docker configurations for models like Qwen and GLM.

A guide showing how to build a system under $2500 using used server components to run GLM5.2 and other large AI models locally, with trade-offs in speed.

A user proposes a hardware setup using four RTX 5060 Ti GPUs and 512 GB of DDR3 server RAM to run GLM2 at a decent quantization and seeks feedback on the idea's viability.

A setup using RTX 5080 and RTX 3090 GPUs achieves 80 tokens per second on the Qwen 3.6 27B Q8 model.

A user describes their home data center setup with multiple high-end systems for ML experiments, training, and agentic coding.

Andrej Karpathy notes that a common first step in his nanochat tutorial (booting up an 8XH100 GPU) would stump beginners, highlighting a barrier to entry in AI development.

A user built a private AI lab under his desk using RTX 5090 and RTX 4090 GPUs, running local open-source models like Qwen, DeepSeek, and Llama to avoid API costs.

A user shares their experience setting up a dual-GPU local AI lab with RTX 4080 Super and 5060 Ti, running Qwen 3.6 models via llama.cpp and llama-swap to reduce API costs and enable unrestricted experimentation.

A user shares their experience setting up a dual 3090 GPU system to run the Qwen 3.6 27b model locally, achieving over 100 tokens/second after switching to Ubuntu and using the club-3090 tool with custom patches. They express excitement about the future of local AI.

A technical guide on setting up local LLM autocomplete (Qwen2.5-Coder-7B) and agentic coding (Qwen3.6-35B-A3B) on a single 16GB GPU with 64GB+ RAM using llama.cpp, including commands and performance benchmarks.

A user debates whether investing in a high-end private local LLM setup with 5×3090 GPUs can match cloud services like Claude or GPT while ensuring data privacy.