gpu-setup

A software developer shares their experience upgrading a local setup with two RTX Pro GPUs, troubleshooting power issues, and achieving high performance for running LLMs like Qwen and Deepseek. They discuss configuration details and seek advice on optimizations and model suggestions.

This article introduces local large model hardware configurations from $2,000 to $40,000, including detailed setups from dual RTX 3090 to quad RTX PRO 6000, covering PCIe switches, GPU communication, Docker configuration, and speech-to-text.

A comprehensive guide to building a local setup for running state-of-the-art LLMs, including hardware recommendations (from $2k to $40k), PCIe switching, and Docker configurations for models like Qwen and GLM.

A guide showing how to build a system under $2500 using used server components to run GLM5.2 and other large AI models locally, with trade-offs in speed.

A user proposes a hardware setup using four RTX 5060 Ti GPUs and 512 GB of DDR3 server RAM to run GLM2 at a decent quantization and seeks feedback on the idea's viability.

A setup using RTX 5080 and RTX 3090 GPUs achieves 80 tokens per second on the Qwen 3.6 27B Q8 model.

A user describes their home data center setup with multiple high-end systems for ML experiments, training, and agentic coding.

Andrej Karpathy notes that a common first step in his nanochat tutorial (booting up an 8XH100 GPU) would stump beginners, highlighting a barrier to entry in AI development.

A user built a private AI lab under his desk using RTX 5090 and RTX 4090 GPUs, running local open-source models like Qwen, DeepSeek, and Llama to avoid API costs.

A user shares their experience setting up a dual-GPU local AI lab with RTX 4080 Super and 5060 Ti, running Qwen 3.6 models via llama.cpp and llama-swap to reduce API costs and enable unrestricted experimentation.

A user shares their experience setting up a dual 3090 GPU system to run the Qwen 3.6 27b model locally, achieving over 100 tokens/second after switching to Ubuntu and using the club-3090 tool with custom patches. They express excitement about the future of local AI.

A technical guide on setting up local LLM autocomplete (Qwen2.5-Coder-7B) and agentic coding (Qwen3.6-35B-A3B) on a single 16GB GPU with 64GB+ RAM using llama.cpp, including commands and performance benchmarks.

A user debates whether investing in a high-end private local LLM setup with 5×3090 GPUs can match cloud services like Claude or GPT while ensuring data privacy.