Tag
该论文提出 LeMA，一种用于多 GPU 大规模神经网络训练的分布式 Levenberg-Marquardt 算法，通过精确对偶公式与 Jacobian 分片策略，将内存开销从参数量平方降低到批量大小平方，并在约 10 亿参数 CNN 上实现 8 GPU 近线性扩展。
一位用户分享了对本地 LLM 推理工具 Strata 的深度性能调优经验，通过调整 --pcie-frac、--pool-workers、--trim-stage-weights、--pipeline-windows 等参数并在双 RTX 3060 上优化 GPU 优先级，实现了约 35 tps 到 60+ tps 的解码速度提升。
一位开发者分享了他用二手零件组装的双 RTX 3090 本地 LLM 主机，并询问为何多 GPU 推理性能提升不及预期，以及如何定位瓶颈和优化配置。
A user shares a 4x AMD Radeon R9700 AI Pro workstation build (128GB total VRAM, Threadripper 9970x) serving Qwen 3.8 Next Flash/27B and DSV4 Flash for a 10-person startup, reporting strong aggregate prefill/decode throughput across 16 concurrent sessions.
作者发布了自制的 Qwen 3.8 27B Swift 1.5 量化模型（8-bit AWQ + GPTQ 混合量化），在 4 张 RTX 3090 / 48GB VRAM 上实现了速度与质量的良好平衡，在 2 张 RTX 3090 上配合 vLLM 与 MTP 可达到近 100 tok/s 的解码速度。
Infermeld is an experimental open-source Linux companion kit (v0.1.0) for running one GGUF model across mixed AMD and NVIDIA GPUs using llama.cpp, packaging device selection, build instructions, preflight checks, and a thermal guard. Tested on an RX 6000 XT + RTX 3080 with Qwen3.6-35B-A3B, with the maintainer seeking other hardware combos for reproduction.
Strata 0.1.39 has been released with multi-GPU optimization. According to user feedback, inference speeds in some scenarios have doubled — benchmarks on a 4090 48G running the hundred-billion-parameter Qwen3.8-Flash-Next showed a significant boost in tokens/s.
A developer recounts scaling their personal local-AI setup from a single RTX 3090 to a 16+ DGX Spark (GB10) cluster spanning two houses, enabling 300k-context inference of models like Kimi K3 (2.8T params) and MiMo 2.5 Pro entirely on-premises without commercial model subscriptions.
A user reports solid results running a $5400 eBay 8x V100 server for local LLM inference, using nvfp4 NVIDIA checkpoints via a heavily optimized vLLM fork (1CatAI/1Cat-vLLM) that unpacks to fp16 on the fly, achieving over 200 tok/s on a 27B model with TP=4 and strong prefill throughput.
A user optimized a 4x3060ti GPU rig for AI inference using tensor parallelism with Exl3 and vllm, achieving up to 120 tokens per second with large context windows.
A user with a local AI server running one RTX 3090 asks for advice on utilizing a second card effectively as a single user.
DwarfStar is a specialized inference engine designed to run large language models like DeepSeek and GLM efficiently on consumer hardware, supporting multiple platforms and advanced features like SSD streaming.
A user shares their experience building a multi-GPU system with RTX 5070 Ti and 5060 Ti for running AI models like Qwen3.8-27B-FP8 using vLLM on Linux, detailing hardware setup, benchmarks, and challenges.
The user has implemented nvfp4 KV cache support for the Qwen3.8 model on a heterogeneous GPU setup using custom CUDA kernels and quantization to optimize performance.
Android Studio now natively supports running the Gemma 4 AI model using llama.cpp, with features like multi-GPU support and a 128k context length.
The article explores optimizing AI language model performance on a home server built from e-waste GPUs, with explanations of transformer models and multi-GPU techniques.
The article details a validated hardware configuration using 16 RTX 5060 Ti GPUs with PLX switches to run the Deepseek V4 Flash model, achieving specific performance metrics for context handling and throughput.
The author benchmarked llama.cpp flags on a hybrid GPU setup with an RTX 4090 laptop and AMD XTX 7900 eGPU, achieving 70% faster generation, 40% faster prefill, and discovering a bug related to MTP in multi-GPU configurations.
A user seeks advice on building a budget home AI server with 32-48GB VRAM, debating between AMD RX 9060 XT and Nvidia RTX 5060 Ti GPUs, and whether to use AM5 or used EPYC platforms for local LLM inference and large MoE model offloading.
Technical post sharing performance stats for running Qwen 27B on 2x RTX 5070 Ti GPUs with vLLM cu129-nightly, achieving up to 94-87 tps decode and 170k GPU KV cache.