21 GPU's benchmarked running a small TTS model (vram peak: 5GB)
Summary
A user benchmarks 21 consumer GPUs on vast.ai running a small TTS model (OmniVoice) with peak VRAM of 5GB, comparing performance relative to real-time and to an RTX 3090.
Similar Articles
Benchmarks of 20 small LLMs on a 6GB RTX 4050
A detailed benchmark of 20 small LLMs quantized for a 6GB GPU, measuring speed and VRAM usage at various context lengths, with qualitative probing for tool-use and instruction following. The report aims to help users with modest hardware choose models for local, private automation tasks.
4090 + 5060 Ti + 64GB RAM: 206 t/s on a 35B-A3B, and a 122B at 37 t/s
A user shares benchmark results for running large language models (Qwen 27B-122B) on a dual-GPU setup with RTX 4090 and RTX 5060 Ti, achieving high token generation speeds (e.g., 206 t/s on 35B-A3B, 37-41 t/s on 122B). The post includes setup details and a link to a GitHub repo with scripts and raw data.
Ran the same models across Strix Halo, RTX 3090, and RTX 5070 because I wanted my own numbers
The author ran 55 inference benchmark runs across Strix Halo, RTX 3090, and RTX 5070 with multiple backends, revealing that memory bandwidth dominates decode speed, the RTX 5070 beats the 3090 on small models, and reasoning models appear ~5x slower due to hidden reasoning content.
Local agent workspace on a 4GB laptop GPU (RTX 3050 Ti): the tok/s and where a small model struggles once it has to call tools, build artifacts, and RAG
The author benchmarks local Qwen models of various sizes on a 4GB RTX 3050 Ti laptop GPU within the Bike4Mind workspace, finding the 2B model at Q4_K_M quantization is the sweet spot for fitting in VRAM, achieving 96 tok/s. Smaller models struggle with tool selection, artifact generation requiring multiple models, and RAG embeddings causing model swap overhead.
Qwen 35B-A3B is very usable with 12GB of VRAM
A user benchmarks Qwen 35B-A3B (a 35B MoE model) on a 12GB RTX 3060, finding that 12GB VRAM is a practical sweet spot for running the model with 32k context, achieving ~47 t/s generation.