Tag
A creator shares benchmark experiments with the Strata inference framework, systematically limiting VRAM budgets for the expert cache on an RTX 4090 to measure how speed degrades across different simulated GPU memory tiers, using Qwen3.8-Flash-Next Q2_0 at 8192 context and temperature 0.
A user configured a mismatched pair of Tesla V100 GPUs (16GB and 32GB) into a capable local LLM lab using llama.cpp with tensor split and other optimizations, achieving high prompt and decode speeds with the Qwen3.8 27B model.
Benchmark results for Apple's M5 Ultra and M6 chips show significant GPU performance gains, with the M5 Ultra achieving up to 59% higher Metal scores on Geekbench compared to the M3 Ultra, and the M6 chip scoring 34% higher than the M5.
A user reports achieving 60 tokens per second with the Ornith-1.5-35B-A3B MOE model on an NVIDIA RTX 4070 Ti, demonstrating efficient local inference optimization.
The RTX 5060 Ti local LLM repository has been refreshed with tested, copyable presets and a new high-context testing harness, providing validated configurations and benchmark results for models like Qwen3.8 27B on single and dual GPU setups.
The author benchmarks local Qwen models of various sizes on a 4GB RTX 3050 Ti laptop GPU within the Bike4Mind workspace, finding the 2B model at Q4_K_M quantization is the sweet spot for fitting in VRAM, achieving 96 tok/s. Smaller models struggle with tool selection, artifact generation requiring multiple models, and RAG embeddings causing model swap overhead.
Primate Labs releases Geekbench 7 with new video/audio encoding tests, a redesigned multi-core test, and larger datasets for more accurate benchmarking of modern hardware.
Demonstrates achieving 256k context on a 16GB RTX 4080 Super using Ternary Bonsai 27B Q2_0 model with llama.cpp, achieving up to 141 tok/s generation speed.
Sharon Zhou proposes a vendor-agnostic, kernel-level GPU performance benchmark to help AI agents optimize compute efficiency for frontier models, and highlights AMD's AgentKernelArena as a starting point.
Running the quantized Gemma-4-31B model on an RTX 5090 increases context length from 35k to 80k, showcasing significant performance improvement.
User benchmarks the APEX quantized version of Gemma4 26B A4B model on AMD RX 9060 XT, achieving 38 tps at 90k context with no quality degradation, finding it better than previous quantizations.
A user benchmarks 21 consumer GPUs on vast.ai running a small TTS model (OmniVoice) with peak VRAM of 5GB, comparing performance relative to real-time and to an RTX 3090.