Tag
Apple's M5 Ultra chip shows up to 1.5x faster token generation and 4x faster prompt processing for local LLMs compared to the M3 Ultra, with improved memory bandwidth and storage speeds, but at a high cost.
A developer shares a fork of FreeToken, an edge-native MoE serving engine, with added support for DeepSeek-V4.1, vision capabilities for Qwen models, and speculative decoding, including benchmarks on dual RTX 3090 hardware.
The author plans to spend $100 on cloud GPUs to benchmark Qwen3.8-27B models with different quantization levels and KV cache settings, focusing on coding and agentic tasks to provide useful data for local AI enthusiasts, and is seeking community input on the methodology.
The author compares various GPUs for LLM inference, critiquing common benchmarks and emphasizing the importance of prefill performance over generation speed, offering recommendations for different budgets and use cases.
The article breaks down memory bandwidth as the critical metric for local AI hardware performance, comparing current GPUs and unified memory systems from NVIDIA, Apple, AMD, Intel, and others across different performance tiers.