@0xSero: Locally Part 1 - Apple Silicon Macs give you large pools of memory to run big models, but the token generation speed wi…
Summary
Apple Silicon Macs offer large memory pools for running big models but with slower token generation, performing best with large MoEs that have low active parameters.
View Cached Full Text
Cached at: 04/22/26, 03:00 PM
Locally Part 1 - Apple Silicon Macs give you large pools of memory to run big models, but the token generation speed will be lower than most are used to. Macs are best with large MoEs that have low ACTIVE params. Basically when you see a model like Qwen3.5-397B-A17B this
Similar Articles
SwiftLM: Pure-Swift Apple Silicon LLM inference server—no Python, runs big models on low-RAM Macs
SwiftLM is a Swift-native LLM inference server for Apple Silicon that runs large models without Python, using SSD streaming to load MoE weights and enabling 122B models on 64 GB Macs.
@nicebabycat: https://x.com/nicebabycat/status/2091726637155103126
This article provides a detailed test of the local deployment and performance of the Ling-3.0-tiny model on an Apple M5 chip Mac, demonstrating the feasibility of running a 7.9B parameter model at 47 tokens per second without a discrete GPU.
@svpino: DeepSeek-V4-Flash running at 5.71 token/s on a Mac M5 Pro. Every day, we get better models running on consumer hardware…
The tweet highlights DeepSeek-V4-Flash running at 5.71 tokens per second on a Mac M5 Pro, emphasizing advancements in local AI inference on consumer hardware, with a mention of Tencent's open-source Palm-Infra for Apple Silicon optimization.
@julien_c: and is Apple Silicon the King of Local AI?
Discussion on whether Apple Silicon is the best hardware for running local AI models, referencing a linked article or thread.
Optimizing On-Device Inference for Apple Silicon (20 minute read)
Apple's Lily engine optimizes on-device LLM inference for Apple silicon by leveraging unified memory and hardware, outperforming MLX-LM, and is tuned for the Qwen3.6-35B-A3B model's architecture.