SOTA Apple Silicon Inference (August 15, 2026)

This is a HANDWRITTEN post. I spent way too much time trying to get fast inference on Apple Silicon. This post is for people who want to know what's the latest on running local models on their mac, and why they may not be seeing the performance others in the community claim. TL;DR I've spent the last 2 weeks full-time looking into the state of inference optimization on Apple Silicon, and honestly, the software stack is a mess. There is no framework that has all the inference optimizations that are available on CUDA/NVIDIA for the latest Qwen models: prefix caching, speculative decoding, paged KV cache, continuous batching, dynamic scheduling, flash attention, etc. On CUDA/NVIDIA, a lot of this stuff is already mature and integrated into the inference stacks people actually use. On Apple Silicon, the pieces are scattered across mlx-lm, vllm-metal, forks, custom model conversions, and a bunch of other frameworks, and a lot of them only implement part of the stack. The biggest issue I've found is that the newer Qwen models use a hybrid KV/recurrent state, which makes prefix caching and speculative decoding much harder to combine. On top of that, mlx-lm currently drops the built-in MTP heads during model conversion, so even the models that have built-in speculative decoding support are getting converted without the thing you need. From everything I've tested, vllm-metal is the closest thing I've found to a proper Apple Silicon inference optimization stack right now. I think we should stop making another fork every time something is missing and instead get one stack working properly, then upstream the pieces into mlx-lm and vllm. The Long Version I spent the last 2 weeks digging into this. It should not have taken me this long to understand the space of inference optimizations on Apple Silicon, and I think that's a sign of how bad the space is right now. First of all, llama.cpp on CUDA has everything built in and working. You go on Reddit, you look at LocalLLaMA, and you're like, okay, how do I run this on Mac? And suddenly there are a thousand different flavors of projects and they're all saying they're the fastest. And you're looking at it like, what the fuck is going on? What are all these things? Why do I need all these things? First you hear "everyone's using LM Studio". So you try LM Studio, and you're like, okay, this isn't quite right. It's a little slow, there's some issues, whatever. So then you start digging...You go framework after framework, test after test, trying to figure out why inference on your mac never pans out to what others say it should. This is exactly what I experienced, and I'm hoping this post can work towards cleaning up this giant mess. Some "Context" on Inference Optimizations For anyone who isn't familiar with this stuff, there are basically two major parts to local inference: prefill and decode. Prefill is when your model processes the context you gave it and fills up the KV cache. So say you send your server a 10k token prompt. It processes all of that text and gets the model into the state represented by that conversation. Then you go into decode mode. Typically that's autoregressive, so you're predicting one token at a time, serially, until you hit the end of the response. There are a ton of optimizations around these two stages. You've got prefix caching, paged KV caching, speculative decoding, flash attention, flash decoding, continuous batching, dynamic scheduling, all of this stuff. And all of these things matter in different situations. If you're running a giant multi-user inference server, obviously some of these optimizations matter more than they do for one person sitting there talking to a model. But for the kind of local, long-running, agentic use case I'm talking about, I think two of the really important ones are prefix caching and speculative decoding. If you don't have both of these, I guarantee you, you will be disappointed at the performance of local models on your mac. Prefix caching helps the prefill side while speculative decoding helps the decode side. Prefix caching Remember that your server is stateless, so if you're having a long agentic conversation and your context keeps getting bigger, every request normally means the model has to process that entire conversation again before it can start generating the next response. So let's say you have a 10k context and you send a message. The model processes 10k tokens, does some inference, sends a response. Then you send another message and now you've got 10.5k tokens. Without prefix caching, it has to process that whole thing again and again. Obviously that gets worse and worse as the conversation gets longer. So you can see a benchmark saying 45 tokens/sec, but then you actually open the thing up in a real long-running session and suddenly you're getting 9 tokens/sec. If that's the case, most likely, the framework isn't doing prefix caching properly, so you're just paying that prefill cost over and over. A lot of frameworks will tell you, yes, prefix caching is implemented, but there is a huge fucking nuance here. The Qwen problem The models everyone wants to run right now, like Qwen3.8, aren't just using a normal KV cache. They have a hybrid cache architecture. You've got your normal KV cache, but you've also got a recurrent state, so it's a hybrid Gated DeltaNet/GDN model (yes, this is a rabbit hole). This recurrent state is tricky because it doesn't work like a flat KV cache. With the KV cache, you can think of it as preserving your history. The recurrent state is more like the "current" state of the model, as it overwrites the previous state as you process tokens. This is why prefix caching is hard for these models. vllm-metal recently (within the last week) added prefix caching support for these hybrid models, but the support comes with a pretty significant limitation: you have to choose between either prefix caching or speculative decoding, not both. The reason why they cut the feature to that point is because it's actually difficult to implement and they needed more time to sit on it. Speculative Decoding There are different kinds of speculative decoding implementations, and more come out each year via white papers. Some of these implementations uses a separate model, called a draft model, which is the option a lot of mlx engine forks are doing (for a particular reason that I will explain). DFlash is an example of a very recent draft model idea recently published. As you can imagine, there are pros and cons to every implementation. The con to these draft model implementations is that they take up much more memory, and they are more susceptible to predicting the next tokens incorrectly. Another implementation of speculative decoding is multi-token prediction, or MTP for short. In this implementation, models are trained with speculative decoding as a built-in feature. As you can imagine, this implementation has some decent benefits. Qwen3.6/3.8 have built-in MTP heads. You can see this on their huggingface page. In case it isn't clear, speculative decoding means you're predicting multiple tokens ahead during decode. So you might guess the next three tokens instead of the usual 1, and maybe 2/3 of those tokens were inferred correctly. This leads us back to the Qwen models that everyone wants to run, which as we know, has MTP built-in. When loading Qwen3.8-27B on llama.cpp, this just works. For mlx-lm, it doesn't. Why? Two parts. Well first of all, as I said before, these Qwen models have recurrent state that essentially "loses" history, so now you need to roll the model back to the state before the bad token was predicted in order to move forward. With a normal KV cache, that's relatively straightforward because you can think of the previous states as being available. With the recurrent state, you actually have to restore the previous recurrent state, so that means a bunch of code/logic to keep snapshots and restore from them. This isn't trivial. An even bigger issue: mlx-lm's lack of support This is where I think a lot of the framework fragmentation comes from. mlx-lm is basically the base layer, provided by Apple, that the whole Apple Silicon inference ecosystem is sitting on top of. Regarding MLX models, models are generally released on Hugging Face as SafeTensors. mlx-lm has the conversion tooling to turn those into MLX models. The problem is that, as of the current main/master state of mlx-lm, when you run the conversion, it silently removes the MTP weights and heads. This is visible in any mlx-community model.safetensors.index.json file. There has been a PR from AirRunner trying to add this support for months to mlx-lm, and it still hasn't been merged because the maintainers are AFK. The state of mlx-lm means these Qwen models are all neutered of their MTP capabilities on Apple Silicon without some additional changes. This is why you see so many flavors of MLX models on huggingface, and so many flavors of inference engines on github. Now we get a new post on localLLama weekly about yet another framework that is "BlAzInGlY F4sT." Pick any MLX inference project you can think of and try running it. With Qwen3.5+ it either doesn't actually support prefix caching or it doesn't support built-in speculative decoding. Please ignore the dumb benchmarks and actually try a couple turns on it with your code base and you'll see the result. Going back to my earlier point about inference optimizations, if you don't have both of these working on macos, you will not be happy with the performance. Either you have speculative decoding only, and the benchmarks look amazing but the agentic performance significantly degrades, or you have prefix caching working (vllm-metal today), which gives you stable tps over the context window but is too slow to be useful. So where does that leave us? I've tested most of the big projects people are talking about. I tried llama.cpp on Apple Silicon and the performance just isn't there for me. I've also noticed my laptop working significantly harder and running hotter with it. I know it's using MPS under the hood, but I think mlx-lm has Apple-specific optimizations that llama.cpp just doesn't have in the same way. So at least from my testing, I don't think `llama.cpp` is the right inference environment for Apple Silicon right now. The most developed and polished thing I've found is vllm-metal, which makes sense considering vllm was developed out of UC Berkeley. It already has a lot of the inference optimizations build in: continuous batching, dynamic scheduling, paged KV cache, flash attention, prefix caching, and so on. The problem is that we're back to the same issue. It has prefix caching. It has speculative decoding. But because the hybrid recurrent-state problem is hard, you can't currently have both together on these models. So what should we actually do? PLEASE don't make another framework. Until the MTP support gets upstreamed into mlx-lm, I think we should have one community fork based on AirRunner's work and everybody should just use that. Instead of somebody spinning up another Claude-coded wrapper every time they find a missing feature, put that effort into the same codebase. Get the built-in MTP support working, get proper prefix caching for the hybrid recurrent state, get prefix caching and speculative decoding working together, and keep the other stuff that's already working, like continuous batching, dynamic scheduling, paged KV cache, flash attention, etc. We do not need another fork for this. Yes, we can experiment with D-Flash and other speculative decoding approaches, I am not saying don't do this, but we need a good baseline first. Right now the Apple Silicon inference optimization space feels like everyone has built one piece of the puzzle and then decided to make their own fucking puzzle box around it. I don't want to spend two weeks figuring out which fork has which PR, which model conversion preserves which weights, whether "speculative decoding" means MTP or a separate draft model, and whether prefix caching actually works once my context gets long. As of August 15, 2026, after all of this testing, I still haven't found anything on apple silicon that can compete with the overall inference optimization stack that's available on CUDA/NVIDIA. So my ask to the community: Let's all stop with the forks and literally just patch the 1-2 projects that matter. Hopefully this post gives a guiding light to people who are as confused as I was 2 weeks ago.