DeepSeek v4 Flash 0731 4bit ~50tps prefill, ~1tps decode on M5 Air 32gb

Currently running some experiments using the streamed experts trick that's been floating around this sub as well as some of my own trickery to get prefill to run a bit faster. It's been quite a bit of fun so far - just getting a 300b model to run at all on an Air is itself equal parts silly and satisfying Repo isn't in a tidy enough state to share - nothing about it is anywhere close to one-click serve yet. Especially reluctant to share since my naïve implementation of streamed experts incurred something like a ~30s tax between turns before any actual KV cache generation began. It's better now - more like ~3s last I benched -- Also fun discovery I've made in the meantime - you can actually run less experts than default at prefill time and the KV caches are still perfectly serviceable. Run conservatively and you get >95% the same top logits at each position. Run more aggressively and you lose that, but it doesn't always seem catastrophic e.g. needle in haystack perf can still be retained