moе

#moе

@yibie: strata-mlx update: ・Long conversations interrupted by other requests no longer reread. For a 10,000 token conversation, the time to output the first character in the next round drops from 19.7 seconds to 0.26 seconds (M4 Max, requires adding --conversation-cache-mib)

X AI KOLs Timeline ↗ · 2026-10-12 Cached

strata-mlx update: interrupted long conversations no longer get reread — time to first character for a 10K-token session drops from 19.7s to 0.26s (M4 Max), and memory management on 32 GB Macs is optimized to avoid swap. The project is an Apple Silicon MLX inference engine for running 125B MoE models like Qwen3.8-Flash-Next, with throughput significantly better than llama.cpp and LM Studio.

0 favorites 0 likes

#moе

$2800 rig with 8x Radeon Pro V620 (256 GB VRAM) + custom vLLM fork = Qwen3.8-Flash-Next at 60 to 100 t/s decode and 3000+ t/s prefill

Reddit r/LocalLLaMA ↗ · 2026-10-08

一位 Reddit 用户用 8 张 Radeon Pro V620（256GB VRAM）和自定义 vLLM fork，让 Qwen3.8-Flash-Next 达到 60-100 t/s 解码速度和 3000+ t/s 预填充，比 llama.cpp 快约 800%。

0 favorites 0 likes

moе

@yibie: strata-mlx update: ・Long conversations interrupted by other requests no longer reread. For a 10,000 token conversation, the time to output the first character in the next round drops from 19.7 seconds to 0.26 seconds (M4 Max, requires adding --conversation-cache-mib)

$2800 rig with 8x Radeon Pro V620 (256 GB VRAM) + custom vLLM fork = Qwen3.8-Flash-Next at 60 to 100 t/s decode and 3000+ t/s prefill

Submit Feedback