Tag
John Ternus congratulates the team on efficiently running Muse Spark Open, a large model, on Mac Studio, highlighting advances in on-device AI performance.
BitRun 是一个可在普通笔记本 CPU 上零编译运行 1-bit 三值 LLM 的独立工具，在 Apple M4 Pro 上可达 98–127 t/s、Intel i5 上 40–45 t/s，并修复了微软 bitnet.cpp 代码库中的多个底层 C++ 内核 bug 并提交了上游 PR。
Qwen3.8-27B is shown running locally on an M1 MacBook with 16GB unified memory using UD-Q2_K_XL quantization and full Metal offload, achieving ~4 tokens/sec generation (~5.5 with speculative decoding) — demonstrating that large open-weight models can run on hardware many people already own.
A detailed technical writeup on pairing an M5 Ultra 256 GB for prefill with an M3 Ultra 512 GB for decode, using disaggregated inference over Thunderbolt 5 RDMA with GLM-5.3-Flash on MLX, cutting time-to-first-token on long prompts from ~40s to ~18.5s.
Apple researchers present Minions, a system that aggregates idle Apple Silicon GPUs across macOS devices into a dynamic, fault-tolerant distributed virtual GPU cloud for large-scale ML inference, offering a cost-effective alternative to centralized GPU infrastructure.
Google's EmbeddingGemma 2 embedding model has been repurposed into a speech-to-text tool that transcribes an hour of audio in about 13 seconds entirely on-device on a base MacBook Air, using a small trained reader on top of the frozen audio encoder.
作者系统整理了本地部署模型的加速方案，包括在 macOS 上结合 Turbo、MLX、VAE Q8 和前缀 KV cache 让 Qwen-Image 2.1 提速近 5 倍，以及用 Uzu 的投机解码把 Qwen3.5-9B 从 22 tok/s 提升到 92 tok/s、用 TensorRT 让 Qwen 在 RTX 5090 上比 llama.cpp 快 3 倍等案例。
The author describes building sift-light, an evidence-first local search layer for coding agents that prioritizes exact search and verifiable sources over plausible semantic guesses, with new visual retrieval for images and scanned PDFs on Apple Silicon.
strata-mlx update: interrupted long conversations no longer get reread — time to first character for a 10K-token session drops from 19.7s to 0.26s (M4 Max), and memory management on 32 GB Macs is optimized to avoid swap. The project is an Apple Silicon MLX inference engine for running 125B MoE models like Qwen3.8-Flash-Next, with throughput significantly better than llama.cpp and LM Studio.
Crest 是一款受 Arc 启发的开源浏览器，支持 Mac、iPad 和 iPhone，提供 Spaces、侧边栏、Split View、Peek 等功能，并可从 Arc、Zen、Chrome、Safari 等浏览器导入数据，目前处于早期访问阶段。
Decision 3.0 has been converted to Core ML for Apple silicon, triaging 1,000 pull requests into vllm-sr workgroups entirely on device at 0.12 s per PR on an M5 Pro, with lite, nano and mini variants unified in one repo.
这篇帖子通俗解释了投机解码（Speculative Decoding）的工作原理——草稿模型预测、目标模型并行验证并以概率方式接受或纠正——并介绍了开源引擎 Uzu，它在 Apple Silicon 上结合并行起草、连贯草稿路径与免回滚验证，使 4-bit Qwen3.5 9B 的推理速度从 llama.cpp 的 22 tokens/s 提升到 92–117 tokens/s。
strata-mlx 更新：在低内存 Mac 上提升推理速度但增加内存占用，新增 --check 性能预检命令，并评估了专家预测预取方案的收益（仅 1.35x 上限，未实施）。
用户惊叹于在 M3 Max 64GB Mac 上通过 oMLX 成功运行 Qwen3.8-Flash-Next-oQ4e-mtp 这个约 100GB 的模型，58GB 分配给 GPU，并支持最高 130k 上下文窗口，展示了本地大模型推理性能的显著提升。
Benchmark thread showing Uzu, an open-source inference engine from Mirai, running Qwen3.5-9B at 92.1 tokens/s on a 16GB M5 MacBook Pro — about 4x faster than llama.cpp (22 tok/s) and MLX (25.1 tok/s) — by using tree-based speculative decoding with a Weaver drafter to break the memory-bandwidth ceiling of ordinary local decoding.
PocketWebTools for Mac 推出，一款在 Apple Silicon 上本地运行最高 35B 参数开源模型的付费应用，提供聊天、带说话人标注的转录、视频剪辑、图片超分和 PDF 文字提取等功能，所有文件保留在本地设备上，一次付费 99 美元含终身更新。
文章介绍开源推理引擎 Uzu（Mirai 出品），通过最多提前草拟 16 个 token 的推测解码与为 Apple M5 优化的 kernel，在 M5 芯片上运行 4-bit Qwen3.5 9B 达到 92-117 tokens/s，远超 llama.cpp 与 MLX 的 22-25 tokens/s，并解释了 batch-one 解码的内存带宽瓶颈原理。
Lithos 开源了 lithos-metal，一个为 Apple Silicon 生成 Metal megakernels 的推理引擎，配合 DSpark 投机解码可在单台 M5 Max 上以每用户 200+ tokens/s 的峰值速度运行 Qwen3.8-27B，支持一行命令接入各类 coding agent。
A community MLX quantization of UkisAI's Swift 1.5 (based on Qwen 3.8 Flash Next) at 4.7bpw, tuned for the 96GB Mac Studio with M5 Ultra, delivering strong prefill/decode speeds and 91% top-1 agreement with the BF16 model.
Udon is a control deck for self-hosting apps, containers, storage, and monitoring on Mac mini or Apple Silicon Macs, managed via browser, mobile companion, or a built-in AI sysadmin assistant.