INT3 compression+fused metal kernels [R]
Summary
Solo researcher released Spiral, a tool that compresses LLMs to INT3 and KV-cache to INT2 with custom fused Metal kernels for Apple Silicon, currently shipping Qwen-7B preview.
Similar Articles
I ported EXL3 to run well on Apple Silicon - PonyExl3
Ported the EXL3 LLM codec to run on Apple Silicon via Metal, achieving high prefill and generation speeds on M5 Max (e.g., ~600 tok/s prefill, 17-80 tok/s gen on various models).
Llama cpp metal moe optimization
A developer shared a Metal optimization for llama.cpp that improves decode speed for IQ3_XXS models on Apple Silicon, with a GitHub pull request and a request for community testing.
Metal-Sci: A Scientific Compute Benchmark for Evolutionary LLM Kernel Search on Apple Silicon
Metal-Sci introduces a 10-task benchmark for optimizing scientific computing kernels on Apple Silicon, paired with an evolutionary search framework driven by large language models. The study evaluates models like Claude Opus 4.7, Gemini 3.1 Pro, and GPT 5.5, demonstrating significant speedups while using out-of-distribution testing to catch silent performance regressions.
Qwen3.6-35B-A3B-Abliterated-Heretic-MLX-4bit
The user reviews a quantized and fine-tuned version of the Qwen3.6-35B model optimized for Apple Silicon via MLX, praising its speed, intelligence, and lack of safety disclaimers.
Optimizing On-Device Inference for Apple Silicon (20 minute read)
Apple's Lily engine optimizes on-device LLM inference for Apple silicon by leveraging unified memory and hardware, outperforming MLX-LM, and is tuned for the Qwen3.6-35B-A3B model's architecture.