We are releasing our first quantized checkpoints for the Qwen3.5 series of models, co-designed jointly with our inference engine to achieve maximum possible performance on Apple hardware

Starting from 0.8B, 2B and 4B models

https://t.co/2R8BdhAfzv

Introducing Mirai Quantization: Redefining the speed-quality frontier for local LLMs on Apple silicon.

Source: https://trymirai.com/blog/quantization By Artur Chakhvadze, Ryan Mathieu, Roman Knyazhitskiy, Nikolai Voinilenko, Chen-Chen Yeh, Artur Mullakhmetov, Eugene Bokhan, in collaboration with others at Mirai Labs.

Jun 4, 2026

We are excited to release our first quantized checkpoints for the Qwen3.5 series of models. Our co-designed quantization and inference-engine stack dramatically outperforms Unsloth/llama.cpp and MLX on the precision-vs-generation-speed tradeoff,delivering 40–60% more tokens per secondat the same quality level across Apple devices ranging from smartphones to high-end desktops.

This is the first of many upcoming releases from our Model Optimization project, and a small part of a larger engineering effort to bring capable fully local agents running at over 1,000 tokens per second to consumer hardware. Many of the choices we made here are strategic: the quantization format was carefully chosen to maximize hardware utilization when used together with our future speculative decoding pipeline.

We are starting with4\-bit \(Mirai\-M\)and8\-bit \(Mirai\-L\)versions of Qwen3.5-0.8B, Qwen3.5-2B, and Qwen3.5-4B. Larger Qwen3.5/Qwen3.6 models, as well as the Gemma 4 family, are coming soon. Notably, on top-tier chips, our 8-bit modelsrun faster than llama\.cpp modelsquantized to 4-bit, while remaining virtually identical to the original unquantized bf16 versions.

Results

991411842262693110.0010.0030.010.030.10KL (log scale)Tokens / SecondMiraiUnsloth / Llama.cppMLXGoodIQ4_NLIQ4_XSQ3_K_MQ3_K_SQ4_1Q4_K_MQ4_K_SQ5_K_MQ5_K_SQ6_KUD_IQ3_XXSUD_Q3_K_XLUD_Q4_K_XLUD_Q5_K_XLUD_Q6_K_XLUD_Q8_K_XLMLX 4bitMLX 8bitMirai • MMirai • L Our M checkpointshave the size and accuracy comparable to “Q4_K_S” Unsloth/Llama.cpp versions, while outperforming all llama.cpp checkpoints by 30-50% in terms of decoding speed across all devices.

Our L checkpointsare statistically indistinguishable from the full-precision versions, yet they achieve decoding speed comparable to 4- or 5-bit llama.cpp versions, and even outperform 4-bit llama.cpp checkpoints on high-end chips

Format

We designed our quantization format with the goal of maximising decode performance, and we deliberately traded off rate/distortion profile for hardware friendliness. Despite those tradeoffs, our quantization still remains highly competitive in terms of accuracy for a given compression rate.

We chose a conservative4\-bit asymmetric integer quantizationformat with 4-bit zero points and bf16 scales for our 4-bit checkpoints, and8\-bit symmetric quantizationfor the 8-bit versions. We use block-diagonal Random Hadamard Transforms with a block size of 32 as pre- and post-processing steps to suppress outliers.

The RHT block size was chosen to match the GPU warp width. This allows pre- and post-processing to be implemented using warp-level shuffle operations and fused into the prologues and epilogues of RMSNorm and GEMM/GEMV kernels, incurring zero additional memory traffic and only a minimal performance penalty.

The integer quantization format, while suboptimal from a rate-distortion perspective, can be implemented extremely efficiently on the GPU. When combined with dynamic 8-bit activation quantization, it can also use the hardware-accelerated int8 GEMM path for high-performance speculative decoding on Apple M5+ chips.

We experimented with advanced vector-quantizer designs and found that, while they can easily achieve much better rate-distortion characteristics, their GPU implementations suffer from shared-memory congestion and bank conflicts when reading from lookup tables. This, along with the desire to use the int8 GEMM path on neural accelerators, convinced us to stick with integer quantization.

For similar reasons,we avoid “intermediate” 3\-, 5\-, and 6\-bit formats,despite the potential for considerable quality gains from fine-grained per-layer precision allocation.

Algorithm

We use a two-stage approach:Post-Training Quantizationfollowed by a lightweightQuantization-Aware Distillationstep for accuracy recovery.

For Post-Training Quantization, we use a custom JAX implementation of the YAQA second-order quantization algorithm, with minor modifications that improve numerical stability. We use an internal calibration dataset to estimate the Fisher Information Matrix.

We follow the PTQ stepwith a small quantization-aware distillation run on a dataset of rollouts generated by the unquantized teacher model. We use Muon combined with a deterministic straight-through estimator to optimize the quantized weights, and AdamW to optimize the group scales and zero points.

We found the quality of the distillation step to be highly sensitive to the quality and domain coverage of the dataset, and we are working on improving and expanding it. We expect significant further quality improvements as we continue to iterate on our QAD recipe.

Lalamo, our custom model-optimization framework, provides highly efficient abstractions that allow us to easily experiment with novel quantization formats and optimization algorithms, and to apply them to any LLM architecture out of the box.

Evaluation

The two metrics we focused on areMMLU-Pro scorewith thinking enabled and an 81,920-token reasoning budget, and themean Kullback-Leibler divergencebetween the logits of the unquantized model and the quantized model.

We computed KL scores on a set of 100 randomly sampled questions sampled from the Lmsys-Chat-1M dataset. Answers to those questions were generated with the unquantized model. We computed the mean KL divergence between teacher and student logits over the generated answers. The Lmsys-Chat-1M dataset was not used during either PTQ or QAD stages.

Notably, the scores for most 8-bit quantizations from any provider generally fall below the noise floor of these evaluations, indicating that these evals do not provide a way to distinguish the quality of different 8-bit quantizations. We estimated the noise floor as the mean KL divergence between a llama.cpp full-precision model and our full-precision checkpoints. In this case, the only differences are due to numerical precision and reduction-order, yielding approximately8 × 10⁻⁴mean KL.

The issue with KL evaluation is that, while it is extremely precise, it is plausible for a good quantization to have a large KL. It also does not detect long-run performance degradation. Therefore, we also use MMLU-Pro evaluations, ran with lm-evaluation-harness byEleutherAI, as an additional anchor.

We used a row-wise binomial parametric bootstrap to estimate a lower bound on the standard deviation of a single MMLU-Pro score. This lower bound is around 0.3 percentage points, making any differences below 1 percentage point statistically insignificant. This also explains how Unsloth’s 6-bit quantization can outperform its 8-bit quantization on MMLU-Pro.

Our 4-bit Qwen3.5-0.8B and Qwen3.5-2B checkpoints are comparable to Q4_K_M checkpoints in quality,**while being about 20% smaller.**The Qwen3.5-4B checkpoints are between Q3_K_M and Q4_0 in both size and quality. Our 8-bit checkpoints are indistinguishable in quality from MLX 8-bit and Unsloth UD-Q8_K_XL.

Try our quantized models

Model

Size

Quantization

Runs, M4 Pro

Resident Memory

Qwen3.5-4B-L

4B

Mirai Large

~ 48 tok/s

Under 4.75 GB

Qwen3.5-4B-M

4B

Mirai Medium

~ 84 tok/s

Under 2.71 GB

Qwen3.5-2B-L

2B

Mirai Large

~ 108 tok/s

Under 2.22 GB

Qwen3.5-2B-M

2B

Mirai Medium

~ 171 tok/s

Under 1.32 GB

Qwen3.5-0.8B-L

0.8B

Mirai Large

~ 222 tok/s

Under 1.01 GB

Qwen3.5-0.8B-M

0.8B

Mirai Medium

~ 314 tok/s

Under 0.65 GB

Future Work

• **In the coming weeks,**we will release quantized checkpoints for larger Qwen3.5/Qwen3.6 models, including the 27B and 35B-A3B variants, as well as the Gemma 4 family of models.
• **Next,**we will release our own take on diffusion-based speculative decoding, which will unlock the full potential of our inference architecture. Additionally, we are experimenting with novel approaches to 2- and 3-bit quantization, which would allow us to efficiently run 300B-parameter MoE models on high-end consumer hardware.

By Artur Chakhvadze, Ryan Mathieu, Roman Knyazhitskiy, Nikolai Voinilenko, Chen-Chen Yeh, Artur Mullakhmetov, Eugene Bokhan, in collaboration with others at Mirai Labs.

Jun 4, 2026

We are excited to release our first quantized checkpoints for the Qwen3.5 series of models. Our co-designed quantization and inference-engine stack dramatically outperforms Unsloth/llama.cpp and MLX on the precision-vs-generation-speed tradeoff,delivering 40–60% more tokens per secondat the same quality level across Apple devices ranging from smartphones to high-end desktops.

This is the first of many upcoming releases from our Model Optimization project, and a small part of a larger engineering effort to bring capable fully local agents running at over 1,000 tokens per second to consumer hardware. Many of the choices we made here are strategic: the quantization format was carefully chosen to maximize hardware utilization when used together with our future speculative decoding pipeline.

We are starting with4\-bit \(Mirai\-M\)and8\-bit \(Mirai\-L\)versions of Qwen3.5-0.8B, Qwen3.5-2B, and Qwen3.5-4B. Larger Qwen3.5/Qwen3.6 models, as well as the Gemma 4 family, are coming soon. Notably, on top-tier chips, our 8-bit modelsrun faster than llama\.cpp modelsquantized to 4-bit, while remaining virtually identical to the original unquantized bf16 versions.

Results

991411842262693110.0010.0030.010.030.10KL (log scale)Tokens / SecondMiraiUnsloth / Llama.cppMLXGoodIQ4_NLIQ4_XSQ3_K_MQ3_K_SQ4_1Q4_K_MQ4_K_SQ5_K_MQ5_K_SQ6_KUD_IQ3_XXSUD_Q3_K_XLUD_Q4_K_XLUD_Q5_K_XLUD_Q6_K_XLUD_Q8_K_XLMLX 4bitMLX 8bitMirai • MMirai • L Our M checkpointshave the size and accuracy comparable to “Q4_K_S” Unsloth/Llama.cpp versions, while outperforming all llama.cpp checkpoints by 30-50% in terms of decoding speed across all devices.

Our L checkpointsare statistically indistinguishable from the full-precision versions, yet they achieve decoding speed comparable to 4- or 5-bit llama.cpp versions, and even outperform 4-bit llama.cpp checkpoints on high-end chips

Format

We designed our quantization format with the goal of maximising decode performance, and we deliberately traded off rate/distortion profile for hardware friendliness. Despite those tradeoffs, our quantization still remains highly competitive in terms of accuracy for a given compression rate.

We chose a conservative4\-bit asymmetric integer quantizationformat with 4-bit zero points and bf16 scales for our 4-bit checkpoints, and8\-bit symmetric quantizationfor the 8-bit versions. We use block-diagonal Random Hadamard Transforms with a block size of 32 as pre- and post-processing steps to suppress outliers.

The RHT block size was chosen to match the GPU warp width. This allows pre- and post-processing to be implemented using warp-level shuffle operations and fused into the prologues and epilogues of RMSNorm and GEMM/GEMV kernels, incurring zero additional memory traffic and only a minimal performance penalty.

The integer quantization format, while suboptimal from a rate-distortion perspective, can be implemented extremely efficiently on the GPU. When combined with dynamic 8-bit activation quantization, it can also use the hardware-accelerated int8 GEMM path for high-performance speculative decoding on Apple M5+ chips.

We experimented with advanced vector-quantizer designs and found that, while they can easily achieve much better rate-distortion characteristics, their GPU implementations suffer from shared-memory congestion and bank conflicts when reading from lookup tables. This, along with the desire to use the int8 GEMM path on neural accelerators, convinced us to stick with integer quantization.

For similar reasons,we avoid “intermediate” 3\-, 5\-, and 6\-bit formats,despite the potential for considerable quality gains from fine-grained per-layer precision allocation.

Algorithm

We use a two-stage approach:Post-Training Quantizationfollowed by a lightweightQuantization-Aware Distillationstep for accuracy recovery.

For Post-Training Quantization, we use a custom JAX implementation of the YAQA second-order quantization algorithm, with minor modifications that improve numerical stability. We use an internal calibration dataset to estimate the Fisher Information Matrix.

We follow the PTQ stepwith a small quantization-aware distillation run on a dataset of rollouts generated by the unquantized teacher model. We use Muon combined with a deterministic straight-through estimator to optimize the quantized weights, and AdamW to optimize the group scales and zero points.

We found the quality of the distillation step to be highly sensitive to the quality and domain coverage of the dataset, and we are working on improving and expanding it. We expect significant further quality improvements as we continue to iterate on our QAD recipe.

Lalamo, our custom model-optimization framework, provides highly efficient abstractions that allow us to easily experiment with novel quantization formats and optimization algorithms, and to apply them to any LLM architecture out of the box.

Evaluation

The two metrics we focused on areMMLU-Pro scorewith thinking enabled and an 81,920-token reasoning budget, and themean Kullback-Leibler divergencebetween the logits of the unquantized model and the quantized model.

We computed KL scores on a set of 100 randomly sampled questions sampled from the Lmsys-Chat-1M dataset. Answers to those questions were generated with the unquantized model. We computed the mean KL divergence between teacher and student logits over the generated answers. The Lmsys-Chat-1M dataset was not used during either PTQ or QAD stages.

Notably, the scores for most 8-bit quantizations from any provider generally fall below the noise floor of these evaluations, indicating that these evals do not provide a way to distinguish the quality of different 8-bit quantizations. We estimated the noise floor as the mean KL divergence between a llama.cpp full-precision model and our full-precision checkpoints. In this case, the only differences are due to numerical precision and reduction-order, yielding approximately8 × 10⁻⁴mean KL.

The issue with KL evaluation is that, while it is extremely precise, it is plausible for a good quantization to have a large KL. It also does not detect long-run performance degradation. Therefore, we also use MMLU-Pro evaluations, ran with lm-evaluation-harness byEleutherAI, as an additional anchor.

We used a row-wise binomial parametric bootstrap to estimate a lower bound on the standard deviation of a single MMLU-Pro score. This lower bound is around 0.3 percentage points, making any differences below 1 percentage point statistically insignificant. This also explains how Unsloth’s 6-bit quantization can outperform its 8-bit quantization on MMLU-Pro.

Our 4-bit Qwen3.5-0.8B and Qwen3.5-2B checkpoints are comparable to Q4_K_M checkpoints in quality,**while being about 20% smaller.**The Qwen3.5-4B checkpoints are between Q3_K_M and Q4_0 in both size and quality. Our 8-bit checkpoints are indistinguishable in quality from MLX 8-bit and Unsloth UD-Q8_K_XL.

Try our quantized models

Model

Size

Quantization

Runs, M4 Pro

Resident Memory

Qwen3.5-4B-L

4B

Mirai Large

~ 48 tok/s

Under 4.75 GB

Qwen3.5-4B-M

4B

Mirai Medium

~ 84 tok/s

Under 2.71 GB

Qwen3.5-2B-L

2B

Mirai Large

~ 108 tok/s

Under 2.22 GB

Qwen3.5-2B-M

2B

Mirai Medium

~ 171 tok/s

Under 1.32 GB

Qwen3.5-0.8B-L

0.8B

Mirai Large

~ 222 tok/s

Under 1.01 GB

Qwen3.5-0.8B-M

0.8B

Mirai Medium

~ 314 tok/s

Under 0.65 GB

Future Work

• **In the coming weeks,**we will release quantized checkpoints for larger Qwen3.5/Qwen3.6 models, including the 27B and 35B-A3B variants, as well as the Gemma 4 family of models.
• **Next,**we will release our own take on diffusion-based speculative decoding, which will unlock the full potential of our inference architecture. Additionally, we are experimenting with novel approaches to 2- and 3-bit quantization, which would allow us to efficiently run 300B-parameter MoE models on high-end consumer hardware.