推测解码领域的重大突破！在我的 MacBook 上，一个 9B LLM 的推理速度提升了 4 倍，比常规推测解码还快 2.6 倍。我在 M5 MacBook Pro 上用相同的 prompt 运行 Qwen3.5 9B，分别测试了三种方式：

• 不使用推测解码：耗时 42 秒
• 常规推测解码：耗时 30 秒
• DFlash + Weaver：耗时 11 秒

下面的视频展示了三种方式的并排对比。

其原理如下：

推测解码让一个轻量的草稿模型预先猜测接下来的几个 token，然后由主模型一次性批量验证。最快的草稿模型（如 DFlash）会并行猜测每个位置的 token。但每个位置的猜测是独立进行的——位置 5 看不到位置 4 选择了什么。一旦某个位置出错，之后的所有猜测都会被丢弃。

Weaver 解决了这个问题。它是一个叠加在 DFlash 之上的极小适配器（仅一层 Transformer），功能如下：

↳ 从左到右依次从 DFlash 的候选中进行挑选，使得每次挑选都依赖上一次的结果 ↳ 构建一棵可能的续写树，而不是单一链式结构 ↳ 让主模型一次通过验证整棵树

在我的 Mac 上，模型每一轮推理能产出 7.5 个 token，而不是 1 个。

论文报告称，在 B200 GPU 上，聊天、数学和代码任务的解码速度均提升 4 倍。

@TryMirai 将整套方案开源在他们的引擎 uzu 中，该引擎专为 Apple Silicon 打造。

GitHub 仓库：https://github.com/trymirai/uzu（别忘了点个 star 🌟）

只需两条命令即可体验：

brew install mirai
mirai --model trymirai/Qwen3.5-9B-M -m "Write a python function to merge two sorted lists"

我还写了一篇完整的文章，解释为什么本地推理受内存带宽限制，以及 uzu 如何突破这一瓶颈，附带可复现的代码。文章全文如下。

回复中还附有一张图，对比了标准解码、推测解码和 uzu 解码的区别。

trymirai/uzu

Source: https://github.com/trymirai/uzu

License · Build (https://github.com/trymirai/uzu/actions) · Python Package (https://pypi.org/project/uzu/) · Python (https://pypi.org/project/uzu/) · TypeScript Package (https://www.npmjs.com/package/@trymirai/uzu) · Downloads (https://www.npmjs.com/package/@trymirai/uzu) · Swift SPM · Platforms · Swift (https://swift.org)

uzu

一个面向 AI 模型的高性能推理引擎。它支持将 AI 直接部署在你的应用中，实现零延迟、完整数据隐私和零推理成本。

核心特性：

• 简洁的高层 API
• 统一的模型配置，方便添加对新模型的支持
• 可追踪的计算过程，确保与参考实现一致
• 充分利用 Apple 设备的统一内存架构
• 广泛的模型支持 (https://trymirai.com/local-models)

快速开始

Rust

添加依赖：

[dependencies]
uzu = { git = "https://github.com/trymirai/uzu", branch = "main", package = "uzu" }

运行以下代码：

use std::io::{self, Write};
use uzu::{
    engine::{Engine, EngineConfig},
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;
    let model = engine.model("alibaba:qwen3.5:0.8b:mirai:mirai-m:4".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();
    let session = engine.chat(model, ChatConfig::default()).await?;
    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let replies = session.reply(messages, ChatReplyConfig::default()).await?;
    if let Some(reply) = replies.last() {
        println!("Reasoning: {}", reply.message.reasoning().unwrap_or_default());
        println!("Text: {}", reply.message.text().unwrap_or_default());
    }
    Ok(())
}

Python

添加依赖：

uv add uzu==0.6.14

运行以下代码：

import asyncio
from uzu import ChatConfig, ChatMessage, ChatReplyConfig, Engine, EngineConfig

async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)
    model = await engine.model("alibaba:qwen3.5:0.8b:mirai:mirai-m:4")
    if model is None:
        return
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()
    session = await engine.chat(model, ChatConfig.create())
    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    replies = await session.reply(messages, ChatReplyConfig.create())
    if not replies:
        return
    message = replies[-1].message
    print(f"Reasoning: {message.reasoning}")
    print(f"Text: {message.text}")

if __name__ == "__main__":
    asyncio.run(main())

Swift

添加依赖：

dependencies: [
    .package(url: "https://github.com/trymirai/uzu.git", from: "0.6.14")
]

运行以下代码：

import Foundation
import Uzu

public func runQuickStart() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    guard let model = try await engine.model(identifier: "alibaba:qwen3.5:0.8b:mirai:mirai-m:4") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    let session = try await engine.chat(model: model, config: .create())
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let reply = try await session.reply(input: messages, config: .create())
    guard let message = reply.last?.message else {
        return
    }
    print("Reasoning: \(message.reasoning() ?? "empty")")
    print("Text: \(message.text() ?? "empty")")
}

TypeScript

添加依赖：

pnpm add @trymirai/[email protected]

运行以下代码：

import { ChatConfig, ChatMessage, ChatReplyConfig, Engine, EngineConfig } from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);
    let model = await engine.model('alibaba:qwen3.5:0.8b:mirai:mirai-m:4');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();
    let session = await engine.chat(model, ChatConfig.create());
    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let reply = await session.reply(messages, ChatReplyConfig.create());
    let message = reply[0]?.message;
    if (message) {
        console.log('Reasoning: ', message.reasoning);
        console.log('Text: ', message.text);
    }
}

main().catch((error) => {
    console.error(error);
});

从模型下载到推理配置，一切均自动处理。详细信息请参阅文档，了解如何自定义每个步骤。

示例

你可以通过以下命令运行任意示例：

cargo tools example <**rust** | **python** | **swift** | **typescript**> <**chat** | **chat-cloud** | **chat-shared-instance** | **chat-structured-output** | **quick-start** | **tool-calls**>

Chat

本示例将下载一个模型并对指定的消息列表获取回复：

Rust

use std::io::{self, Write};
use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;
    let model = engine.model("alibaba:qwen3.5:0.8b:mirai:mirai-m:4".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();
    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::ToolResults { .. } => {},
            ChatSessionStreamChunk::Replies { replies } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error { error } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }
    Ok(())
}

Python

import asyncio
from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)

async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)
    model = await engine.model("alibaba:qwen3.5:0.8b:mirai:mirai-m:4")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()
    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")

if __name__ == "__main__":
    asyncio.run(main())

Swift

import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    guard let model = try await engine.model(identifier: "alibaba:qwen3.5:0.8b:mirai:mirai-m:4") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        case .toolResults:
            break
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

TypeScript

import { ChatConfig, ChatMessage, ChatReplyConfig, ChatSessionStreamChunkError, ChatSessionStreamChunkReplies, Engine, EngineConfig } from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);
    let model = await engine.model('alibaba:qwen3.5:0.8b:mirai:mirai-m:4');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();
    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

加载完成后，同一个 ChatSession 可以在释放前被多次复用。每个模型可能占用大量内存，因此同时只加载一个会话非常重要。

对于 iOS 应用，建议添加 Increased Memory Capability (https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.developer.kernel.increased-memory-limit) 权限，以确保应用能够分配所需内存。

Chat with the cloud model

本示例将对指定的消息列表从云端模型获取回复：

Rust

use uzu::{
    engine::{Engine, EngineConfig},
    types::{
        basic::ReasoningEffort,
        session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
    },
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default().with_openai_api_key("OPENAI_API_KEY".to_string());
    let engine = Engine::new(engine_config).await?;
    let model = engine.model("gpt-5".to_string()).await?.ok_or("Model not found")?;
    let messages = vec![
        ChatMessage::system().with_reasoning_effort(ReasoningEffort::Low),
        ChatMessage::user().with_text("How LLMs work".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let replies = session.reply(messages, ChatReplyConfig::default()).await?;
    if let Some(reply) = replies.first() {
        println!("Reasoning: {}", reply.message.reasoning().unwrap_or_default());
        println!("Text: {}", reply.message.text().unwrap_or_default());
    }
    Ok(())
}

Python

import asyncio
from uzu import ChatConfig, ChatMessage, ChatReplyConfig, Engine, EngineConfig, ReasoningEffort

async def main() -> None:
    engine_config = EngineConfig.create().with_openai_api_key("OPENAI_API_KEY")
    engine = await Engine.create(engine_config)
    model = await engine.model("gpt-5")
    if model is None:
        raise RuntimeError("Model not found")
    messages = [
        ChatMessage.system().with_reasoning_effort(ReasoningEffort.Low),
        ChatMessage.user().with_text("How LLMs work"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    replies = await session.reply(messages, ChatReplyConfig.create())
    if replies:
        message = replies[0].message
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")

if __name__ == "__main__":
    asyncio.run(main())

Swift

import Uzu

public func runChatCloud() async throws {
    let engineConfig = EngineConfig.create().withOpenaiApiKey(openaiApiKey: "OPENAI_API_KEY")
    let engine = try await Engine.create(config: engineConfig)
    guard let model = try await engine.model(identifier: "gpt-5") else {
        return
    }
    let messages = [
        ChatMessage.system().withReasoningEffort(reasoningEffort: .low),
        ChatMessage.user().withText(text: "How LLMs work")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let reply = try await session.reply(input: messages, config: .create())
    // ...
}