用苹果电脑跑本地大模型的朋友，有个工具值得盯上——Rapid-MLX。它在 M 系列芯片上的推理速度比 Ollama 快 2 到 4 倍，因为它是直接基于苹果的 MLX 框架开发的，对芯片架构的压榨更彻底。

几个关键点：

KV 缓存裁剪加 DeltaNet 状态快照，多轮对话的首 token 延迟能做到 0.08 秒左右，体感上基本没有等待 工具调用支持 17 种解析器，能自动识别 Qwen、DeepSeek、Gemma、GLM 等模型的输出格式，量化后的内容如果出岔子还能自动修复 兼容 OpenAI API 规范，Cursor、Claude Code、Aider、LangChain 都可以直接接入，基本不用改代码 此外还支持推理链分离、云端路由、视觉音频多模态、V 缓存压缩等功能。 如果你有 M 系列 Mac，又觉得 Ollama 跑起来不够利索，Rapid-MLX 值得试一把。

https://github.com/raullenchai/Rapid-MLX…

raullenchai/Rapid-MLX

Source: https://github.com/raullenchai/Rapid-MLX

Rapid-MLX

Run AI on your Mac. Faster than anything else.

Run local AI models on your Mac — no cloud, no API costs. Works with Cursor, Claude Code, and any OpenAI-compatible app.

_{rapidmlx.com · Desktop app · Community benchmarks · Model mirror}

pip install → serve Gemma 4 26B → chat + tool calling → works with PydanticAI, LangChain, Aider, and more.

_{Single-user end-to-end throughput (B=1: one request at a time, 256 max output tokens, output_tokens / wall-clock incl. first-token latency), median of 3 rounds. chat_template_kwargs.enable_thinking=False passed where the engine honours it. Tested on M3 Ultra 256 GB / rapid-mlx v0.6.83 (fused top-p sampler). ¹ carried over from 2026-04 bench — disk-constrained on this refresh.}

Quick Start

Step 1 — Install (pick one):

# uv (recommended — one command, isolated env, auto-manages Python)
uv tool install rapid-mlx@latest
# Don't have uv yet? Install it first: curl -LsSf https://astral.sh/uv/install.sh | sh

# Or one-liner with auto-setup (installs Python if needed)
curl -fsSL https://raullenchai.github.io/Rapid-MLX/install.sh | bash

# Homebrew (Mac-native — needs tap + trust before install on Homebrew 4.x)
brew tap raullenchai/rapid-mlx
brew trust raullenchai/rapid-mlx
brew install rapid-mlx

# pip (requires Python 3.10+ — macOS ships 3.9, so install Python first if needed)
pip install rapid-mlx

Upgrade later: uv tool upgrade rapid-mlx / brew upgrade rapid-mlx / pip install -U rapid-mlx.

Vision/multimodal models (Gemma 4, Qwen-VL, etc.) need extras: pip install 'rapid-mlx[vision]'. Text-only install is ~460 MB; vision adds ~322 MB. See Optional Extras for the full list.

“No matching distribution” error? Your Python is too old. Run python3 --version — if it says 3.9, install a newer Python: brew install [email protected] then python3.12 -m pip install rapid-mlx

Refusing to load formula ... from untrusted tap? Homebrew 4.x requires third-party taps to be explicitly trusted before install. The brew trust raullenchai/rapid-mlx line above is what marks the tap as trusted — without it, even after brew tap, the install is refused. Trust is per-machine and persists across upgrades.

Tapping homebrew/core / Operation not permitted during brew install? Brew 5.x’s install sandbox can’t auto-tap homebrew/core mid-install. Pre-tap it once, then retry:

brew tap homebrew/core --force   # ~1.3 GB, one-time
brew tap raullenchai/rapid-mlx
brew trust raullenchai/rapid-mlx
brew install rapid-mlx

Step 2 — Talk to a model right now (one command, no second terminal):

rapid-mlx chat

Defaults to qwen3.5-4b-4bit. First run downloads the model (~2.5 GB) — you’ll see a progress bar. Drops you into a REPL when it’s ready. Type /help for slash commands, /exit to quit. Pass --think to surface chain-of-thought.

Step 2b — Or serve a model for use from other apps:

rapid-mlx serve qwen3.5-4b-4bit

Same model, same download — but this starts an OpenAI-compatible HTTP server instead of a REPL. Wait for Ready: http://localhost:8000/v1.

Want vision? pip install 'rapid-mlx[vision]' then rapid-mlx serve gemma-4-26b-4bit (~14 GB).

Step 3 — Hit the API (from a second terminal tab):

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"default","messages":[{"role":"user","content":"Say hello"}]}'

That’s it — you now have an OpenAI-compatible AI server on localhost:8000. Point any app at http://localhost:8000/v1 and it just works.

Step 4 — Share it publicly (optional — get a https:// URL anyone can hit):

rapid-mlx share qwen3.6-27b-8bit

This spawns the same local serve and tunnels it through rapidserver.quicksilverpro.io over a WebSocket. Your terminal prints a public OpenAI-compatible endpoint plus a bearer key — point any chat UI or OpenAI SDK at it. Bearer auth, a locked-down CORS allowlist, and a default 120 RPM rate-limit are wired on the spawned child; closing the terminal tears the tunnel down.

The default chat surface is our hosted Big-AGI fork (tool calling, personas, voice — no signup); any OpenAI-compatible client also works, e.g. OPENAI_API_BASE_URL=<share-url>/v1 OPENAI_API_KEY=<bearer> open-webui serve.

Pick a 27B-class model or larger for a usable share experience — 4B is fine for local dev but too small for live chat (rapid-mlx models lists all aliases).

Want a Claude Code-like TUI? Rapid-MLX is the backend — pair it with an open-source agent CLI like OpenCode or codex for the full slash-commands / tool-use / multi-turn experience. Run rapid-mlx agents opencode --setup (or codex --setup) to wire it up automatically.

Tip: Run rapid-mlx models to see all available model aliases. For a smaller/faster model, try rapid-mlx serve qwen3.5-9b-4bit (~5 GB).

git clone https://github.com/raullenchai/Rapid-MLX.git
cd Rapid-MLX && pip install -e .

pip install 'rapid-mlx[vision]'

pip install 'rapid-mlx[audio]'

Not into the terminal? Rapid-MLX Desktop is a Mac app that bundles the same rapid-mlx engine inside a one-click GUI — drag to Applications, pick a model, chat. No Python, no pip, no brew. The CLI here is still the source of truth for serving and scripting; the desktop app is the friendlier on-ramp.

Try it with Python (make sure the server is running, then pip install openai):

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")  # any value works, no real key needed

response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Say hello"}],
)
print(response.choices[0].message.content)

Works With

Agent Harnesses (MHI-tested)

UI / IDE Clients

Claude Code

Claude Code (Anthropic’s web-based code editor) works with Rapid-MLX via the Anthropic Messages API endpoint (/v1/messages).

Terminal 1 — Start the Rapid-MLX server:

rapid-mlx serve qwen3.5-9b-4bit

Wait for: Ready: http://localhost:8000/v1

Terminal 2 — Launch Claude Code pointing to your local server:

export ANTHROPIC_BASE_URL="http://localhost:8000"
export ANTHROPIC_API_KEY="not-needed"
claude --model claude-opus-4-5

The server accepts any claude-* or gpt-* model name in requests and routes them to the loaded engine (configured on the server). The response always reflects the actual loaded model, not the client-requested name. This means:

• Claude Code can use --model claude-opus-4-5 or any other alias
• The server runs whatever model you specified with rapid-mlx serve <model>
• Tool calling and streaming work out of the box with Qwen3.5 / Qwen3.6 models

Tip: For the best Claude Code experience on a 24 GB MacBook Pro, use qwen3.5-9b-4bit — it’s smart enough for coding tasks while staying responsive.

Model-Harness Index (MHI)

MHI measures how well a model works with a specific agent harness. It combines three dimensions:

MHI = 0.50 × ToolCalling + 0.30 × HumanEval + 0.20 × MMLU (scale 0-100)

Quick setup for popular apps:

Cursor: Settings → Models → Add Model:

OpenAI API Base:  http://localhost:8000/v1
API Key:          not-needed
Model name:       default          (or qwen3.5-9b-4bit — either works)

Cursor’s agent/composer mode uses tool calls automatically — Rapid-MLX handles them natively with Qwen3.5 models, no extra flags needed.

Claw Code:

export OPENAI_BASE_URL=http://localhost:8000/v1
export OPENAI_API_KEY=not-needed
claw --model "openai/default" prompt "summarize this repo"

OpenClaude:

CLAUDE_CODE_USE_OPENAI=1 OPENAI_BASE_URL=http://localhost:8000/v1 \
OPENAI_API_KEY=not-needed OPENAI_MODEL=default openclaude -p "hello"

Hermes Agent (~/.hermes/config.yaml):

model:
  provider: "custom"
  default: "default"
  base_url: "http://localhost:8000/v1"
  context_length: 32768

Goose:

GOOSE_PROVIDER=ollama OLLAMA_HOST=http://localhost:8000 \
GOOSE_MODEL=default goose run --text "hello"

Claude Code:

OPENAI_BASE_URL=http://localhost:8000/v1 claude

models:
  - name: rapid-mlx
    provider: openai
    model: default
    apiBase: http://localhost:8000/v1
    apiKey: not-needed

aider --openai-api-base http://localhost:8000/v1 --openai-api-key not-needed

[profiles.rapidmlx]
provider = "generic"
base_url = "http://127.0.0.1:8000"
model = "default"

swival --profile rapidmlx "summarize this repo"

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e ENABLE_OLLAMA_API=False \
  -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \
  -e OPENAI_API_KEY=not-needed \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

{
  "provider": {
    "openai": {
      "api": "http://localhost:8000/v1",
      "models": {
        "default": {
          "name": "rapid-mlx local",
          "limit": { "context": 32768, "output": 8192 }
        }
      },
      "options": { "apiKey": "not-needed" }
    }
  }
}

model = "default"
model_provider = "rapid-mlx"

[model_providers.rapid-mlx]
name = "Rapid-MLX (local)"
base_url = "http://localhost:8000/v1"
# If rapid-mlx was started with --api-key, add env_key = "RAPID_MLX_API_KEY"
# and `export RAPID_MLX_API_KEY=...`. Don't use api_key = "..." — Codex
# CLI's --strict-config rejects inline literals.

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIChatModel(
    model_name="default",
    provider=OpenAIProvider(
        base_url="http://localhost:8000/v1",
        api_key="not-needed",
    ),
)
agent = Agent(model)
print(agent.run_sync("What is 2+2?").output)

from smolagents import CodeAgent, OpenAIServerModel

model = OpenAIServerModel(
    model_id="default",
    api_base="http://localhost:8000/v1",
    api_key="not-needed",
)
agent = CodeAgent(tools=[], model=model)
agent.run("What is 5 multiplied by 7?")

- name: "Rapid-MLX"
  apiKey: "rapid-mlx"
  baseURL: "http://localhost:8000/v1/"
  models:
    default: ["default"]
    fetch: true
  titleConvo: true
  titleModel: "current_model"
  modelDisplayLabel: "Rapid-MLX"

from anthropic import Anthropic
client = Anthropic(base_url="http://localhost:8000", api_key="not-needed")

message = client.messages.create(
    model="default",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Say hello"}],
)
print(message.content[0].text)

Choose Your Model

What fits my Mac?

The model has to fit in your Mac’s RAM. If your Mac slows down or Activity Monitor shows red memory pressure, pick a smaller model from the table below.

Browse the full catalog at models.rapidmlx.com — 80+ MLX-quantised models on a free R2 mirror with resumable downloads, no HuggingFace rate limits. rapid-mlx pull <alias> fetches from there automatically.

_{Speed = single-user end-to-end throughput (B=1: one request, 256 max output tokens, output_tokens / wall-clock including first-token latency), median of 3 rounds. rapid-mlx v0.6.83 (fused top-p sampler) on M3 Ultra 256 GB, 2026-06-09. ¹ Carried over from prior bench (disk-constrained on this refresh).}

4bit vs 8bit: 4bit models are compressed to use less memory (recommended for most users). 8bit models are higher quality but need more RAM. “mxfp4” is a high-quality 4bit format.

Naming convention

Every alias follows the same template so you can read off the model family, parameter count, training technique, and quantization at a glance:

<family>-<version>-<params>-<modality?>-<technique?>-<quant>

The quantization suffix is mandatory on every alias — qwen3.5-4b-4bit not qwen3.5-4b, gemma-4-12b-qat-8bit not gemma-4-12b-qat. This mirrors LM Studio’s …-MLX-4bit / …-MLX-8bit HuggingFace convention so you never have to guess the bit width.

-qat is a technique suffix, not a quant — it stacks before the quant. So a QAT-trained Gemma 4 12B in 4-bit is gemma-4-12b-qat-4bit, and the 8-bit variant is gemma-4-12b-qat-8bit.

Decoded examples:

• gemma-4-12b-qat-4bit = Gemma 4 · 12B params · QAT-trained · 4-bit quant
• qwen3.5-35b-8bit = Qwen 3.5 · 35B params (3B active MoE) · 8-bit quant
• gpt-oss-20b-mxfp4-q8 = GPT-OSS · 20B params · MXFP4 weights + Q8 head
• bonsai-1.7b-unpacked = Bonsai · 1.7B params · no quantization

Full model lineup

72 explicit aliases across 13 families ship today. Run rapid-mlx models for the live list with parser, hybrid / MoE flags, and DFlash eligibility.

Copy-paste commands

Pick the one that matches your Mac. Run rapid-mlx models to see all available aliases.

# 16 GB — lightweight, fast
rapid-mlx serve qwen3.5-4b-4bit --port 8000

# 24 GB — best small model
rapid-mlx serve qwen3.5-9b-4bit --port 8000

# 32 GB — solid coding model
rapid-mlx serve qwen3.5-27b-4bit --port 8000

# 32 GB — Gemma 4 12B (vision-capable, 64 tok/s)
rapid-mlx serve gemma-4-12b-4bit --port 8000

# 32 GB — GPT-OSS 20B (harmony-native, 100% tool calling, 119 tok/s)
rapid-mlx serve gpt-oss-20b-mxfp4-q8 --port 8000

# 32+ GB — Qwen 3.6 35B-A3B (256 experts, 262K context, 93 tok/s)
rapid-mlx serve qwen3.6-35b-4bit --port 8000

# 48+ GB — sweet spot (Qwen3.5-35B-A3B 8bit, 80 tok/s)
rapid-mlx serve qwen3.5-35b-8bit --prefill-step-size 8192 --port 8000  # faster first response

# 96+ GB — frontier (Qwen3.5-122B mxfp4)
rapid-mlx serve qwen3.5-122b-mxfp4 --prefill-step-size 8192 --port 8000

# Coding agent — fast MoE, great for Claude Code / Cursor
rapid-mlx serve qwen3-coder-4bit --prefill-step-size 8192 --port 8000  # MoE = only uses part of the model, so it's fast

# Vision — image understanding (see note below)
rapid-mlx serve qwen3-vl-4b-4bit --mllm --port 8000

# 🆕 Text-diffusion — DiffusionGemma 26B-A4B (block denoising, not autoregressive)
rapid-mlx serve diffusion-gemma-26b-4bit --port 8000  # needs [vision] extras for mlx-vlm 0.6.3+

Vision deps: Install into the same environment where rapid-mlx lives:

• install.sh users: ~/.rapid-mlx/bin/pip install 'rapid-mlx[vision]'
• pip users: pip install 'rapid-mlx[vision]' (in the same venv)
• brew users: $(brew --prefix)/opt/rapid-mlx/libexec/bin/pip install 'rapid-mlx[vision]'

🆕 Text-Diffusion (DiffusionGemma 26B-A4B)

DiffusionGemma is a non-autoregressive language model — instead of emitting one token at a time, it denoises whole blocks of tokens in parallel via a diffusion process. Rapid-MLX wraps it behind the standard OpenAI Chat Completions API, so any client (chat UIs, agent harnesses, your own scripts) talks to it the same way it talks to Qwen / Gemma / GPT-OSS.

pip install 'rapid-mlx[vision]'       # mlx-vlm 0.6.3+ provides the diffusion runtime
rapid-mlx serve diffusion-gemma-26b-4bit --port 8000

B=1 single-user benchmark (M3 Ultra 256 GB, mlx-community/diffusiongemma-26B-A4B-it-4bit, median of 3 runs + 1 warmup):

Diffusion models emit tokens in whole denoising blocks, so the conventional decode_tok/s = tokens / (e2e − ttft) metric isn’t meaningful here (ttft ≈ e2e for short outputs). The table reports aggregate throughput — tokens / total_wall_time — i.e. how many tokens actually land in the chat window per second. Throughput climbs with output length because the per-step denoising cost amortizes across more emitted tokens.

Reproduce the table: python3.12 scripts/bench_diffusion_gemma.py --port 8000.

Benchmarks

Tested on Mac Studio M3 Ultra (256 GB), 2026-06-06. Workload is B=4 sustained concurrent streaming (four parallel chat requests, 256 max output tokens each), median of 3 measured rounds after one warmup discard. Engines were swapped sequentially with an 8 s Metal cooldown so contention never crossed engine boundaries.

chat_template_kwargs.enable_thinking=False is passed to all engines that honour it (rapid-mlx, mlx-lm, mlx-vlm). Ollama 0.24 ignores that hook for Qwen3 and keeps streaming reasoning chunks — those decode at the same model rate as content tokens, so we count them, and the Qwen3 Ollama numbers reflect chain-of-thought-on throughput in practice. Token counts come from the streaming usage chunk (authoritative), not from counting SSE frames.

Versions: rapid-mlx v0.6.80, mlx-lm 0.31.3, Ollama 0.24.0 (latest stable).

Aggregate throughput = sum of output tokens across all four streams ÷ wall-clock seconds — the metric that matters for a server fronting multiple users or a TUI firing parallel sub-agents. Per-user decode is roughly aggregate ÷ 4 on a true batching engine; on Ollama 0.24 (no in-flight batching) the four streams effectively serialize, so the per-stream decode-only rate (output_tokens / (e2e − ttft), recorded as median_per_stream_tps in the raw JSON) sits at or slightly above the aggregate. That is expected and not a metric mismatch — decode-only excludes prefill while aggregate spans the entire wall-clock window. The Ollama daemon also caches the previously loaded model in memory across rows; OllamaEngine.stop() only unloads the row’s own tag, so cross-row Metal residency effects are possible — ollama ps between rows shows what’s actually resident.

✅ Direct apples-to-apples: identical weights both sides.

_{¹ Ollama Qwen3 base, not Qwen3.5 — DeltaNet hybrid arch isn’t on llama.cpp yet. ² Closest dense Qwen3; Unsloth Qwen3.6-27B GGUF fails to load on Ollama 0.24. ³ mlx-lm 0.31.3 has no Gemma 4 loader (it lives in mlx-vlm). ⁴ Gemma 4 not yet on llama.cpp — Gemma 3 is the closest. ⁵ Closest MoE A3B available; Qwen3.5/3.6-35B-A3B don’t have a llama.cpp build yet.}

Different Mac? Numbers above are one M3 Ultra. See community-submitted runs across M1/M2/M3/M4 Apple Silicon at rapidmlx.com/performance — sortable by chip × model × version. Submit your own with rapid-mlx bench <alias> --submit.

Full benchmark data with all models, TTFT tables, DeltaNet snapshots, and engine comparison below.

Reproduce the throughput table:

python3.12 scripts/bench_readme_refresh.py \
  --models qwen3.5-4b-4bit,qwen3.5-9b-4bit,qwen3.5-27b-4bit,gemma-4-12b-4bit,gpt-oss-20b-mxfp4-q8,qwen3.6-35b-4bit,qwen3.5-35b-8bit \
  --engines rapid-mlx,mlx-lm,ollama

Raw JSON per round + per-stream tok/s land in reports/benchmarks/readme-refresh/.

Features

Tool Calling

Full OpenAI-compatible tool calling with 17 parser formats and automatic recovery when quantized models break. Models at 4-bit degrade after multiple tool rounds — Rapid-MLX auto-detects broken output and converts it back to structured tool_calls.

Reasoning Separation

Models with chain-of-thought (Qwen3, DeepSeek-R1) output reasoning in a separate reasoning_content field — cleanly separated from content in streaming mode. Works with Qwen3, DeepSeek-R1, MiniMax, and GPT-OSS reasoning formats.

Prompt Cache

Persistent cache across requests — only new tokens are prefilled on each turn. For standard transformers, KV cache trimming. For hybrid models (Qwen3.5 DeltaNet), RNN state snapshots restore non-trimmable layers from memory instead of re-computing. 2-5x faster TTFT on all architectures. Always on, no flags needed.

Smart Cloud Routing

Large-context requests auto-route to a cloud LLM (GPT-5, Claude, etc.) when local prefill would be slow. Routing based on new tokens after cache hit. --cloud-model openai/gpt-5 --cloud-threshold 20000

Multimodal

Vision, audio (STT/TTS), video understanding, and text embeddings — all through the same OpenAI-compatible API.

DFlash Speculative Decoding (single-user)

z-lab’s block-diffusion drafter (via mlx-vlm) accelerates single-stream generation on validated Qwen3.5/3.6 27B aliases. Opt in with --enable-dflash:

pip install 'rapid-mlx[dflash]'
rapid-mlx info qwen3.5-27b-8bit       # check per-gate eligibility
rapid-mlx serve qwen3.5-27b-8bit --enable-dflash

Workload sensitivity: speedup varies by entropy. Coding / math / summarization typically see 1.5-2.7×; high-entropy creative writing and long-form chat can dip to 0.6-0.9× because the drafter’s training distribution diverges from open-ended generation. This is a known pattern in spec-decode literature (arXiv 2604.14682, AdaEDL) — not a bug. Other Qwen3.5/3.6 sizes (35B-A3B MoE, 122B-A10B MoE) were benched and rejected because their average speedup was below the gate.

v1 limitations: DFlash mode runs a dedicated single-user server (mlx-vlm doesn’t expose a batched DFlash kernel yet). Tool calling, MCP, and embeddings aren’t available in DFlash mode — restart without --enable-dflash for those.

Also: logprobs API, structured JSON output (response_format), continuous batching, KV cache quantization (--kv-cache-quantization), and 3300+ tests.

Optional Extras

The base pip install rapid-mlx is ~460 MB and covers all text-only models. Vision, audio, and other features ship as opt-in extras:

If you installed via Homebrew and want vision/audio support, use pip install 'rapid-mlx[vision]' (or [audio]) inside your own Python 3.10+ venv — that gives you the full feature set without rebuilding the brew formula.

Troubleshooting

Run the built-in environment-health probe (works from pip install, no dev tools needed, no model load, ≤5 s):

rapid-mlx doctor

┌─────────────────────────────────────────────────────────┐
│                  🩺 Rapid-MLX Doctor                    │
└─────────────────────────────────────────────────────────┘

◆ System
  ✓ Apple Silicon (Apple M3 Pro, 36 GB)
  ✓ macOS 14.3 (Darwin 23.3.0)
  ✓ Free disk: 162 GB
◆ Python
  ✓ Python 3.12.13
◆ Required Packages
  ✓ mlx 0.29.x
  ✓ mlx-lm 0.31.x
  ✓ transformers 5.x
  ✓ fastapi 0.x
  ✓ uvicorn 0.x
  ✓ rapid-mlx 0.7.22
◆ HuggingFace Cache / Network / Shell Integration / Optional Tools
  ...
────────────────────────────────────────
Summary: 18 ok, 4 warnings, 0 issues

Use rapid-mlx doctor --verbose to see the underlying probe detail (exact path, version, response code) for each check.

Want to validate model inference instead? That lives at rapid-mlx bench <model> --tier {smoke,check,full,benchmark} — doctor is purely env-health now.

Telemetry

Rapid-MLX can send anonymous usage data to help us prioritise the right models and catch regressions. It is off by default and never starts collecting without your explicit opt-in.

What we collect (only if you opt in)

• Subcommand names (serve / chat / agents / bench / doctor)
• Model alias names (qwen3.5-9b-4bit) or canonical HF repo IDs (mlx-community/...) — local paths are redacted to <local>
• Bucketed counts: prompt/completion tokens, TTFT, tokens/sec — never exact values
• Error categories + a hash fingerprint of the failure site (exception class name + per-frame file:function:lineno only — never the message text or absolute paths)
• OS, arch, Apple chip name, RAM (rounded to GB), Python major.minor

What we never collect

• Prompts, completions, tool-call arguments, file contents, or any user-generated text
• Local file paths, working directory, or model paths beyond their HF repo ID
• IPs or hostnames (Phase 2 will route through a Cloudflare Worker that strips IPs before forwarding to the aggregator; Phase 1 ships no transport at all)
• API keys, environment variable values, auth headers
• Stack trace messages or argument values

Manage it

rapid-mlx telemetry status     # show current state and why
rapid-mlx telemetry preview    # print the exact JSON payload that would be sent
rapid-mlx telemetry enable     # opt in
rapid-mlx telemetry disable    # opt out
rapid-mlx telemetry reset      # delete consent + client-id files (re-prompts on next run)

Force-disable in scripts / CI

Either of these always wins, regardless of stored consent:

RAPID_MLX_TELEMETRY=0 rapid-mlx serve qwen3.5-9b-4bit
rapid-mlx --no-telemetry serve qwen3.5-9b-4bit

There is intentionally no env-var equivalent for force-on — opting in must be an explicit one-time rapid-mlx telemetry enable. CI agents will never silently contribute.

Where the code lives

Everything is in vllm_mlx/telemetry/ — read it. Phase 1 (this release) ships the consent mechanism and CLI surface; no network code is in the codebase yet. Phase 2 will add the transport behind the same opt-in gate; the schema is documented in vllm_mlx/telemetry/schema.py. Tracking issue: #236.

Development

Quick start

git clone https://github.com/raullenchai/Rapid-MLX.git
cd Rapid-MLX
pip install -e ".[dev]"

Testing

Two layers: user-facing doctor (ships with pip) and dev test suite (source checkout only).

Dev test commands

For stress/soak, start a server first:

rapid-mlx serve mlx-community/Qwen3.5-4B-MLX-4bit --enable-auto-tool-choice --tool-call-parser hermes
# In another terminal:
make stress

Or use the script directly for more options:

python scripts/dev_test.py smoke              # lint + unit
python scripts/dev_test.py stress --port 8000 # custom port
python scripts/dev_test.py full               # everything

Regression harness (multi-model)

make check              # 1 model (~10 min, auto starts server)
make full               # 3 models + 12 agent profiles (~1 hr)
make benchmark          # all local models (overnight)

Architecture

vllm_mlx/
  server.py              # App factory + model loading + CLI entry
  config/                # ServerConfig singleton
  service/
    helpers.py           # Shared request helpers
    postprocessor.py     # Streaming pipeline (100% test coverage)
  routes/
    chat.py              # /v1/chat/completions
    completions.py       # /v1/completions
    anthropic.py         # /v1/messages (Anthropic API)
    health.py, models.py, embeddings.py, audio.py, mcp_routes.py
  engine/                # BatchedEngine (continuous batching)
  reasoning/             # 7 reasoning parsers (Qwen3, DeepSeek, MiniMax, ...)
  tool_parsers/          # 17 tool call parsers
  speculative/           # DFlash, SuffixDecoding, MTP drafters
  agents/                # 12 agent profiles (YAML)
  runtime/               # Model registry, cache persistence
  doctor/                # Environment-health probe (rapid-mlx doctor)
  bench/tiers/           # Model-validation tiers (rapid-mlx bench --tier ...)
scripts/                 # Dev-only (NOT shipped with pip)
  dev_test.py            # Unified test entry point
  stress_test.py         # 8-scenario stress test
  agent_soak_test.py     # 10-min agent soak test
  mhi_eval.py            # Compute MHI scores against a running server
tests/                   # pytest unit tests (3300+)
harness/                 # Regression baselines + thresholds

Roadmap

Contributing

We welcome contributions of all sizes! See CONTRIBUTING.md for setup and guidelines.

Easy first contributions (no model download needed):

• Add a model alias — map a short name to a HuggingFace model ID
• Request model support — tell us which model you want

Testing contributions (needs a Mac with Apple Silicon):

• Share your hardware’s benchmark numbers — one command:

  rapid-mlx bench qwen3.5-9b-4bit --submit
  

  Runs the standardized B=1 bench (greedy, 128 + 512 token buckets, 5 rounds each), shows you the JSON payload, asks for consent, and opens the PR for you via gh. If you don’t have gh, it prints the JSON path + a deep-link to GitHub’s compare page so you can open the PR in your browser. Submitted rows land in community-benchmarks/submissions/ and show up on https://rapidmlx.com once merged.
• Test with your favorite AI client (Cursor, Aider, LangChain, etc.)
• Report a bug

Contributors

Star History

License

Apache 2.0 — see LICENSE.