GitHub 又一款本地跑 AI 模型的开源新工具：Shimmy，直指 Ollama 的痛点。

单文件仅 5MB，却能提供快速、稳定的本地推理，并带完整的 OpenAI 兼容 API，接入成本几乎为零。

它用 Rust 打底，把性能榨到极致：启动不到 100ms、内存占用约 50MB。

项目地址：http://github.com/Michael-A-Kuykendall/shimmy…

更省心的是开箱即用：无需配置、自动分配端口，还能自动发现模型来源——Hugging Face、Ollama 以及本地目录都支持。

对比 Ollama，在体积、启动速度、内存占用等关键指标上基本是“全线领先”。想体验更轻更快的本地推理，可以装一个试试。

Michael-A-Kuykendall/shimmy

Source: https://github.com/Michael-A-Kuykendall/shimmy

The Lightweight OpenAI API Server

🔒 Local Inference Without Dependencies 🚀

Languages: 简体中文 · 繁體中文

Shimmy will be free forever. No asterisks. No “free for now.” No pivot to paid.

💝 Support Shimmy’s Growth

🚀 If Shimmy helps you, consider sponsoring — 100% of support goes to keeping it free forever.

• $5/month: Coffee tier ☕ - Eternal gratitude + sponsor badge
• $25/month: Bug prioritizer 🐛 - Priority support + name in SPONSORS.md
• $100/month: Corporate backer 🏢 - Logo placement + monthly office hours
• $500/month: Infrastructure partner 🚀 - Direct support + roadmap input

🎯 Become a Sponsor | See our amazing sponsors 🙏

Table of Contents

• What Is Shimmy?
• 🔥 Airframe Engine (v2.0)
• ⚡ TurboShimmy INT4 KV (v2.1)
• 🎯 Supported Models
• 📦 Migrating from v1.x
• ⚡ Quick Start (30 seconds)
• 🚀 OpenAI SDK Compatibility
• 🔧 Extended Context
• 📥 Download & Install
• 🔗 Integration Examples
• 📖 API Reference
• ❓ FAQ
• 🏛️ Technical Architecture
• 📚 Documentation Hub
• 🌍 Community & Support
• ⚡ Performance
• License

Drop-in OpenAI API Replacement for Local LLMs

Shimmy is a single-binary that provides 100% OpenAI-compatible endpoints for GGUF models. Point your existing AI tools to Shimmy and they just work — locally, privately, and free.

🎉 NEW in v2.0.0: Shimmy now runs on Airframe, a pure-Rust WGSL GPU engine. No C++ toolchain, no backend flags, no compilation required.

⚡ NEW in v2.1.0: TurboShimmy INT4 KV — ~7× less KV cache VRAM with one flag. Run Llama-3.2-3B on 4 GB GPUs.

🔥 Airframe Engine (v0.2.7)

Starting in v2.0.0, Shimmy’s default inference engine is Airframe — a pure-Rust WebGPU (WGSL) transformer runtime built from scratch. v0.2.7 brings the Inference Saturation Fabric (ISF) refit and TDR transport integration to production.

See airframe CHANGELOG for full release notes.

Why this matters:

• No C++ toolchain required — Rust only, top to bottom
• F32 precision throughout for deterministic, high-quality output
• WGSL compute shaders work on any GPU via WebGPU (NVIDIA, AMD, Intel, integrated)
• Model spec auto-derived from GGUF metadata — no hardcoded per-model constants
• YaRN RoPE scaling for extended context via SHIMMY_MAX_CTX — engine allocates KV cache and sets RoPE scale automatically (see Extended Context below)

Quick start with Airframe (v2.0.0+):

# Default: 2048-token context
SHIMMY_BASE_GGUF=/path/to/TinyLlama-1.1B-Chat-v1.0.Q4_0.gguf ./shimmy serve

# Extended context (4096 tokens — YaRN RoPE enabled automatically, KV cache resized)
SHIMMY_BASE_GGUF=/path/to/model.gguf SHIMMY_MAX_CTX=4096 ./shimmy serve

⚡ TurboShimmy INT4 KV

TurboShimmy is Shimmy’s on-GPU INT4 KV-cache compression system, shipping in v2.1.0. It squeezes the KV cache from 32-bit floats down to per-head-vector 4-bit integers — entirely in WGSL compute shaders with no CPU roundtrips — delivering ~7× less KV VRAM with no measurable quality loss at normal context lengths.

One flag. ~7× less KV VRAM. Same output quality.

# Enable TurboShimmy on any GGUF model
./shimmy serve --kv-quant int4

# Or via environment variable (docker-compose, systemd, etc.)
SHIMMY_KV_QUANT=int4 ./shimmy serve

# Windows GPU + long prompts: reduce per-dispatch work to prevent TDR resets
./shimmy serve --kv-quant int4 --prefill-chunk 8

Why it matters — TurboShimmy changes what fits on your GPU:

VRAM comparison (Llama-3.2-3B, ctx=2048):

VRAM comparison (TinyLlama 1.1B, ctx=2048):

When to use TurboShimmy:

How it works: Each K/V head vector is independently quantized to 4-bit integers with a per-vector F32 scale factor, encoded as packed nibbles by WGSL compute shaders. Dequantization happens on-the-fly when computing attention scores — also on GPU. The Airframe engine’s helical context-shift operates directly on the packed INT4 representation. No CPU roundtrips at any step. Full architecture details in the Airframe engine.

Quality validation: Needle-in-a-haystack benchmarks on Llama-3.2-3B show zero retrieval degradation vs F32 at ctx≤2048 across all tested depths (15%, 50%, 85%). Full benchmark data and setup guide: TurboShimmy on the wiki.

Windows stability: Airframe v0.2.1 ships a device.on_uncaptured_error handler so GPU validation errors surface as clean HTTP 500 responses instead of crashes. Use --prefill-chunk 8 to prevent Windows TDR resets during long prefills on older GPUs (GTX 10xx/16xx series). v0.2.7 adds TDR transport with GPU timestamp pools for accurate dispatch timing, fixing TDR watchdog crashes during long prefill sequences.

🎯 Supported Models

Airframe v2.0 ships with GPU-verified support across 7 model architectures and 5 quantization types, covering the models most commonly run on consumer hardware. Context window is read directly from each model’s GGUF metadata — no hardcoded limits.

✅ Locally Validated (GPU Math Verified)

* Llama-3.2’s native context is 131072 tokens. Airframe reads this from GGUF and allocates KV cache accordingly. Use SHIMMY_MAX_CTX=8192 for a practical 8K window on consumer hardware (~256 MB KV cache for the 1B model).

GPU Math Verified means the Airframe GPU dequantization shader produces results matching the CPU reference implementation, independently confirmed for every tensor type in each model. This is done via quant_verify, which tests 512 elements per quantization type per model.

⏳ Roadmap — Larger Models (Require ≥16 GB VRAM)

✅ Supported Quantization Types

All types are implemented in both the GPU inference shader and a CPU reference implementation. GPU vs CPU agreement is validated for every type.

Auto-discovery is enabled. If Shimmy finds GGUF models in your HuggingFace cache, Ollama directory, LM Studio cache (~/.cache/lm-studio/models), or local ./models/ folder, it will register and serve them automatically. See docs/MODEL_EXPANSION.md for the full compatibility matrix.

📦 Migrating from v1.x

The llama.cpp backend is removed in v2.0.0. The Airframe engine is the only inference path. See docs/MIGRATION_v2.md for the step-by-step migration guide.

Developer Tools

Whether you’re forking Shimmy or integrating it as a service, we provide complete documentation and integration templates.

Try it in 30 seconds

# 1) Download pre-built binary
# Windows:
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe
set SHIMMY_BASE_GGUF=C:\path\to\model.gguf && ./shimmy.exe serve &

# Linux / macOS:
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-x86_64 -o shimmy && chmod +x shimmy
SHIMMY_BASE_GGUF=/path/to/model.gguf ./shimmy serve &

# 2) See registered models
./shimmy list

# 3) Smoke test the OpenAI API
curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
        "model":"tinyllama-1.1b",
        "messages":[{"role":"user","content":"Say hi in 5 words."}],
        "max_tokens":32
      }' | jq -r '.choices[0].message.content'

🚀 Compatible with OpenAI SDKs and Tools

No code changes needed - just change the API endpoint:

• Any OpenAI client: Python, Node.js, curl, etc.
• Development applications: Compatible with standard SDKs
• VSCode Extensions: Point to http://localhost:11435
• Cursor Editor: Built-in OpenAI compatibility
• Continue.dev: Drop-in model provider

Use with OpenAI SDKs

• Node.js (openai v4)

import OpenAI from "openai";

const openai = new OpenAI({
  baseURL: "http://127.0.0.1:11435/v1",
  apiKey: "sk-local", // placeholder, Shimmy ignores it
});

const resp = await openai.chat.completions.create({
  model: "REPLACE_WITH_MODEL",
  messages: [{ role: "user", content: "Say hi in 5 words." }],
  max_tokens: 32,
});

console.log(resp.choices[0].message?.content);

• Python (openai>=1.0.0)

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11435/v1", api_key="sk-local")

resp = client.chat.completions.create(
    model="REPLACE_WITH_MODEL",
    messages=[{"role": "user", "content": "Say hi in 5 words."}],
    max_tokens=32,
)

print(resp.choices[0].message.content)

⚡ Zero Configuration Required

• Automatically finds models from Hugging Face cache, Ollama, LM Studio (~/.cache/lm-studio/models), and local dirs
• Auto-allocates ports to avoid conflicts
• Auto-detects LoRA adapters for specialized models
• Just works - no config files, no setup wizards

🧠 Advanced MOE (Mixture of Experts) Support

Note: MoE (Mixture of Experts) CPU offloading is on the Airframe roadmap. See docs/AIRFRAME_MOE_ROADMAP.md for the implementation plan.

Run 70B+ models on consumer hardware — coming to the Airframe engine. Track progress on the roadmap.

Perfect for: Large models (70B+), limited VRAM systems, cost-effective inference

🎯 Perfect for Local Development

• Privacy: Your code never leaves your machine
• Cost: No API keys, no per-token billing
• Speed: Local inference, sub-second responses
• Reliability: No rate limits, no downtime

Quick Start (30 seconds)

Installation

v2.0.0: Download pre-built binaries with Airframe WebGPU engine included!

📥 Pre-Built Binaries (Recommended — Zero Dependencies)

Pick your platform and download — no compilation needed, GPU acceleration included:

# Windows x64 (Airframe WebGPU engine)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe

# Linux x86_64 (Airframe WebGPU engine)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-x86_64 -o shimmy && chmod +x shimmy

# macOS ARM64 (Airframe with Metal backend via wgpu)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-arm64 -o shimmy && chmod +x shimmy

# macOS Intel
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-intel -o shimmy && chmod +x shimmy

# Linux ARM64 (huggingface engine; Airframe cross-compilation not yet supported)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-aarch64 -o shimmy && chmod +x shimmy

That’s it! The Airframe WebGPU adapter is selected automatically at runtime.

🛠️ Build from Source / cargo install

# Install from crates.io
cargo install shimmy

# Build from source (huggingface engine, no GPU)
git clone https://github.com/Michael-A-Kuykendall/shimmy
cd shimmy
cargo build --release

Note: The Airframe GPU engine is a private dependency and cannot be built from source by public users. The pre-built release binaries already include Airframe compiled in — download those to get full GPU acceleration. cargo install shimmy installs the huggingface engine variant from crates.io.

GPU Acceleration

v2.0.0: Airframe uses WebGPU (wgpu) for GPU acceleration. No backend flags, no driver installation beyond standard OS graphics drivers.

📥 Download Pre-Built Binaries (Recommended)

Release binaries include the Airframe engine with WebGPU support compiled in:

🎯 How GPU Selection Works

Airframe uses wgpu’s adapter enumeration. On first launch it selects the best available GPU adapter for your system — discrete GPU preferred over integrated, integrated over CPU fallback. No configuration needed.

# Check selected adapter
shimmy gpu-info

# Start serving (GPU adapter auto-selected)
shimmy serve

🔧 Extended Context

SHIMMY_MAX_CTX overrides the context window at the engine level. When set above the model’s native window, Airframe automatically engages YaRN RoPE scaling and resizes the KV cache accordingly.

# 4096-token context with YaRN (2x native window for TinyLlama)
SHIMMY_BASE_GGUF=/path/to/model.gguf SHIMMY_MAX_CTX=4096 shimmy serve

# 8192 tokens (4x native, higher RoPE compression)
SHIMMY_BASE_GGUF=/path/to/model.gguf SHIMMY_MAX_CTX=8192 shimmy serve

Note: Extended context beyond 4096 is functional but not yet as deeply validated as the native 2048-token window. Accepted range is 512–131072. Values outside that range are silently ignored and 2048 is used.

💾 VRAM Sizing Reference

Airframe allocates VRAM at load time: weights + KV cache. The KV cache is F32 and scales linearly with context length (n_layers × n_kv_heads × head_dim × ctx × 2 × 4 bytes).

TinyLlama 1.1B Q4_0 — the v2.0 validated path:

Integrated graphics (Intel Iris, Apple M-series unified memory, AMD Vega) running at 2048 context is ~800 MB — comfortably inside the 2 GB allocation most integrated GPUs share with system RAM.

Scaling up to larger models (architecture and quant support required — see docs/MODEL_EXPANSION.md):

The KV cache formula for any model: n_layers × n_kv_heads × head_dim × ctx × 2 × 4 bytes. Multiply the 2048 baseline by your SHIMMY_MAX_CTX multiplier to get the extended context allocation.

Get Models

Shimmy auto-discovers models from:

• Hugging Face cache: ~/.cache/huggingface/hub/
• Ollama models: ~/.ollama/models/
• Local directory: ./models/
• Environment: SHIMMY_BASE_GGUF=path/to/model.gguf

# Primary validated model for Airframe v2.0
huggingface-cli download TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF \
  --include "tinyllama-1.1b-chat-v1.0.Q4_0.gguf" --local-dir ./models/

# Alternative 1B — also fits in the same hardware envelope
huggingface-cli download bartowski/Llama-3.2-1B-Instruct-GGUF \
  --include "*Q4_K_M*" --local-dir ./models/

Start Server

# Auto-allocates port to avoid conflicts
shimmy serve

# Or use manual port
shimmy serve --bind 127.0.0.1:11435

Point your development tools to the displayed port — VSCode Copilot, Cursor, Continue.dev all work instantly.

📦 Download & Install

Package Managers

• Rust: cargo install shimmy (installs huggingface engine; for Airframe GPU, use GitHub Releases binaries)
• VS Code: Shimmy Extension
• npm: npm install -g shimmy-js (planned)
• Python: pip install shimmy (planned)

Direct Downloads

• GitHub Releases: Latest binaries
• Docker: docker pull shimmy/shimmy:latest (coming soon)

🍎 macOS Support

Full compatibility confirmed! Shimmy works on macOS with Metal GPU acceleration via wgpu.

# Install from crates.io (huggingface engine)
cargo install shimmy

# For Airframe GPU engine, download the macOS binary from GitHub Releases:
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-arm64 -o shimmy && chmod +x shimmy

✅ Verified working:

• Intel and Apple Silicon Macs
• Metal GPU acceleration via wgpu (automatic on Apple Silicon)
• Xcode 17+ compatibility

Integration Examples

VSCode Copilot

{
  "github.copilot.advanced": {
    "serverUrl": "http://localhost:11435"
  }
}

Continue.dev

{
  "models": [{
    "title": "Local Shimmy",
    "provider": "openai",
    "model": "your-model-name",
    "apiBase": "http://localhost:11435/v1"
  }]
}

Cursor IDE

Works out of the box - just point to http://localhost:11435/v1

Why Shimmy Will Always Be Free

I built Shimmy to retain privacy-first control on my AI development and keep things local and lean.

This is my commitment: Shimmy stays MIT licensed, forever. If you want to support development, sponsor it. If you don’t, just build something cool with it.

💡 Shimmy saves you time and money. If it’s useful, consider sponsoring for $5/month — less than your Netflix subscription, infinitely more useful for developers.

API Reference

Endpoints

• GET /health - Health check
• POST /v1/chat/completions - OpenAI-compatible chat (streaming supported)
• POST /v1/completions - OpenAI-compatible text completions
• GET /v1/models - List available models
• POST /api/generate - Shimmy native API
• GET /ws/generate - WebSocket streaming

Environment Variables

CLI Commands

shimmy serve                              # Start server (auto port allocation)
shimmy serve --bind 127.0.0.1:8080        # Manual port binding
shimmy serve --gpu-backend auto           # WebGPU adapter auto-select (default)
shimmy serve --gpu-backend cpu            # Force CPU (disable GPU)
shimmy serve --kv-quant int4              # Enable TurboShimmy INT4 KV cache compression
shimmy serve --kv-quant int4 --prefill-chunk 8  # INT4 + Windows TDR prevention
shimmy list                               # Show available models
shimmy discover                           # Refresh model discovery
shimmy generate --name X --prompt "Hi"   # Test generation
shimmy probe model-name                   # Verify model loads
shimmy gpu-info                           # Show selected WebGPU adapter

Technical Architecture

• Rust + Tokio: Memory-safe, async performance
• Airframe engine: Pure-Rust WGSL GPU inference — no C++ toolchain, deterministic output, GGUF-native
• OpenAI API compatibility: Drop-in replacement
• Dynamic port management: Zero conflicts, auto-allocation
• Zero-config auto-discovery: Just works™

🚀 Advanced Features

• 🧠 MOE CPU Offloading: Hybrid GPU/CPU processing for large models (70B+)
• 🎯 Smart Model Filtering: Automatically excludes non-language models (Stable Diffusion, Whisper, CLIP)
• 🛡️ 6-Gate Release Validation: Constitutional quality limits ensure reliability
• ⚡ Smart Model Preloading: Background loading with usage tracking for instant model switching
• 💾 Response Caching: LRU + TTL cache delivering 20-40% performance gains on repeat queries
• 🚀 Integration Templates: One-command deployment for Docker, Kubernetes, Railway, Fly.io, FastAPI, Express
• 🔄 Request Routing: Multi-instance support with health checking and load balancing
• 📊 Advanced Observability: Real-time metrics with self-optimization and Prometheus integration
• 🔗 RustChain Integration: Universal workflow transpilation with workflow orchestration

❓ FAQ

Does Shimmy work on my GPU? Shimmy uses WebGPU (via the Airframe engine) which runs on Vulkan, D3D12, and Metal — covering NVIDIA, AMD, Intel, and Apple Silicon. No CUDA required. See GPU requirements in TROUBLESHOOTING.md if you hit adapter errors.

What’s the difference between Shimmy and llama.cpp / Ollama? Shimmy is written in pure Rust with no C++ toolchain dependency. The Airframe engine runs WGSL compute shaders compiled at startup — no pre-built binaries, no driver version pinning. The result is faster startup, lower memory overhead, and deterministic output. See the GPU Pipeline doc for internals.

Why do I need SHIMMY_BASE_GGUF or LIBSHIMMY_MODEL_PATH? If you don’t set these, Shimmy auto-discovers models in standard directories (~/.cache/huggingface, ~/.ollama, ~/lm-studio/models, ~/.cache/lm-studio/models, ~/Library/Application Support/LMStudio). Set SHIMMY_BASE_GGUF to override and point directly at a specific GGUF file.

Can I run multiple models at once? Not currently — Shimmy loads one model per server instance. To serve multiple models, run multiple server instances on different ports. Hot-swapping models (reload without restart) is on the roadmap.

Why does generation stop before max_tokens? The model reached a natural end-of-sequence token. For chat models this is expected behavior — the model signals it’s done. If you want to force longer output, increase max_tokens and set temperature > 0. If generation stops on the wrong token, the model may be using the wrong chat template — see CHAT_TEMPLATES.md.

Is there streaming support? Set "stream": true in your request. Shimmy returns Server-Sent Events in the standard OpenAI streaming format.

Q4_K_M vs Q4_0 — which should I use? Q4_K_M (K-quant) is consistently better quality than Q4_0 for the same file size. Use Q4_0 only when you need maximum compatibility or the model isn’t available in K-quant. See QUANTIZATION.md for the full analysis.

Can I extend the context window beyond what the model was trained on? Yes — set SHIMMY_MAX_CTX to any value. Airframe applies YaRN scaling automatically when the requested context exceeds the model’s native context. Quality degrades gradually beyond 2× the native context. See EXTENDED_CONTEXT.md.

📚 Documentation Hub

Full documentation lives in docs/. Use this table to find what you need:

Getting Started

API & Integration

Engine Deep Dives

Troubleshooting & Reference

Development & Methodology

Community & Support

• 🐛 Bug Reports: GitHub Issues
• 💬 Discussions: GitHub Discussions
• 📖 Documentation: Full Documentation Hub • Migration Guide v1→v2 • Engineering Methodology • OpenAI Compatibility Matrix • Architecture • GPU Pipeline • Troubleshooting
• 💝 Sponsorship: GitHub Sponsors

Star History

🚀 Momentum Snapshot

🌟 stars and climbing fast ⏱ <1s startup 🦀 100% Rust, no Python

📰 As Featured On

🔥 Hacker News • Front Page Again • IPE Newsletter

Companies: Need invoicing? Email [email protected]

⚡ Performance Comparison

Quality & Reliability

Shimmy maintains high code quality through comprehensive testing:

• Comprehensive test suite with property-based testing
• Automated CI/CD pipeline with quality gates
• Runtime invariant checking for critical operations
• Cross-platform compatibility testing

Development Testing

Run the complete test suite:

# Using cargo aliases
cargo test-quick           # Quick development tests

# Using Makefile  
make test                  # Full test suite
make test-quick            # Quick development tests

See our testing approach for technical details.

License & Philosophy

MIT License - forever and always.

Philosophy: Infrastructure should be invisible. Shimmy is infrastructure.

Testing Philosophy: Reliability through comprehensive validation and property-based testing.

Forever maintainer: Michael A. Kuykendall Promise: This will never become a paid product Mission: Making local model inference simple and reliable