GitHub has a new open-source tool for running AI models locally: Shimmy, directly targeting Ollama’s pain points. At just 5MB for a single file, it delivers fast, stable local inference with a fully OpenAI-compatible API, making integration virtually zero-cost. Built on Rust, it squeezes every drop of performance: starts in under 100ms and uses only about 50MB of RAM. Project link: http://github.com/Michael-A-Kuykendall/shimmy… Even better, it’s truly plug-and-play: no configuration needed, automatic port allocation, and it can discover model sources on its own — supporting Hugging Face, Ollama, and local directories. Compared to Ollama, it leads across key metrics like size, startup speed, and memory usage. If you want a lighter, faster local inference experience, give it a try.

Michael-A-Kuykendall/shimmy

Source: https://github.com/Michael-A-Kuykendall/shimmy

The Lightweight OpenAI API Server

🔒 Local Inference Without Dependencies

🚀 License: MIT (https://opensource.org/licenses/MIT) Security (https://github.com/Michael-A-Kuykendall/shimmy/security) Crates.io (https://crates.io/crates/shimmy) Downloads (https://crates.io/crates/shimmy) Rust (https://rustup.rs/) GitHub Stars (https://github.com/Michael-A-Kuykendall/shimmy/stargazers)

💝 Sponsor this project (https://github.com/sponsors/Michael-A-Kuykendall)

Languages: 简体中文 · 繁體中文

Shimmy will be free forever. No asterisks. No “free for now.” No pivot to paid.

💝 Support Shimmy’s Growth

🚀 If Shimmy helps you, consider sponsoring (https://github.com/sponsors/Michael-A-Kuykendall) — 100% of support goes to keeping it free forever.

• $5/month: Coffee tier ☕ — Eternal gratitude + sponsor badge
• $25/month: Bug prioritizer 🐛 — Priority support + name in SPONSORS.md
• $100/month: Corporate backer 🏢 — Logo placement + monthly office hours
• $500/month: Infrastructure partner 🚀 — Direct support + roadmap input

🎯 Become a Sponsor (https://github.com/sponsors/Michael-A-Kuykendall) | See our amazing sponsors 🙏

Table of Contents

• What Is Shimmy?
• 🔥 Airframe Engine (v2.0)
• ⚡ TurboShimmy INT4 KV (v2.1)
• 🎯 Supported Models
• 📦 Migrating from v1.x
• ⚡ Quick Start (30 seconds)
• 🚀 OpenAI SDK Compatibility
• 🔧 Extended Context
• 📥 Download & Install
• 🔗 Integration Examples
• 📖 API Reference
• ❓ FAQ
• 🏛️ Technical Architecture
• 📚 Documentation Hub
• 🌍 Community & Support
• ⚡ Performance
• License

Drop-in OpenAI API Replacement for Local LLMs

Shimmy is a single-binary that provides 100% OpenAI-compatible endpoints for GGUF models. Point your existing AI tools to Shimmy and they just work — locally, privately, and free.

🎉 NEW in v2.0.0: Shimmy now runs on Airframe, a pure-Rust WGSL GPU engine. No C++ toolchain, no backend flags, no compilation required.

⚡ NEW in v2.1.0: TurboShimmy INT4 KV — ~7× less KV cache VRAM with one flag. Run Llama-3.2-3B on 4 GB GPUs.

🔥 Airframe Engine (v0.2.7)

Starting in v2.0.0, Shimmy’s default inference engine is Airframe — a pure-Rust WebGPU (WGSL) transformer runtime built from scratch.

v0.2.7 brings the Inference Saturation Fabric (ISF) refit and TDR transport integration to production.

See airframe CHANGELOG (https://github.com/Michael-A-Kuykendall/airframe/blob/master/CHANGELOG.md) for full release notes.

Why this matters:

• No C++ toolchain required — Rust only, top to bottom
• F32 precision throughout for deterministic, high-quality output
• WGSL compute shaders work on any GPU via WebGPU (NVIDIA, AMD, Intel, integrated)
• Model spec auto-derived from GGUF metadata — no hardcoded per-model constants
• YaRN RoPE scaling for extended context via SHIMMY_MAX_CTX — engine allocates KV cache and sets RoPE scale automatically (see Extended Context below)

Quick start with Airframe (v2.0.0+):

# Default: 2048-token context
SHIMMY_BASE_GGUF=/path/to/TinyLlama-1.1B-Chat-v1.0.Q4_0.gguf ./shimmy serve

# Extended context (4096 tokens — YaRN RoPE enabled automatically, KV cache resized)
SHIMMY_BASE_GGUF=/path/to/model.gguf SHIMMY_MAX_CTX=4096 ./shimmy serve

⚡ TurboShimmy INT4 KV

TurboShimmy is Shimmy’s on-GPU INT4 KV-cache compression system, shipping in v2.1.0. It squeezes the KV cache from 32-bit floats down to per-head-vector 4-bit integers — entirely in WGSL compute shaders with no CPU roundtrips — delivering ~7× less KV VRAM with no measurable quality loss at normal context lengths.

One flag. ~7× less KV VRAM. Same output quality.

# Enable TurboShimmy on any GGUF model
./shimmy serve --kv-quant int4

# Or via environment variable (docker-compose, systemd, etc.)
SHIMMY_KV_QUANT=int4 ./shimmy serve

# Windows GPU + long prompts: reduce per-dispatch work to prevent TDR resets
./shimmy serve --kv-quant int4 --prefill-chunk 8

Why it matters — TurboShimmy changes what fits on your GPU:

VRAM comparison (Llama-3.2-3B, ctx=2048):

VRAM comparison (TinyLlama 1.1B, ctx=2048):

When to use TurboShimmy:

How it works: Each K/V head vector is independently quantized to 4-bit integers with a per-vector F32 scale factor, encoded as packed nibbles by WGSL compute shaders. Dequantization happens on-the-fly when computing attention scores — also on GPU. The Airframe engine’s helical context-shift operates directly on the packed INT4 representation. No CPU roundtrips at any step.

Full architecture details in the Airframe engine (https://github.com/Michael-A-Kuykendall/airframe).

Quality validation: Needle-in-a-haystack benchmarks on Llama-3.2-3B show zero retrieval degradation vs F32 at ctx≤2048 across all tested depths (15%, 50%, 85%). Full benchmark data and setup guide: TurboShimmy on the wiki (https://github.com/Michael-A-Kuykendall/shimmy/wiki/TurboShimmy).

Windows stability: Airframe v0.2.1 ships a device.on_uncaptured_error handler so GPU validation errors surface as clean HTTP 500 responses instead of crashes. Use --prefill-chunk 8 to prevent Windows TDR resets during long prefills on older GPUs (GTX 10xx/16xx series). v0.2.7 adds TDR transport with GPU timestamp pools for accurate dispatch timing, fixing TDR watchdog crashes during long prefill sequences.

🎯 Supported Models

Airframe v2.0 ships with GPU-verified support across 7 model architectures and 5 quantization types, covering the models most commonly run on consumer hardware. Context window is read directly from each model’s GGUF metadata — no hardcoded limits.

✅ Locally Validated (GPU Math Verified)

* Llama-3.2’s native context is 131072 tokens. Airframe reads this from GGUF and allocates KV cache accordingly. Use SHIMMY_MAX_CTX=8192 for a practical 8K window on consumer hardware (~256 MB KV cache for the 1B model).

GPU Math Verified means the Airframe GPU dequantization shader produces results matching the CPU reference implementation, independently confirmed for every tensor type in each model. This is done via quant_verify, which tests 512 elements per quantization type per model.

⏳ Roadmap — Larger Models (Require ≥16 GB VRAM)

✅ Supported Quantization Types

All types are implemented in both the GPU inference shader and a CPU reference implementation. GPU vs CPU agreement is validated for every type.

Auto-discovery is enabled. If Shimmy finds GGUF models in your HuggingFace cache, Ollama directory, LM Studio cache (~/.cache/lm-studio/models), or local ./models/ folder, it will register and serve them automatically. See docs/MODEL_EXPANSION.md for the full compatibility matrix.

📦 Migrating from v1.x

The llama.cpp backend is removed in v2.0.0. The Airframe engine is the only inference path. See docs/MIGRATION_v2.md for the step-by-step migration guide.

Developer Tools

Whether you’re forking Shimmy or integrating it as a service, we provide complete documentation and integration templates.

Try it in 30 seconds

# 1) Download pre-built binary
# Windows:
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe
set SHIMMY_BASE_GGUF=C:\path\to\model.gguf && ./shimmy.exe serve &

# Linux / macOS:
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-x86_64 -o shimmy && chmod +x shimmy
SHIMMY_BASE_GGUF=/path/to/model.gguf ./shimmy serve &

# 2) See registered models
./shimmy list

# 3) Smoke test the OpenAI API
curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"tinyllama-1.1b",
    "messages":[{"role":"user","content":"Say hi in 5 words."}],
    "max_tokens":32
  }' | jq -r '.choices[0].message.content'

🚀 Compatible with OpenAI SDKs and Tools

No code changes needed - just change the API endpoint:

• Any OpenAI client: Python, Node.js, curl, etc.
• Development applications: Compatible with standard SDKs
• VSCode Extensions: Point to http://localhost:11435
• Cursor Editor: Built-in OpenAI compatibility
• Continue.dev: Drop-in model provider

Use with OpenAI SDKs

• Node.js (openai v4)

import OpenAI from "openai";

const openai = new OpenAI({
  baseURL: "http://127.0.0.1:11435/v1",
  apiKey: "sk-local", // placeholder, Shimmy ignores it
});

const resp = await openai.chat.completions.create({
  model: "REPLACE_WITH_MODEL",
  messages: [{ role: "user", content: "Say hi in 5 words." }],
  max_tokens: 32,
});

console.log(resp.choices[0].message?.content);

• Python (openai>=1.0.0)

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11435/v1", api_key="sk-local")
resp = client.chat.completions.create(
    model="REPLACE_WITH_MODEL",
    messages=[{"role": "user", "content": "Say hi in 5 words."}],
    max_tokens=32,
)
print(resp.choices[0].message.content)

⚡ Zero Configuration Required

• Automatically finds models from Hugging Face cache, Ollama, LM Studio (~/.cache/lm-studio/models), and local dirs
• Auto-allocates ports to avoid conflicts
• Auto-detects LoRA adapters for specialized models
• Just works - no config files, no setup wizards

🧠 Advanced MOE (Mixture of Experts) Support

Note: MoE (Mixture of Experts) CPU offloading is on the Airframe roadmap. See docs/AIRFRAME_MOE_ROADMAP.md for the implementation plan.

Run 70B+ models on consumer hardware — coming to the Airframe engine. Track progress on the roadmap.

Perfect for: Large models (70B+), limited VRAM systems, cost-effective inference

🎯 Perfect for Local Development

• Privacy: Your code never leaves your machine
• Cost: No API keys, no per-token billing
• Speed: Local inference, sub-second responses
• Reliability: No rate limits, no downtime

Quick Start (30 seconds)

Installation

v2.0.0: Download pre-built binaries with Airframe WebGPU engine included!

📥 Pre-Built Binaries (Recommended — Zero Dependencies)

Pick your platform and download — no compilation needed, GPU acceleration included:

# Windows x64 (Airframe WebGPU engine)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe

# Linux x86_64 (Airframe WebGPU engine)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-x86_64 -o shimmy && chmod +x shimmy

# macOS ARM64 (Airframe with Metal backend via wgpu)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-arm64 -o shimmy && chmod +x shimmy

# macOS Intel
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-intel -o shimmy && chmod +x shimmy

# Linux ARM64 (huggingface engine; Airframe cross-compilation not yet supported)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-aarch64 -o shimmy && chmod +x shimmy

That’s it! The Airframe WebGPU adapter is selected automatically at runtime.

🛠️ Build from Source / cargo install

# Install from crates.io
cargo install shimmy

# Build from source (huggingface engine, no GPU)
git clone https://github.com/Michael-A-Kuykendall/shimmy
cd shimmy
cargo build --release

Note: The Airframe GPU engine is a private dependency and cannot be built from source by public users. The pre-built release binaries already include Airframe compiled in — download those to get full GPU acceleration. cargo install shimmy installs the huggingface engine variant from crates.io.

GPU Acceleration

v2.0.0: Airframe uses WebGPU (wgpu) for GPU acceleration. No backend flags, no driver installation beyond standard OS graphics drivers.

📥 Download Pre-Built Binaries (Recommended)

Release binaries include the Airframe engine with WebGPU support compiled in: