Describes Iris Ai, a system that routes queries across 8 specialized LLMs on consumer hardware, achieving large-model performance with low memory by keeping only one model active at a time and dynamic model swapping.
I've spent the last year convinced the industry is solving the wrong problem. We don't need bigger models. We need smarter deployment. I would love to name my model "Iris Ai" Iris runs 8 specialized LLMs (Triage, Router, Control, Math, Code, Reasoning, General, Vision) on a single consumer machine with one hard constraint: M\_active ≤ 1 at any execution timestamp. The peak memory footprint is bounded by the single largest active model, not the aggregate cluster. How routing works: - Phase 1: Zero-cost regex scanner catches unambiguous inputs (AST patterns, math operands, file paths) and routes directly — sub-millisecond, no neural inference - Phase 2: A small triage model (T=0.2, Top-P=0.1) handles ambiguous inputs and emits a single structured routing token from a closed vocabulary On model swaps, we run an explicit 3-phase flush: free KV cache → gc.collect() + cuda.empty\_cache() / mlx.metal.clear\_cache() → load new GGUF. Skipping the Metal cache flush on Apple Silicon causes the new model load to fight retained Metal allocations Benchmarks on our Medium tier (66B Total, ≤14B active, 16GB RAM): - 92.0% on HumanEval (pass@1, greedy decoding) Large tier (193B Total, ≤32B active, 24GB VRAM): - 95.0% HumanEval, 94.0% MATH Also A 40B math specialist beats a 200B generalist on math. A 14B code specialist hits 92% HumanEval on a MacBook Air. The proposition isn't "bigger is better." It's that specialization + dynamic loading beats monolithic scaling on consumer hardware. Specialists were trained with LoRA (r=16/32, alpha=2r) targeting W\_q/W\_k/W\_v/W\_o, then delta matrices merged back into base weights — zero adapter overhead at inference. Math and Reasoning specialists used GRPO instead of RLHF, with reward signals for format compliance (<think> tags), compiler-verified syntax, and verbosity penalties. A few things worth noting for context: All benchmark scores are measured with dynamic output harnesses enabled (AST truncation repair, automatic import injection, LaTeX normalizer). The harness is part of the system — not a post-processing trick — but worth being transparent about. Current roadmap: - Overlapping model load with preceding token stream (eliminating perceived swap latency) - Online quantization hot-patches based on runtime memory pressure - Expanding GRPO training to Vision and Control specialists Here is the Source Code with Training Code and Data, The Routing Mechanism and and Benchmarks script: https://www.github.com/ahmedbarakat207/Iris-Ai MIT licensed. Everything runs on llama.cpp / GGUF.
AirLLM is an open-source tool that optimizes inference memory usage, enabling 70B LLMs to run on a single 4GB GPU without quantization, and supports 405B models on 8GB VRAM.
A 744B parameter mixture-of-experts model boots on a laptop with 25GB RAM by storing expert weights on SSD and only loading the active ~40B parameters per token, enabling local inference despite the model's size.
A guide on running local AI models like Qwen 3.5-9B on an M4 MacBook with 24GB RAM using tools like LM Studio, Ollama, and pi, including specific configuration tips for optimal performance.
Demonstrates running an 80B Qwen model in just 4.3 GB of RAM on a Mac and a 35B model on an iPhone, showcasing extreme memory optimization for local LLM inference.
Ternary Bonsai 27B, a large language model, is demonstrated running locally on an NVIDIA RTX 5090 GPU, requiring under 6GB of memory and enabling end-to-end agentic workflows on consumer hardware.