New MLX LM Server From Apple

TL;DR: Apple MLX team releases MLX LM Server, enabling fully local AI agent workflows on Mac with continuous batching, distributed inference, and M5 neural acceleration — no cloud or API keys required. ## From Chat to Agent: Local AI Workflows Over the past year, AI agents have evolved from research prototypes into everyday productivity tools. In traditional chat experiences, you send a prompt to the model, it returns a response, and then you manually execute follow-up actions — run commands, check files, fix errors. The agent loop changes this: user → agent → model → tool → result fed back to the model, iterating until the task is complete. On Apple silicon, this loop can run entirely locally: data stays on the device, AI is always available, with no usage costs. ## The Four-Layer Architecture of the Local Agent Stack The technology stack powering local agent AI on Mac consists of four layers, from bottom to top: 1. **MLX** — An open-source array framework built specifically for Apple silicon, handling low-level computation, Metal acceleration, and memory management. 2. **Language Model Layer (MLX‑LM)** — Provides everything needed to load, run, quantize, and fine-tune large language models. Supports thousands of models on HuggingFace, and offers CLI tools and a Python API. 3. **MLX-LM Server** — An OpenAI-compatible HTTP server that exposes local models through standard APIs, supporting structured tool calls and reasoning models (step-by-step analysis of complex problems). It is a drop-in replacement for any cloud LLM API. 4. **Agent Layer** — Any framework or tool that supports the OpenAI chat completions protocol, such as Xcode, OpenCode, Pi agent, custom scripts, etc. Because MLX-LM Server provides a standard interface, all agent frameworks can use it directly. Many popular applications (Ollama, LM Studio, vLLM) are also built on top of MLX and MLX-LM, and the ecosystem continues to grow. ## Three Steps to Set Up a Local Agent Getting started from scratch takes just three steps: 1. **Install MLX-LM**: One `pip install` command. 2. **Start the server**: Run `mlx_lm.server`, specifying a model that supports tool calls (start with a small model for testing). The server loads the model and is ready to accept localhost requests. 3. **Point your agent to the local server**: In your agent framework, set the base URL to the local server address (e.g., `http://localhost:8080`) and specify the model name. The agent doesn't care whether the model is local or cloud-based. For example, OpenCode configuration only requires defining a local provider, setting the URL and model name — then all interactions run through the local model. ## Hardware Acceleration: Making Agents Fly ### Neural Accelerator: Up to 4x Faster Prompt Processing In the agent loop, every time the model receives tool output, it must process a large amount of new context. The M5 chip's dedicated neural accelerator makes matrix multiplication four times faster than the M4. Combined with MLX's specialized multiplication and attention kernels, this translates almost directly into a near 4x speedup in prompt processing. No special parameters are needed — MLX automatically selects the best kernel for the available hardware. ### Continuous Batching: Handling Concurrent Requests Agents often spawn multiple sub-agents that work in parallel (e.g., one reads documentation, one searches code, one writes tests). MLX-LM Server uses continuous batching to dynamically group incoming requests and process them together on the GPU. New requests can join an ongoing batch, so sub‑agents don’t have to wait in line — keeping the workflow smooth. ### Distributed Inference: Running Giant Models When a model exceeds local memory (e.g., DeepSeek’s 1.6 trillion parameters require 800 GB+), MLX’s distributed support allows distributing the model across multiple Macs connected via Thunderbolt or Ethernet. This not only makes larger models usable but also speeds up the agent loop by parallelizing prompt processing. Set it up using `mlx.launch` with a node hosts file. Starting from macOS 26.2, Thunderbolt RDMA support enables low‑latency, high‑bandwidth communication, achieving up to 3x speedup with four nodes. ## Live Demos: Building an App from Scratch & Fixing Bugs with Xcode ### Demo 1: Building a SwiftUI Drawing App The agent starts from a blank Xcode project, examines the existing structure, creates a plan, writes code, then builds and fixes errors. Within minutes it creates a fully functional drawing app. When asked to add rounded endpoints, the agent edits the code and recompiles successfully. > Everything runs locally: the model is served by MLX-LM on this Mac, and the agent uses standard development tools like xcodebuild to verify the build. ### Demo 2: Xcode Integration – Fixing a Bug Set up the MLX server in Xcode (Settings → Intelligence → Add Chat Provider → Locally Hosted, port 8080). After introducing a bug, the model identifies the issue in seconds, checks the relevant code, writes a fix, then builds and runs the app. The code never leaves the Mac. ## Summary Today we demonstrated the complete technology stack for running agent AI locally on a Mac: from MLX to MLX-LM Server to agent frameworks, and how to leverage the M5 neural accelerator, continuous batching, and distributed inference to make everything faster. All code is open source and available now. Source: https://www.youtube.com/watch?v=wykPErJ8M-8