I kept rewriting parameters every time I swapped models on vLLM and llama.cpp, so I built a tool to manage them (llmux, MIT)

Reddit r/LocalLLaMA Tools
model-management vllm llama-cpp docker gpu open-source cli

Summary

A developer built llmux, an open-source tool to simplify managing model profiles across different engines like vLLM and llama.cpp, allowing one-click switching with Docker and NVIDIA GPU support.

llmux dashboard I test a lot of models on vLLM and llama.cpp. Every swap meant editing parameters again, bringing each model up its own way, taking it down, then bringing the next one up. It was tedious and easy to get wrong. So I built **llmux** to manage it. Each model you use is a profile you write once, and after that you pick it from a list and it starts on whichever engine that profile belongs to. Running the same engine at a different version is just another profile pinned to a different image tag: a release, a nightly, or an image you built from source. My team and I use it for our actual work, and most of what's in it came out of that: Every action also has a headless CLI twin (`llmux up`, `ps --json`, `logs`, `bench`), so you can script it or run it over SSH. Requirements: Linux, an NVIDIA GPU, and Docker. It uses the NVIDIA Container Toolkit for GPU passthrough, so macOS and AMD/ROCm aren't supported yet. Install (clones the repo, sets up uv, puts `llmux` on your PATH): curl -fsSL https://raw.githubusercontent.com/Bae-ChangHyun/llmux/main/install.sh | sh Or by hand: git clone https://github.com/Bae-ChangHyun/llmux.git| cd llmux uv tool install --editable . && uv tool update-shell Repo: https://github.com/Bae-ChangHyun/llmux Docs: https://Bae-ChangHyun.github.io/llmux/ It's my own project, MIT licensed. English isn't my first language, so I used an LLM to help write this post.
Original Article

Similar Articles

vllm-project/vllm v0.19.1

GitHub Releases Watchlist

vLLM v0.19.1 release - a fast and easy-to-use open-source library for LLM inference and serving with state-of-the-art throughput, supporting 200+ model architectures and diverse hardware including NVIDIA/AMD GPUs and CPUs.

llama.cpp is the linux of llm

Reddit r/LocalLLaMA

The article draws a parallel between llama.cpp and Linux, positioning the open-source library as foundational infrastructure for running large language models.

ggml-org/llama.cpp

GitHub Trending (daily)

llama.cpp is an open-source C/C++ library for efficient LLM inference on local hardware, supporting various quantization methods and multiple backends (CPU, GPU, etc.).