Tag
Olla v0.1.0 releases a lightweight open-source Go proxy that unifies multiple local LLM inference runtimes (Ollama, vLLM, SGLang, llama.cpp, MLX, etc.) behind a single endpoint with model-aware routing, load balancing, failover, sticky sessions, and OpenAI/Anthropic-compatible APIs. The update adds Hugging Face TEI support, a reworked dashboard, configurable retry behavior, and improved model capability detection.
作者主张开发者应自建并控制 LLM 代理（proxy），以获得真正的 token 用量、请求日志、缓存命中等可观测性数据，而不是依赖各大模型厂商提供的无用仪表盘。
Stuntd is an open-source local proxy that records LLM decision calls, trains a lightweight model to handle them locally, reducing costs while maintaining high agreement with the teacher model.
LiteLLM enables using over 100 AI models through a single connection, supporting major providers like OpenAI and Anthropic, with open-source and self-hosted options.
The article discusses whether AI agents should have internal or external spending authority, advocating for external policy enforcement through tools like gateways to manage budgets across multiple agents.
OpenClawMachines is an open-source platform for running OpenClaw AI agents in hardware-isolated Firecracker microVMs on your own infrastructure, with a control plane, host agent, and LLM proxy.
A list of six powerful but lesser-known AI developer tools: Instructor for structured JSON output, Octopoda for agent memory, E2B for secure sandboxes, Firecrawl for website-to-markdown, Composio for app integrations, and LiteLLM for multi-model API.
Tokenwise is a smart LLM proxy that helps users identify where they are overpaying for LLM usage.
A solo founder introduces Orqen, a proxy that sits between your SDK and LLM providers to optimize outbound requests by compressing tool results, managing history, and reducing token costs, without changing agent code.