We built a small gateway for multi-model agents — the hard part was streaming and tool calls

I kept treating provider switching as a model-selection problem. It turned into an integration problem. Our coding agents and automation workflows use several providers depending on the task: Claude for long context, GPT for some coding jobs, and other models for cost, latency, or availability. At first, each application had its own provider adapter. That worked until we had enough agents running that the edge cases became more expensive than the model choice itself. The problems were mostly around the boundary: - SSE chunks do not always line up with message or JSON boundaries; - OpenAI `tool_calls` and Claude `tool_use` blocks carry similar information in different shapes; - providers return different status codes, error bodies, and finish reasons for similar failures; and - usage accounting becomes hard to reconcile when every project has a separate key and billing view. So we built as a small gateway for our own workflows. The application talks to one contract, and provider-specific behavior stops at the gateway boundary. The current setup has two compatible request paths: - OpenAI-compatible Chat Completions for existing SDKs and agent frameworks; - Claude Messages-compatible requests for applications that need Anthropic’s content-block format. Behind those endpoints, the gateway currently handles: - normalized SSE events and usage metadata; - translation between OpenAI tool calls and Claude tool-use blocks; - provider-specific error normalization; - dynamic model configuration and routing; and - API keys, prepaid balance, and usage tracking in one console. The part that still needs the most testing is not the happy path. It is reconnecting after a partial stream, preserving raw provider events for debugging, and making sure retries or fallbacks do not make cost accounting misleading.