Our doc-QA agent runs four small models as four separate services - has anyone actually consolidated this?

Our agent does document Q&A. Before the final answer it runs four small models: bge-m3 to embed the query a cross-encoder to rerank the chunks GLiNER to pull a few fields like dates and org names a small Qwen to draft the routine parts before anything hard goes to the frontier LLM Until recently that was four separate services, one per model, each with its own container and scaling rules. It works, but the reranker OOM'd twice last month sharing a node it shouldn't have, and maintaining four deployments is most of my week for something that isn't the product. I moved them onto one server with SIE (Superlinked's inference engine, not affiliated) to cut that down. Two things that helped: bge-m3 returns dense and sparse from one encode call, so two of my "services" were one call I'd split for no reason the Qwen step runs on the same server via a generate call instead of its own box Under the hood it uses a shared queue and batches per model and operation, so they share a GPU without starving each other. Tradeoffs are real. It evicts least-recently-used, so on a 24GB card you keep 2-3 models hot and the rest reload on hit, and their docs say a latency-critical model at steady QPS is better in its own pool. Our traffic is bursty so sharing was fine. Frontier LLM stays outside SIE. The one thing I'm unsure about is blast radius. Four services means a bad deploy kills one model, but one server means it can take the whole pipeline down. Anyone who consolidated a stack like this, did it cut the ops load or just concentrate it?