Massive breakthrough here!

Self-hosting LLMs just got ~75% cheaper:

Most agent pipelines now run 4-5 small models under the hood: an embedder for retrieval, a reranker for precision, an extractor for entities, and often a small LLM for routine generation.

The standard way to serve them is one server per model. vLLM holds the LLM, TEI holds the embedder, and everything else gets a custom FastAPI wrapper.

Each server reserves its own slice of GPU memory and holds it whether traffic arrives or not. GPUs are billed by the hour, so idle time costs the same as busy time.

This is why switching to small models rarely reduces the bill. The cost was never in the calls but rather in the servers.

The structural fix is serving every model from one process that loads and evicts models based on traffic.

Superlinked just open-sourced exactly that. SIE (Superlinked Inference Engine) is an Apache 2.0 server that runs 85+ models behind one API.

Four calls cover the whole pipeline:

• encode() returns vectors
• score() returns relevance scores
• extract() returns entity spans
• and generate() runs small open LLMs.

Models load on first request and are evicted least-recently-used, so one GPU serves a rotating set of models instead of sitting siloed behind one.

It runs anywhere from a laptop to a Kubernetes cluster, and it plugs into Qdrant, Weaviate, Chroma, LanceDB, LangChain, and LlamaIndex.

You can find the repo here: https://github.com/superlinked/sie

(don’t forget to star )

I also wrote a hands-on guide to using SIE. It walks through a real-world example where we deploy and manage five models on a single GPU, including a small language model, an OCR model, an NER model, a reranker, and an object detector.

The article is quoted below.

superlinked/sie

Source: https://github.com/superlinked/sie

SIE: Superlinked Inference Engine

Self-hosted inference for agents. Every open model your agents call, served from one cluster in your cloud.

Docs | Quickstart | API Reference | Models

⭐ Help us reach more developers and grow the SIE community. Star this repo!

About

SIE is an open-source inference engine that runs the models behind every agent task through one API: search and retrieval, document-to-markdown conversion, structured output, content safety, and the agent loop itself. It replaces the patchwork of a separate model server per task with one system that serves 100+ models, loading each on demand.

• OpenAI-compatible API for drop-in migration: /v1/embeddings, /v1/chat/completions, /v1/completions, /v1/responses
• Pre-configured model catalog: Stella, SPLADE, Qwen3, GLiNER, SigLIP, and more; embedding and retrieval models benchmarked on MTEB
• Serves multiple models simultaneously with on-demand loading and LRU eviction
• Ships the full production stack: load-balancing gateway, KEDA autoscaling, Grafana dashboards, Terraform for GKE, EKS, and AKS
• Integrates with LangChain, LlamaIndex, Haystack, DSPy, CrewAI, Chroma, Qdrant, Weaviate, and LanceDB

Development

The repository root is a virtual Python workspace. From the repository root, install and verify every workspace member with the committed lock (the audio-prep member requires its documented native build prerequisites):

uv python install 3.12
uv lock --check
uv sync --frozen --all-packages
uv run --frozen --project . --no-sync pytest -c pyproject.toml

Package membership is explicit in the root pyproject.toml; a package joins the workspace only in the same change that adds its complete source.

Tasks

One SIE cluster runs the inference behind a whole agent. Each task is a handful of swappable models; browse packages/sie_server/models/ for the full set.

Quickstart

Prefer a notebook? examples/quickstart.ipynb runs this same flow, on your machine or a free Colab GPU.

1. Start the server

# macOS (Apple Silicon) or Linux, native (requires Python 3.12)
pip install "sie-server[local]" && sie-server serve

# Linux, NVIDIA GPU
docker run --gpus all -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cuda12-default

# Linux, NVIDIA GPU — Transformers 5 OCR models (LightOnOCR and GLM-OCR)
docker run --gpus all -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cuda12-transformers5

# Linux, CPU
docker run -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cpu-default

Docker images are bundle-specific so dependency-incompatible model families stay isolated. Use the transformers5 image for LightOnOCR or GLM-OCR; the default image intentionally does not advertise them.

# in a second terminal
curl http://localhost:8080/readyz   # expect: ok

The server speaks the OpenAI API out of the box, embeddings and generation alike (the cluster gateway serves /v1/chat/completions, /v1/completions, and /v1/responses). Your first call needs nothing but curl:

curl http://localhost:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model": "sentence-transformers/all-MiniLM-L6-v2", "input": "Hello world"}'
# {"object": "list", "data": [{"object": "embedding", "embedding": [-0.0344, 0.0310, ...

Each model’s first call downloads its weights (progress appears in the server terminal). Later calls skip the download; inference latency depends on the model, task, hardware, and batch size.

2. Install the SDK

pip install sie-sdk                # Python
npm install @superlinked/sie-sdk   # TypeScript (pnpm and yarn work too)

3. Generate embeddings, rerank, and extract entities

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")

# Generate embeddings
result = client.encode("sentence-transformers/all-MiniLM-L6-v2", Item(text="Hello world"))
print(result["dense"].shape)  # (384,)

# Rerank search results
scores = client.score(
    "cross-encoder/ms-marco-MiniLM-L-6-v2",
    Item(text="What is machine learning?"),
    [Item(text="ML learns from data."), Item(text="The weather is sunny.")],
)
print(scores["scores"][0])  # {'item_id': 'item-0', 'score': -7.1, 'rank': 0}

# Extract entities
result = client.extract(
    "urchade/gliner_multi-v2.1",
    Item(text="Tim Cook is the CEO of Apple."),
    labels=["person", "organization"],
)
print(result["entities"][0])
# {'text': 'Tim Cook', 'label': 'person', 'score': 0.992, 'start': 0, 'end': 8, ...}

Text generation runs on the GPU generation image; stop the first server, then start this one on the same port:

# Linux, NVIDIA GPU (for generation on Apple Silicon via MLX, see the docs below)
docker run --gpus all -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cuda12-sglang

result = client.generate(
    "Qwen/Qwen3-0.6B",
    "Reply with a single word: the capital of France.",
    max_new_tokens=16,
    temperature=0.0,
)
print(result["text"])  # Paris

For generation on Apple Silicon (MLX), the TypeScript walkthrough, and every configuration in between, see the quickstart guide, TypeScript SDK docs, and SDK reference.

Production

The same code works against a production cluster. SIE ships a load-balancing gateway, KEDA autoscaling (scale to zero), Grafana dashboards, and Terraform modules for GKE, EKS, and AKS. Not just the server, the whole stack. All Apache 2.0.

# pick one values overlay: values-gke.yaml / values-aws.yaml / values-aks.yaml
# (pin a chart version for reproducible installs, e.g. --version 0.6.18)
helm upgrade --install sie-cluster oci://ghcr.io/superlinked/charts/sie-cluster \
  --namespace sie --create-namespace \
  --set hfToken.create=true \
  --set hfToken.value=YOUR_HF_TOKEN \
  -f https://raw.githubusercontent.com/superlinked/sie/main/deploy/helm/sie-cluster/values-gke.yaml

See the deployment guide.

Telemetry: SIE collects anonymous usage data (version, OS, architecture, GPU type) to understand adoption. No IP addresses, hostnames, or request data are collected. Disable with SIE_TELEMETRY_DISABLED=1 or DO_NOT_TRACK=1.

Explore

Model catalog: every model is a config in packages/sie_server/models/; pass its Hugging Face ID to the SDK.

Integrations: setup guides for all nine framework and vector-store integrations, in Python and TypeScript.

Examples: A quickstart notebook and an end-to-end project gallery.

MCP edge: offload document work from Claude and other MCP clients to your cluster and save agent tokens.

Why we built SIE: The motivation, told at AI Engineer Europe 2026.

superlinked.com/docs | Apache 2.0