If you use LLM-as-judge, this one is for you.

(bookmark it)

Most teams validate their agent’s outputs by calling a frontier model as the judge. It works, until it doesn’t.

Three problems stack up fast:

→ Cost: you’re hitting a frontier API on every turn, every tool call, every response. In production that burns millions.

→ Latency: bigger models, remote calls, slow reasoning on every check.

→ Blind spots: frontier models don’t actually know your domain. In finance, insurance, or healthcare, they miss the keywords and principles your work depends on.

So I walk through a different approach: train your own small LLM judge.

Instead of a giant model, you start with a small one and let the system generate the training data for you. It decomposes your domain, samples synthetic examples, runs them through a debate arena where judges reach consensus, then trains on the refined set.

The result is a judge that’s cheaper, faster, and more accurate on your data than Gemini, Claude, or GPT, with an OpenAI-compatible endpoint you can even deploy on-prem.

I show the whole thing end to end, using a Claude Code plugin and a web interface, with a real insurance RAG grounding evaluator as the example.

You can get the plugin here: https://github.com/plurai-ai/plurai-plugins…

Here’s the full breakdown:

00:00 - Intro 00:12 - Three problems with using frontier LLMs as judges 01:05 - A different approach: train your own small judge 01:31 - How it works (synthetic data and a debate arena) 02:50 - Installing the Claude Code plugin 04:03 - Defining your task with /eval 04:34 - Example: an insurance RAG grounding evaluator 05:51 - Kicking it off and giving early feedback 06:26 - Choosing labels, domain, and strictness 08:30 - The web interface and dashboard 09:52 - Bringing your own example data (optional) 10:26 - The finished model: endpoint, accuracy, and speed 11:16 - Control, on-prem deployment, and interpretability 11:57 - Benchmarks vs frontier models and the GitHub repo 12:30 - Outro

I worked with the @pluraiAI team on this. Thanks for sponsoring the video.

plurai-ai/plurai-plugins

Source: https://github.com/plurai-ai/plurai-plugins

Evals & Guardrails SLMs — Claude Code Plugin

Turn a simple task description (or a few examples) into a deployed SLM for online evals or guardrails, directly from Claude Code. You write a description; it handles data generation, labeling, fine-tuning, and serving, returning a live HTTPS endpoint in minutes.

The resulting SLM runs in real time at sub-100ms, with up to 93% lower latency, 43% lower failure rate, and 87% cost savings versus frontier LLM judges. Backed by our ICML 2026 research paper, BARRED.

Requirements

• Claude Code
• Python 3.11+ with uv on your PATH
• A free Plurai account

Quickstart

1. Get your API key. Create a free Plurai account, generate a key, and paste it into the Claude console on first run. Your key is stored locally (~/.config/evals/credentials.json) and used only to authenticate with Plurai’s API.

2. Install — run these one at a time in any Claude Code session:

/plugin marketplace add plurai-ai/plurai-plugins

/plugin install evals@plurai-plugins

/reload-plugins

VS Code / JetBrains: run /plugins, add plurai-ai/plurai-plugins in the Marketplace tab, install the evals plugin, then Restart.

3. Run it.

/evals:eval Evaluate whether my RAG responses are grounded in the retrieved context

The plugin optionally asks refinement questions to sharpen what “good” looks like, fine-tunes an SLM-based eval or guardrail tailored to your use case, and returns an endpoint you call with the same API key.

Troubleshooting

• API key invalid or missing — the plugin links you to generate a new one; paste it into the console.
• /evals:eval doesn’t appear — the MCP server didn’t start. Reload (/reload-plugins or Restart), and confirm uv is on your PATH with Python 3.11+. Without uv, the tools fail to load silently.
• Requests hang / “Network error reaching Plurai” — allowlist app.plurai.ai, api.plurai.ai, run.plurai.ai.