The winner of my 16-model eval fabricated 5 times in the audit. Second place, a tenth of a point back: zero fabrications. That one ships. https://t.co/nTdxsKzuH5

The model that won the leaderboard isn’t the one I shipped

How I evaluated 16 models on truthfulness, accuracy, and precision against condensr’s real workload - and why the top score lost.

Condensr has one job - read videos for you and tell you what they actually said. Which means the model underneath has one job: don’t make anything up. A summary that invents facts is worse than no summary, because you can’t tell which sentence is the invented one.

Back in June I stopped assuming the model in prod was fine and made 16 models prove themselves on my own data. This is the full process - the eval bug that nearly poisoned the results, the four-judge panel, the blind lineup, the adversarial audit at the end, and why the model with the highest score is not the one running in prod today.

Start from your own data

Public benchmarks measure benchmark performance. I needed to know how models handle condensr’s actual workload: real videos I’d run through the product - tutorials, business breakdowns, health explainers, interviews, commentary - with transcripts from 40k to 148k characters of messy speech-to-text. That mess turns out to be where the bodies are buried.

30 videos. 16 models across 5 providers (OpenAI, Anthropic, Google, xAI, and DeepSeek / Mistral / Llama / Qwen through OpenRouter). Every model summarizes every video.

One detail that matters more than it looks: no candidate ever got a generic “summarize this video” prompt. Every generation ran through condensr’s actual production prompt pipeline - the real system prompt plus the per-category template that video would get in prod. An eval that auditions a model in a different seat than the one it will occupy tells you about the wrong seat. The 30 videos are also category-stratified, round-robin across categories, so the set isn’t secretly 30 tech videos.

Step one: catch your judge lying

Nobody hand-grades 480 summaries, so an LLM judge scores each one against its transcript. The judge failed before any model did.

An early version of the judge prompt truncated the transcript to 8,000 characters to save tokens. The tell was that the most detailed summaries were scoring the worst - down around 65% for output that read as clearly faithful. Digging in: the judge went looking for the summary’s claims in the slice of transcript it could see, couldn’t find them, and counted them as fabrications. Judged against the full transcript, the same summaries scored 97%. The judge was accusing the best output of lying because I had hidden the evidence.

That’s lesson one, and it’s the big one: an LLM judge with partial evidence produces confident, wrong verdicts. Pass the full source, every time. If that makes the eval expensive, that’s the price of an eval you can believe.

The bug is now a permanent rule in the judge prompt, and it’s my favorite line in the harness: the judge may not claim fabrication unless it can name the invented phrase and confirm it is absent from the transcript. Paraphrase, cleaned-up speech-to-text, and correct inference are not fabrication. When unsure, treat as supported. The judge has to do the same work I had to do when I caught it lying.

What a judge actually does

“LLM judge” sounds hand-wavy, so here’s the mechanism. Each video category has a rubric - a short list of criteria describing what a good summary of that category contains. The judge scores every criterion 0, 1, or 2, and the summary’s score is points earned over points possible.

Two rules keep the rubric honest:

• A criterion only counts as met if it’s satisfied with content grounded in the transcript. A beautifully structured section that invents its facts scores zero, and fails the faithfulness criterion on top.

• A criterion can be marked N/A only when the video genuinely lacks that content - not when the summary forgot it. N/A criteria drop out of the denominator, so a summary isn’t punished because a commentary video has no ordered steps to extract. Faithfulness itself can never be N/A.

Judges run at temperature 0 wherever the API allows it, with forced-JSON output and backoff retries on rate limits. And there’s a hard rule for garbage: a judge response that doesn’t parse or is missing scores gets retried, then discarded. A judge that silently returns nonsense must not corrupt the eval.

Four judges, because one opinion is a bias

Every summary got scored by a panel of four judges from four different model families: gpt-5.4-mini, grok-4.3, claude-opus-4.8, and gemini-3.1-pro. Two ran per-cell through APIs; two ran batched through CLI subscriptions I already pay for, which cut the projected API bill from about $146 to about $54.

Two numbers come out of the panel:

• Consensus - the mean across the four judges. The quality score.

• Spread - max judge minus min judge. The bias alarm. A model at 96 with a spread of 2 is a real 96. A model at 96 with a spread of 12 means the judges are fighting, and you should go read the summaries yourself.

(Yes, two judges were also contestants. The panel is the mitigation - and for what it’s worth, gpt-5.4-mini’s own judge handed it a 98 while Opus gave it a 99. Nobody got to grade their own homework generously.)

The blind lineup

The two subscription judges are where the harness gets fun. Opus and Gemini Pro run on plans I already pay for, not API credits - the harness literally strips the API key out of the subprocess environment so a CLI call can’t silently bill the API instead of the plan. Both plans rate-limit, so instead of one call per summary, one call grades all 16 summaries of a video at once.

That batching bought an accidental methodological win: the lineup is blind. The 16 summaries go in labeled A through P, shuffled with a seed derived from the video id - deterministic, so a resumed run reproduces the same lineup, but shuffled, so no model always sits first and collects position bias. No judge ever sees a model name, and the batch prompt tells the judge to grade each summary on its own faithfulness and never reward one for agreeing with the others.

(One dumb-sounding, very real constraint: a 180k-character transcript does not fit in a command-line argument. The prompt goes in over stdin. You find these things out exactly once.)

Built to be killed

The run state lives in three JSON files - generations, API judge scores, CLI judge scores - and every write is a temp-file-then-atomic-rename, so an interrupt mid-write can’t corrupt state and force a full, paid regeneration. Each phase is independently runnable, and re-running skips anything already finished.

That design got used the same day it was written: API credits ran dry mid-run (topped up, resumed), one judge started throwing 429s at 8-way concurrency (dropped to 4, resumed, mopped up), and three cells died permanently to per-request token caps on the longest transcripts. The concurrency constant still carries the scar in a comment: “lowered from 8.” If your eval harness can’t resume, you will pay for the same tokens twice.

The leaderboard

Three things jumped out:

• The incumbent came 9th. gemini-2.5-flash, the model actually summarizing prod videos, scored 94.5 - behind an open-weight DeepSeek running at roughly a third of the price.

• Price bought nothing at the top. gpt-5.5 at $154 per thousand summaries tied gpt-5.4-mini at $15.59. Ten times the cost and five times the latency for a tenth of a point.

• grok-4.3 won on points, at 8 seconds a summary. Ship it, right?

Scores can’t see lying

A mean quality score dilutes exactly the thing I care most about. One invented company name in an otherwise excellent summary barely moves an average - and it’s the whole reason a reader would stop trusting the product.

So the final phase went adversarial. I took the 6 hardest videos and dispatched 18 independent auditor agents - one per model-video pair, 3 models by 6 videos, so no auditor anchors on another’s verdict - to check every claim in each summary against the full transcript, hunting specifically for fabrications. Three models got audited: the incumbent and the top two.

The incumbent fabricated the most, and its signature move is the scariest failure mode in this product category: name completion. The speech-to-text heard “ComFAC”; the summary said Compaq. “Knight Ritter” became Knight Ridder. A rocket thrust figure gained precision the speaker never gave. Every one of those reads as competence - and every one is the model asserting something the source never said. It kept doing it despite a prompt guard I had added specifically against name completion. Prompts reduce the tendency. They don’t remove it.

grok-4.3, the points winner, fabricated five times - including the exact same Compaq mistake.

gpt-5.4-mini fabricated zero times across all six videos, and handled the same garbled speech-to-text the right way: TrueNAS, WireGuard, and “Louis Ignarro” all cleaned up without asserting anything the context didn’t support.

The pick

gpt-5.4-mini. Second on the leaderboard by a tenth of a point, first on the thing that matters:

• 0 fabrications in the adversarial audit (the incumbent had 9)

• tightest judge agreement in the whole matrix (spread of 2)

• faster than the incumbent - 14s vs 19s

• $0.0156 per summary vs $0.0085 - less than a cent more to stop making things up

It shipped in June, and it’s the model summarizing every condensr video today.

The process, if you want to steal it

• Eval on your own data, not benchmarks. The failure modes live in your data’s mess.

• Run candidates through your real production prompts. Audition the model in the seat it will actually occupy.

• Eval the judge before the models. Full source, always. A truncated judge convicts the innocent.

• Use a judge panel across model families, blind the lineup, and track the spread, not just the mean.

• Accuracy and truthfulness are different measurements. The leaderboard tells you whether summaries are good. Only a claim-level adversarial audit tells you whether they’re true.

• The incumbent is a contestant, not a baseline. Mine came 9th and fabricated the most.

• Make the harness resumable before you need it to be.

• Price is not a proxy for quality. The 10x model tied. The winner costs a cent and a half per summary.

The whole harness is about 1,300 lines of Python: a generation adapter per provider, the judge panel, and the resumable matrix runner. Nothing in it is clever. All of it is paranoid. That’s the right ratio for an eval.

Picking a model used to be reading benchmarks and vibes on X. Now it’s just running the eval.