Been picking frontier models on benchmarks that don't match our deployment conditions

Turns out Opus is better at research, while Gemini is better at judgment! When each model does its own web research before making predictions on a 1,417-question forecasting benchmark, Opus outperforms (0.131 Brier vs Gemini's 0.143). But when both models are given the same starting research on each question (via a pre-gathered dossier), Gemini wins by the same margin (0.141 vs Opus's 0.153), suggesting that Opus's edge is in the research stage: figuring out what to search for, which pages to read, what details matter. Strip that away and Gemini's judgment over fixed evidence is sharper. Calibration scores corroborate this. Opus’s calibration drops noticeably when it’s no longer tasked with conducting its own research. And Gemini’s actually improves when provided with the standardized dossier, suggesting that its own agent’s research was leaving signal on the table. The asymmetry implies that Opus might be using its search trace as scaffolding for probability assignment (i.e., the act of going through the search loop is itself doing some of the epistemic work, separately from the information it surfaces.) To figure this out, we ran 4 models: Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro, and Grok 4.20, on a benchmark of 1,417 binary forecasting questions resolving Oct–Dec 2025 with two evaluation conditions: agentic (each model does its own web research with tools) and fixed-evidence (every model receives the same \~12k-character research dossier). Note, one limitation is that the fixed-evidence dossiers are themselves LM-produced, so we may be measuring how well each model interprets a particular standardized version of the evidence rather than judgment in the abstract. But that would indicate all four models drifting in the same direction. They didn't. GPT-5.4 and Grok 4.20 barely moved between conditions while Opus and Gemini swapped rank order (the opposite of what a broken or biased eval would produce.) We’ve been picking frontier models on benchmarks that don't match our deployment conditions. And to my knowledge this is the first direct evaluation of frontier models that decomposes performance into these research vs judgment stages. The rank-order flip is one specific instance of that mismatch, the one we measured; and there are probably others. If you've found similar splits on your own deployments (retrieval vs synthesis, summarization vs reasoning, anything where the model has to do two distinct things in sequence), I’d love to hear what you’re seeing/doing about it.