Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery

# Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery
Source: [https://arxiv.org/html/2607.15766](https://arxiv.org/html/2607.15766)
1\]University of Chinese Academy of Sciences 2\]Institute of Software, Chinese Academy of Sciences 3\]Alibaba Group\\contribution\[\*\]Equal contribution\. Work done during internship at Alibaba Group\. , †\{\\dagger\}Corresponding authors\.

Wangyi JiangWei WangXuanang ChenYaojie LuShiwei YeYuzhen ShiBoyu YangJinghang WangHan LiWeiqi ZhaiBing ZhaoHu WeiHaiyang YuYongbin LiHongyu LinLe SunXianpei Han\[\[\[[zhongtianyun2023@iscas\.ac\.cn](https://arxiv.org/html/2607.15766v1/mailto:[email protected])[jiangwangyi2020@iscas\.ac\.cn](https://arxiv.org/html/2607.15766v1/mailto:[email protected])

###### Abstract

Large language models \(LLMs\) excel at answering given questions, yet their ability to navigate open\-ended, pre\-conclusion discovery remains largely unmeasured\. We introduceProspective Hypothesis Discovery \(PHD\), which asks models to autonomously construct grounded, discriminative, and testable hypothesis spaces from inconclusive evidence, including anomalous observations and fragmented records, to guide subsequent investigation\. To evaluate this, we presentHypoArena, comprisingHypoData\(988 cases across six domains\) andHypoEval\(an evaluation framework for open\-ended hypotheses\)\. We constructHypoDataviaRetrospective Context Regression, a pipeline that reconstructs pre\-conclusion contexts from expert documents by removing explicit conclusions while preserving factual substrates\. Since PHD admits multiple valid outputs,HypoEvalcombines bidirectional pairwise judgments with Bradley–Terry–Davidson aggregation for ranking, alongside six\-dimensional rubric scoring for diagnosis\. Experiments on 15 frontier LLMs reveal clear capability stratification and show that arena evaluation resolves finer\-grained model differences while aligning strongly with human expert judgment\. Together, these results support treating PHD as a distinct target for evaluating the investigative reasoning of LLMs\. All code, data, and evaluation tools are publicly available at[github\.com/SKYLENAGE\-AI/HypoArena](https://github.com/SKYLENAGE-AI/HypoArena)and[huggingface\.co/datasets/HypoArena/HypoData](https://huggingface.co/datasets/HypoArena/HypoData)\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.15766v1/x1.png)Figure 1:From reactive QA to prospective hypothesis discovery\.Conventional QA hands the model a pre\-formed question \(*reactive*\)\.HypoArenainstead reconstructs a real\-world context in pre\-conclusion form and asks the model to*proactively*construct a plausible hypothesis space—the capability we termProspective Hypothesis Discovery\(PHD\)\.Real\-world discovery rarely starts from a well\-posed question\. In scientific research, accident investigation, and financial analysis, the first encounter with a problem is typically a tangle of anomalous observations, fragmented facts, and incomplete records\. The defining challenge at this stage is not retrieving a known answer but constructing a plausible hypothesis space: a set of grounded, verifiable, and mutually discriminative claims worth investigating next\. We call this capabilityProspective Hypothesis Discovery\(PHD\), and argue that it is a distinct and largely unmeasured competence of LLMs\. For instance, given sudden server errors and fragmented logs, a capable system should propose several verifiable directions such as memory leaks or microservice deadlocks before any cause has been confirmed\. Figure[1](https://arxiv.org/html/2607.15766#S1.F1)contrasts this setting with conventional QA\-style evaluation\.

Existing benchmarks do not directly target this capability\. QA and agent benchmarks\[egg2025dabstep,jingdsbench,mitchener2025kosmos\]score models against predefined goals or task\-completion criteria\. Scientific discovery and reasoning benchmarks\[discovery\-bench,insight\-bench,hypo\-bench,sci\-arena,inno\-eval\]typically supply structured data or pre\-formulated research questions\. Open\-ended idea\-generation benchmarks\[wang2026fire,idea\-bench,ai\-idea\-bench,research\-bench\]start from paper abstracts or already\-refined research backgrounds\. None of these settings asks whether a model, given raw heterogeneous facts and no formed conclusion, can identify several plausible hypotheses, distinguish among them, and argue for each\. A benchmark that isolates this question is missing\.

Building such a benchmark is hard for two reasons, one on the data side and one on the metric side\. On the data side, expert documents such as scientific papers, accident reports, and analyst notes are written after the conclusion is reached\. They carry final claims, hypothesis statements, and explicit causal links, all of which are strong suggestive cues\. Feeding such documents to a model collapses PHD into restating the known answer, while authoring conclusion\-free contexts by hand is expensive and does not scale\. On the metric side, a high\-quality PHD output is not a single reference but an open hypothesis space\[sican\]\. The same context typically supports multiple legitimate directions, so reference\-matching metrics underestimate plausible hypotheses that happen to fall outside the gold set\[research\-bench,papineni2002bleu,lin2004rouge\], while absolute rubric scoring struggles to discriminate among open\-ended outputs of comparable surface quality\[liu2023g\]\.

To address these challenges, we constructHypoArena, a benchmark comprising 988 cases across six domains: Biomedical Science, Machine Learning, Social Science, Financial Analysis, IT Operations, and Safety Investigation\. Each case is derived from human\-authored documents such as research papers, investigation reports, or professional blogs\. Our core motivation stems from the observation that when humans author these reports, they must generate plausible hypotheses, yet these intermediate hypotheses often remain unrecorded in the final artifacts\. To approximate this intermediate reasoning stage, we construct our cases through Retrospective Context Regression\. Rather than recovering the exact historical information state, this process reconstructs a model\-visible context by preserving temporally admissible factual content while filtering explicit conclusions, target hypotheses, and retrospective causal attributions\. We operationally refer to contexts satisfying these criteria as “conclusion\-free\.” The corresponding hypotheses and evidence are kept on the reference side for verification and diagnostic evaluation\. Table[1](https://arxiv.org/html/2607.15766#S1.T1)positionsHypoArenarelative to prior benchmarks in scientific discovery and idea generation\.

Table 1:Comparison ofHypoArenawith related benchmarks\. We categorize existing benchmarks into three groups: QA & Agents, Scientific Discovery, and Idea Generation\. Here, D denotes database and Q denotes query\.BenchmarkInputOutputRetro\.OpenMulti\-ScaleContextHypo\.domainCategory 1: QA & AgentsDABstep\[egg2025dabstep\]Q \+ DataStr/Num/List×\\times×\\times×\\times450DSBENCH\[jingdsbench\]Q \+ DataCode / Num×\\times×\\times×\\times540Category 2: Sci\. Discovery & ReasoningDiscoveryBench\[discovery\-bench\]D \+ GoalSingle Hypo\.×\\times×\\times×\\times1\.1kInsightBench\[insight\-bench\]D \+ GoalInsights×\\times×\\times×\\times100HypoBench\[hypo\-bench\]DatasetsSingle Hypo\.×\\times×\\times✓\\checkmark194SciArena\[sci\-arena\]Q \+ DocsLit\. Response×\\times✓\\checkmark✓\\checkmark20KInnoEval\[inno\-eval\]IdeaEval Report✓\\checkmark×\\times×\\times761Category 3: Idea GenerationFIRE\-Bench\[wang2026fire\]QFull Research✓\\checkmark✓\\checkmark×\\times30IdeaBench\[idea\-bench\]AbsIdea×\\times✓\\checkmark×\\times2\.4KAI Idea BenchTopic\+PapersIdea×\\times✓\\checkmark×\\times3\.5KResearchBench\[research\-bench\]Q \+ SurveyInspiration×\\times✓\\checkmark✓\\checkmark1\.4KHypoArena \(Ours\)ContextOpen Hypos✓\\checkmark✓\\checkmark✓\\checkmark988Since a single “conclusion\-free context” often supports multiple plausible exploration directions, high\-quality output is no longer a single answer but an open hypothesis space\. To address issues such as the underestimation of innovative hypotheses and scoring biases inherent in exact match or rubric scoring, we designHypoEval, an arena\-style evaluation protocol based on pairwise comparison\. Under the same context,HypoEvalmoves away from a reliance on a single reference answer\. Instead, it compares the hypothesis sets generated by two models to judge which side performs better across six dimensions: Contextual Grounding, Inferential Insight, Evidential Justification, Hypothesis\-Space Breadth, Directional Distinctness, and Analytical Utility\. Subsequently, we incorporate the Bradley–Terry–Davidson model to aggregate these pairwise judgments and calculate a more robust model ranking\. Meanwhile, we retain rubric dimensions as diagnostic tools for analyzing model performance along the same six dimensions\.

We conduct systematic empirical evaluations on 15 contemporary language models across six domains\. OurHypoEvalpairwise evaluation protocol addresses drawbacks of conventional rubric\-based scoring\. Across domains, arena evaluation produces a clearly stratified leaderboard, while rubric scores remain concentrated within sub\-one\-point bands on the 1–5 scale\. Agent\-mode structured analytic skills produce heterogeneous and model\-dependent effects\. We further assess the reliability of the arena rankings through human\-expert alignment and cross\-judge triangulation\.

Our contributions are:

- •Task Definition & Benchmark: We define the task ofProspective Hypothesis Discovery\(PHD\) and introduceHypoArena, a benchmark of 988 cases across six scientific and analytical domains\.
- •HypoDataand Retrospective Context Regression: We propose a scalable Forge–Audit construction method that reconstructs pre\-conclusion contexts from completed expert documents under explicit leakage, temporal, faithfulness, and supportability controls\.
- •HypoEvaland Systematic Empirical Analysis: We propose a dual\-track evaluation framework that pairs an arena protocol with the Bradley–Terry–Davidson model as the primary ranking, complemented by rubric scoring as diagnostics\. A study of 15 contemporary LLMs shows that arena evaluation provides finer\-grained separation than rubric scoring and reveals protocol\-dependent differences in model ordering\.

## 2HypoData: Benchmarking Discovery

This section presentsHypoData, the foundational data component ofHypoArena, designed to systematically evaluate Prospective Hypothesis Discovery \(PHD\)\. We formalize hypothesis generation as a unified task across six domains, decoupling the model\-visibleContextfrom the hiddenHypothesisand its associatedEvidence\. Figure[2](https://arxiv.org/html/2607.15766#S2.F2)illustrates the full data construction pipeline\. We formally define the task components and the mathematical formulation below\.

![Refer to caption](https://arxiv.org/html/2607.15766v1/x2.png)Figure 2:TheHypoArenapipeline\. \(1\) Multi\-domain data collection from human\-authored documents; \(2\)HypoData: Retrospective Context Regression rolls source documents back to a pre\-conclusion factual state and extracts hidden hypothesis\-evidence pairs as the reference; \(3\) Structured Context\-Hypothesis\-Evidence output, with the Context model\-visible and the rest held out; \(4\)HypoEval: pairwise arena ranking aggregated by Bradley–Terry–Davidson, complemented by rubric scoring as diagnostics\.### 2\.1Task Definition

Prospective Hypothesis Discovery\(PHD\) is defined as the mapping from a reconstructed pre\-conclusion context𝒞\\mathcal\{C\}to aHypothesis Setℋ=\{\(hi,ei\)\}i=1K\\mathcal\{H\}=\\\{\(h\_\{i\},e\_\{i\}\)\\\}\_\{i=1\}^\{K\}\. This task requires models to autonomously construct a plausible hypothesis space from inconclusive premises\. Unlike traditional QA, which focuses on retrieving a pre\-existing answer, PHD requires models to identify unresolved tensions and propose verifiable directions worth investigating next\.

##### Context \(𝒞\\mathcal\{C\}\)\.

The context𝒞\\mathcal\{C\}approximates the raw, heterogeneous information state prior to conclusion formation, comprising anomalous observations, fragmented facts, and incomplete records\. To preserve task validity,𝒞\\mathcal\{C\}must satisfy two constraints:\(i\)informational sufficiency, meaning it provides enough factual substrate to support non\-trivial hypothesis generation; and\(ii\)operational conclusion absence, meaning that explicit final conclusions, target hypotheses, and retrospective causal attributions are withheld from the model\-visible input\. We refer to contexts satisfying the second criterion as operationally “conclusion\-free\.”

##### Hypothesis Set \(ℋ\\mathcal\{H\}\)\.

The outputℋ\\mathcal\{H\}consists of\(hi,ei\)\(h\_\{i\},e\_\{i\}\)pairs, where each pair represents an atomic unit of reasoning:

- •Hypothesis \(hih\_\{i\}\): An explanatory or predictive claim that addresses the anomalies within𝒞\\mathcal\{C\}\. Eachhih\_\{i\}must begroundedin the provided facts andverifiablethrough subsequent testing\. WhenK\>1K\>1, the hypotheses should additionally bemutually discriminative\.
- •Evidence \(eie\_\{i\}\): The domain\-specific justification for the hypothesis\. In scientific research, this manifests as averification plan\(e\.g\., experimental design\); in investigative analysis, it takes the form of adiagnostic package\(e\.g\., actionable checks and supporting evidence logs\) that substantiates the claim and guides further exploration\.

The cardinalityKKofℋ\\mathcal\{H\}is domain\-dependent\. Scientific cases elicit a single primary conjecture, emphasizing depth and testability, whereas analytical cases allow an open\-cardinality set, emphasizing breadth and differentiation\. A single\-conjecture case can still admit multiple valid outputs across systems, so the source\-derived Reference is not treated as a unique answer\.

### 2\.2Data Collection

To provide broad domain coverage and grounding in expert\-authored sources, we curate source documents from six domains across two categories:Scientific Domains\(Biomedical Science, Machine Learning, Social Science\) andAnalytical Domains\(Financial Analysis, IT Operations, Safety Investigation\)\. Domain\-specific temporal cutoffs are enforced to minimize data contamination\. Detailed collection protocols are provided in Appendix[8](https://arxiv.org/html/2607.15766#S8)\.

### 2\.3Data Construction with the Forge–Audit Agent Loop

To balance the trade\-off between information leakage and context richness, our framework employs an iterativeForge–Auditloop\. In this pipeline, theForgeagent generates candidate contexts, while theAuditagent validates them against benchmark constraints, repeating the cycle until criteria are met or the budget is exhausted\. Detailed construction protocols and prompts are provided in Appendix[9](https://arxiv.org/html/2607.15766#S9)and Appendix[16](https://arxiv.org/html/2607.15766#S16)\. TheForge agentoperates through two sequential components:

##### Context Forge\.

TheContext Forgeconstructs a model\-visible pre\-conclusion context𝒞\\mathcal\{C\}from source\-side factual material and, when appropriate, externally retrieved background\. External search is constrained by source\-specific timestamp boundaries that exclude materials dated after the relevant publication, release, or incident cutoff\. These controls reduce the risk of introducing post\-cutoff material through external retrieval\. We apply different strategies based on the domain:

- •Scientific Domain:To avoid reliance on any single, conclusion\-bearing document, we retrieve a corpus of literature predating the discovery and useDocument Mergingto synthesize multi\-faceted sources into a coherent factual substrate\.
- •Analytical Domain:To preserve raw evidence while stripping analytical bias, we employStructural De\-conclusion\. This process retains granular facts \(e\.g\., timestamps and measurements\) while systematically excising expert verdicts and explicit causal determinations\.

##### Hypothesis Forge\.

TheHypothesis Forgederives a source\-grounded reference\-side hypothesis set from the completed document while expressing each candidate in a form supportable from the reconstructed context\. In scientific domains, this captures a central research conjecture or proposal core; in analytical domains, it yields distinct explanatory or diagnostic claims grounded in the factual record\.

##### Audit Agent\.

After each Forge round, theAudit Agentverifies the candidate against three checks:Leakage Checktests whether the context inadvertently reveals the held\-out hypothesis,Faithfulness Checktests whether the hypothesis remains supported by the source document, andSupportability Checktests whether the evidence provides sufficient grounding\. Failed checks are returned to the Forge agent as actionable feedback for the next round\.

### 2\.4Human Quality Audit

Table 2:Human Quality Audit Results\.DimensionPass Rate\(a\) Informativeness95%\(b\) Openness100%\(c\) Completeness100%\(d\) Supportiveness97%Overall92%To validate the fidelity of our automatically constructed instances, we sampled 20 cases from each of the three domains: Biomedical Science, Financial Analysis, and Social Science\. For each domain, we recruited 2–3 expert annotators who are either senior doctoral candidates or practicing professionals\. Each \(Context, Hypothesis, Evidence\) triple was evaluated against four criteria: Informativeness and Openness for Context, Completeness for Hypothesis, and Supportiveness for Evidence \(detailed in Table[5](https://arxiv.org/html/2607.15766#S10.T5)\)\. Our results in Table[2](https://arxiv.org/html/2607.15766#S2.T2)show that 92% of the sampled cases passed all criteria, providing evidence for construction quality on the audited subset\. The full annotation guidelines are available in Appendix[10](https://arxiv.org/html/2607.15766#S10)\.

The final benchmark represents each case as a model\-visible, reconstructed pre\-conclusionContextand a reference\-sideHypothesis Set \(ℋ\\mathcal\{H\}\)comprising hypothesis–evidence pairs\(hi,ei\)\(h\_\{i\},e\_\{i\}\)\. The reference side is withheld during generation and is used for verification and diagnostic analysis or, in the arena, as an anonymous competitor for calibration\. This separation supports evaluating a model’s ability to synthesize disparate facts into grounded and verifiable hypotheses without exposing the source\-derived Reference during generation\.

## 3HypoEval: The Evaluation Framework

HypoEvalevaluates submitted hypothesis sets against the shared model\-visible context\. The source\-derived Reference is withheld from generation and enters the arena only as an anonymous competitor for calibration, not as a unique gold answer\. This separation accommodates the open\-ended nature of PHD, where a grounded hypothesis may remain valid even if it is absent from the Reference\. Accordingly,HypoEvalcombines a primaryRubric\-based Arenafor relative ranking withRubric\-based Scoringfor diagnostic analysis\.

### 3\.1Evaluation Rubrics

Both tracks ofHypoEvalshare a unified set of six metrics, with detailed definitions in Appendix[12\.2](https://arxiv.org/html/2607.15766#S12.SS2), designed to assess hypothesis set quality across two levels\.

- •Pair\-level Fidelity: For each\(hi,ei\)\(h\_\{i\},e\_\{i\}\)pair, we scoreContextual Grounding,Inferential Insight, andEvidential Justification\.
- •Set\-level Quality: For submissions in domains that admit multiple pairs, we additionally scoreHypothesis\-Space Breadth,Directional Distinctness, andAnalytical Utility;Directional Distinctnessis not applicable whenK=1K=1\.

### 3\.2Rubric\-based Arena \(Primary Metric\)

To overcome the “reference\-matching” bottleneck and the biases of absolute scoring such as score compression and ceiling effects, we adopt a pairwise comparison protocol\. In each matchup, an LLM\-as\-a\-judge is presented with a shared conclusion\-free context and the hypothesis sets generated by two models, A and B\.

- •Position Debiasing\.To mitigate position bias, each matchup is judged twice with A/B assignments swapped\. The two verdicts are averaged into a single debiased score, and only matchups where both directions agree in polarity are marked as consistent\.
- •Aggregating Pairwise Judgments\.The judge provides a 5\-level categorical verdict:A≫BA\\gg B,A\>BA\>B,A≈BA\\approx B,B\>AB\>A,B≫AB\\gg A, where≫\\ggindicates “significantly better” and≈\\approxindicates a tie\. These verdicts are mapped to win\-sharesw∈\{1\.0,0\.75,0\.5,0\.25,0\.0\}w\\in\\\{1\.0,0\.75,0\.5,0\.25,0\.0\\\}, respectively\. To derive a global leaderboard, we aggregate outcomes using the Bradley–Terry–Davidson \(BTD\) model\[davidson1970extending\], which models ties as a distinct epistemic state via a tie parameterθ\\theta\.

### 3\.3Rubric\-based Scoring \(Secondary Metric\)

We complement the arena protocol with Rubric\-based Scoring for diagnostic interpretability\. In this track, the judge assigns independent absolute scores on a 1–5 scale to a model’s output across the same six dimensions\. We aggregate these into pair\-levelQpairQ\_\{\\text\{pair\}\}and set\-levelQsetQ\_\{\\text\{set\}\}scores, with full formulas in Appendix[12\.2](https://arxiv.org/html/2607.15766#S12.SS2)\. This granular evaluation serves two purposes:

- •Multi\-dimensional Profiling: It produces per\-dimension profiles that identify models strong inContextual Groundingbut weak inInferential Insight, informing iterative model development\.
- •Complementary Absolute\-Scale Context\.It provides an absolute\-scale view of output quality, helping distinguish meaningful performance differences from relative advantages within an otherwise uniformly strong or weak model pool\.

By synthesizing the discriminative power of competitive ranking with the diagnostic granularity of absolute scoring,HypoEvalprovides an assessment framework for evaluating discovery\-oriented LLMs\.

## 4Experiments

Table 3:Main leaderboard under reference\-independent pairwise judging byseed\-2\.0\-pro\. BTD initializes each model at a baseline rating of 1500 and updates ratings based on pairwise match outcomes\. WinRate \(WR\) denotes each model’s win fraction across all pairwise matchups\. The Effort column reports each model’s thinking\-effort setting\. Reference participates as an anonymous competitor for calibration\.Scientific DomainsAnalytical Domains\#ModelEffortBioMLSocialFinITSafetyAvgWR1claude\-sonnet\-4\.6high1601\.21579\.31582\.71739\.21699\.81721\.61654\.071\.3%2claude\-opus\-4\.6high1608\.91618\.91671\.51670\.31678\.31668\.41652\.771\.4%3gpt\-5\.4high1627\.21650\.61602\.31636\.21641\.21630\.71631\.470\.7%4reference1568\.81561\.71502\.21631\.11604\.81674\.31590\.557\.8%5kimi\-k2\.6✓1570\.51562\.41559\.61601\.01569\.11633\.51582\.758\.2%6glm\-5\.1✓1592\.81595\.01591\.61563\.71588\.31560\.01581\.959\.1%7deepseek\-v4\-prohigh1525\.01520\.11564\.81533\.21517\.51551\.41535\.446\.1%8qwen\-3\.6\-max✓1499\.11518\.61503\.61513\.21499\.61520\.01509\.042\.2%9minimax\-m2\.7✓1508\.31505\.21492\.51517\.21516\.11453\.91498\.937\.0%10deepseek\-v4\-flashhigh1464\.31522\.11513\.81493\.81424\.61489\.31484\.636\.3%11glm\-5✓1469\.11450\.21452\.61485\.01483\.51445\.21464\.332\.8%12minimax\-m2\.5✓1458\.41428\.91427\.41424\.81433\.41369\.71423\.822\.9%13gpt\-5\.4\-minihigh1424\.01437\.41393\.71440\.71397\.71437\.71421\.923\.7%14gemini\-3\.1\-prohigh1429\.91407\.51415\.71335\.71388\.21387\.21394\.020\.7%15gemini\-3\-flashhigh1420\.01422\.21418\.21250\.71301\.01262\.11345\.714\.7%16kimi\-k2\.5✓1281\.81246\.51322\.71266\.11330\.31289\.41289\.59\.2%

This section evaluatesHypoArenaalong three axes: \(i\) the baseline leaderboard and the conditional effect of structured analytic skills \(§[4\.2](https://arxiv.org/html/2607.15766#S4.SS2)\); \(ii\) arena vs\. rubric scoring \(§[4\.4](https://arxiv.org/html/2607.15766#S4.SS4)\); \(iii\) alignment with human annotation preference and external quality \(§[4\.3](https://arxiv.org/html/2607.15766#S4.SS3)\)\.

### 4\.1Experimental Setup

We evaluate 15 contemporary LLMs on all 988HypoDatacases under two generation modes\. These models span frontier closed\-source and open\-weight families, includingclaude\-sonnet\-4\.6,claude\-opus\-4\.6, andgpt\-5\.4\. The full model list appears in Table[3](https://arxiv.org/html/2607.15766#S4.T3)\.

The primary evaluation follows the arena protocol \(§[3\.2](https://arxiv.org/html/2607.15766#S3.SS2)\) usingseed\-2\.0\-proas judge\. The source\-derived Reference participates as an anonymous competitor for calibration\. Cross\-judge consistency withmimo\-v2\-prois verified in Appendix[15](https://arxiv.org/html/2607.15766#S15)\.

- •Baseline Mode: The model generates its final Hypothesis Set directly from the input context in a single pass without invoking any explicit analytic skills\. This mode serves as a zero\-shot control, producing a concise single hypothesis–evidence pair for scientific domains, whereas for analytical domains, it yields an open\-cardinality set of plausible hypotheses, serving as a performance floor for comparison\.
- •Agent Mode: To mimic professional inquiry, models in this mode can select and execute a sequence from a library of 12 structured analytic skills \(see Appendix[17](https://arxiv.org/html/2607.15766#S17)for the full library\)\. These skills are adapted from the methodology ofStructured Analytic Techniques for Intelligence Analysis\[pherson2019structured\], including techniques such as the Analysis of Competing Hypotheses \(ACH\), Structured Brainstorming, and Chronology Analysis\. Selected skills are composed into a sequential pipeline, where intermediate analytical artifacts are passed forward to refine the final submission\.

### 4\.2Main Results

##### Clear performance stratification\.

![Refer to caption](https://arxiv.org/html/2607.15766v1/x3.png)Figure 3:Per\-model Agent vs\. Baseline ratings on the same full\-pool scale, sorted by baseline strength\. Annotations showΔ=Agent−Baseline\\Delta=\\text\{Agent\}\-\\text\{Baseline\}\(green positive, red negative\)\.As shown in Table[3](https://arxiv.org/html/2607.15766#S4.T3), the baseline leaderboard spans more than 360 BTD points, producing clear separation across the model pool\. Three frontier models,claude\-sonnet\-4\.6,claude\-opus\-4\.6, andgpt\-5\.4, form a dominant top tier, leading across most domains\. The protocol also resolves large within\-family differences; for example,kimi\-k2\.6outscoreskimi\-k2\.5by nearly 300 BTD points\.

##### Reference as a domain\-asymmetric baseline\.

The source\-derived Reference outperforms most models under the primary judge but falls below the top frontier tier\. Its strength is domain\-dependent: it is more competitive in procedure\-heavy domains such as Safety Investigation and Financial Analysis, but ranks lower in Social Science\. This indicates that a fixed source\-derived baseline is more competitive in domains with established procedural structure than in those admitting multiple analytical trajectories\.

##### Model\-Dependent Effects of Structured Analytic Skills\.

As illustrated in Figure[3](https://arxiv.org/html/2607.15766#S4.F3), Agent mode does not provide uniform gains\. The shifts are widely dispersed, ranging from an 88\-point gain forkimi\-k2\.5to a nearly 60\-point decline forclaude\-opus\-4\.6, and show little monotonic association with baseline strength \(Spearman’sρ=−0\.10\\rho=\-0\.10\)\. These results indicate that structured analytic skills act as a model\-dependent intervention, with candidate compression as one observed failure mode \(Appendix[11](https://arxiv.org/html/2607.15766#S11)\)\.

### 4\.3Alignment with Human and External Signals

##### Human Alignment and Judge Robustness\.

![Refer to caption](https://arxiv.org/html/2607.15766v1/x4.png)Figure 4:Human vs\. LLM Ranking Alignment\.Expert and primary\-judge rankings show high overlap \(cf\. Table[10](https://arxiv.org/html/2607.15766#S14.T10)\)\.To validateHypoEval, domain experts and our primary judge independently evaluated 1,500 pairwise comparisons\. As visualized in Figure[4](https://arxiv.org/html/2607.15766#S4.F4), the resulting rankings exhibit strong global alignment \(Kendall’sτ=0\.90\\tau=0\.90, Spearman’sρ=0\.98\\rho=0\.98\)\. This consistency confirms that our automated arena protocol serves as a high\-fidelity proxy for expert judgment\. While we observe some per\-domain variance, ranging fromτ=0\.97\\tau=0\.97in Financial Analysis toτ=0\.53\\tau=0\.53in Biomedical Science due to interdisciplinary noise, the top\-tier hierarchy remains identical across both evaluators\. See Appendix[14\.1](https://arxiv.org/html/2607.15766#S14.SS1)for the full leaderboard\.

##### Association with Peer\-Review Outcomes\.

The Machine Learning subset was sampled from ICLR 2026 and stratified by acceptance outcome, an external signal not used by the arena judge\. As detailed in Appendix[14\.2](https://arxiv.org/html/2607.15766#S14.SS2), the source\-derived Reference is more competitive on accepted than rejected cases across all three reported measures\. Its bucket\-specific BTD rating is 47 points higher, while the median per\-case debiased score and win rate are higher by 0\.06 and 0\.04, respectively\. This consistent directional pattern provides a complementary external check without treating acceptance outcome as ground truth for case\-level hypothesis quality\. This convergence with an independent quality signal supports the external validity of our protocol\.

### 4\.4Pairwise Arena versus Direct Scoring

![Refer to caption](https://arxiv.org/html/2607.15766v1/x5.png)Figure 5:Resolution contrast: Arena scaling vs\. Rubric score compression\.While the arena protocol \(left axis\) provides high discriminative resolution across hundreds of points, traditional rubric scores \(right axis\) exhibit extremeconcentration, with most models clustered within anarrow 1\-point band\.##### Score compression and ceiling effect\.

Both protocols agree on coarse ordering but diverge sharply at fine\-grained resolution\. As Figure[5](https://arxiv.org/html/2607.15766#S4.F5)shows, rubric scoring produces tighter model distributions: arena spreads range 345–490 points per domain, whereas rubric spreads stay under one point on the 1–5 scale, a compression ratio of 430×\\times–725×\\times\. This compression is driven by a pronounced ceiling effect: in scientific domains, 95%–98% of assessments land at or above 4\.0\. Absent a single canonical answer, judges find it easier to declare one submission better in a head\-to\-head comparison than to commit to a low absolute score\.

##### Protocol\-dependent ranking differences\.

Score compression is accompanied by changes in fine\-grained ordering, with per\-domain Kendallτ\\taubetween arena and rubric rankings ranging from 0\.52 to 0\.82\. In the baseline pool, the Reference rises by three positions under rubric scoring in IT Operations, Financial Analysis, and Social Science, whileclaude\-opus\-4\.6falls from its top\-two arena placement in each of these domains\. These shifts demonstrate sensitivity to the evaluation protocol, but rank changes alone do not identify the underlying mechanism\. Together with the score compression and ceiling effects described above, the protocol sensitivity motivates using arena comparisons for primary ranking while retaining rubric scores for diagnostics\.

## 5Related Work

### 5\.1Benchmarks for Discovery\-Oriented Tasks

Recent benchmark\-oriented work has begun to study scientific discovery and idea generation more systematically, moving beyond early generation systems toward explicit task formulation and standardized evaluation\. Recent examples include inspiration\-based scientific discovery benchmarks\[research\-bench\], principled benchmarking for hypothesis generation\[hypo\-bench\], research\-idea generation benchmarks\[ai\-idea\-bench\], and domain\-specific fine\-grained rediscovery settings\[moose\-chem\-2\]\. Although these works differ in input format and scope, they collectively shift the field from demonstrating that LLMs can generate ideas or hypotheses to asking how such capabilities should be benchmarked, compared, and stress\-tested\. In parallel, another line of work has improved hypothesis\-generation systems through literature\-grounded or hybrid pipelines, while beginning to examine reliability issues such as grounding\[literature\-meets\-data\], truthfulness\[hyper\], and hallucination in generated scientific hypotheses\[truth\-hypo\]\. Relative to this literature,HypoArenacenters on a particular combination of reconstructed pre\-conclusion contexts under temporal and leakage controls, open\-ended hypothesis generation without a unique reference answer, and unified coverage of scientific and analytical domains\.

### 5\.2Evaluation for Open\-Ended Scientific Outputs

Recent work has increasingly recognized that evaluating open\-ended scientific outputs requires task\-specific and knowledge\-grounded protocols\. Recent examples include arena\-style evaluation platforms for non\-verifiable scientific tasks\[sci\-arena\], multi\-perspective frameworks for research\-idea evaluation\[inno\-eval\], graph\-based evaluators for idea assessment\[graph\-eval\], and work probing how grounded LLM critiques of scientific papers actually are\[claim\-check\]\. Collectively, these efforts move beyond treating scientific outputs as ordinary text\-generation targets and instead emphasize evaluator reliability, grounding, and structured comparison\.HypoArenaaddresses these by introducingRetrospective Context Regressionto filter explicit conclusion\-bearing and retrospective causal content from source documents andHypoEvalwithout relying on a single gold\-standard reference\.

## 6Conclusion

We introduceHypoArena, a benchmark forProspective Hypothesis Discovery \(PHD\)that uses Retrospective Context Regression to reconstruct pre\-conclusion discovery tasks from approximately 1,000 completed expert documents\. OurHypoEvalframework employs a pairwise arena and the Bradley–Terry–Davidson model to robustly evaluate open\-ended hypothesis sets\. Systematic testing of 15 frontier LLMs reveals heterogeneous, model\-dependent effects of structured analytical skills, while qualitative generation traces identify candidate compression as a possible failure mode\.

## References

\\beginappendix

## 7Limitations

Despite its comprehensive framework,HypoArenahas several limitations that open avenues for future research\. First, while our benchmark spans six diverse domains, it remains primarily text\-centric\. ExpandingHypoArenato include multimodal or structured\-data\-driven discovery settings, such as genomics and medical imaging, represents a critical next step\. Second, the current construction pipeline relies on a single model \(gpt\-5\.4\) for theForgeprocess\. Future iterations could incorporate a more diverse ensemble of construction models or human\-authored cases to enhance stylistic variety and mitigate potential model\-specific biases\. Third, although we verified evaluation consistency with a second judge, the reliability of our framework could be further strengthened by employing a broader judge panel or a hybrid human\-LLM scoring system\. Finally, theAgent Modecurrently operates within a fixed library of analytic skills\. Moving toward a more flexible paradigm where models can dynamically compose or define novel strategies would better reflect the fluid nature of human scientific inquiry and investigative reasoning\.

## 8Details of Data Collection

To provide broad domain coverage and grounding in expert\-authored sources,HypoDataspans six domains organized into two high\-level categories:Scientific Domains\(Biomedical Science, Machine Learning, and Social Science\) andAnalytical Domains\(Financial Analysis, IT Operations, and Safety Investigation\)\. Across all domains, we prioritize publicly accessible and authoritative source materials that are rich enough to support context reconstruction and source\-grounded construction of reference\-side hypothesis–evidence pairs\. Where temporal control is important, we further restrict collection to recent artifacts or apply domain\-appropriate release\-date constraints to reduce contamination while preserving sufficient source diversity\. Unless otherwise noted, each collected source artifact is processed into a single final benchmark case, whose reference side may contain one or more hypothesis–evidence pairs\.

##### Biomedical Science\.

Biomedical Science sources are primary research articles retrieved from PubMed111[https://pubmed\.ncbi\.nlm\.nih\.gov](https://pubmed.ncbi.nlm.nih.gov/)via a systematic query pipeline, restricted to open\-access full texts published between July 2025 and April 2026 from high\-impact journals includingNature Communications,Cell Reports,eLife,Nucleic Acids Research,PLOS Biology,PNAS, andScience Advances\. We retain only empirical papers with explicit hypothesis framing, excluding reviews, case reports, clinical trials, and letters, yielding 244 benchmark cases\.

##### Machine Learning\.

Machine Learning draws on ICLR 2026222[https://openreview\.net/group?id=ICLR\.cc/2026](https://openreview.net/group?id=ICLR.cc/2026)submissions with released final decisions\. To diversify research maturity and benchmark difficulty, we stratify by decision outcomes, yielding 218 cases\. Decision categories shape sampling only and are not used as supervision targets\.

##### Social Science\.

##### Financial Analysis\.

Financial Analysis instances are anchored on the quarterly disclosure cycle of large\-cap U\.S\.\-listed companies, combining primary 10\-Q filings \(2025–2026\) sourced from the SEC101010[https://www\.sec\.gov/search\-filings](https://www.sec.gov/search-filings)with matched publicly accessible professional analyses\. Context is reconstructed primarily from the filing, while paired analyses supply hypothesis\-level interpretation and supporting evidence\. Cases are retained only when the quarterly disclosure is substantive and the analytical coverage is deep, yielding 114 cases\.

##### IT Operations\.

IT Operations sources are public post\-mortem reports from widely referenced incident repositories111111[https://github\.com/danluu/post\-mortems](https://github.com/danluu/post-mortems)and engineering blogs of major technology companies\. Unlike other domains, no strict temporal cutoff is imposed: post\-mortem reports describe unique incident configurations and system\-specific failure chains that are unlikely to appear verbatim in model training data, making temporal contamination a lower risk than in publication\-based domains\. We retain technically detailed cases with explicit descriptions of failure mechanisms, contributing factors, or remediation\-relevant reasoning, yielding 146 cases\.

##### Safety Investigation\.

Safety Investigation cases are drawn from official U\.S\. government accident investigation reports, including transportation accident reports121212[https://www\.ntsb\.gov/investigations/AccidentReports/Pages/Reports\.aspx](https://www.ntsb.gov/investigations/AccidentReports/Pages/Reports.aspx)and industrial chemical incident investigations131313[https://www\.csb\.gov/news/incident\-report\-rule\-form\-](https://www.csb.gov/news/incident-report-rule-form-), restricted to reports released in 2025 or later\. We retain only cases with detailed event narratives and causal analyses to support mechanistic reasoning, yielding 103 cases\.

Table 4:Statistics and source overview ofHypoData\. Each hypothesis–evidence \(H\-E\) pair is the atomic evaluation unit\. Scientific domains yield focused, single\-direction hypotheses; real\-world domains yield multi\-directional hypothesis sets\. Detailed collection protocols are in Appendix[8](https://arxiv.org/html/2607.15766#S8)\.DomainPrimary Source\# Cases\# H\-E\# CategoryResearchBiomedical ScienceJournal Articles244244Machine LearningConference Papers218218Social ScienceJournal Articles163163Real\-WorldFinancial AnalysisSEC 10\-Q \+ Analyst Reports1144563\.99IT OperationsPost\-mortems Blogs1465273\.29Safety InvestigationCSB & NTSB Reports1034043\.87Total9882,0123\.67

## 9Details of Data Construction

This appendix provides the full implementation details of the two\-stage Forge–Audit pipeline used to constructHypoData\(Section[2](https://arxiv.org/html/2607.15766#S2)\)\.

### 9\.1Forge–Audit Pipeline

Each benchmark instance is constructed through an iterative multi\-agent pipeline\. A coordinating controller dispatches specialized Forge and Audit agents across both construction stages\. The Forge agent generates candidate outputs \(contexts in Stage 1; hypothesis–evidence pairs in Stage 2\), while the Audit agent evaluates whether the candidates satisfy benchmark requirements\. If the candidate passes audit, construction terminates; otherwise, the Audit agent returns explicit and actionable feedback, which is used as direct input to the next Forge round\. Construction proceeds through repeated cycles until the audit passes or a fixed iteration budget is reached\. Under this protocol, each collected source artifact is processed exactly once and yields a single final benchmark instance\.

### 9\.2Stage 1: Context Construction under Temporal Control

##### Domain\-Specific Construction Policies\.

Context construction follows different default policies across domains\. In scientific domains, where source papers often provide limited background and related work, the Forge agent may enrich the context through temporally controlled search over academic and web sources\. In real\-world domains, where source materials are typically more information\-dense, context is constructed primarily from the source materials themselves, with minimal or no external augmentation\. In these domains, the construction process places greater emphasis on preserving source\-side factual detail and, when possible, the original writing texture\.

##### Strict Temporal Cutoff\.

When external retrieval is used, all search tools are constrained by source\-specific timestamp boundaries so that only information available prior to the publication, release, or incident date of the source materials can be accessed\. This reduces leakage from future knowledge and helps preserve the temporal realism of the original reasoning setting\.

##### Dynamic Context Reconstruction\.

Directly extracting localized arguments from the source materials often yields overly prescriptive inputs and leads to homogenized hypothesis generation\. To preserve exploratory reasoning, the Forge agent reconstructs context by combining source\-side factual content with temporally admissible external background when appropriate\. During this process, it removes explicit hypothesis statements, conclusion\-side claims, citation anchors, and other cues that would reveal the original line of reasoning\. The resulting context is designed to be information\-rich, multidimensional, coherent, and pre\-conclusion in character, while avoiding answer\-side guidance\.

### 9\.3Stage 2: Hypothesis and Evidence Extraction

While the main text characterizes thetargetof Stage 2 along two patterns—vertical deepening and horizontal association—theextraction methodvaries by source structure\.

##### Hypothesis Extraction Strategies\.

Because hypothesis presentation varies substantially across domains, the Forge agent applies one of three strategies based on the full source materials:

- •Direct Extraction:When the source explicitly states a single central hypothesis, it is extracted with minimal modification\.
- •Synthesis and Rewriting:When multiple distributed sub\-hypotheses appear across the source, they are consolidated into a unified core claim\.
- •Latent Distillation:When no explicit hypothesis is stated, the underlying claim is inferred from the source’s methods, results, and conclusions\.

##### Evidence Construction\.

Each evidence fieldeie\_\{i\}provides the domain\-specific support associated with hypothesishih\_\{i\}\. As an umbrella field, it may take either prospective form \(verification sketches in scientific domains\) or retrospective form \(evidence packages in real\-world domains\)\. By constructing this field explicitly, we require benchmark targets to be not only plausible but also supportable\.

### 9\.4Input–Reference Boundary Enforcement

Throughout construction, we strictly separate the model\-input side from the reference side of each benchmark case\. The context must not leak final conclusions, posterior explanations, or direct restatements of the source’s claims; all hypothesis–evidence pairs remain exclusively on the reference side\. This boundary is enforced explicitly in the Audit loop and is central to maintaining the benchmark’s pre\-conclusion setting\.

## 10Human Annotation Details

To assess the construction quality ofHypoDataand the alignment of our automated evaluator with expert preferences, we designed two distinct annotation tasks,Quality AuditandPreference Evaluation\. All tasks were performed in English by domain experts\. This section describes the annotation guidelines and setup\.

### 10\.1General Experimental Setup

The annotation setup was as follows\.

- •Expertise\. We recruited 2–3 annotators per domain, consisting of senior Ph\.D\. candidates or practicing professionals in Biomedical Science, Financial Analysis, and Social Science\.
- •Anonymization\. To reduce identity\- and provenance\-related bias, model identities and reference provenance were hidden from annotators where applicable\.
- •Data Scale\. Contexts averaged∼\\sim3,000 tokens, while Hypothesis–Evidence pairs averaged∼\\sim1,000 tokens\.

### 10\.2Quality Audit

##### Objective\.

Assess whether the automatically constructed \(Context, Hypothesis, Evidence\) triples provide informative and open\-ended contexts, complete source\-grounded hypotheses, and supportive evidence\.

##### Input Format\.

Annotators are presented with a Context and its corresponding \(Hypothesis, Evidence\) tuples\. In scientific domains, each case contains a single tuple; in real\-world analytical domains, multiple tuples may be provided\.

##### Evaluation Criteria\.

Annotators perform four binary checks across three core dimensions \(detailed in Table[5](https://arxiv.org/html/2607.15766#S10.T5)\)\.

A key qualitative metric is Context Openness: This measures whether the Context allows for multiple plausible analytical directions\. While a "No" \(indicating a singular reasonable direction\) is acceptable for specific deterministic cases, a high "Yes" rate across the dataset is preferred to ensure the benchmark’s challenge level\.

### 10\.3Preference Evaluation

##### Objective\.

Collect pairwise human preference judgments as an expert reference signal for assessing the alignment of our automated LLM\-as\-a\-judge system\.

##### Input Format\.

For each instance, annotators are provided with:

1. 1\.Shared Context\. The conclusion\-free background information provided to both models\.
2. 2\.Model Outputs\. Two candidate sets of hypotheses and evidence \(Output A and Output B\), randomized and stripped of model metadata\.
3. 3\.Evaluation Metrics\. The standardized rubric used for assessment \(see Appendix[12\.2](https://arxiv.org/html/2607.15766#S12.SS2)\)\.

##### Labeling Requirements\.

For each pairwise matchup, annotators must provide:

- •Preference\. Select from A\>\>B, Tie, B\>\>A\.
- •Confidence Score\. Annotators rated their confidence as High, Medium, or Low\.

### 10\.4Results of Human Annotation

Results of the Quality Audit and Preference Evaluation are reported in Table[6](https://arxiv.org/html/2607.15766#S10.T6)and Appendix[14\.1](https://arxiv.org/html/2607.15766#S14.SS1), respectively\.

Table 5:Summary of Quality Audit Criteria\. These questions guide human annotators in assessing whether the synthesized Context, Hypothesis, and Evidence meet the intended quality standards\.IDCriterionDim\.Core Audit Question\(a\)InformativenessContextDoes the context provide sufficient background and key facts to guide hypothesis generation?\(b\)OpennessContextCan the context elicit diverse and plausible alternative hypotheses beyond the target?\(c\)CompletenessHypothesisDoes the hypothesis capture the core claims originally presented in the source document?\(d\)SupportivenessEvidenceDoes the evidence provide a sound verification plan or factual basis for the hypothesis?Table 6:Human Quality Audit Results\. Criteria \(a–d\) correspond to Informativeness, Openness, Completeness, and Supportiveness respectively\. The92%overall pass rate validates the robustness of our HypoData construction\.DomainabcdPass RateBiomedical85%100%100%95%80%Financial100%100%100%100%100%Social Science100%100%100%95%95%Overall95%100%100%97%92%

## 11Generation\-Side Diagnostics

Qualitative inspection of the generation\-side logs suggests three recurrent distinctions among systems that may help explain why models appearing similarly plausible on the surface can diverge substantially under the primary evaluation protocol\.

##### Coverage breadth\.

In multi\-pair domains, the strongest systems do not simply produce more pairs; they produce more meaningfully distinct pairs\. Their advantage comes from broader exploration of the supported hypothesis space while maintaining non\-redundancy and local quality\. Weaker systems often generate only a small set of familiar directions, even when the context supports a wider range of mechanistic, latent, or intervention\-oriented possibilities\.

##### Candidate compression\.

In several inspected traces, particularly under Agent Mode, systems generate rich intermediate analyses but compress them into overly narrow final submissions\. These systems identify multiple plausible lines of reasoning internally yet fail to preserve them in the submitted set\. This possible failure mode is especially costly in open\-ended investigative settings, where omitted candidates may correspond to important supported directions rather than merely stylistic alternatives\.

##### Mechanism granularity\.

Strong systems formulate hypotheses at an appropriate level of explanatory resolution: they identify sharper mechanisms, separate distinct causal pathways, and attach concrete support to each direction\. Weaker systems often conflate multiple mechanisms into a single diffuse hypothesis or remain at a level of generality too broad to be meaningfully testable\. Such differences may be muted under direct scoring but become highly visible under arena comparison and set\-level evaluation\.

Taken together, these observations suggest that failure in deep hypothesis generation is not always reducible to obvious factual mistakes or low fluency\. It may instead reflect insufficient breadth, insufficient decomposition, or loss of structure between intermediate reasoning and the final submitted set\.

## 12Details of Evaluation

### 12\.1Submission Cardinality

HypoEvaldoes not impose a hard cap on submission size in multi\-pair domains\. This reflects the fact that breadth of supported hypothesis exploration is itself part of the target capability\. At the same time, larger submissions are not automatically preferred: additional pairs improve evaluation only insofar as they expand meaningful coverage without reducing distinctness or local quality\. All formal evaluation is therefore conducted on the final submitted set\.

### 12\.2Details of the Rubric

##### Quality Dimensions\.

Evaluation operates at two granularities to handle both local plausibility and global coherence\. At thepair level, each submitted\(h^i,e^i\)\(\\hat\{h\}\_\{i\},\\hat\{e\}\_\{i\}\)is assessed on three dimensions:

- •Contextual Grounding: whether the hypothesis is anchored in specific facts, observations, or unresolved tensions in𝒞\\mathcal\{C\}, and whether the inference is warranted—not merely topically relevant\.
- •Inferential Insight: whether the hypothesis goes beyond surface restatement by synthesizing dispersed information into a non\-obvious, context\-constrained explanatory, predictive, or mechanistic claim\.
- •Evidential Justification: whether the evidence concretely and proportionately supports the hypothesis, with a coherent reasoning chain and without overclaiming\.

These dimensions operationalize the requirement that hypotheses be grounded, non\-trivial, and supportable \(Section[2](https://arxiv.org/html/2607.15766#S2)\)\. WhenK\>1K\>1, the submission is additionally evaluated at theset level:

- •Hypothesis\-Space Breadth: coverage of multiple distinct, context\-supported directions such as mechanisms, risk axes, or causal pathways\.
- •Directional Distinctness: non\-redundancy across hypotheses—not overlapping or trivially reformulated versions of the same claim\.
- •Analytical Utility: actionability for downstream investigation, testing, or prioritization\.

Set\-level evaluation is necessary because a locally plausible but narrow or repetitive submission may still be materially incomplete\.

### 12\.3Rubric Scoring Formulas

For each submitted pair\(h^i,e^i\)\(\\hat\{h\}\_\{i\},\\hat\{e\}\_\{i\}\), the judge assigns scores on Contextual Grounding \(gig\_\{i\}\), Inferential Insight \(ℓi\\ell\_\{i\}\), and Evidential Justification \(jij\_\{i\}\)\. We define the local pair score as

qi=gi\+ℓi\+ji3\.q\_\{i\}=\\frac\{g\_\{i\}\+\\ell\_\{i\}\+j\_\{i\}\}\{3\}\.Pair\-level quality is aggregated over all submitted pairs as

Qpair=1K​∑i=1Kqi\.Q\_\{\\text\{pair\}\}=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}q\_\{i\}\.This average reflects the overall quality of the submitted hypothesis–evidence pairs and prevents larger submissions from being rewarded solely for containing more candidates\.

For domains that admit multi\-pair submissions, the judge additionally assigns set\-level scores on Hypothesis\-Space Breadth \(bb\), Directional Distinctness \(dd\), and Analytical Utility \(uu\)\. For submissions withK\>1K\>1, we define

Qset=b\+d\+u3\.Q\_\{\\text\{set\}\}=\\frac\{b\+d\+u\}\{3\}\.If a system submits only a single pair in a domain that admits multi\-pair submissions, Directional Distinctness is not applicable, and we compute

Qset=b\+u2\.Q\_\{\\text\{set\}\}=\\frac\{b\+u\}\{2\}\.
In reporting rubric\-based results, we primarily presentQpairQ\_\{\\text\{pair\}\}andQsetQ\_\{\\text\{set\}\}separately\. When a compact summary is useful, we reportSrubric=q1S\_\{\\text\{rubric\}\}=q\_\{1\}for singleton domains andSrubric=\(Qpair\+Qset\)/2S\_\{\\text\{rubric\}\}=\(Q\_\{\\text\{pair\}\}\+Q\_\{\\text\{set\}\}\)/2for domains that admit multi\-pair submissions\. This summary is used only for concise diagnostic reporting and not for primary model ranking\.

## 13Full\-Pool Arena and Rubric Leaderboards

This appendix reports both protocols at the \(model, mode\) level on a single full\-pool BTD scale, so that Section[4\.4](https://arxiv.org/html/2607.15766#S4.SS4)’s cascade can be inspected row by row against specific numbers\. Table[7](https://arxiv.org/html/2607.15766#S13.T7)ranks every \(model, mode\) cell by arena BTD; Table[8](https://arxiv.org/html/2607.15766#S13.T8)reports the per\-\(model, mode\) mean rubric scoreSSin the same row order, with aRubric Rankcolumn showing how the same set is re\-ranked under rubric AvgSS; Table[9](https://arxiv.org/html/2607.15766#S13.T9)reports the detailed results of different Rubrics in Financial Analysis\.

Table 7:Full\-pool arena leaderboard underseed\-2\.0\-pro: every \(model, mode\) cell on the same BTD scale across the six domains\. Rows are sorted by Avg BTD descending;referenceparticipates as a single anonymous competitor \(no agent counterpart\)\. This table is the arena\-side counterpart to Table[8](https://arxiv.org/html/2607.15766#S13.T8)and the source of theArena \#ordering used there\.Scientific DomainsAnalytical Domains\#ModelModeBioMLSocialFinITSafetyAvgWR1claude\-sonnet\-4\.6Agent1580\.51594\.51612\.21754\.11672\.91700\.21652\.473\.6%2claude\-sonnet\-4\.6Baseline1609\.41582\.71583\.91713\.91683\.11697\.61645\.172\.0%3claude\-opus\-4\.6Baseline1615\.51617\.31662\.11646\.41664\.71654\.41643\.471\.8%4gpt\-5\.4Agent1607\.01628\.61603\.11645\.11651\.81638\.71629\.073\.1%5gpt\-5\.4Baseline1627\.31638\.21604\.81623\.01630\.81611\.31622\.671\.0%6glm\-5\.1Agent1542\.11562\.41580\.01649\.51601\.31649\.51597\.560\.2%7kimi\-k2\.6Agent1553\.31570\.71552\.61601\.81611\.41629\.91586\.659\.7%8reference1578\.91565\.61516\.01615\.31598\.11645\.61586\.659\.9%9claude\-opus\-4\.6Agent1540\.71518\.61533\.21628\.41647\.71645\.41585\.757\.1%10glm\-5\.1Baseline1597\.61595\.01594\.51554\.11582\.01547\.01578\.460\.0%11kimi\-k2\.6Baseline1580\.01566\.41565\.81585\.71560\.31609\.41577\.959\.2%12deepseek\-v4\-proBaseline1543\.71528\.61573\.31525\.71510\.51538\.21536\.748\.0%13qwen\-3\.6\-maxBaseline1519\.01525\.31516\.11508\.21493\.91503\.11510\.944\.3%14qwen\-3\.6\-maxAgent1502\.91506\.51495\.61527\.31518\.41512\.31510\.542\.2%15minimax\-m2\.7Agent1492\.21485\.61481\.01534\.01497\.61535\.11504\.339\.0%16minimax\-m2\.7Baseline1526\.51515\.61509\.21515\.41509\.71432\.51501\.539\.7%17deepseek\-v4\-proAgent1480\.51504\.71499\.31475\.01483\.11514\.61492\.938\.3%18deepseek\-v4\-flashBaseline1482\.81529\.61525\.51488\.61427\.61475\.91488\.338\.2%19glm\-5Agent1462\.31479\.01476\.91485\.21501\.71484\.51481\.636\.0%20glm\-5Baseline1496\.21469\.31474\.71480\.71481\.81443\.91474\.436\.1%21gpt\-5\.4\-miniAgent1455\.81463\.41436\.01493\.01452\.01466\.11461\.132\.2%22deepseek\-v4\-flashAgent1417\.11451\.11463\.91446\.61427\.11488\.01449\.029\.8%23gpt\-5\.4\-miniBaseline1448\.41458\.81416\.81446\.51409\.11428\.11434\.627\.3%24minimax\-m2\.5Baseline1481\.31444\.31453\.51422\.61435\.31360\.51432\.926\.4%25minimax\-m2\.5Agent1446\.11415\.51397\.71445\.31458\.21401\.81427\.523\.9%26gemini\-3\.1\-proAgent1443\.61427\.11418\.21354\.71413\.61409\.61411\.123\.6%27gemini\-3\.1\-proBaseline1457\.31428\.31440\.51338\.91389\.01377\.91405\.324\.2%28kimi\-k2\.5Agent1333\.81351\.41380\.81465\.31369\.81475\.81396\.121\.1%29gemini\-3\-flashBaseline1445\.31441\.31439\.31257\.91301\.41262\.51357\.918\.0%30gemini\-3\-flashAgent1397\.31408\.01415\.41215\.41305\.21238\.21329\.913\.2%31kimi\-k2\.5Baseline1328\.11285\.11355\.21258\.61337\.81282\.51307\.911\.9%Table 8:Rubric scoring diagnostics underseed\-2\.0\-pro: per\-\(model, mode\) mean rubricSSon the 1–5 scale, complementing Figure[5](https://arxiv.org/html/2607.15766#S4.F5)of Section[4\.4](https://arxiv.org/html/2607.15766#S4.SS4)and Table[7](https://arxiv.org/html/2607.15766#S13.T7)\. Rows are sorted by arena rank \(Arena \#\) on the same full\-pool scale as that table;Rubric Rankis the position when the same cells are re\-ranked by AvgSS\(descending\), withΔ=\\Delta=Rubric Rank−\-Arena Rank\. The narrow AvgSSspread and frequent non\-zeroΔ\\Deltainstantiate the compression and protocol\-dependent reordering analyzed in Section[4\.4](https://arxiv.org/html/2607.15766#S4.SS4)\.Scientific DomainsAnalytical DomainsArena \#ModelModeBioMLSocialFinITSafetyAvgSSRubric Rank \(Δ\\Delta\)1claude\-sonnet\-4\.6Agent4\.704\.924\.894\.774\.374\.714\.734\(Δ\\Delta\+3\)2claude\-sonnet\-4\.6Baseline4\.794\.924\.894\.544\.404\.624\.695\(Δ\\Delta\+3\)3claude\-opus\-4\.6Baseline4\.764\.894\.904\.254\.294\.574\.616\(Δ\\Delta\+3\)4gpt\-5\.4Agent4\.754\.954\.914\.634\.464\.694\.733\(Δ\\Delta\-1\)5gpt\-5\.4Baseline4\.764\.964\.914\.574\.534\.664\.732\(Δ\\Delta\-3\)6glm\-5\.1Agent4\.574\.764\.774\.294\.244\.524\.5310\(Δ\\Delta\+4\)7kimi\-k2\.6Agent4\.594\.794\.714\.234\.254\.454\.5012\(Δ\\Delta\+5\)8reference4\.764\.884\.754\.954\.864\.934\.851\(Δ\\Delta\-7\)9claude\-opus\-4\.6Agent4\.624\.814\.784\.204\.214\.584\.539\(Δ\\Delta0\)10glm\-5\.1Baseline4\.754\.864\.824\.254\.274\.414\.568\(Δ\\Delta\-2\)11kimi\-k2\.6Baseline4\.674\.854\.834\.374\.254\.564\.597\(Δ\\Delta\-4\)12deepseek\-v4\-proBaseline4\.454\.674\.624\.044\.004\.364\.3618\(Δ\\Delta\+6\)13qwen\-3\.6\-maxBaseline4\.554\.814\.744\.234\.254\.494\.5111\(Δ\\Delta\-2\)14qwen\-3\.6\-maxAgent4\.514\.754\.704\.174\.164\.404\.4515\(Δ\\Delta\+1\)15minimax\-m2\.7Agent4\.444\.624\.593\.933\.924\.204\.2821\(Δ\\Delta\+6\)16minimax\-m2\.7Baseline4\.604\.754\.724\.093\.964\.164\.3817\(Δ\\Delta\+1\)17deepseek\-v4\-proAgent4\.284\.564\.483\.873\.944\.224\.2324\(Δ\\Delta\+7\)18deepseek\-v4\-flashBaseline4\.264\.574\.544\.023\.804\.284\.2422\(Δ\\Delta\+4\)19glm\-5Agent4\.374\.574\.614\.074\.084\.284\.3319\(Δ\\Delta0\)20glm\-5Baseline4\.524\.724\.684\.164\.204\.384\.4416\(Δ\\Delta\-4\)21gpt\-5\.4\-miniAgent4\.464\.764\.634\.354\.174\.414\.4614\(Δ\\Delta\-7\)22deepseek\-v4\-flashAgent4\.104\.444\.453\.923\.834\.294\.1725\(Δ\\Delta\+3\)23gpt\-5\.4\-miniBaseline4\.484\.764\.624\.344\.144\.444\.4613\(Δ\\Delta\-10\)24minimax\-m2\.5Baseline4\.444\.634\.593\.853\.954\.014\.2423\(Δ\\Delta\-1\)25minimax\-m2\.5Agent4\.214\.434\.323\.883\.854\.074\.1328\(Δ\\Delta\+3\)26gemini\-3\.1\-proAgent4\.274\.474\.453\.683\.934\.204\.1726\(Δ\\Delta0\)27gemini\-3\.1\-proBaseline4\.334\.484\.463\.964\.134\.364\.2920\(Δ\\Delta\-7\)28kimi\-k2\.5Agent4\.014\.314\.283\.733\.794\.024\.0230\(Δ\\Delta\+2\)29gemini\-3\-flashBaseline4\.174\.424\.353\.703\.994\.164\.1327\(Δ\\Delta\-2\)30gemini\-3\-flashAgent4\.004\.334\.303\.513\.804\.034\.0031\(Δ\\Delta\+1\)31kimi\-k2\.5Baseline3\.994\.314\.344\.013\.844\.014\.0829\(Δ\\Delta\-2\)Table 9:Per\-rubric\-dimension breakdown forFinancial Analysisunderseed\-2\.0\-pro\(full pool\)\. Rows sorted by AvgSSdescending; all dimensions on the 1–5 scale\.\#ModelModeGround\.InsightJustif\.BreadthDistinct\.UtilityAvgSS1reference4\.994\.714\.985\.005\.005\.004\.952claude\-sonnet\-4\.6Agent5\.004\.254\.884\.505\.005\.004\.773gpt\-5\.4Agent4\.774\.134\.664\.464\.924\.884\.634gpt\-5\.4Baseline4\.754\.064\.674\.394\.774\.794\.575claude\-sonnet\-4\.6Baseline4\.614\.094\.514\.554\.824\.684\.546kimi\-k2\.6Baseline4\.553\.904\.414\.224\.734\.384\.377gpt\-5\.4\-miniAgent4\.613\.814\.494\.114\.684\.394\.358gpt\-5\.4\-miniBaseline4\.623\.814\.514\.114\.684\.344\.349glm\-5\.1Agent4\.373\.944\.294\.164\.544\.414\.2910claude\-opus\-4\.6Baseline4\.283\.894\.154\.234\.614\.374\.2511glm\-5\.1Baseline4\.413\.844\.304\.154\.644\.274\.2512qwen\-3\.6\-maxBaseline4\.313\.854\.274\.044\.654\.254\.2313kimi\-k2\.6Agent4\.233\.884\.134\.204\.614\.324\.2314claude\-opus\-4\.6Agent4\.173\.824\.084\.244\.564\.324\.2015qwen\-3\.6\-maxAgent4\.173\.894\.093\.994\.614\.264\.1716glm\-5Baseline4\.393\.774\.293\.914\.504\.134\.1617minimax\-m2\.7Baseline4\.213\.664\.044\.054\.524\.064\.0918glm\-5Agent4\.173\.744\.063\.964\.464\.044\.0719deepseek\-v4\-proBaseline4\.113\.733\.944\.074\.444\.044\.0420deepseek\-v4\-flashBaseline4\.063\.743\.884\.024\.384\.084\.0221kimi\-k2\.5Baseline4\.523\.094\.374\.194\.213\.614\.0122gemini\-3\.1\-proBaseline4\.033\.643\.903\.824\.453\.903\.9623minimax\-m2\.7Agent3\.953\.553\.894\.004\.273\.913\.9324deepseek\-v4\-flashAgent3\.913\.713\.743\.934\.333\.923\.9225minimax\-m2\.5Agent3\.883\.493\.743\.954\.253\.953\.8826deepseek\-v4\-proAgent3\.803\.633\.703\.894\.343\.863\.8727minimax\-m2\.5Baseline3\.923\.423\.743\.984\.153\.883\.8528kimi\-k2\.5Agent3\.663\.403\.514\.083\.983\.793\.7329gemini\-3\-flashBaseline3\.703\.453\.483\.784\.223\.613\.7030gemini\-3\.1\-proAgent3\.433\.533\.313\.824\.323\.683\.6831gemini\-3\-flashAgent3\.333\.333\.143\.684\.243\.373\.51Three regularities emerge directly from the comparison\.

Compression\.Every AvgSSfalls within a single rubric point; the strongest and weakest cells in the paper’s evaluation set are separated by less than the distance between4\.04\.0and5\.05\.0, while the same cells span several hundred BTD points under arena\. The compression is visible in every domain column\.

Ceiling effect\.Almost every row sits above4\.04\.0, with the majority above4\.34\.3\. Rubric judges default to near\-top scores for anything not manifestly deficient, and this behaviour concentrates at the top of the scale rather than the middle\. The concentration is structural: absent a single canonical answer, committing to a low absolute score is harder than declaring one of two submissions better in head\-to\-head comparison\.

Protocol\-dependent reordering\.Relative to the arena ordering, 28 of 31 full\-pool entries receive a different rubric rank\. The top three arena entries each rank three positions lower under rubric scoring, while the Reference moves from rank 8 under arena to rank 1 under rubric\. These shifts demonstrate substantial protocol sensitivity in fine\-grained ordering, but they do not identify the underlying mechanism\.

Together, score compression, ceiling effects, and widespread reordering support using arena BTD for primary comparative ranking and rubric scores for diagnostic analysis\.

## 14Extended Analysis of Evaluator Alignment

We examine two validation signals that complement the cross\-judge analysis in Appendix[15](https://arxiv.org/html/2607.15766#S15)\. Human preference judgments assess alignment between automated rankings and expert preferences, while ICLR acceptance outcomes provide a source\-side signal not used by the arena judge\. Because these analyses operate at different granularities, we interpret each within the scope of the evidence it provides\.

### 14\.1Detailed Analysis of Human\-Judge Alignment

To assess alignment between the primary arena judge and expert preferences, we compare rankings induced byseed\-2\.0\-prowith rankings aggregated from 1,500 expert pairwise judgments across Financial Analysis, Social Science, and Biomedical Science\. Detailed annotation procedures are provided in Appendix[10\.3](https://arxiv.org/html/2607.15766#S10.SS3)\.

Table 10:Per\-domain leaderboard agreement between human experts and theseed\-2\.0\-projudge across 16 models\.Δ\\Deltais the rank shift from human to judge; per\-domain Kendall’sτ\\tauand Spearman’sρ\\rhoare reported at the bottom\. Most models show small displacements, with the largest spread concentrated in Biomedical Science\.ModelBiomedicalFinancial AnalysisSocial ScienceRankWin%Δ\\DeltaRankWin%Δ\\DeltaRankWin%Δ\\Deltaclaude\-sonnet\-4\.6\-high290\.90197\.50193\.80gpt\-5\.4\-high181\.80488\.50271\.40claude\-opus\-4\.6\-high386\.70573\.50475\.0\-1qwen\-3\.6\-max\-thinking844\.4\+4275\.80571\.70kimi\-k2\.6\-thinking566\.7\+1381\.20758\.3\-1glm\-5\.1\-thinking470\.60663\.40373\.9\+1qwen\-3\.6\-plus\-thinking657\.1\+9854\.80664\.3\+2glm\-5\-thinking1518\.2\-6763\.901423\.1\-3gemini\-3\.1\-pro\-high754\.501223\.5\+2951\.9\-2kimi\-k2\.5\-thinking1428\.6\-1944\.301143\.2\+1gpt\-5\.4\-mini\-high1237\.5\-71040\.001326\.00minimax\-m2\.7\-thinking1138\.5\-31129\.001051\.90deepseek\-v4\-flash\-high1040\.0\+41320\.0\-11611\.3\-1minimax\-m2\.5\-thinking944\.4\+21418\.8\-1850\.0\+1gemini\-3\-flash\-high1330\.0\-3152\.901517\.3\+1deepseek\-v4\-pro\-high165\.90160\.001242\.0\+2Correlations\(Kendall’sτ\\tau/ Spearman’sρ\\rho\)Biomedical: 0\.53 / 0\.67 Financial: 0\.97 / 0\.99 Social Science: 0\.85 / 0\.96As shown in Table[10](https://arxiv.org/html/2607.15766#S14.T10), agreement is strong but varies by domain\. The top three positions match exactly in Biomedical Science and Financial Analysis, whereas Social Science differs by a swap between ranks 3 and 4\. Most rank shifts are small, although Biomedical Science contains larger displacements among lower\-ranked models, includingΔ=\+9\\Delta=\+9forqwen\-3\.6\-plusandΔ=−7\\Delta=\-7forgpt\-5\.4\-mini\.

##### Lower alignment in Biomedical Science\.

The lower correlation in Biomedical Science \(τ=0\.53\\tau=0\.53\) indicates greater evaluator disagreement than in Financial Analysis \(τ=0\.97\\tau=0\.97\)\. The current analysis does not isolate the source of this difference, so we do not attribute it to a particular property of the cases or annotators\. Nevertheless, the matching top\-three positions in Biomedical Science and the concentration of the largest shifts among lower\-ranked models show that disagreement does not extend uniformly across the leaderboard\. The stronger correlations in Financial Analysis and Social Science further indicate that alignment varies by domain\.

### 14\.2Association with Peer\-Review Outcomes

The Machine Learning source papers were sampled from ICLR 2026 and stratified by acceptance category, providing an external outcome not used by the arena judge\. If arena evaluation is sensitive to source\-side differences associated with peer\-review outcomes, the source\-derived Reference constructed by Forge from each source paper should be more competitive on accepted cases than on rejected ones\.

Table 11:Reference arena strength underseed\-2\.0\-pro\(baseline pool\) on themachine\_learningsubset, split by ICLR 2026 acceptance\.Debiased: per\-case median of Reference’s\[0,1\]\[0,1\]debiased win share\.Win Rate: per\-case median of the AND\-of\-both win indicator, where Reference wins both forward and reverse, and ties score0\.50\.5\.BTD: Reference’s BTD rating fit on the bucket’s matches alone\.GroupDebiased \(↑\\uparrow\)Win Rate \(↑\\uparrow\)BTD \(↑\\uparrow\)Accepted0\.6020\.6541577\.6Rejected0\.5380\.6151530\.4Table[11](https://arxiv.org/html/2607.15766#S14.T11)shows a consistent directional pattern across all three measurements\. Accepted cases exceed rejected cases by0\.060\.06in median per\-case debiased score and0\.040\.04in median per\-case win rate, while the bucket\-specific BTD rating is 47 points higher\. This agreement provides a complementary external check using information unavailable to the arena judge\. Because acceptance reflects factors beyond hypothesis quality, we do not treat it as case\-level ground truth\.

## 15Inter\-Judge Agreement Analysis

The headline analyses in Section[4](https://arxiv.org/html/2607.15766#S4)useseed\-2\.0\-proas the sole arena judge\. To test whether the resulting rankings are an artifact of that single model’s idiosyncrasies, we re\-run the entire arena pipeline with an independent second judge,mimo\-v2\-pro, on the same submissions and the same case set\. This appendix reports five complementary agreement signals between the two judges, spanning three granularities\.

##### Three levels of agreement\.

We measure cross\-judge agreement at three granularities, all restricted to the baseline\-pool model set used for the main leaderboard:

- •Full\-ranking agreement: per\-domain Spearmanρ\\rhoand Kendallτ\\taubetween the two judges’ BTD ratings \(one value per \(model, domain\) cell\), andMean Rank Shift, the average\|Δ​rank\|\|\\Delta\\text\{rank\}\|per model when switching judges\.
- •Top\-tier identity: per\-domainTop\-5 Overlap, i\.e\. the size of the intersection of the two judges’ top\-5 ranked models\.
- •Individual verdicts: per\-domainPairwise Agreement, i\.e\. the fraction of shared \(case, modela, modelb\) matches where both judges’ forward verdicts select the same outcome among A wins, B wins, and tie\.

Table[12](https://arxiv.org/html/2607.15766#S15.T12)reports all five statistics per domain plus a pooled row across all \(model×\\timesdomain\) cells\.

Table 12:Inter\-judge agreement betweenseed\-2\.0\-proandmimo\-v2\-proon the 16 baseline\-mode submissions evaluated by both judges in each domain\. Spearmanρ\\rhoand Kendallτ\\tauare rank correlations between the two judges’ BTD ratings; Rank Shift is the mean\|Δ​rank\|\|\\Delta\\text\{rank\}\|per model; Top\-5 is the intersection size of each judge’s top\-5 models; Pairwise is the fraction of shared \(case, modela, modelb\) matches where both judges’ forward verdicts agree\.DomainSpearmanρ\\rho\(↑\\uparrow\)Kendallτ\\tau\(↑\\uparrow\)Rank Shift \(↓\\downarrow\)Top\-5 \(↑\\uparrow\)Pairwise \(↑\\uparrow\)Biomedical Science0\.9150\.7501\.505/563\.3%Machine Learning0\.9500\.8331\.125/567\.0%Social Science0\.9740\.9000\.755/566\.8%Financial Analysis0\.9090\.8331\.004/568\.1%IT Operations0\.9000\.7831\.504/567\.5%Safety Investigation0\.9440\.8171\.124/565\.6%Overall0\.9200\.7641\.174\.50/566\.3%
##### Rankings replicate; single\-match verdicts are noisier\.

The two judges produce essentially the same leaderboard\. Per\-domain rank correlations are uniformly strong \(Spearmanρ∈\[0\.90,0\.97\]\\rho\\in\[0\.90,0\.97\]; Kendallτ∈\[0\.75,0\.90\]\\tau\\in\[0\.75,0\.90\]\), the pooledρ=0\.920\\rho=0\.920has a tight bootstrap 95% CI of\[0\.871,0\.949\]\[0\.871,0\.949\], and theMean Rank Shiftof1\.171\.17on a 16\-model scale translates this into concrete terms: switching judges moves a model’s leaderboard position by only about one place on average\.Top\-5 Overlapaverages4\.5/54\.5/5across domains, with three domains in perfect agreement and the remaining three differing by a single model\. The top tier’s identity is robust to judge choice\. Pairwise verdict agreement ranges from63%63\\%to68%68\\%across domains, indicating substantial but incomplete agreement at the individual\-match level\. Despite disagreement on roughly one third of individual matches, the BTD\-aggregated rankings remain closely aligned across judges\. This rank\-level stability supports the use of arena aggregation for the primary leaderboard\.

Table 13:Main leaderboard under reference\-independent pairwise judging bymimo\-v2\-proas the independent second judge \(companion to Table[3](https://arxiv.org/html/2607.15766#S4.T3)\)\. All BTD and WR conventions match Table[3](https://arxiv.org/html/2607.15766#S4.T3); Reference participates as an anonymous competitor for calibration\.Scientific DomainsAnalytical Domains\#ModelEffortBioMLSocialFinITSafetyAvgWR1claude\-sonnet\-4\.6high1642\.31618\.91589\.31640\.21632\.31609\.91622\.267\.7%2claude\-opus\-4\.6high1632\.21634\.71620\.21610\.71589\.41574\.31610\.362\.6%3gpt\-5\.4high1578\.21602\.91570\.01556\.01548\.31542\.11566\.349\.1%4glm\-5\.1✓1583\.91593\.21562\.71549\.41543\.21543\.81562\.749\.0%5kimi\-k2\.6✓1530\.31547\.11542\.81555\.71549\.71569\.31549\.144\.3%6deepseek\-v4\-prohigh1514\.01519\.31556\.11535\.61534\.01547\.71534\.541\.5%7reference1494\.71538\.01485\.81558\.51532\.11567\.21529\.439\.3%8deepseek\-v4\-flashhigh1489\.01505\.81513\.41564\.31533\.51548\.71525\.840\.0%9qwen\-3\.6\-max✓1519\.81520\.41517\.61523\.31499\.71509\.71515\.135\.1%10minimax\-m2\.7✓1487\.91484\.01461\.81509\.91514\.41470\.81488\.126\.9%11glm\-5✓1467\.61456\.61452\.61496\.21482\.01488\.21473\.925\.2%12gpt\-5\.4\-minihigh1475\.81422\.81429\.31461\.11496\.31458\.41457\.320\.2%13gemini\-3\.1\-prohigh1438\.61426\.31448\.01442\.71426\.21457\.51439\.919\.7%14minimax\-m2\.5✓1423\.31412\.61434\.41451\.81436\.71435\.21432\.315\.1%15gemini\-3\-flashhigh1432\.71436\.41455\.21373\.91386\.81385\.81411\.816\.0%16kimi\-k2\.5✓1310\.31300\.01383\.41193\.81324\.81320\.61305\.57\.2%Table[13](https://arxiv.org/html/2607.15766#S15.T13)reports the baseline leaderboard undermimo\-v2\-proas a companion to Table[3](https://arxiv.org/html/2607.15766#S4.T3)\. Both judges rankclaude\-sonnet\-4\.6first,claude\-opus\-4\.6second, andgpt\-5\.4third, withkimi\-k2\.5last\. They also yield the same within\-family ordering for the DeepSeek, GLM, and Kimi model pairs\.

Fine\-grained disagreement is concentrated in the middle of the leaderboard\. The Reference moves from rank 4 underseed\-2\.0\-proto rank 7 undermimo\-v2\-pro, the largest displacement, while no other system moves by more than two positions\. Because this analysis does not isolate the source of judge disagreement, we interpret the Reference shift as judge sensitivity in its relative placement rather than evidence of a specific stylistic mechanism\. The mean rank shift of1\.171\.17and pooled Spearmanρ=0\.920\\rho=0\.920with 95% CI\[0\.871,0\.949\]\[0\.871,0\.949\]show that the overall ranking structure remains closely aligned\.

##### Complementary validation signals\.

We consider cross\-judge agreement alongside two additional checks\. The first measures alignment with human expert preferences in three domains \(§[4\.3](https://arxiv.org/html/2607.15766#S4.SS3), Kendallτ=0\.90\\tau=0\.90\), and the second examines the directional association between the competitiveness of the source\-derived Reference and ICLR acceptance outcome \(Appendix[14\.2](https://arxiv.org/html/2607.15766#S14.SS2), with a 47\-point higher bucket\-specific BTD rating on accepted cases\)\. These analyses operate at different granularities and do not constitute three replications of the same leaderboard\. Instead, they probe sensitivity to judge choice, expert alignment, and association with a peer\-review outcome not used by the arena judge\. Together, they broaden the evidence base forHypoEvalwithout treating any single signal as definitive\.

## 16Prompt Templates

This appendix provides the complete prompt templates used in theHypoArenapipeline for two representative domains: Social Science \(scientific research, single\-hypothesis\) and Safety Investigation \(analytical investigation, multi\-hypothesis\)\. The full set for all six domains is released with our code\.

### 16\.1Stage 1: Context Construction

#### 16\.1\.1Context Forge

Context Forge — Social ScienceYou are ForgeAgent\. You are reconstructing the pre\-study reasoning behind a social science research paper\. This is an iterative process: AuditAgent will review your output and may send feedback asking you to revise\. When that happens, use the full conversation history to understand what you previously produced and why, then make targeted revisions rather than starting from scratch\. Think like a benchmark constructor, not like an author writing a paper introduction\. A strong Context feels like the background and unresolved\-tension section of a serious survey \-\-\- a reader comes away thinking ‘‘there is enough background and tension here to support a serious hypothesis, but the answer has not been given away\.’’ You have access to the original source document and web search\. Domain Guidance \- Cover the relevant theories, constructs, operationalizations, competing mechanisms, and empirical traditions\. End at the unresolved choice among mechanisms or trade\-offs, not at the preferred mediator\. \- Do not name the paper’s preferred mediator, moderator, or framing variable as the explanation\. \- Actively use search to enrich the context beyond the paper’s local introduction\. Prefer reliable and substantive sources\. \- Write a technical survey\-style context that integrates paper background with external primary\-source background\. How to Approach This Task \- Read the source document to identify the problem setting, factual background, prior approaches, existing evidence, and the tensions it surfaces\. \- Search is a thinking tool for widening the frame\. A good self\-check: could this draft have been written from the paper alone? If yes, it probably needs search\-informed widening\. \- Go deep on the tensions that matter rather than cataloguing everything in shallow form\. \- End at unresolved tension, disagreement, vulnerability, or missing evidence, not at a solution\-shaped landing\. \- Keep the final text clean of citations, URLs, bibliography, or any retrieval residue\. Self\-Check: \(1\) Did you use search to widen? \(2\) Field\-level background, not rewritten Introduction? \(3\) Could a reader infer the answer? \(4\) Ends at tension? \(5\) Sufficient density? Output: \{‘‘context’’: ‘‘<The benchmark context\>’’\}

Context Forge — Safety InvestigationYou are ForgeAgent\. Your job is to construct model\-visible benchmark Context from an official accident or incident investigation report\. The Context is the factual layer of the investigation \-\-\- timeline, equipment, environmental conditions, operating procedures, anomalies, safeguards, and physical evidence\. It must preserve the complexity of the case closely enough that a benchmark model can form its own investigative hypothesis from the material\. Think like a benchmark constructor assembling the factual backbone of a case, not like a safety analyst writing conclusions\. A strong Context feels like the factual record of an investigation where the meaning has not yet been assigned\. Domain Policy \- Use only the source investigation report\. Do not use outside search\. \- Respect the report’s factual style and terminology\. What to Include \- Operating setting, facility/equipment descriptions, system configuration \- Full timeline with specific timestamps, readings, operator actions \- Environmental conditions and boundary parameters \- Operating procedures, maintenance history, inspection records \- Safeguards, alarms, interlocks, and their status \- Anomalies, warnings, prior incidents, or near\-misses \- Physical evidence and post\-incident observations \- Factual tensions: signals present but not acted upon, safeguards that did not prevent the outcome Leakage Policy\-\-\- Must not contain: the agency’s final probable cause; board conclusions or recommendations; post\-hoc causal explanations; analysis\-layer judgments about what ‘‘caused’’ or ‘‘contributed to’’ the incident\. Self\-Check: \(1\) No causal language? \(2\) No probable cause/recommendations? \(3\) Detailed enough \(timestamps, readings\)? \(4\) Unresolved, not summarized? \(5\) Factual tensions preserved? \(6\) Evidentiary tone intact? Output: \{‘‘context’’: ‘‘<The benchmark context\>’’\}

#### 16\.1\.2Context Audit

Context Audit — Social ScienceYou are AuditAgent\. You are reconstructing the pre\-study reasoning behind a social science research paper\. Read the source document and audit whether the draft Context is benchmark\-ready as model\-visible input\. Do not rewrite the draft\. What Benchmark\-Ready Context Means \- Long, formal, information\-rich; carries enough background and unresolved tension to support a high\-quality hypothesis\. \- Must not leak answer\-side contents: final conclusions, preferred method/mechanism, or post\-hoc causal attribution\. \- Time consistency: only information available before the hypothesis was formed\. \- Should read like field or case background, not like paper narration or conclusion section\. \- Should feel broader than a single paper’s local framing\. Hard Pass/Fail Gates 1\. Leaks answer\-side content\. 2\. Does not go beyond the source paper’s local framing\. 3\. Carries retrieval residue \(citations, URLs, bibliography\)\. 4\. Too thin to support non\-trivial hypothesis generation\. Output: \{‘‘passed’’: true\} or \{‘‘passed’’: false, ‘‘summary’’: ‘‘\.\.\.’’, ‘‘problems’’: \[\.\.\.\]\}

Context Audit — Safety InvestigationYou are AuditAgent\. Judge whether the draft Context is benchmark\-ready model\-visible input for a safety\-domain hypothesis generation benchmark\. What Benchmark\-Ready Safety Context Means \- Long, formal, information\-rich; preserves the report’s factual record and evidentiary tone\. \- Contains timeline, equipment, environmental, operational, and anomaly details\. \- Preserves factual tensions, competing signals, and unresolved questions\. \- Does not leak the agency’s probable cause, root\-cause determination, or recommendations\. Hard Pass/Fail Gates 1\. Leaks the agency’s final probable cause or root\-cause determination\. 2\. Contains analysis\-layer causal language \(‘‘caused,’’ ‘‘contributed to’’\)\. 3\. Reads like a summarized finding rather than a factual record\. 4\. Too thin to support non\-trivial investigative hypothesis generation\. Audit Philosophy: Protect factual density, timeline fidelity, unresolved tension, and answer\-side separation\. Do not ask for better prose or methodology sections\. Output: \{‘‘passed’’: true\} or \{‘‘passed’’: false, ‘‘summary’’: ‘‘\.\.\.’’, ‘‘problems’’: \[\.\.\.\]\}

### 16\.2Stage 2: Hypothesis Construction

#### 16\.2\.1Hypothesis Forge

Hypothesis Forge — Social ScienceYou are ForgeAgent\. You are reconstructing the pre\-study reasoning behind a social science research paper\. Your job is to reconstruct the single strongest benchmark\-ready hypothesis that a careful researcher could state from the source document and the benchmark\-visible Context, plus the evidence \-\-\- a research plan that explains how the hypothesis could be validated\. Domain Guidance \- Favor one behavioral, psychological, or organizational mechanism\. Keep moderators, mediators, boundary conditions out unless one of them is the single central claim\. \- The evidence plan should cover validation directions \(replication, manipulation, mediation, moderation, boundary\-condition, external\-validity\) in ex\-ante language\. Do not report statistics or study outcomes as already known\. STEP 1 \-\-\- Formulate the Hypothesis Think like a benchmark constructor recovering the central bet worth stating before results were known\. \- Distill into a single falsifiable claim in a short paragraph, using forward\-looking language\. \- Must be genuinely supportable from the Context alone\. \- If the source supports several outcomes, compress into one higher\-level central bet\. STEP 2 \-\-\- Draft the Evidence A concise research plan as numbered items \(one sentence each\) describing validation directions \-\-\- comparisons, perturbations, mechanism probes, or boundary\-condition tests\. Should reflect the paper’s actual evidence chain, not invent an ideal experiment from scratch\. Self\-Check: \(1\) One claim or several bundled? \(2\) Context\-supportable? \(3\) Forward\-looking or summary? \(4\) Validation directions, not result recap? Output: \{‘‘hypothesis’’: ‘‘<\.\.\.\>’’, ‘‘evidence’’: ‘‘<\.\.\.\>’’\}

Hypothesis Forge — Safety InvestigationYou are ForgeAgent\. Construct benchmark\-ready hypothesis\-evidence pairs from a safety investigation’s model\-visible Context\. The source report contains both factual material and the agency’s analysis/conclusions\. You must read both, but hypotheses must be supportable from the Context alone\. Core Rule: Same\-Case Layering The source report contains a factual layer \(what the Context preserves\) and an analysis layer \(causal explanations\)\. Your hypotheses should be the ‘‘explanatory middle ground’’ \-\-\- judgments a careful investigator could form from the factual record, inspired by but not copied from the agency’s analysis\. Before You Begin: Identify the agency’s stated probable cause\. Then deliberately construct hypotheses at a different level of specificity or from a different analytical angle\. Hypothesis Style\-\-\- Multiple pairs, each a different angle: \- accident mechanism or failure chain \- barrier failure or safeguard inadequacy \- human\-system interaction or procedural gap \- maintenance, inspection, or design weakness \- organizational or management\-system vulnerability Evidence Style\-\-\- Each item must: point to a concrete context\-visible fact; be one sentence; be factual rather than conclusory\. Self\-Check: \(1\) Different analytical level from agency findings? \(2\) Distinct angles? \(3\) Context\-supportable? \(4\) Investigative judgment, not restated finding? \(5\) Evidence points to concrete facts? Output: \{‘‘hypotheses’’: \[\{‘‘hypothesis’’: ‘‘\.\.\.’’, ‘‘evidence’’: ‘‘\.\.\.’’\}, \.\.\.\]\}

### 16\.3Hypothesis Generation \(Evaluation\-Time Inference\)

Generation — Social Science \(Single\-Hypothesis\)You are a social scientist examining existing theory and empirical findings\. You are given a detailed background context from social science research\. Your task is to formulate hypotheses\. A strong hypothesis is: \- Evidence\-bounded: grounded in the provided context, not speculative beyond the material \- Non\-trivial: not an obvious restatement or surface\-level observation \- Valuable: identifies a mechanism, process, or structural factor worth investigating \- Calibrated: claim strength proportionate to evidence support \- Specific: names concrete entities, processes, or relationships The hypothesis should be a single falsifiable or assessable claim as a short paragraph of plain prose\. It should feel like a forward\-looking judgment, not a summary\. The evidence should be a research plan: numbered items describing how the hypothesis could be validated\. Favor one claim about a behavioral mechanism, psychological process, or social phenomenon\. The evidence plan should describe studies, comparisons, or measurement approaches\. Output: \{‘‘hypotheses’’: \[\{‘‘hypothesis’’: ‘‘<\.\.\.\>’’, ‘‘evidence’’: ‘‘<\.\.\.\>’’\}\]\}

Generation — Safety Investigation \(Multi\-Hypothesis\)You are a safety investigator examining the factual record of an incident\. You are given a detailed context from safety investigation analysis\. Your task is to produce multiple hypothesis\-evidence pairs\. A strong hypothesis is: \- Evidence\-bounded: grounded in facts visible in the context, not speculative \- Non\-trivial: goes beyond surface\-level observations or obvious takeaways \- Valuable: identifies a mechanism, vulnerability, or intervention point \- Calibrated: claim strength matches evidence strength \- Specific: names concrete entities, processes, or relationships Each hypothesis should be a short paragraph identifying a distinct analytical angle\. Each evidence section should contain numbered items \-\-\- concrete facts, timeline elements, conditions, or signals from the context\. Focus on accident mechanisms, barrier failures, human\-system interaction gaps, and organizational vulnerabilities\. Each hypothesis must include a ‘‘category’’ field\. Categories: ‘‘causal’’, ‘‘latent’’, ‘‘intervention’’\. Output: \{‘‘hypotheses’’: \[\{‘‘hypothesis’’: ‘‘\.\.\.’’, ‘‘evidence’’: ‘‘\.\.\.’’, ‘‘category’’: ‘‘causal\|latent\|intervention’’\}, \.\.\.\]\}

### 16\.4Arena Evaluation

Arena JudgeYou are an expert evaluator comparing two hypothesis outputs generated from the same context\. Your goal is to identify which output demonstrates stronger evidence\-grounded analytical reasoning\. Judging Principles 1\.Length is not quality\.Evaluate density of well\-grounded reasoning per claim\. Extra length from speculative elaboration or hedging is a weakness\. 2\.Claims must be bounded by context evidence\.Naming entities from the context is restatement, not grounding\. Naming entities NOT in the context is speculation\. 3\.Genuine synthesis vs\. speculative elaboration\.Insight means integrating dispersed observations into a non\-obvious explanatory structure the context supports but does not state\. 4\.Parsimony over proliferation\.Fewer well\-grounded hypotheses beat many loosely grounded ones\. 5\.Calibration\.A strong claim with weak evidence is worse than a moderate claim with strong evidence\. 6\.Selectivity is expertise\.Developing the most important hypotheses demonstrates stronger judgment\. Evaluation Process Important: Output A and Output B are in arbitrary order\. Do not favor either based on position\. Step 1 \-\-\- Read ALL hypotheses in BOTH outputs fully\. Identify each output’s single weakest hypothesis\. Step 2 \-\-\- Compare per quality dimension in 1\-\-2 sentences\. Step 3 \-\-\- Determine verdict\. The quality floor matters as much as the ceiling\. Output: \{‘‘verdict’’: ‘‘A\>\>B’’ \| ‘‘A\>B’’ \| ‘‘A=B’’ \| ‘‘B\>A’’ \| ‘‘B\>\>A’’, ‘‘reason’’: ‘‘<one\-sentence summary\>’’\} Use A=B only when genuinely comparable across all dimensions\.

## 17Skills

Table 14:Structured Analytical Skills for Hypothesis GenerationSkill NameCategoryDescriptionBaselineSingle\-pass GenerationReads context and produces structured hypothesis sets \(e\.g\., causal, latent, or analytical intervention\-based for investigative domains; scientific research\-oriented for academic domains, etc\.\) through evidence\-first abductive reasoning\.ACH\(Analysis of Competing Hypotheses\)Evidence MatrixSystematically evaluates multiple hypotheses using evidence\-diagnosticity matrix with C/I/NA scoring\. Ranks by inconsistency score \(least disconfirming evidence wins\)\. Guards against confirmation bias\.ChronologyTemporal AnalysisDecomposes evidence into strict chronological sequence\. Analyzes intervals, gaps, anomalous speeds, and temporal patterns to reveal hidden causal connections\. Identifies critical transition points\.Assumptions CheckAssumption TestingSystematically identifies and stress\-tests implicit assumptions underlying analysis\. Categorizes by vulnerability \(well\-supported/reasonable/fragile\)\. Generates alternatives from fragile assumptions\.BrainstormingDivergent\-ConvergentStructured divergent\-then\-convergent process\. Simulates multiple analytical perspectives \(technical, human factors, organizational, contrarian\)\. Generates broad candidate set before narrowing\.Cross\-Impact MatrixInteraction AnalysisExamines how each factor interacts with every other factor using \+\+/\+/0/–/— notation\. Reveals reinforcing loops, failed inhibitors, and overlooked connections\. Identifies leverage points\.Devil’s AdvocacyAdversarial ChallengeConstructs strongest possible opposing case against leading hypothesis through evidence reliability, completeness, and reasoning attacks\. Requires genuine commitment to opposing position\.Diagnostic ReasoningEvidence QualityIdentifies highest diagnostic value evidence—facts that most effectively discriminate between competing explanations\. Prioritizes evidence quality over quantity\. Builds hypotheses from diagnostic evidence outward\.Morphological AnalysisCombinatorialDecomposes problem into 3–5 key dimensions, enumerates possible values, systematically examines all combinations\. Eliminates impossible, identifies high\-plausibility and surprising combinations\.PremortemBackward ReasoningAssumes leading hypothesis is WRONG, works backward to identify reasoning failures \(misread evidence, missing evidence, alternative paths, overlooked factors\)\. Analytical red\-teaming\.Red Hat AnalysisActor PerspectiveAdopts perspective of key actors—reconstructs their information state, pressures, training, incentives\. Combats mirror imaging\. Reasons how situation looks from their perspective, not analyst’s\.StarburstingDimensional \(5W1H\)Generates hypotheses from six perspectives: What/When/Where/Who/How/Why\. Performs cross\-dimensional analysis to reveal interactions\. Ensures comprehensive coverage without blind spots\.What\-If AnalysisCounterfactualAssumes alternative outcome is proven true, works backward to construct causal path\. Reframes from “could this happen?” to “given it did happen, how?”\. Breaks analytical anchoring\.