Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation

# Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation
Source: [https://arxiv.org/html/2605.21491](https://arxiv.org/html/2605.21491)
Srujan P Mule1,2Aniketh Garikaparthi2Manasi Patwardhan2

1IISER Pune2TCS Research srujan\.mule@students\.iiserpune\.ac\.in \{aniketh\.g, manasi\.patwardhan\}@tcs\.com

###### Abstract

As language models accelerate scientific research by automating hypothesis generation and implementation, a new bottleneck emerges: evaluating and filtering hundreds of AI\-generated ideas without exhaustive experimentation\. We ask whether LMs can learn to forecast the empirical success of research ideas before any experiments are run\. We study*comparative empirical forecasting*: given a benchmark\-specific research goal and two candidate ideas, predict which will achieve better benchmark performance\. We construct a dataset of 11,488 idea pairs grounded in objective outcomes from PapersWithCode\. While off\-the\-shelf 8B\-parameter models struggle \(30% acc\.\), SFT dramatically boosts performance to 77\.1%, outperforming GPT\-5 \(61\.1%\)\. By framing evaluation as a reasoning task via Reinforcement Learning with Verifiable Rewards \(RLVR\), we train models to discover latent reasoning paths, achieving 71\.35% acc\. with interpretable justifications\. Through additional ablations and out\-of\-distribution tests, we show robustness to surface\-level heuristics and transfer to both a cross\-domain time\-split test set and an independently constructed test set\. Our results demonstrate that compute\-efficient small language models can serve as effective, objective verifiers, offering a scalable path for autonomous scientific discovery\.

Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation

Srujan P Mule1,2Aniketh Garikaparthi2Manasi Patwardhan21IISER Pune2TCS Researchsrujan\.mule@students\.iiserpune\.ac\.in\{aniketh\.g, manasi\.patwardhan\}@tcs\.com

## 1Introduction

![Refer to caption](https://arxiv.org/html/2605.21491v1/x1.png)Figure 1:We explore various methods to fine\-tune 8B Parameter Language Models using our constructed dataset\. The figure illustrates the potential of our fine\-tuned model to accurately judge the quality of scientific ideas\. For the two given benchmarks and an overlapping method, our model robustly predicts which idea will perform better in context of the benchmark, while providing insightful reasoning\.Language Models are starting to function as autonomous research agents that can generate hypotheses, run experiments, and analyze results\(Luet al\.,[2024](https://arxiv.org/html/2605.21491#bib.bib5); Yamadaet al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib25); Gridachet al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib26)\)\. A recurring pattern in these systems is high\-throughput ideation, where the model generates hundreds of candidate methods for a given scientific goal\(Baeket al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib28); Siet al\.,[2024](https://arxiv.org/html/2605.21491#bib.bib12); Garikaparthiet al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib1)\)\. This scale makes filtering “good ideas” crucial, as running hundreds of experiments is infeasible\. However, current evaluation approaches rely on language\-model judgments over*subjective*criteria like “excitement”, “innovativeness” or “novelty”\(Wanget al\.,[2024](https://arxiv.org/html/2605.21491#bib.bib11); Baeket al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib28); Huet al\.,[2024](https://arxiv.org/html/2605.21491#bib.bib27)\)\. While helpful, these metrics are often just proxies; an idea can be novel and well\-argued but still fail to work in practice\(Siet al\.,[2024](https://arxiv.org/html/2605.21491#bib.bib12); Zhuet al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib34)\)\.

This gap motivates our study of*comparative empirical forecasting*: given a research goal and two candidate ideas, predict which idea will achieve better performance when evaluated on a benchmark\. While these objective outcomes are very hard to predict, researchers routinely form useful intuitions from patterns across prior work to do so\. We ask whether language models can be trained to internalize such priors and discriminate between two competing ideas*before*running experiments\. Potentially, such a verifier model could complement the scale of generator models by shortlisting stronger ideas through pairwise comparison, minimizing the pool of candidates for implementation\.

While recent work has begun exploring this direction by constructing datasets of idea comparisons\(Wenet al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib17)\), our goal is to push this setting toward \(i\)*fine\-grained prediction*over specific benchmarks rather than coarse aggregation; \(ii\)*compute\-efficient*models that are broadly accessible, and \(iii\)*interpretable reasoning*which can clarify the intuition behind predictions\. Figure[1](https://arxiv.org/html/2605.21491#S1.F1)illustrates how our fine\-tuned 8B parameter model can make fine\-grained predictions in context of the benchmark, while producing insightful reasoning to support its prediction\.

To support this task, we construct a large\-scale dataset by scraping public benchmark leaderboards to retrieve linked papers, and extract \(a\) benchmark\-specific research goals, \(b\) descriptions representing competing ideas, and \(c\) empirical scores determining winners\. This produces 11,488 labeled idea pairs across 724 valid benchmark leaderboards, grounded in objective outcomes\.

As a first step, we cast*comparative empirical forecasting*as a direct preference\-prediction problem and fine\-tune language models to output a binary winner label\. This black\-box formulation serves as a simple baseline following prior work\(Wenet al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib17)\)\. Further to encourage and capture the intermediate reasoning, we use a two\-stage training process\. First, we use Supervised Fine\-Tuning \(SFT\) on two curated dataset with reasoning traces alongside labels: a synthetically obtained subset from a large teacher model; a much smaller set grounded in contents of papers\. Then fine\-tune using Reinforcement Learning \(RL\) variants\(Jiaet al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib16); Shaoet al\.,[2024](https://arxiv.org/html/2605.21491#bib.bib21); Yuet al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib23)\), to let the model explore and discover the reasoning paths which lead to the correct prediction\.

Our results show that while base models struggle \(Qwen3\-8B achieving only 20\.13% accuracy\), supervised fine\-tuning dramatically improves performance \(77\.1%\)\. Variants trained to output interpretable reasoning achieve 71\.35% accuracy, outperforming GPT\-5 \(61\.10%\) by over 10 percentage points while being substantially more compute\-efficient and interpretable\. Our models remain robust to stress tests on paraphrasing and recency, length and position bias, suggesting genuine task understanding rather than learning superficial heuristics\. Our main contributions are:

- •We introduce a large\-scale dataset of research idea pairs with*benchmark\-specific*research goals and outcomes, enabling fine\-grained comparative forecasting \(§[4](https://arxiv.org/html/2605.21491#S4)\)\.
- •We demonstrate that 8B\-parameter models can outperform even frontier models on comparative scientific forecasting after fine\-tuning \(§[7\.1](https://arxiv.org/html/2605.21491#S7.SS1)\) and show non\-trivial cross\-domain generalization to non\-NLP benchmarks and externally constructed datasets\.
- •We show that through careful training with reinforcement learning, models can produce coherent explanations to justify their predictions \(§[7\.3](https://arxiv.org/html/2605.21491#S7.SS3)\)\.
- •We analyze robustness under paraphrasing and presentation shifts to assess whether models rely on brittle heuristics \(§[7\.4](https://arxiv.org/html/2605.21491#S7.SS4)\)\.

Finally, we present insights into the strengths and weaknesses of our work highlighting directions for future work on scientific forecasting\.

## 2Problem Statement

We define this problem formally as follows: Letℋ\\mathcal\{H\}be the space of scientific hypotheses \(ideas\),𝒢\\mathcal\{G\}be the space of research goals and𝒞\\mathcal\{C\}be the space of reasoning traces\. We construct a dataset𝒟=\{g,hA,hB,\(c\),y\}\\mathcal\{D\}=\\\{g,h\_\{A\},h\_\{B\},\(c\),y\\\}, where bothhAh\_\{A\}andhBh\_\{B\}∈ℋ\\in\\mathcal\{H\}, are textual descriptions of two competing ideas,g∈𝒢g\\in\\mathcal\{G\}is the specific research goal for which the ideas are implemented \(e\.g using One class SVM vs CNN\-BiLSTM with the goal of detecting modern cyber threats in high\-volume network traffic while minimizing false positives\), andy∈\{0,1\}y\\in\\\{0,1\\\}is a binary label wherey=0y=0implies thathAh\_\{A\}outperformshBh\_\{B\}on goalggandc∈𝒞c\\in\\mathcal\{C\}denotes an optional chain\-of\-thought explaining why one idea outperforms the other\. Our objective is to learn a parameterized policyπθ\\pi\_\{\\theta\}that accurately predictsyygiven the context of the ideas and the goal, while generating chain of thought reasoning traceccbefore prediction\.

## 3Related Work

##### Research Ideation

Research ideation, being inherently language\-intensive, benefits significantly from advances in LLMsWanget al\.\([2024](https://arxiv.org/html/2605.21491#bib.bib11)\); Baeket al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib28)\); Siet al\.\([2024](https://arxiv.org/html/2605.21491#bib.bib12)\)\. Recent efforts leverage frontier LLMs via retrievalLiet al\.\([2024](https://arxiv.org/html/2605.21491#bib.bib39)\), test\-time computeHuet al\.\([2024](https://arxiv.org/html/2605.21491#bib.bib27)\), or multi\-agent debateSuet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib40)\)\. In contrast, few works fine\-tune open\-source models for better research ideationO’Neillet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib14)\)\. Crucially, these ideas frequently fail to translate into real\-world empirical improvementsZhuet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib34)\); Siet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib41)\)\.

##### Evaluation Methodologies

Most systems evaluate candidates using LLM\-judges augmented with retrieval or agentsBaeket al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib28)\); Garikaparthiet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib1)\)\. Assessments are typically rubric\-driven, focusing on novelty, feasibility, and clarityLiet al\.\([2024](https://arxiv.org/html/2605.21491#bib.bib39)\), occasionally calibrated via human studiesSiet al\.\([2024](https://arxiv.org/html/2605.21491#bib.bib12)\)\. Methodologically, scoring relies on absolute ratingsBaeket al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib28)\)or aggregated pairwise rankingsSiet al\.\([2024](https://arxiv.org/html/2605.21491#bib.bib12)\); Garikaparthiet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib1)\)\. Verifiers that reward objective performance beyond surface plausibility remain largely underexploredWenet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib17)\)\.

##### LLMs for Forecasting

LLMs have demonstrated potential as forecasters of real\-world events, approaching competitive human crowd benchmarksHalawiet al\.\([2024](https://arxiv.org/html/2605.21491#bib.bib37)\); Kargeret al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib36)\)\. Several works employ specialized training for such tasksLeeet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib29)\); Chandaket al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib38)\)\. Closest to our setting are efforts to forecastempirical ML outcomes:Wenet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib17)\)train GPT\-4\.1 to predict better\-performing ideas from pairs, whileParket al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib35)\)estimate benchmark scores from textual descriptions without experimentation\.

![Refer to caption](https://arxiv.org/html/2605.21491v1/x2.png)Figure 2:Dataset Construction PipelineWe use raw entries from 1,918 NLP leaderboards to construct a statistically grounded idea pairs with a benchmark specific research goal, while difficulty stratification ensures robust evaluation across diverse research goals\.

## 4Benchmark

A sample in our benchmark consists of: \(i\)Idea Pair:Detailed descriptions of two competing methods \(i​d​e​aAidea\_\{A\},i​d​e​aBidea\_\{B\}\), grounded in their scientific publications\. \(ii\)Research Goal:A clear statement of the specific evaluation objective of a benchmark for which the ideas are implemented \(iii\)Binary Label:A label \(0 or 1\) indicating which idea achieved a higher empirical score on that specific benchmark\. We develop a pipeline to construct a benchmark dataset of idea pairs, transforming raw leaderboards into statistically grounded comparisons\. The process involves:

##### Scraping and Paper Collection\.

We extract ideas from entries in live leaderboards, this allows us to build comparisons specific for each benchmark\. Thus our evaluation becomes more fine\-grained in comparison to parallel workWenet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib17)\), which can potentially conflate evaluations due to aggregation of scores across various benchmarks via majority voting\.

We first scrape all available NLP leaderboards frompaperswithcode\.comthat have at least two entries\. This yields 1,918 benchmark leaderboards\. For each entry in a leaderboard we identify a referenced paper , resulting in 5,713Result\-Reporting \(RR\)papers \(excluding 7 behind paywall\)\. We observe for some instances, the RR paper is not theOriginalpaper which introduced the method, but rather the paper reporting results on the benchmark using that method\. Relying on such papers for idea extraction \(to be done in the later stage\) would result in generic or incomplete descriptions\. Therefore, we prompt an LLM \(Gemini\-2\.5\-pro, prompt in Appendix[C](https://arxiv.org/html/2605.21491#A3)\) to verify whether each RR paper is the one that originally introduces the idea , and if not, look at the full content of RR paper for citations and the reference section to find the original paper\. We also ask the LLM to report the confidence of its analysis \(high/medium/low\)\. Two NLP experts then manually process the low\-confidence entries to verify the identified original paper citation and correct if necessary\.We download an additional 908 Original papers based on this analysis\. All downloaded papers are parsed usings2orc\-doc2json111[https://github\.com/allenai/s2orc\-doc2json](https://github.com/allenai/s2orc-doc2json)to convert the full text into Markdown format, providing clean and structured input for subsequent processing\. Papers with unresolvable parsing errors are discarded, resulting in 5,695 RR and 832 Original markdown papers\.

##### Research Goal and Idea Extraction\.

For each one of the 1918 leaderboards, we extract a single canonical research goal from official benchmark descriptions in the following order of sources: \(1\) the dataset page on[paperswithcode\.com](https://arxiv.org/html/2605.21491v1/paperswithcode.com), \(2\) the corresponding dataset file from thepwc\-archive222[https://huggingface\.co/datasets/pwc\-archive/datasets](https://huggingface.co/datasets/pwc-archive/datasets), or \(3\) the RR paper when the above sources are unavailable \(for 278 benchmarks\)\. The extracted textual description is provided as input to an LLM \(Prompt in Appendix[C](https://arxiv.org/html/2605.21491#A3)\), which generates a clear, comprehensive research goal including what the benchmark evaluates\. 327 such benchmarks with missing or unusable sources are skipped\.

We process each RR and original markdown paper corresponding to the leaderboard entries with an LLM to extract the detailed idea, excluding any details, empirical results, comparisons, unique identifier like author/model names, year etc\. The LLM \(Prompt in Appendix[C](https://arxiv.org/html/2605.21491#A3)\) has access to the complete paper context capturing all necessary details like algorithms, mathematical details etc\., whenever present\. This results in 9,722 total ideas\.

##### Train–Test Split\.

We construct the train–test division based on the ideas from the extracted papers belonging to each leaderboard entry by iterating over the leaderoboards\. Within a leaderboard, ideas are first grouped by their publication year into respective time buckets\. Buckets with fewer than five unique papers are merged with adjacent years\. Ideas in each time bucket are then split in an 80/20 ratio\. As we iterate through leaderboards, we ensure that if an idea is already assigned to a split in a previous iteration, this assignment is strictly maintained\. Leaderboards with fewer than four total ideas are assigned entirely to the training set\. This approach helps prevent an idea appearing in both the train and test avoiding information leakage, while also ensuring similar temporal distribution of the ideas\. This process yields 892 leaderboards, with 2,893 ideas in the training set and 693 in the test set\.

##### Details of manual verification of the test set:

To ensure that the LLM based idea extractions are accurate, we manually verify the correctness of the idea summary extracted from the parsed papers by consulting the original PDFs\.

We observe that the errors/inaccuracies can be classified into 2 main categories:\(i\)Incomplete \(∼\\sim4%\): When the summary falls short of the full detail necessary to correctly summarize the idea\. For example, in one instance the description did not include the special loss function that was introduced as part of the idea\. In such case we add the necessary details\. \(ii\)Incorrect \(∼\\sim8%\): When the details of the ideas doesn’t correspond to actual ideas/methods\. These can be further classified into 2 cases:\(a\)Minor:Minor mistakes like wrong output dimension etc\.\. Necessary changes are made in such cases\. \(b\)Major:When the summary of the idea is completely incorrect\. For example, hallucinated details like adversarial component in a BERT based system, when the original method involves simple fine\-tuning\. We remove such ideas\. Importantly, the verification criteria explicitly included checking for successful exclusion of empirical results and outcome statements from idea descriptions \(lines of exclusion criteria in the extraction prompt\)\. Ideas that contained residual performance comparisons or result statements were corrected or removed accordingly\.

##### Unified Score and Pair Formation

We next construct idea pairs within each benchmark\. Relying solely on the benchmark leaderboard ranks to decide the winner of an idea pair can be misleading, as the performance gap between ranks can vary significantly across leaderboards\. To obtain a consistent and quantitative basis for comparison, we compute aUnified Scorefor every idea within a benchmark\. For each benchmark, we first apply min–max normalization for the results of each metric, across all entries and handle “lower\-is\-better” metrics \(e\.g\., perplexity\) by inversion\. Approximately 85% of benchmarks report only a single metric\. If there is more than one metric, the normalized results are averaged across all to yield a Unified Score for each idea\. This procedure captures the relative performance distribution within a benchmark, regardless of metric scale or density\.

All possible\(n2\)\\binom\{n\}\{2\}ideas pairs are generated for each benchmark within each split\. We calculate the standard deviation \(σ\\sigma\) of the Unified Scores across all entries in that benchmark and use it to define normalized score differences \(Δ\\Delta\) for each pair\. Based onΔ\\Delta, we categorize pairs into three mutually exclusive difficulty tiers based on how close their unified scores are:1σ\\sigma\(hard\),2σ\\sigma\(medium\), and3σ\\sigma\(easy\), using a 20% tolerance margin \(e\.g\., 0\.8σ\\sigma–1\.2σ\\sigmafor 1σ\\sigma\)\. This categorization enables controlled difficulty evaluation based on empirical performance separation\.

##### Labeling and Final Dataset\.

Given a pair\(ideaA,ideaB\)\(\\text\{idea\}\_\{A\},\\text\{idea\}\_\{B\}\), we assigny=0y=0ifideaA\\text\{idea\}\_\{A\}has better Unified Score andy=1y=1otherwise\. Each pair is annotated with the corresponding research goal of its benchmark\. Benchmark leaderboards without a research goal and all the pairs from such benchmarks are removed\. Given the asymmetric removal due to the lack of research goals, we end up with≈90/10\\approx 90/10train/test split in the end\. We manually verify each sample in test We release our dataset333[https://anonymous\.4open\.science/r/Benchmark\-Dataset\-81B0](https://anonymous.4open.science/r/Benchmark-Dataset-81B0)\.

σ\\sigma\-CategoryTrain/w Reasoning\.TestTotal1\-sigma612012049466142\-sigma34614528437453\-sigma10385911129Total1061917086911488

Table 1:Stratified Dataset Statistics of Train, subset of train with reasoning traces and Test Idea pairs
##### Reasoning extraction

Predicting empirical outcomes is challenging because research does not always follow clean, deductive logic; often, the explanatory “reasoning” consists of insights gained onlyafterresults are observed\. Given the reasoning intensive nature of the task, we want Chain\-of\-Thought \(CoT\) reasoning traces to train the models\. We extract CoT in 2 ways based on 2 opposite observations made in the current literature\.

SyntheticRM\-R1\(Chenet al\.,[2026](https://arxiv.org/html/2605.21491#bib.bib2)\)proposes and shows effectiveness of adistillation\-then\-RLVRpipeline for training reasoning reward models: a smaller model is first trained via SFT on structured “Chain\-of\-Rubrics” traces distilled from larger teacher, and then refined through RLVR\. Motivated by this, we extract synthetic Chain\-of\-Rubrics from GPT\-5 \(high reasoning\)\. We randomly sample a small subset of 2125 idea pairs from train\. We first prompt with the research goal and both idea descriptions, similar to the prediction task, asking the model to produce structured rubric\-style reasoning traces that evaluate each idea before concluding which is better\. Next, we retain only traces where GPT\-5’s predicted label matches the ground truth, yielding 1,369 pairs; after swap\-augmentation this produces 2,738 training examples\.

Literature Groundedshow that usingself\-generatedCoT reasoning leads to performance degradation compared to the zero\-shot setting for comparative prediction tasks\. Following this, we take another approach to extract CoT reasoning\. We consider all the idea pairs such that both of the ideas within each pair have the same RR paper\. This way, we can be sure that such comparisons actually exist and are presented within the paper and not a case of inferred reasoning\. We then prompt an LLM \(Prompt in Appendix[C](https://arxiv.org/html/2605.21491#A3)\) to look for the presence of any explanation for the better performance of one method over the other and extract this as a paragraph reflecting the grounded reasoning\. In case such reasoning or justification is not present, the LLM simply has to state that such reasoning is not available\. The input prompt contains the full RR paper and a list of all the methods that were reported in this paper\.

##### Cross domain Test set

We employ similar pipeline as described above to curate a new test set of idea pairs fromnon\-NLPleaderboards \(e\.g\. Speech synthesis, Molecular property prediction etc\.\) from PapersWithCode that have at least 3 entries, with the years of RR papers≥2024\\geq 2024\. Unlike the previous case, we use GPT\-5 with high reasoning for idea extraction to introduce linguistic diversity relative to the training distribution\. Additionally, we do not categorise the pairs based on difficulty, instead use individual metrics directly to form all\(n2\)\\binom\{n\}\{2\}pairs\. This results in 705 idea pairs across 46 leaderboards\. This dataset allows us to test if the trained models can pay attention to metric based nuances \(i\.e\. characterize sensitivity to metric based nuances\), while testing for their ability to generalize to cross domains and taking care of the potential knowledge\-cutoff confound\.

##### Independently Constructed Test Set\.

We obtained access to an independently constructed test setWenet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib17)\); sourced from diverse domains and no overlap with any ideas in the train set\. This resulted in 1750 idea pairs with majority voting labels across multiple benchmarks\.

## 5Methodology

We first treat our task as a binary classification problem, and later probe models to also output interpretable reasoning for their judgments\. Hence we train under two distinct settings:Supervised Fine\-Tuning \(SFT\)for direct prediction andReinforcement Learning \(RL\)for latent reasoning\.

### 5\.1Supervised Fine\-Tuning \(SFT\)

Base models \(LMs\) typically lack the comparative intuition needed to map idea differences to benchmark performance\. To bridge this gap, we employ standard SFT to train the model so that it can learn and do better at this task\.

The model is provided with the research goalggand the descriptions of two ideashA,hBh\_\{A\},h\_\{B\}\. The target output is simply the binary labely∈\{0,1\}y\\in\\\{0,1\\\}corresponding to the empirically superior idea\. We apply the following SFT loss:

ℒS​F​T=−log⁡P​\(y∣g,hA,hB\)\\mathcal\{L\}\_\{SFT\}=\-\\log P\(y\\mid g,h\_\{A\},h\_\{B\}\)\\\\ \\vskip\-5\.0pt\(1\)
This phase utilizes the full train dataset to ground the model in the “scientific intuition” of identifying successful ideas\.

### 5\.2Reinforcement Learning \(RL\)

Additionally, we want the model to reason before making a prediction\. We treat reasoning as a latent variable to be optimized via Reinforcement Learning\. We structure this as a two\-step process within the RL framework\.

##### 1\. Cold Start Finetuning \(SFT\-Reasoning\)

Initial experimentation with pure RL \(DAPO\) applied to the LM, using the reward and objective function defined in the next paragraph, revealed consistent reward hacking\(Amodeiet al\.,[2016](https://arxiv.org/html/2605.21491#bib.bib10)\)and generation of incomplete reasoning traces\. To address these problems and to ensure that the model can generate coherent reasoning structures, we fine\-tune the model on the small subset of available reasoning traces\. We apply Cross Entropy \(CE\) loss during fine\-tuning:

ℒCE=−∑t=1Tlog⁡P​\(ot∣g,hA,hB,o<t\)\\mathcal\{L\}\_\{\\mathrm\{CE\}\}=\-\\sum\_\{t=1\}^\{T\}\\log P\(o\_\{t\}\\mid g,h\_\{A\},h\_\{B\},o\_\{<t\}\)\(2\)whereg,hA,hBg,h\_\{A\},h\_\{B\}constitute the input context and the target sequence\(o1,…,oT\)\(o\_\{1\},\\dots,o\_\{T\}\)represents the reasoning traceccfollowed by the labelyy\.

This step aims to teach models astyleof scientific argumentation \(e\.g\., ”Idea A reduces variance by… therefore it is likely to outperform Idea B”\) and align well with human or Chain\-of\-Rubrics from a strong teacher model\.

##### 2\. Variants of Group Relative Policy Optimization \(GRPO\)

We initialize our policyπθ\\pi\_\{\\theta\}and train on the remaining dataset using two variants of Group Relative Policy Optimization \(GRPO\) that address some of it’s limitations\(Yuet al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib23); Liuet al\.,[2025b](https://arxiv.org/html/2605.21491#bib.bib24)\)\. For a given input contextx=\(g,hA,hB\)x=\(g,h\_\{A\},h\_\{B\}\), the model samples a group ofGGoutputs\{o1,…,oG\}\\\{o\_\{1\},\\dots,o\_\{G\}\\\}, where each outputoi∈Go\_\{i\}\\in Gcomprises a generated reasoning tracecic\_\{i\}and a predictiony^i\\hat\{y\}\_\{i\}\. The policy is optimized using the advantage estimateAiA\_\{i\}derived from the group rewards\. By optimizing for the final outcome, the model is incentivized to discover reasoning tracesccthat lead to the correct empirical predictionyy\.

##### Reward Function

The total rewardR​\(o\)R\(o\)is the sum of a correctness scorercorr\_\{\\text\{cor\}\}and a formatting scorerfmtr\_\{\\text\{fmt\}\}\. We define these components as:

rcor​\(o\)\\displaystyle r\_\{\\text\{cor\}\}\(o\)=\{\+3\.0if​y^=y−3\.0otherwise\\displaystyle=\\begin\{cases\}\+3\.0&\\text\{if \}\\hat\{y\}=y\\\\ \-3\.0&\\text\{otherwise\}\\end\{cases\}\(3\)rfmt​\(o\)\\displaystyle r\_\{\\text\{fmt\}\}\(o\)=0\.5​\(𝕀think−𝕀¬think\)⏟“⟨t​h​i​n​k⟩”​\(±0\.5\)\+0\.5​\(𝕀ans−𝕀¬ans\)⏟“Answer:”​\(±0\.5\)\\displaystyle=0\.5\\underbrace\{\(\\mathbb\{I\}\_\{\\text\{think\}\}\-\\mathbb\{I\}\_\{\\neg\\text\{think\}\}\)\}\_\{\\text\{\`\`$\\langle think\\rangle$'' \}\(\\pm 0\.5\)\}\+0\.5\\underbrace\{\(\\mathbb\{I\}\_\{\\text\{ans\}\}\-\\mathbb\{I\}\_\{\\neg\\text\{ans\}\}\)\}\_\{\\text\{\`\`Answer:'' \}\(\\pm 0\.5\)\}\(4\)where𝕀​\(x\)=1\\mathbb\{I\}\(x\)=1if x is true and 0 otherwise\.

##### Objective Functions

The objective functions of DAPO and Dr\. GRPO are based on the probability ratio

ρi,t=πθ​\(oi,t\|x,oi,<t\)πθold​\(oi,t\|x,oi,<t\)\\rho\_\{i,t\}=\\frac\{\\pi\_\{\\theta\}\(o\_\{i,t\}\|x,o\_\{i,<t\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(o\_\{i,t\}\|x,o\_\{i,<t\}\)\}\(5\)

##### DAPO

utilizes global token normalization and decoupled clipping to encourage exploration, while also addressing the known length bias of GRPO during training\.

##### Dr\. GRPO

To fix the length bias \(during training\) in standard GRPO, we use Dr\. GRPO , where the advantage centered but not scaled: the standard deviation term in the denominator is removed\.

## 6Experiments

### 6\.1Metric

We design our evaluation to mitigate LLM’s vulnerabilities to position bias \(e\.g\., strong preference the first item in a tuple\)\. To achieve this, we consider a predictionconsistentif the model predicts the same idea to be better for both original and swapped position pairs\. For accuracy, an idea pair is considered correctly classified only if the prediction is consistentandcorrect\.

### 6\.2Language Models

We evaluate two open\-source models:Qwen3\-8B\(Yanget al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib15)\), hereafter called Qwen3 andLlama3\.1\-8B\-Instruct\(AI@Meta,[2024](https://arxiv.org/html/2605.21491#bib.bib31)\)hereafter called Llama3\.1\. We also use Gemini 2\.5 FlashComaniciet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib32)\)and GPT\-5\(OpenAI,[2025](https://arxiv.org/html/2605.21491#bib.bib30)\)for comparison\. All training was done with BF16 precision on NVIDIA\-A100\-40GB GPUs\.

We name the fine tuned models in the following pattern: \[Direct/\(Synthetic\-\)Reason\]\-\[\(SFT\)\]\-\[\(DAPO/DrGRPO\)\], where the first block tells us type of output: Direct\- just the label prediction; \(Synthetic\-\)Reason\- generate CoT before making final prediction; second block tells us if the models was SF\-Tuned or not; third block corresponds to the RL method used for training\.

Direct label prediction is done with ”reasoning” mode turned off \(no think tokens\) and reasoning models with ”reasoning” turned on in case of Qwen3\.

### 6\.3Training and Hyperparameter

We use the train/test split defined in our benchmark section §[4](https://arxiv.org/html/2605.21491#S4)\(Table[1](https://arxiv.org/html/2605.21491#S4.T1)\)\. We further split the train set into 90% actual train and 10% validation following theσ\\sigma\-category distribution\. SFT:We use LoRA rank of 64, 128α\\alpha, 0\.1 dropout, batch size of 2, learning rate of 2e\-4, weight decay of 0\.01,cosinelearning rate scheduler, 1 epochs and default temperature\. We tune these hyper\-parameters using the validation set\. RL:We use LoRA rank 64, 128α\\alpha, 0 dropout, batch size 1, learning rate 5e\-6, weight decay 0\.01, group size 4,β=1​e−5\\beta=1e\-5, max output token length 3600 and default temperature\(Yanget al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib15); AI@Meta,[2024](https://arxiv.org/html/2605.21491#bib.bib31)\)\. We use Unsloth library along withvllmfor fast inference\. Very small, yet nonzeroβ\\beta, corresponding to the KL\-Divergence penalty allow us to have a regularization effect for consistent grammar\(Liuet al\.,[2025a](https://arxiv.org/html/2605.21491#bib.bib9)\)\.

## 7Results and Discussion

### 7\.1Can LMs predict better ideas based on their likely empirical outcomes?

[Table2](https://arxiv.org/html/2605.21491#S7.T2)compares the untrained models against their SFT counterparts\. We observe that untrained models perform poorly with Qwen3\-8B scoring 25\.31% and Llama3 scoring a 30\.02%, since we consider inconsistent predictions to be wrong by default, the accuracies account for position bias and remain around random guessing \(25%\)\.

However, Direct\-SFT yields dramatic improvements\.Qwen3reaches 77\.10% accuracy\. This crucial result demonstrates that even SLMs can predict the better idea based on their likely empirical outcome, and this doesn’t require frontier models as hypothesized by\(Wenet al\.,[2025](https://arxiv.org/html/2605.21491#bib.bib17)\)\.

We also observe thatσ\\sigmabased categorization captures comparison difficulty well\. In most fine\-tuned models, the accuracies on1​σ<2​σ<3​σ1\\sigma<2\\sigma<3\\sigma\.

##### Cross Domain Test

We observe that all trainedQwen3models perform at par or better than GPT\-5 \(under all the reasoning effort, and zero\-shot setting\) except Synthetic\-Reason\-SFT\-DAPO\. Furthermore, we see that RL tuned models likeReason\-SFT\-DAPOandReason\-SFT\-DrGRPOperform better thanDirect\-SFT\(≈3%\\approx 3\\%\) showing their learning is more robust \(Table[2](https://arxiv.org/html/2605.21491#S7.T2)\) in general\.

##### Independently Constructed Test Set\.

To further validate generalization on a dataset with zero methodological overlap to our pipeline, we evaluate onWenet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib17)\)’s test set\. Our fine\-tunedReason\-SFT\-DrGRPOachieves67\.49%, outperforming the Wen et al\. zero\-shot GPT\-4\.1 \+ retrieval system \(51\.4%\) by over 16 points despite using a model 50×\\timessmaller and*no*retrieval augmentation\. This result on an independently constructed dataset with a different construction methodology confirms that our trained models capture transferable comparative reasoning priors about scientific methods\.

Model / Method1\-σ\\sigma2\-σ\\sigma3\-σ\\sigmaOverallCD TestQwen3Base18\.4226\.0510\.9920\.143\.55Base \(Reasoning\)15\.3827\.1126\.1125\.3112\.62Direct\-SFT70\.8585\.5684\.6277\.1045\.67Reason\-SFT35\.3238\.9045\.0537\.5129\.31Reason\-DAPO69\.4375\.0083\.5272\.7345\.96Reason\-SFT\-DAPO64\.5779\.2383\.5271\.3548\.37Synthetic\-Reason\-SFT\-DAPO65\.7972\.5374\.7268\.9341\.10Reason\-SFT\-DrGRPO66\.1976\.4183\.5271\.3549\.08Llama3\.1Base37\.3631\.3327\.9330\.033\.83Base \(Reasoning\)26\.5230\.6321\.9827\.3918\.22Direct\-SFT53\.6458\.1067\.0356\.5031\.20GPT\-5Reasoning \(low\)58\.7058\.4549\.4557\.6542\.84Reasoning \(med\)59\.1061\.6256\.0459\.6145\.25Reasoning \(high\)61\.9461\.2756\.0461\.1045\.96Gemini 2\.5 FlashBase \(Reasoning\)41\.9040\.1436\.2640\.73\-

Table 2:Accuracy \(%\) breakdown across different difficulty subsets i\.e\. \(σ\\sigma\)\-categories, on CD test set and models\.Bold: Best,Underline: Second Best

### 7\.2The Role of Reasoning

To assess the importance of reasoning for our task, we study how allowing a LM to think before answering affects predictive performance\. We operationalize “reasoning” differently across model families\. For Qwen3, we compare the standard \(non\-thinking\) variant against its thinking counterpart\. For GPT\-5, we vary the thinking budget \(low, medium, high\)\. For Llama 3\.1, we prepend a CoT instruction and ask the model to reason before producing its final prediction\.

Across these settings, models that are explicitly trained to reason benefit more reliably from deliberation\. Qwen3 improves by∼5\\sim 5% points with thinking variant\. GPT\-5 shows a consistent \(but diminishing\) accuracy gain as the thinking budget increases\. In contrast, naïvely prompting CoT can be counterproductive: Llama 3\.1 reduces accuracy from30\.02%30\.02\\%to27\.38%27\.38\\%\.

ModelAccuracy \(%\)Qwen3Base2\.69Base \(Reasoning\)20\.06Direct\-SFT63\.43Reason\-DAPO65\.94Synthetic\-Reason\-SFT\-DAPO56\.46Reason\-SFT\-DAPO61\.83Reason\-SFT\-DrGRPO67\.49Llama3\.1Base12\.80Base \(Reasoning\)36\.29Direct\-SFT41\.94GPT\-4\.1 \(Wen et al\., w/ retrieval\)51\.4

Table 3:Accuracy \(%\) on the Wen et al\. \(2025\) independently constructed test set\. Our fine\-tuned 8B models are evaluated zero\-shot \(no retraining\)\. GPT\-4\.1 result fromWenet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib17)\)\.![Refer to caption](https://arxiv.org/html/2605.21491v1/images/heatmap1.png)Figure 3:Differential Analysis \(Δ\\Deltabased\) with Bootstrap statistical significance tests across different Difficulty Subsets \(σ\\sigma\) and Overall Performance\.∗∗:p<0\.01p<0\.01;∗:p<0\.05p<0\.05\.
### 7\.3Does RL Induce Reasoning?

We further explore whether we can induce correct reasoning\. The initial SFT with reasoning \(Reason\-SFT/Synthetic\-Reason\-SFT\) achieves an overall accuracy of37\.51%37\.51\\%and25\.54%25\.54\\%respectively\.While Reason\-SFT improves over the Base model, Sythetic\-Reason\-SFT shows negligible improvement\. These results confirm that synthetic reasoning traces—even when filtered for correctness—do not provide learnable grounding for this task, agreeing with the findings of[Wenet al\.](https://arxiv.org/html/2605.21491#bib.bib17)\. RL, however proved highly effective in recovering predictive performance in both cases\.Reason\-SFT\-DAPOandReason\-SFT\-DrGRPOachieve accuracies of≈71%\\approx 71\\%\(Table[2](https://arxiv.org/html/2605.21491#S7.T2)\), narrowing the gap with theDirect\-SFT\.

While RL mostly restored the accuracy, an inspection of the generated reasoning traces revealed a divergence between theoutcomeand theprocess: \(i\)Reason\-DAPOdemonstrate incoherent reasoning,partway through the training\. The model stops generating any reasoning, while still receiving format rewards and attending only towards the final answer\. To prevent this, we introduce a penalty for responses shorter than 600 characters and tweakβ\\betato1​e−41e\-4\. This results in the model repeating 3\-4 sentences before predicting the label\. \(ii\)Reason\-SFT\-DAPOandSynthetic\-Reason\-SFT\-DAPOproduce consistent and coherent reasoning traces prior to the final answer, while being resilient to the form of reward hacking seen inReason\-DAPO, showcasing that it is possible to induce interpretable reasoning in LMs for our task\. \(iii\)Reason\-SFT\-DrGRPOtends to minimize the reasoning trace over the training iterations\. The outputs often devolve into superficial justifications \(e\.g\., stating one idea is better because it is “more recent”, even when ideas lack any information about their dates\)\.

![Refer to caption](https://arxiv.org/html/2605.21491v1/images/consistency_bars.png)Figure 4:Consistency \(%\) across different stages and training paradigm of Qwen3 Model\.
### 7\.4Robustness analysis

We test the robustness of the trained models for some features they might be exploiting: \(i\)Length:categorize the idea pairs based on cases where longer idea is better and otherwise, \(ii\)Recency:categorize the idea pairs based on cases where the newer idea \(published later\) is better or worse, \(iii\)Paraphrasing:test for possible bias based on sentence structure by carefully restating the same \(winning\) idea using Gemini\-2\.5\-pro \(Prompt in Appendix[C](https://arxiv.org/html/2605.21491#A3)\) in a new way in each pair\.

We analyze the model robustness for these potential biases in Figure[3](https://arxiv.org/html/2605.21491#S7.F3)based on the deviation \(Δ\\Delta\) of accuracies on the corresponding subset and bootstrap statistical test of these accuracies, as : \(i\)Length: larger positive value indicates a bias towards longer ideas, \(ii\)Recency: larger positive value indicating a bias towards newer ideas, andParaphrase: larger negative value indicates worse accuracy after paraphrasing\.

All the trained models show minimal variations in performance due to paraphrasing while being statistically robust p\-values\. All models except Reason\-DAPO show high recency bias on the easiest subset\. We hypothesise this is likely due to the imbalanced data in respective subsets and argue that recent ideas are, often stronger\.

We also plot the percentage of consistent pairs \(§[6\.1](https://arxiv.org/html/2605.21491#S6.SS1)\) various models generate post inference on our test set \(Figure[4](https://arxiv.org/html/2605.21491#S7.F4)\)\. We observe that the trained models are more robust towards the position bias i\.e\. are not influenced by the order of presentation, with consistency exceeding 85%\.

## 8Conclusion

In this work we demonstrate that language models can be taught to forecast research success through comparative evaluation of ideas, with fine\-tuned compute efficient models performing at par or better than frontier models\. By constructing a large\-scale dataset, we enable fine\-grained prediction grounded in objective outcomes\. The predictive models developed here offer a path toward closing the loop on autonomous scientific discovery by serving as reward models for AI Agents\. Finally, our data collection approach enables continual improvement, establishing a scalable foundation for the next generation of AI\-assisted science\.

## Limitations

Our benchmark could potentially inherit noise from upstream leaderboard sources\. In particular, the dataset is based onpaperswithcode\.comrecords that were scraped shortly before the site shut down\. While a similar SOTA index is available via HyperAI444[https://hyper\.ai/en/sota/category/natural\-language\-processing](https://hyper.ai/en/sota/category/natural-language-processing), our core methodology \(aligning paper claims with live leaderboard entries and constructing outcome\-supervised comparisons\) is not tied to any single provider and should transfer to other actively maintained leaderboards\.

While we motivate the fine\-tuned model as a potential component for shortlisting ideas via pairwise ranking, we do not yet provide sufficient experiments to quantify its effectiveness in an ideation workflow\.

Scope\-wise, the current dataset is restricted to NLP benchmarks, reflecting the manual effort, cost, and time required for data construction and validation, as well as the scope of our experiments\. Extending the approach to additional domains and task families beyond NLP remains future work\.

## Acknowledgement

We thank Jiaxin Wen and authors ofWenet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib17)\)for providing us with their private test set for evaluation, which helped us demonstrate the transfer of our trained models to new domains and heterogeneous data sources\.

## References

- AI@Meta \(2024\)The llama 3 herd of models\.ArXivabs/2407\.21783\.External Links:[Link](https://arxiv.org/abs/2407.21783)Cited by:[§6\.2](https://arxiv.org/html/2605.21491#S6.SS2.p1.1),[§6\.3](https://arxiv.org/html/2605.21491#S6.SS3.p1.5)\.
- D\. Amodei, C\. Olah, J\. Steinhardt, P\. Christiano, J\. Schulman, and D\. Mané \(2016\)Concrete problems in ai safety\.External Links:1606\.06565,[Link](https://arxiv.org/abs/1606.06565)Cited by:[§5\.2](https://arxiv.org/html/2605.21491#S5.SS2.SSS0.Px1.p1.5)\.
- J\. Baek, S\. K\. Jauhar, S\. Cucerzan, and S\. J\. Hwang \(2025\)ResearchAgent: iterative research idea generation over scientific literature with large language models\.External Links:2404\.07738,[Link](https://arxiv.org/abs/2404.07738)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p1.1),[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px2.p1.1)\.
- N\. Chandak, S\. Goel, A\. Prabhu, M\. Hardt, and J\. Geiping \(2025\)Scaling open\-ended reasoning to predict the future\.External Links:2512\.25070,[Link](https://arxiv.org/abs/2512.25070)Cited by:[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px3.p1.1)\.
- X\. Chen, G\. Li, Z\. Wang, B\. Jin, C\. Qian, Y\. Wang, H\. WANG, Y\. Zhang, D\. Zhang, T\. Zhang, H\. Tong, and H\. Ji \(2026\)RM\-r1: reward modeling as reasoning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=1ZqJ6jj75q)Cited by:[§4](https://arxiv.org/html/2605.21491#S4.SS0.SSS0.Px7.p2.1)\.
- G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§6\.2](https://arxiv.org/html/2605.21491#S6.SS2.p1.1)\.
- A\. Garikaparthi, M\. Patwardhan, L\. Vig, and A\. Cohan \(2025\)IRIS: interactive research ideation system for accelerating scientific discovery\.External Links:2504\.16728,[Link](https://arxiv.org/abs/2504.16728)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p1.1),[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px2.p1.1)\.
- J\. Geng, F\. Cai, Y\. Wang, H\. Koeppl, P\. Nakov, and I\. Gurevych \(2024\)A survey of confidence estimation and calibration in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 6577–6595\.External Links:[Link](https://aclanthology.org/2024.naacl-long.366/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.366)Cited by:[§B\.11](https://arxiv.org/html/2605.21491#A2.SS11.p9.1)\.
- M\. Gridach, J\. Nanavati, K\. Z\. E\. Abidine, L\. Mendes, and C\. Mack \(2025\)Agentic ai for scientific discovery: a survey of progress, challenges, and future directions\.External Links:2503\.08979,[Link](https://arxiv.org/abs/2503.08979)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p1.1)\.
- D\. Halawi, F\. Zhang, C\. Yueh\-Han, and J\. Steinhardt \(2024\)Approaching human\-level forecasting with language models\.External Links:2402\.18563,[Link](https://arxiv.org/abs/2402.18563)Cited by:[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px3.p1.1)\.
- X\. Hu, H\. Fu, J\. Wang, Y\. Wang, Z\. Li, R\. Xu, Y\. Lu, Y\. Jin, L\. Pan, and Z\. Lan \(2024\)Nova: an iterative planning and search approach to enhance novelty and diversity of llm generated ideas\.External Links:2410\.14255,[Link](https://arxiv.org/abs/2410.14255)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p1.1),[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px1.p1.1)\.
- R\. Jia, Y\. Yang, Y\. Gai, K\. Luo, S\. Huang, J\. Lin, X\. Jiang, and G\. Jiang \(2025\)Writing\-zero: bridge the gap between non\-verifiable tasks and verifiable rewards\.External Links:2506\.00103,[Link](https://arxiv.org/abs/2506.00103)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p5.1)\.
- E\. Karger, H\. Bastani, C\. Yueh\-Han, Z\. Jacobs, D\. Halawi, F\. Zhang, and P\. E\. Tetlock \(2025\)ForecastBench: a dynamic benchmark of ai forecasting capabilities\.External Links:2409\.19839,[Link](https://arxiv.org/abs/2409.19839)Cited by:[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px3.p1.1)\.
- S\. Lee, S\. Yang, D\. Kwak, and N\. Y\. Siegel \(2025\)Advancing event forecasting through massive training of large language models: challenges, solutions, and broader impacts\.External Links:2507\.19477,[Link](https://arxiv.org/abs/2507.19477)Cited by:[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px3.p1.1)\.
- L\. Li, W\. Xu, J\. Guo, R\. Zhao, X\. Li, Y\. Yuan, B\. Zhang, Y\. Jiang, Y\. Xin, R\. Dang, D\. Zhao, Y\. Rong, T\. Feng, and L\. Bing \(2024\)Chain of ideas: revolutionizing research via novel idea development with llm agents\.External Links:2410\.13185,[Link](https://arxiv.org/abs/2410.13185)Cited by:[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px2.p1.1)\.
- M\. Liu, S\. Diao, X\. Lu, J\. Hu, X\. Dong, Y\. Choi, J\. Kautz, and Y\. Dong \(2025a\)ProRL: prolonged reinforcement learning expands reasoning boundaries in large language models\.External Links:2505\.24864,[Link](https://arxiv.org/abs/2505.24864)Cited by:[§6\.3](https://arxiv.org/html/2605.21491#S6.SS3.p1.5)\.
- Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin \(2025b\)Understanding r1\-zero\-like training: a critical perspective\.External Links:2503\.20783,[Link](https://arxiv.org/abs/2503.20783)Cited by:[§5\.2](https://arxiv.org/html/2605.21491#S5.SS2.SSS0.Px2.p1.10)\.
- C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. Ha \(2024\)The ai scientist: towards fully automated open\-ended scientific discovery\.External Links:2408\.06292,[Link](https://arxiv.org/abs/2408.06292)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p1.1)\.
- C\. O’Neill, T\. Ghosal, R\. Răileanu, M\. Walmsley, T\. Bui, K\. Schawinski, and I\. Ciucă \(2025\)Sparks of science: hypothesis generation using structured paper data\.External Links:2504\.12976,[Link](https://arxiv.org/abs/2504.12976)Cited by:[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px1.p1.1)\.
- OpenAI \(2025\)GPT\-5 system card\.Note:Accessed: 2026\-01\-05External Links:[Link](https://cdn.openai.com/gpt-5-system-card.pdf)Cited by:[§6\.2](https://arxiv.org/html/2605.21491#S6.SS2.p1.1)\.
- J\. Park, E\. Mendes, G\. Stanovsky, and A\. Ritter \(2025\)Look before you leap: estimating llm benchmark scores from descriptions\.External Links:2509\.20645,[Link](https://arxiv.org/abs/2509.20645)Cited by:[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px3.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p5.1)\.
- C\. Si, T\. Hashimoto, and D\. Yang \(2025\)The ideation\-execution gap: execution outcomes of llm\-generated versus human research ideas\.External Links:2506\.20803,[Link](https://arxiv.org/abs/2506.20803)Cited by:[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px1.p1.1)\.
- C\. Si, D\. Yang, and T\. Hashimoto \(2024\)Can llms generate novel research ideas? a large\-scale human study with 100\+ nlp researchers\.External Links:2409\.04109,[Link](https://arxiv.org/abs/2409.04109)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p1.1),[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px2.p1.1)\.
- H\. Su, R\. Chen, S\. Tang, Z\. Yin, X\. Zheng, J\. Li, B\. Qi, Q\. Wu, H\. Li, W\. Ouyang, P\. Torr, B\. Zhou, and N\. Dong \(2025\)Many heads are better than one: improved scientific idea generation by a llm\-based multi\-agent system\.External Links:2410\.09403,[Link](https://arxiv.org/abs/2410.09403)Cited by:[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px1.p1.1)\.
- Q\. Wang, D\. Downey, H\. Ji, and T\. Hope \(2024\)SciMON: scientific inspiration machines optimized for novelty\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 279–299\.External Links:[Link](https://aclanthology.org/2024.acl-long.18/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.18)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p1.1),[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px1.p1.1)\.
- J\. Wen, C\. Si, Y\. Chen, H\. He, and S\. Feng \(2025\)Predicting empirical ai research outcomes with language models\.External Links:2506\.00794,[Link](https://arxiv.org/abs/2506.00794)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p3.1),[§1](https://arxiv.org/html/2605.21491#S1.p5.1),[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2605.21491#S4.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2605.21491#S4.SS0.SSS0.Px7.p3.1),[§4](https://arxiv.org/html/2605.21491#S4.SS0.SSS0.Px9.p1.1),[§7\.1](https://arxiv.org/html/2605.21491#S7.SS1.SSS0.Px2.p1.1),[§7\.1](https://arxiv.org/html/2605.21491#S7.SS1.p2.1),[§7\.3](https://arxiv.org/html/2605.21491#S7.SS3.p1.3),[Table 3](https://arxiv.org/html/2605.21491#S7.T3),[Acknowledgement](https://arxiv.org/html/2605.21491#Sx2.p1.1)\.
- Y\. Yamada, R\. T\. Lange, C\. Lu, S\. Hu, C\. Lu, J\. Foerster, J\. Clune, and D\. Ha \(2025\)The ai scientist\-v2: workshop\-level automated scientific discovery via agentic tree search\.External Links:2504\.08066,[Link](https://arxiv.org/abs/2504.08066)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§6\.2](https://arxiv.org/html/2605.21491#S6.SS2.p1.1),[§6\.3](https://arxiv.org/html/2605.21491#S6.SS3.p1.5)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. Wang \(2025\)DAPO: an open\-source llm reinforcement learning system at scale\.External Links:2503\.14476,[Link](https://arxiv.org/abs/2503.14476)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p5.1),[§5\.2](https://arxiv.org/html/2605.21491#S5.SS2.SSS0.Px2.p1.10)\.
- K\. Zhou, A\. Dejl, G\. Freedman, L\. Chen, A\. Rago, and F\. Toni \(2025\)Evaluating uncertainty quantification methods in argumentative large language models\.External Links:2510\.02339,[Link](https://arxiv.org/abs/2510.02339)Cited by:[§B\.11](https://arxiv.org/html/2605.21491#A2.SS11.p9.1)\.
- M\. Zhu, Q\. Xie, Y\. Weng, J\. Wu, Z\. Lin, L\. Yang, and Y\. Zhang \(2025\)AI scientists fail without strong implementation capability\.External Links:2506\.01372,[Link](https://arxiv.org/abs/2506.01372)Cited by:[§1](https://arxiv.org/html/2605.21491#S1.p1.1),[§3](https://arxiv.org/html/2605.21491#S3.SS0.SSS0.Px1.p1.1)\.

## Appendix AAdditional Benchmark Dataset Construction details

This appendix provides the mathematical and algorithmic details of the benchmark dataset construction pipeline described in the main text\.

### A\.1Metric selection and Normalization

##### Metric Cleaning:

To combine heterogeneous metrics, we only used metric columns that were universally reported for all entries in a benchmark\. If no metric had universal coverage, the benchmark was skipped\.

##### Normalization:

Letmi\(k\)m\_\{i\}^\{\(k\)\}be metrickkfor entryii\. We normalize:

m~i\(k\)=mi\(k\)−minkmaxk−mink\\tilde\{m\}\_\{i\}^\{\(k\)\}=\\frac\{m\_\{i\}^\{\(k\)\}\-\\min\_\{k\}\}\{\\max\_\{k\}\-\\min\_\{k\}\}\(6\)

##### Direction check:

We calculate the Pearson correlation betweenm~\(k\)\\tilde\{m\}^\{\(k\)\}and the rankrr\. Ifcorr​\(m~\(k\),r\)\>0\\text\{corr\}\(\\tilde\{m\}^\{\(k\)\},r\)\>0\(implying higher value = worse rank, e\.g\., perplexity\), we invert:

m^i\(k\)=1−m~i\(k\)\\hat\{m\}\_\{i\}^\{\(k\)\}=1\-\\tilde\{m\}\_\{i\}^\{\(k\)\}\(7\)

##### Unified Score:

The final scoresis\_\{i\}is the arithmetic mean across adjusted metrics:

si=1\|M\|​∑k∈Mm^i\(k\)s\_\{i\}=\\frac\{1\}\{\|M\|\}\\sum\_\{k\\in M\}\\hat\{m\}\_\{i\}^\{\(k\)\}\(8\)

### A\.2Discordance Removal

To ensure the unified scores broadly agree with leaderboard ranks, we use a strict pairwise discordance test\. A pair\(i,j\)\(i,j\)is discordant if:

discordant​\(i,j\)=\{1if​\(ri​<rj​and​si\>​sj\)or​\(ri\>rj​and​si<sj\)0otherwise\\text\{discordant\}\(i,j\)=\\begin\{cases\}1&\\text\{if \}\(r\_\{i\}<r\_\{j\}\\text\{ and \}s\_\{i\}\>s\_\{j\}\)\\\\ &\\text\{or \}\(r\_\{i\}\>r\_\{j\}\\text\{ and \}s\_\{i\}<s\_\{j\}\)\\\\ 0&\\text\{otherwise\}\\end\{cases\}\(9\)
Algorithm 1Iterative discordance removal1:Entries with ranks and unified scores\.

2:Compute discordance fraction

f=D/\(n2\)f=D/\\binom\{n\}\{2\}\.

3:while

f\>0f\>0and at least 2 entries remaindo

4:Identify entry involved in max discordant pairs\.

5:Remove entry\.

6:Recompute

ff\.

7:endwhile

### A\.3Time Bucketing

To handle temporal shifts, we grouped entries by year\. A bucket was valid only if it contained at least 5 unique papers and 2 test papers \(post global\-split\)\.

Algorithm 2Time bucketing and validation1:Entries with years and global split\.

2:Create initial year\-based buckets\.

3:whilebucket has

<<5 papers or

<<2 test papersdo

4:Merge bucket with adjacent one\.

5:endwhile

### A\.4Pair Generation and Augmentation

Pairs are produced within the same benchmark and bucket using the standardized differenceΔi​j=\|si−sj\|/σ\\Delta\_\{ij\}=\|s\_\{i\}\-s\_\{j\}\|/\\sigma\.

##### Sigma Categories:

- •1\-sigma:0\.8≤Δi​j≤1\.20\.8\\leq\\Delta\_\{ij\}\\leq 1\.2
- •2\-sigma:1\.8≤Δi​j≤2\.21\.8\\leq\\Delta\_\{ij\}\\leq 2\.2
- •3\-sigma:2\.8≤Δi​j≤3\.22\.8\\leq\\Delta\_\{ij\}\\leq 3\.2

##### Labeling and Augmentation:

For each valid pair\(i,j\)\(i,j\)wheresi\>sjs\_\{i\}\>s\_\{j\}:

- •Generate record:\{i​d​e​a​\_​A:i,i​d​e​a​\_​B:j,l​a​b​e​l:1\}\\\{idea\\\_A:i,idea\\\_B:j,label:1\\\}
- •Generate swap:\{i​d​e​a​\_​A:j,i​d​e​a​\_​B:i,l​a​b​e​l:0\}\\\{idea\\\_A:j,idea\\\_B:i,label:0\\\}

This augmentation ensures the model is robust to input order and the class distribution is perfectly balanced\.

Algorithm 3Pair generation1:Validated bucket with scores

ssand std

σ\\sigma\.

2:foreach unordered pair

\(i,j\)\(i,j\)do

3:

Δi​j=\|si−sj\|/σ\\Delta\_\{ij\}=\|s\_\{i\}\-s\_\{j\}\|/\\sigma\.

4:if

Δi​j\\Delta\_\{ij\}in sigma\-windowthen

5:Emit pair

\(i,j\)\(i,j\)with label 1\.

6:Emit swapped pair

\(j,i\)\(j,i\)with label 0\.

7:endif

8:endfor

BenchmarkTotal PairsCode Generation On Mbpp864Common Sense Reasoning On Winogrande454Question Answering On Copa381Named Entity Recognition Ner On Conll 2003372Question Answering On Boolq363Question Answering On Piqa331Common Sense Reasoning On Arc Challenge227Math Word Problem Solving On Math217Question Answering On Squad11203Question Answering On Natural Questions200Question Answering On Squad11 Dev198Relation Extraction On Docred179Question Answering On Webquestions176Aspect Based Sentiment Analysis On Semeval170Pose Estimation On Mpii Human Pose163Word Sense Disambiguation On Words In Context157Deblurring On Gopro155Common Sense Reasoning On Arc Easy142Entity Alignment On Dbp15K Zh En134Common Sense Reasoning On Commonsenseqa126

Table 4:Top 20 Benchmarks by Total Pairs \(includes train and test\)Table 5:Dataset distribution across sigma categories\. Pairs classified as “Excluded” were removed due to the lack of a valid research goal\.Train SetTest SetCategoryOriginalExcludedFinalAugmentedOriginalExcludedFinalAugmented1​σ1\\sigma8,8812,7616,12012,2406871934949882​σ2\\sigma4,8271,3663,4616,9224011172845683​σ3\\sigma1,4363981,0382,0761697891182Total15,1444,52510,61921,2381,2573888691,738

## Appendix BAdditional Insights

### B\.1Dataset Details

##### Dataset Release Notes

##### Tasks and Benchmarks

A benchmark in our dataset is defined as a ”Task” \(e\.g\. Question Answering\) on a specific dataset \(e\.g\. PIQA\)\. Table 4 shows the top 20 benchmarks based on the total pairs \(including train and test\) in our dataset\. Question Answering and Common Sense Reasoning are most common NLP tasks in our dataset\. Further, Figure[5](https://arxiv.org/html/2605.21491#A2.F5)shows the top 20 ”Tasks” and the number of pairs from each of them\. Question Answering is the most common task in our dataset \(21\.2% of total pairs\), followed by Common Sense Reasoning with a drastic drop \(8\.5% of total pairs\)\.

![Refer to caption](https://arxiv.org/html/2605.21491v1/images/Task.png)Figure 5:Top 20 Tasks based on the total pairs \(includes both train and test\)![Refer to caption](https://arxiv.org/html/2605.21491v1/images/year_distribution_stacked.png)Figure 6:Distribution of the ideas/methods across the years withσ\\sigma\-wise breakdown in test![Refer to caption](https://arxiv.org/html/2605.21491v1/images/year_distribution_stacked_copy.png)Figure 7:Distribution of the ideas/methods across the years withσ\\sigma\-wise breakdown in train
##### Temporal Distribution

Figure[6](https://arxiv.org/html/2605.21491#A2.F6)and[7](https://arxiv.org/html/2605.21491#A2.F7)shows the temporal distribution of entires of all the leaderboards based on the year of publication of their Result Reporting Paper\. The temporal distribution is uni\-modal in both the train and test set, with the test set more skewed to the left\.

##### Excluded Pairs

Table[5](https://arxiv.org/html/2605.21491#A1.T5)gives us the full statistics of the final dataset\. Even after employing manually verified LLM based Research Goal Sythesis in §\([4](https://arxiv.org/html/2605.21491#S4)\), we miss out on a large chunk of pairs \(close to 30%\)\.

##### Excluded Benchmark Leaderboards

Apart from missing Research Goals, some of the benchmarks are excluded during the Train\-Test as described in Section §\([4](https://arxiv.org/html/2605.21491#S4)\) because of the following scenario: Consider a case where the leaderboard has only 2 entries with 2 corresponding RR papers, and due to the iterative nature of of the train\-test split, if one of it has been assigned to train and the other to test \(based on the splits form the previous leaderboards\) we will be unable to form pairs within the train or test subsequently\. So, this benchmark leaderboard would get skipped in the process\.

### B\.2Knowledge cutoffs and memorization

For Qwen3 with which we run our primary experiments, an official pretraining knowledge cutoff is not publicly documented\. However, we argue that our evaluation is unlikely to be dominated by knowledge cutoff leakage for three reasons\.

The prediction target is not a fact in the input text\.Our labels are derived from benchmark\-specific leaderboard outcomes via a unified score computed from reported metrics, including normalization and direction correction\. At inference time, the model is shown only a benchmark\-specific research goal and an idea description, while we explicitly remove empirical results and outcome statements from the paper text\. Therefore, succeeding on our task requires mapping from a proposed methodological change to its expected empirical impact under a specific benchmark, not simply recalling a numeric result or a rank that appears verbatim in a paper\.

Leakage would have to reconstruct a benchmark\-conditional comparison, not a single\-paper lookup\.Even if a model had seen one or both papers during pretraining, the correct answer depends on \(i\) the specific leaderboard and metric normalization used in our pipeline, and \(ii\) the relative ordering between two ideas within that benchmark\. Memorizing this at scale would require storing a large number of benchmark\-conditioned pairwise outcomes across 1,918 leaderboards, rather than recalling isolated paper facts\. This makes direct memorization an implausible explanation for performance gains\.

Empirical evidence suggests the task is not solved by recall\.Base \(untrained\) 8B models perform poorly \(20–30% accuracy after accounting for position bias\), which is substantially below chance under our consistency\-based evaluation\. Moreover, strong proprietary models \(e\.g\., Gemini\-2\.5\-Flash\) also struggle zero\-shot, indicating that broad pretraining exposure alone does not reliably recover the correct comparative outcome\. These results are more consistent with the task requiring non\-trivial comparative reasoning than with label retrieval\.

Results on unseen CD Test SetExact knowledge cutoff is not available for Qwen3 models\. But Llama3\.1 has a knowledge cutoff of December 2023\. Inference results on the test set constructed from non\-NLP leaderboards, where the entries have also been filtered based on year \(≥2024\\geq 2024\), which is post the knowledge cutoff, show that fine tuned models not only generalize well, but also do well, performing either at par or better than frontier models like GPT\-5 \(which has a much recent knowledge cutoff of September 2024\) \(Table[2](https://arxiv.org/html/2605.21491#S7.T2)\)

### B\.3Ablations on CD Test

To further robustly stress\-test against knowledge cut\-off leakage and sensitivity to individual metric based prediction, we isolate a2025\-only subsetof 52 pairs where all papers are dated≥\\geq2025, making direct memorisation implausible for all models\. On this subsetReason\-SFT\-DAPOachieves57\.69%vs\. GPT\-5\-high at48\.07%\(\+9\.6\+9\.6pp\), andDirect\-SFTreaches53\.85%\. Base models crater to 3–17%, confirming that fine\-tuning—not memorisation—drives the performance gains\.

We additionally break down the 1,410 CD pairs by whether the leaderboard rank ordering and the individual metric ordering*agree*or*disagree*\(102 pairs where they conflict\)\. Table[6](https://arxiv.org/html/2605.21491#A2.T6)shows thatReason\-SFT\-DrGRPOachieves60\.78%on the disagreement subset—the hardest cases where metric weighting matters most—outperforming all other models as well as GPT\-5\. This validates that our model is sensitive to metric\-specific nuances rather than relying on simple rank heuristics\.

ModelRank=MetricRank≠\\neqMetricOverall2025\-Only\(1,308\)\(102\)\(1,410\)\(104\)Qwen3Base3\.671\.963\.553\.85Base \(Reasoning\)13\.145\.8812\.6217\.31Direct\-SFT46\.6433\.3345\.6753\.85Reason\-DAPO46\.6437\.2545\.9651\.92Synthetic\-Reason\-SFT\-DAPO41\.9029\.4141\.1044\.23Reason\-SFT\-DAPO49\.0839\.2148\.3757\.69Reason\-SFT\-DrGRPO48\.1660\.7849\.0853\.85Llama3\.1Base3\.971\.963\.833\.85Base \(Reasoning\)18\.0419\.6118\.2223\.08Direct\-SFT32\.879\.8031\.2036\.54GPT\-5 \(zero\-shot\)Reasoning \(low\)44\.1925\.4942\.8444\.23Reasoning \(med\)46\.1833\.3345\.2551\.92Reasoning \(high\)46\.9433\.3345\.9648\.07

Table 6:Full Cross\-Domain \(CD\) test set results\.Rank=Metric: pairs where the leaderboard rank order agrees with the individual metric order\.Rank≠\\neqMetric: disagreement subset\.2025\-Only: subset of 104 pairs \(52 pre\-augmentation\) with all papers dated≥\\geq2025\. Consistency\-aware accuracy throughout\.Bold: best per column among trained models\.
### B\.4Reward Hacking

![Refer to caption](https://arxiv.org/html/2605.21491v1/images/Chart_5_1_2026_4_38_01_PM.png)Figure 8:The average rewards through the training iterations of Reason\-DAPO![Refer to caption](https://arxiv.org/html/2605.21491v1/images/Chart_5_1_2026_4_39_01_PM.png)Figure 9:The mean output length through the training iterations of Reason\-DAPOFigure[8](https://arxiv.org/html/2605.21491#A2.F8)shows the running average of rewards and Figure[9](https://arxiv.org/html/2605.21491#A2.F9)shows the average length of the output/response generated within the reasoning traces of a group over the train iteration of Qwen3 model being trained directly with DAPO objective and a final answer and format based reward as described in §[5](https://arxiv.org/html/2605.21491#S5)\. As the training closes towards the 24k steps, we see the response length blow up and coupled with a drastic drop in the rewards at the individual steps\. This is followed by a drastic drop in the output length\. It drops down to 9 and continues generating only 9 tokens\. But we see a recovery in the rewards and the models continues to obtain rewards similar to before 23k step\. Thus the model transitions to learning to better predict the correct final label corresponding to the better idea in a pair, while generating only the tokens necessary for the format reward and the final reward, and fully circumventing the reasoning\. A clear case of ”reward hacking”\.

### B\.5Sensitivity to Benchmark\-Specific Research Goals

We evaluate whether the trained model can identify the superior idea within a pair of ideas, conditionally based on the target benchmark i\.e\. research goal\. We observe that all of the Qwen3 trained models demonstrate robust contextual awareness of benchmark specific research goals For instance, theEfficient Audio Transformer \(EAT\)achieves SOTA results on theAudio Classification on Balanced Audio Set, but ranks significantly lower onAudio Classification on ESC\-50, despite a high accuracy of 96% \(vs\. 99\.1% SOTA\)\. We observe that Qwen3 correctly predicts EAT as the superior candidate among the pairs for theAudio Classification on Balanced Audio Setbenchmark and inferior forAudio Classification on ESC\-50\. This indicates that the model does not rely on superficial textual characteristics or large numerical margins\. And exhibitsconditional reasoning, correctly inferring relative utility of an idea based on a given benchmark\.

### B\.6Token Efficiency

![Refer to caption](https://arxiv.org/html/2605.21491v1/images/overall_accuracy_vs_tokens.png)Figure 10:Overall Accuracy \(%\) Vs Mean Number of tokens generated during reasoning\.Figure[10](https://arxiv.org/html/2605.21491#A2.F10)shows the average number of reasoning tokens generated before prediction\. Our RL variants that produce reasoning traces, achieve higher accuracy while using a fraction of the tokens compared to GPT\-5\.

Table 7:Distribution of dataset preferences in percentages in Train\. TheTotalcolumn indicates the number of samples, while other columns show the percentage breakdown of Recency and Length preferences within each split\.Recency \(%\)Length \(%\)Data SplitTotalNewerOlderSameShorterLongerEqualFull Train Set19,11357\.0817\.7325\.1947\.9751\.830\.20Breakdown by Sigma \(σ\\sigma\)σ=1\\sigma=111,01652\.3519\.6428\.0148\.0451\.820\.15σ=2\\sigma=26,22962\.9315\.4921\.5848\.4051\.310\.29σ=3\\sigma=31,86865\.4213\.9720\.6146\.0953\.690\.21Table 8:Detailed Robustness Statistics: Accuracy \(%\) and Total Sample Count \(NN\)\. The sample counts for each category \(e\.g\., Longer, Newer\) remain same across models for the sameσ\\sigma\-subset\.ModelLengthRecencyParaphrasingLongerShorterNewerOlderSameOriginalPara\.1\-σ\\sigma\(NN\)516466480212296988988Direct\-SFT72\.972\.969\.169\.179\.679\.662\.362\.362\.862\.870\.970\.975\.575\.5Reason\-DAPO79\.179\.159\.259\.276\.376\.364\.264\.262\.262\.269\.469\.472\.172\.1Reason\-SFT\-DAPO71\.371\.357\.157\.170\.470\.450\.950\.964\.964\.964\.664\.661\.561\.5Reason\-SFT\-DrGRPO70\.970\.961\.861\.870\.470\.469\.869\.856\.856\.866\.266\.265\.665\.6Synthetic\-Reason\-SFT\-DAPO72\.172\.159\.359\.368\.868\.874\.574\.554\.754\.765\.865\.862\.962\.92\-σ\\sigma\(NN\)27829033813496568568Direct\-SFT85\.685\.685\.585\.590\.590\.583\.683\.670\.870\.885\.685\.687\.087\.0Reason\-DAPO76\.376\.373\.873\.877\.577\.585\.185\.152\.152\.175\.075\.077\.577\.5Reason\-SFT\-DAPO83\.583\.575\.275\.284\.684\.676\.176\.164\.664\.679\.279\.276\.176\.1Reason\-SFT\-DrGRPO75\.575\.577\.277\.278\.778\.783\.683\.658\.358\.376\.476\.475\.075\.0Synthetic\-Reason\-SFT\-DAPO72\.672\.672\.572\.571\.671\.688\.188\.154\.254\.272\.572\.572\.972\.93\-σ\\sigma\(NN\)108701462412182182Direct\-SFT87\.087\.080\.080\.089\.089\.058\.358\.383\.383\.384\.684\.686\.886\.8Reason\-DAPO83\.383\.382\.982\.982\.282\.283\.383\.3100\.0100\.083\.583\.582\.482\.4Reason\-SFT\-DAPO90\.790\.771\.471\.489\.089\.058\.358\.366\.766\.783\.583\.581\.381\.3Reason\-SFT\-DrGRPO83\.383\.382\.982\.989\.089\.041\.741\.7100\.0100\.083\.583\.587\.987\.9Synthetic\-Reason\-SFT\-DAPO76\.076\.071\.771\.780\.880\.866\.766\.716\.716\.774\.774\.773\.673\.6Overall\(NN\)90282696437040417381738Direct\-SFT78\.578\.575\.875\.884\.984\.969\.769\.765\.465\.477\.177\.180\.480\.4Reason\-DAPO78\.778\.766\.366\.377\.677\.673\.073\.060\.960\.972\.772\.774\.974\.9Reason\-SFT\-DAPO77\.477\.464\.764\.778\.278\.260\.560\.564\.964\.971\.471\.468\.468\.4Reason\-SFT\-DrGRPO73\.873\.869\.069\.076\.176\.173\.073\.058\.458\.471\.471\.471\.071\.0Synthetic\-Reason\-SFT\-DAPO72\.772\.764\.964\.971\.671\.678\.978\.953\.553\.568\.968\.967\.367\.3
### B\.7Detailed Robustness Analysis

Table[8](https://arxiv.org/html/2605.21491#A2.T8)shows a detailed breakdown of the performance, measured using accuracies on the respective subsets, of different trained Qwen3 models across different stress tests and difficulty levels of idea pair comparison\.

#### B\.7\.1Difficulty vs\. Length Sensitivity

- •Reason\-DAPO: There is a strong inverse correlation between task difficulty and length sensitivity\. As the difficulty increases \(moving from 3\-σ\\sigmato 1\-σ\\sigma\), the performance gap between “Longer” and “Shorter” inputs widens significantly\. - –3\-σ\\sigma\(Easy\): Gap is small≈\\approx0\.4 %\)\. - –2\-σ\\sigma\(Medium\): Gap increases \(≈\\approx2\.5 %\)\. - –1\-σ\\sigma\(Hard\): Gap maximizes \(≈\\approx19\.9 %\)\.
- •Direct\-SFT: This model shows no correlation between difficulty and length sensitivity\. The performance gap between “Longer” and “Shorter” remains consistently low \(<4<4%\) across all three sigma categories, regardless of task difficulty\.
- •Reason\-SFT\-DAPO: This model maintains a high length dependency across all levels, but unlike Reason\-DAPO, the gap does not widen monotonically with difficulty; it remains large \(≈\\approx14\-19 points\) in both the easiest \(3\-σ\\sigma\) and hardest \(1\-σ\\sigma\) tiers\.

#### B\.7\.2Difficulty vs\. Recency Variance

- •Reason\-DAPO \(Inversion Effect\): The model’s preference for “Newer” vs “Older” data inverts based on difficulty\. - –In 1\-σ\\sigma\(Hard\), it scores higher on “Newer” data \(\+12\.1 %\)\. - –In 2\-σ\\sigma\(Medium\), it scores higher on “Older” data \(\-7\.6 %\)\. - –In 3\-σ\\sigma\(Easy\), it scores slightly higher on “Older” data \(\-1\.1 %\)\.
- •Direct\-SFT: The “Recency Gap” \(Newer minus Older\) exists across all difficulties but is non\-linear\. The model is most robust to recency shifts in the 2\-σ\\sigmacategory \(gap of 6\.9 %\) compared to the 1\-σ\\sigma\(17\.3 %\) and 3\-σ\\sigma\(30\.7 %\) categories\.

#### B\.7\.3Sample Size \(N\) vs\. Accuracy’s Stability

- •Variance Correlation: There is a direct observable correlation between the sample sizeNNand the variance of scores within a model’s row\. - –HighNN\(1\-σ\\sigma\): Accuracies across splits \(Recency/Length\) are generally clustered\. For example,Reason\-SFT\-DrGRPOhas accuracies tightly between 56\.8% and 70\.9%\. - –LowNN\(3\-σ\\sigma\): Accuracies exhibit extreme volatility\. In this tier,Reason\-SFT\-DrGRPOspans from 41\.7% to 100\.0% accuracies\.
- •Paraphrasing Stability: The impact of paraphrasing is relatively uniform across difficulty levels \(NNcounts are high for both Original and Para\)\. For instance,Reason\-DAPO’s gain from paraphrasing remains steady at roughly \+2\.5 to \+2\.7 % in both 1\-σ\\sigmaand 2\-σ\\sigmacategories\.

#### B\.7\.4Length Dominance across Models

- •In the1\-σ\\sigma\(Hard\)category, the hierarchy of models changes depending on the input length\. - –On Longer Inputs:Reason\-DAPO\(79\.1%\)\>\>Direct\-SFT\(72\.9%\)\. - –On Shorter Inputs:Direct\-SFT\(69\.1%\)\>\>Reason\-DAPO\(59\.2%\)\. This indicates that in the hardest difficulty tier, the comparative advantage of the RL model is conditional on the presence of longer context\.

#### B\.7\.5Distribution of train Dataset

We further analyse the distribution of the idea pairs based on Recency and Length as described in §[7\.4](https://arxiv.org/html/2605.21491#S7.SS4)\(Table[7](https://arxiv.org/html/2605.21491#A2.T7)\)\.

We see that even within the differentσ\\sigma\-Categories, the distribution of shorter and longer is relatively balanced\. It also contains a very small fraction of pairs that have equal length \(The test set doesn’t have any\)\. This likely enabled the models to learn much better without introducing such bias due to imbalance\. The distribution is largely imbalanced under the newer, older and same year \(§[7\.4](https://arxiv.org/html/2605.21491#S7.SS4)\) categories, with the number the newer category having almost 2 times the number of pairs in older and same year combined\. Older category has consistently lower representation across allσ\\sigma\. This could additionally explain slightly large variations across different trained models, apart from the largely imbalanced distribution within the test itself\.

#### B\.7\.6Statistical Testing

To rigorously assess whether observed bias patterns are statistically meaningful rather than noise, a non\-parametric bootstrap procedure is employed\. For each robustness dimension, accuracy is computed separately on two complementary subsets \(for example, “longer is better” vs\. “shorter is better”\) and the null hypothesisH0:Δ=0H\_\{0\}\\colon\\Delta=0\(no bias\) is tested against the two\-sided alternativeH1:Δ≠0H\_\{1\}\\colon\\Delta\\neq 0\.

Concretely,B=10,000B=10\{,\}000bootstrap resamples are drawn with replacement from the test set, and the accuracy on each subset is recomputed per resample\. For each resample, the deltaΔ∗=accA∗−accB∗\\Delta^\{\*\}=\\mathrm\{acc\}\_\{A\}^\{\*\}\-\\mathrm\{acc\}\_\{B\}^\{\*\}between the two subsets is recorded, forming the bootstrap distribution\{Δ1∗,…,ΔB∗\}\\\{\\Delta^\{\*\}\_\{1\},\\dots,\\Delta^\{\*\}\_\{B\}\\\}\. The 95% percentile confidence interval is:

CI95%=\[P2\.5​\(Δ∗\),P97\.5​\(Δ∗\)\],\\mathrm\{CI\}\_\{95\\%\}=\\big\[P\_\{2\.5\}\(\\Delta^\{\*\}\),\\;P\_\{97\.5\}\(\\Delta^\{\*\}\)\\big\],\(10\)and the two\-sidedpp\-value is:

p=\{min⁡\(2⋅P​\(Δ∗<0\),1\)if​Δ¯≥0,min⁡\(2⋅P​\(Δ∗\>0\),1\)if​Δ¯<0\.p=\\begin\{cases\}\\min\\\!\\big\(2\\cdot P\(\\Delta^\{\*\}<0\),\\;1\\big\)&\\text\{if \}\\bar\{\\Delta\}\\geq 0,\\\\ \\min\\\!\\big\(2\\cdot P\(\\Delta^\{\*\}\>0\),\\;1\\big\)&\\text\{if \}\\bar\{\\Delta\}<0\.\\end\{cases\}\(11\)H0H\_\{0\}is rejected atp<0\.05p<0\.05\(reported as \* \) andp<0\.01p<0\.01\(reported as \*\* \)\. Resampling is performed at the level of paired units \(original\+\+swapped pair\) rather than individual samples, preserving the dependency structure of the consistency\-based evaluation metric\. A non\-parametric bootstrap is preferred over parametric alternatives such as McNemar’s test because it makes no distributional assumptions about the accuracy difference and extends straightforwardly to the composite consistent accuracy statistic, which does not have a closed\-form null distribution\.

Table 9:Bootstrapped bias significance tests \(B=10,000B=10\{,\}000\)\.Δ\\Deltain percentage points with 95% CIs\.∗∗:p<0\.01p<0\.01;∗:p<0\.05p<0\.05\.ModelTierLengthΔ\\Delta\[95% CI\]RecencyΔ\\Delta\[95% CI\]ParaphraseΔ\\Delta\[95% CI\]Direct\-SFTOverall\+\+2\.7 \[−\-2\.8, 8\.3\]\+\+17\.5∗∗\[11\.8, 23\.2\]\+\+3\.3 \[−\-0\.6, 7\.1\]1\-σ\\sigma\+\+3\.8 \[−\-4\.4, 12\.0\]\+\+17\.1∗∗\[9\.3, 24\.8\]\+\+4\.6 \[−\-0\.8, 10\.1\]2\-σ\\sigma\+\+0\.1 \[−\-8\.3, 8\.5\]\+\+12\.3∗∗\[3\.7, 21\.3\]\+\+1\.4 \[−\-4\.2, 7\.0\]3\-σ\\sigma\+\+7\.2 \[−\-8\.1, 23\.2\]\+\+22\.5∗\[0\.2, 46\.0\]\+\+2\.2 \[−\-7\.7, 12\.1\]Reason\-DAPOOverall\+\+12\.3∗∗\[6\.5, 18\.2\]\+\+10\.9∗∗\[4\.9, 16\.9\]\+\+2\.2 \[−\-2\.0, 6\.3\]1\-σ\\sigma\+\+19\.8∗∗\[11\.7, 27\.7\]\+\+13\.3∗∗\[5\.2, 21\.2\]\+\+2\.6 \[−\-3\.0, 8\.3\]2\-σ\\sigma\+\+2\.5 \[−\-7\.5, 12\.4\]\+\+6\.2 \[−\-4\.1, 16\.7\]\+\+2\.5 \[−\-4\.2, 9\.5\]3\-σ\\sigma\+\+0\.3 \[−\-15\.5, 16\.6\]−\-6\.8 \[−\-21\.9, 11\.3\]−\-1\.1 \[−\-12\.1, 9\.9\]Reason\-SFT\-DAPOOverall\+\+12\.7∗∗\[6\.5, 18\.8\]\+\+15\.4∗∗\[9\.2, 21\.5\]−\-3\.0 \[−\-7\.2, 1\.4\]1\-σ\\sigma\+\+14\.2∗∗\[5\.8, 22\.7\]\+\+11\.3∗∗\[3\.1, 19\.8\]−\-3\.1 \[−\-9\.1, 2\.8\]2\-σ\\sigma\+\+8\.3 \[−\-0\.9, 18\.0\]\+\+13\.3∗∗\[3\.4, 23\.2\]−\-3\.2 \[−\-9\.9, 3\.5\]3\-σ\\sigma\+\+19\.2∗\[2\.3, 36\.3\]\+\+27\.8∗\[4\.3, 51\.5\]−\-2\.2 \[−\-13\.2, 8\.8\]Reason\-SFT\-DrGRPOOverall\+\+4\.8 \[−\-1\.2, 10\.8\]\+\+10\.7∗∗\[4\.7, 16\.8\]−\-0\.3 \[−\-4\.5, 3\.9\]1\-σ\\sigma\+\+9\.1∗\[0\.7, 17\.5\]\+\+8\.2 \[−\-0\.1, 16\.5\]−\-0\.7 \[−\-6\.7, 5\.3\]2\-σ\\sigma−\-1\.8 \[−\-11\.7, 8\.2\]\+\+5\.7 \[−\-4\.3, 16\.1\]−\-1\.4 \[−\-8\.5, 5\.6\]3\-σ\\sigma\+\+0\.4 \[−\-15\.5, 16\.5\]\+\+27\.8∗\[4\.4, 51\.5\]\+\+4\.4 \[−\-5\.5, 14\.3\]Synthetic\-Reason\-SFT\-DAPOOverall\+\+7\.9∗\[1\.7, 14\.0\]\+\+6\.0 \[−\-0\.3, 12\.3\]−\-1\.6 \[−\-6\.1, 2\.8\]1\-σ\\sigma\+\+12\.8∗∗\[4\.3, 21\.2\]\+\+5\.7 \[−\-2\.7, 14\.0\]−\-2\.9 \[−\-8\.7, 3\.0\]2\-σ\\sigma\+\+0\.2 \[−\-10\.2, 10\.7\]−\-2\.3 \[−\-13\.1, 8\.2\]\+\+0\.4 \[−\-7\.0, 7\.7\]3\-σ\\sigma\+\+4\.3 \[−\-14\.3, 23\.3\]\+\+30\.6∗\[5\.8, 55\.7\]−\-1\.1 \[−\-14\.3, 12\.1\]

Paraphrasing robustness\.No model shows a statistically significant change in accuracy due to paraphrasing at*any*difficulty tier \(p\>0\.05p\>0\.05for all models across 1\-σ\\sigma, 2\-σ\\sigma, 3\-σ\\sigma, and Overall\)\. This confirms that the observed minor fluctuations are noise rather than genuine structural sensitivity to surface form—our models’ decisions are anchored to the semantics of the idea, not the specific phrasing\.

Length bias\.Results are model\-specific rather than universal\.Direct\-SFTshows*no*statistically significant length bias, confirming its robustness to idea length variation\. In contrast,Reason\-DAPOandReason\-SFT\-DAPOexhibit highly significant length preferences, indicating these RL\-trained models systematically favour longer ideas\.Reason\-SFT\-DrGRPOsits at the boundary: borderline non\-significant at the overall level \(p=0\.116p=0\.116\) but significant at the hardest 1\-σ\\sigmatier \(p=0\.035p=0\.035\)\. These findings suggest that RL training with purely binary rewards may inadvertently reinforce surface\-level length heuristics, whereas supervised fine\-tuning on balanced data is more effective at preventing this\.

Recency bias\.All models show statistically significant recency bias at the overall level \(p<0\.01p<0\.01\) exceptSynthetic\-Reason\-SFT\-DAPO\. Crucially, however, we argue this is not a spurious artefact: in competitive NLP benchmarking, newer methods genuinely tend to outperform older ones\. Rather than indicating a bias the model needs to overcome, the recency signal reflects a learnable empirical prior about scientific progress\. This interpretation is supported by the fact that even the frontier zero\-shot models pick up on this signal; it is a feature of the task domain\.

### B\.8Few\-Shot Ablation with GPT\-5

To test whether in\-context learning can substitute for task\-specific fine\-tuning, we evaluate GPT\-5 in a 3\-shot setting\. For each test pair, we prepend 3 demonstration examples—one from each difficulty category \(1​σ1\\sigma,2​σ2\\sigma,3​σ3\\sigma\)—as in\-context examples before asking for the final prediction\. The same 3 examples are used for all test pairs\. This ablation directly addresses whether the performance gap between our fine\-tuned 8B models and GPT\-5 can be closed by providing GPT\-5 with task demonstrations\.

ModelZero\-shot3\-shotGPT\-5 Reasoning \(Low\)57\.6557\.89GPT\-5 Reasoning \(Med\)59\.6159\.03GPT\-5 Reasoning \(High\)61\.1060\.41Table 10:Zero\-shot vs\. 3\-shot GPT\-5 accuracy \(%\) on our in\-domain test set\. Few\-shot examples marginally improve low\-reasoning performance but slightly degrade medium and high\.As shown in Table[10](https://arxiv.org/html/2605.21491#A2.T10), few\-shot examples do not meaningfully close the performance gap\. While the low\-reasoning setting shows a marginal improvement \(\+0\.24\+0\.24points\), both medium and high reasoning levels*degrade*\(Med:−0\.58\-0\.58, High:−0\.69\-0\.69points\)\. This suggests that the comparative empirical forecasting task cannot be solved through in\-context pattern matching alone; the performance gap between GPT\-5 and our fine\-tuned models reflects a genuine difference in internalised task priors that arises from optimising on thousands of labelled pairs, rather than a deficiency addressable by a few demonstrations\.

### B\.9Conditional Accuracies

We define conditional accuracy as the accuracy of predictions within all the consistent pairs i\.e\. we consider only the subset of total pairs that show no position bias to compute the accuracy§[6\.1](https://arxiv.org/html/2605.21491#S6.SS1)\. Table[11](https://arxiv.org/html/2605.21491#A2.T11)reports the conditional accuracy of various models\. The number in brackets denote the total number of consistent pairs\. The total here represents the augmented set\.

Model / Method1\-σ\\sigma\(988\)2\-σ\\sigma\(568\)3\-σ\\sigma\(182\)Overall\(1738\)Qwen3Base69\.47 \(262\)72\.55 \(204\)50\.00 \(40\)69\.17 \(506\)Base \(Reasoning\)63\.86 \(404\)64\.71 \(238\)46\.67 \(60\)62\.68 \(702\)Direct\-SFT75\.92 \(922\)88\.69\(548\)90\.59 \(170\)81\.71 \(1640\)Reason\-DAPO80\.90 \(848\)85\.54 \(498\)86\.36 \(176\)83\.05 \(1522\)Reason\-SFT\-DAPO75\.59 \(844\)87\.55 \(514\)87\.36 \(174\)80\.94 \(1532\)Reason\-SFT\-DrGRPO76\.58 \(854\)86\.11 \(504\)85\.39 \(178\)80\.73 \(1536\)Llama3\.1Base52\.87 \(522\)64\.96 \(274\)64\.15 \(106\)57\.87 \(902\)Base \(Reasoning\)65\.17 \(402\)64\.93 \(268\)68\.97 \(58\)65\.38 \(728\)Direct\-SFT78\.17 \(678\)84\.62 \(390\)81\.33 \(150\)80\.62 \(1218\)GPT\-5Reasoning \(low\)67\.13 \(864\)67\.48 \(492\)56\.25 \(160\)66\.09 \(1516\)Reasoning \(med\)69\.03 \(846\)70\.00 \(500\)62\.96 \(162\)68\.70 \(1508\)Reasoning \(high\)70\.67 \(866\)68\.24 \(510\)60\.00 \(170\)68\.69 \(1546\)Gemini 2\.5 FlashBase \(Reasoning\)72\.13 \(574\)66\.67 \(342\)55\.93 \(118\)68\.47 \(1034\)

Table 11:Conditional Accuracy \(%\) breakdown across differentσ\\sigmacategories\. Values in parentheses denote the total number of samples \(NN\) for that category that were consistent\. They are no directly comparable since N changes\.
### B\.10Idea Ranking

We analyze how well the trained models do when used for idea ranking on both in\-domain \(ID\) \(from the train\-test split\) and cross\-domain \(CD\) test set\. We pick all leaderboards/research goals that have at least 3 unique entries/ideas in them\. We do\(n2\)\\binom\{n\}\{2\}many comparison\. We rank the ideas based on the number of times each idea wins when compared to all others\. Ideas with ties are given the same rank\. Comparisons that have inconsistent predictions are dropped and not considered\. We assess the quality of ranking using Top\-1 Accuracy and Root Mean Square Error \(RMSE\) of the true and predicted ranks\.

ModelIn\-Domain \(ID\)Cross\-Domain \(CD\)Con\. \(%\)↑\\uparrowTop\-1 \(%\)↑\\uparrowRMSE↓\\downarrowCon\. \(%\)↑\\uparrowTop\-1 \(%\)↑\\uparrowRMSE↓\\downarrowQwen3Base29\.5540\.001\.875\.3538\.462\.45Base \(Reason\)40\.0931\.431\.7319\.3428\.211\.87Direct\-SFT90\.7144\.761\.2279\.8431\.821\.96Reason\-SFT\-DAPO83\.9642\.861\.1277\.3728\.891\.73Reason\-DAPO84\.8551\.431\.2976\.9533\.331\.83Reason\-SFT\-DrGRPO87\.7250\.481\.1280\.2541\.301\.65Synthetic\-Reason\-SFT\-DAPO85\.2743\.811\.3271\.7436\.361\.80GPT\-5Low85\.8238\.101\.4877\.5036\.961\.76Medium86\.8836\.191\.4181\.4834\.781\.78High85\.7735\.241\.4182\.9943\.481\.77

Table 12:Performance comparison on In\-Domain \(ID\) and Cross\-Domain \(CD\) test sets\. Metrics reported are Overall Consistency Rate \(Con\.\), Top\-1 Accuracy, and Median RMSE\.Bold: Best,Underline: Second Best within each domain\. \(↓\) lower is better; \(↑\) higher is better\.![Refer to caption](https://arxiv.org/html/2605.21491v1/images/Rank5.png)Figure 11:Distribution of consistency rate \(%\) across different research goals/leaderboards for the cross\-domain test set
![Refer to caption](https://arxiv.org/html/2605.21491v1/images/Rank6.png)Figure 12:Distribution of RMSE across different research goals/leaderboards for the cross\-domain test set

![Refer to caption](https://arxiv.org/html/2605.21491v1/images/Rank7.png)Figure 13:Distribution of consistency rate \(%\) across different research goals/leaderboards for the in\-domain test set
![Refer to caption](https://arxiv.org/html/2605.21491v1/images/Rank8.png)Figure 14:Distribution of RMSE across different research goals/leaderboards for the in\-domain test set

- •We observe that the untrained base models ofQwen3, have a high position bias, thus have a poor overall consistency rate \(Table[12](https://arxiv.org/html/2605.21491#A2.T12)\)\. Meanwhile, the fine tuned models show drastic improvement, with consistency almost at par \(or better in case of In\-Domain test\) with GPT\-5\.
- •Reason\-SFT\-DrGRPOachieves better Top\-1 accuracy than GPT\-5 under low and medium reasoning effort on the CD test\. All of the trained models achieve better Top\-1 accuracy than GPT\-5 on the ID test\.
- •Before comparing models for their RMSE, it is important to keep in mind that though median is a more robust statistic which is not affected by outliers, it depends highly on the number of samples one has\. To make meaningful comparisons, one should have at least similar sample size\. Similar consistency rates allow for fair comparisons\. For example if we had 10 leaderboards with each having 4 entries and 0% consistency, all the entries within each leaderboard would be assigned a rank of 1\. So all of them would have an RMSE of 1\.87, which in turn would also be the median\. This is lower than what you see for Direct\-SFT even when you have no meaningful comparisons or predictions anywhere\! This example would also have a 100% Top\-1 accuracy\! To prevent this we only use Top\-1 accuracy from leaderboards that have at least 2 different ranks predicted\. This might still not be enough since we see high Top\-1 accuracy for base models when they actually don’t do very well\.
- •Reason\-SFT\-DAPOandReason\-SFT\-DrGRPOachieve better RMSE than GPT\-5 \(across all reasoning efforts\) on CD test, hence showing the potential of such models in filtration and idea re\-ranking\. All trained models achieve better RMSE compared to GPT\-5 on the ID test\.
- •Figure[12](https://arxiv.org/html/2605.21491#A2.F12)&[14](https://arxiv.org/html/2605.21491#A2.F14)and[12](https://arxiv.org/html/2605.21491#A2.F12)&[14](https://arxiv.org/html/2605.21491#A2.F14)show the distribution of the consistency and RMSE across different leaderboards/research goals for ID and CD test respectively\. The highest outlier of Direct\-SFT and Reason\-SFT\-DrGRPO is lower than that of GPT\-5, while Reason\-DAPO and Reason\-SFT\-DAPO have comparable RMSE on CD test\.
- •The consistency distribution is left\-skewed for all cases except for the base models, with most values at high consistency\. And the RMSE distribution is right\-skewed in all almost all cases\. The skewness is greater on ID test compared to CD test, but has more number of outliers in general as well\.
- •Reason\-SFT\-DrGRPOshows least drastic change in all three metrics for ranking compared to other models showing that it has learnt to generalize better than the others\.

These observations show that fine\-tuned 8B models that are trained for comparative empirical forecasting could be used as an idea re\-ranker, performing better than frontier models under zero\-shot setting\.

### B\.11Probabilistic Calibration

Three metrics are used to quantify calibration\.

Brier Score\(lower is better\) measures the mean squared probability error:

BS=1N​∑i=1N\(pi−yi\)2,\\mathrm\{BS\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\(p\_\{i\}\-y\_\{i\}\)^\{2\},\(12\)wherepip\_\{i\}is the model’s predicted probability for the correct class andyi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}is the ground\-truth label\. The Brier Score is a proper scoring rule that penalizes both overconfidence and underconfidence\.

Expected Calibration Error \(ECE\)measures the mean absolute gap between binned confidence and binned accuracy\. Predictions are partitioned intoB=10B=10equal\-width bins based on confidence, and the calibration error is computed as:

ECE=∑b=1B\|ℬb\|N⋅\|acc​\(ℬb\)−conf​\(ℬb\)\|,\\mathrm\{ECE\}=\\sum\_\{b=1\}^\{B\}\\frac\{\|\\mathcal\{B\}\_\{b\}\|\}\{N\}\\cdot\\big\|\\mathrm\{acc\}\(\\mathcal\{B\}\_\{b\}\)\-\\mathrm\{conf\}\(\\mathcal\{B\}\_\{b\}\)\\big\|,\(13\)whereacc​\(ℬb\)\\mathrm\{acc\}\(\\mathcal\{B\}\_\{b\}\)andconf​\(ℬb\)\\mathrm\{conf\}\(\\mathcal\{B\}\_\{b\}\)are the mean accuracy and mean confidence of predictions in binbb, and\|ℬb\|\|\\mathcal\{B\}\_\{b\}\|is the bin count\. ECE is a binning approximation to the full calibration integral, weighted by the empirical distribution of confidence values\.

Maximum Calibration Error \(MCE\)captures the worst\-case miscalibration across all bins:

MCE=maxb:\|ℬb\|\>0⁡\|acc​\(ℬb\)−conf​\(ℬb\)\|\.\\mathrm\{MCE\}=\\max\_\{b:\\,\|\\mathcal\{B\}\_\{b\}\|\>0\}\\big\|\\mathrm\{acc\}\(\\mathcal\{B\}\_\{b\}\)\-\\mathrm\{conf\}\(\\mathcal\{B\}\_\{b\}\)\\big\|\.\(14\)
Because the primary evaluation metric filters to consistent predictions \(§[6\.1](https://arxiv.org/html/2605.21491#S6.SS1)\), we report thedebiasedvariant: each sample is paired with its swapped counterpart and their probabilities are averaged in a common reference frame to obtain a position\-debiased confidence:

p~i=piorig\+\(1−piswap\)2,\\tilde\{p\}\_\{i\}=\\frac\{p\_\{i\}^\{\\mathrm\{orig\}\}\+\(1\-p\_\{i\}^\{\\mathrm\{swap\}\}\)\}\{2\},\(15\)wherepiorigp\_\{i\}^\{\\mathrm\{orig\}\}is the model’sP​\(class=1\)P\(\\text\{class\}=1\)in the original ordering andpiswapp\_\{i\}^\{\\mathrm\{swap\}\}is the corresponding probability from the swapped ordering, inverted to align reference frames\.

Using these calibration metrics under the*debiased*evaluation regimes, we assess the confidence quality of our trained models\.

FamilyModelDB↓\\downarrowDE↓\\downarrowDM↓\\downarrowQwenBase0\.25070\.09370\.2832Base \(Reason\)0\.26350\.13920\.3276Direct\-SFT0\.16760\.13900\.1915Reason\-DAPO0\.17550\.15840\.5475Reason\-SFT\-DAPO0\.19410\.16880\.3112Reason\-SFT\-DrGRPO0\.19700\.17340\.3641Synthetic\-Reason\-SFT\-DAPO0\.21860\.20090\.4320LlamaBase0\.26850\.15520\.3275Base \(Reason\)0\.28820\.19310\.3373Direct\-SFT0\.21290\.12050\.2184

Table 13:Calibration metrics on the in\-domain test set\.DB: Debiased Brier,DE: Debiased ECE,DM: Debiased MCE\.Bold: best within family,Underline: second best within family\.Fine\-tuning substantially improves Brier scores across both model families: base models score≈0\.25−26\\approx 0\.25\-26whereas fine\-tuned models achieve 0\.16–0\.21, representing a reduction of roughly 30–40%\.Qwen3 Direct\-SFTachieves the best overall calibration, with Consistent Brier of0\.1676and second best Consistent ECE of0\.1463\. This means that on stable, position\-invariant predictions, the model’s confidence levels are well\-aligned with its accuracy\. Additionally it is important to compare only between the non\-reasoning and reasoning models respectively\. This is because the non reasoning models output only 0/1 where as reasoning models generate a certain lenght of tokens before making the prediction which makes the prediction conditional on the tokens generated before making the prediction\.

The RL\-tuned variants \(Reason\-DAPO, Reason\-SFT\-DAPO, Reason\-SFT\-DrGRPO\) show higher MCE values \(0\.31–0\.87 consistent\), indicating that while their average calibration is reasonable, their worst\-case confidence bins are poorly calibrated\. This may reflect that RL training with a binary reward signal encourages the model to be more decisive \(higher entropy collapse\) at the cost of miscalibration in low\-confidence regions\.

Our results largely reflect the findings of recent literature demonstrating that overconfidence and miscalibration are pervasive when utilizing raw logit probabilities as direct proxies for confidence estimation in large language models\. As systematically outlined byGenget al\.\([2024](https://arxiv.org/html/2605.21491#bib.bib47)\), generative LLMs inherently exhibit overconfidence, and relying on extracted token logits often fails to account for the semantic variability of language, ultimately yielding unreliable and uncalibrated confidence estimates\. Furthermore, our observations regarding the inadequacy of logit\-based probabilities are empirically supported by recent work fromZhouet al\.\([2025](https://arxiv.org/html/2605.21491#bib.bib48)\)\. In their evaluation of uncertainty quantification methods, they demonstrate that complex metrics derived directly from token logits frequently yield poorly calibrated confidence scores and are systematically outperformed by simpler strategies such as verbalized direct prompting\. Consequently, these findings reinforce that utilizing logit probabilities directly is a suboptimal approach for reliable confidence estimation, frequently leaving models highly uncalibrated under such settings\.

## Appendix CPrompts

`Idea Extraction Prompt for Paraphrasing the Idea Reasoning Extraction Research Goal Extraction True Original Paper analysis Prompt format used for FT with RL`

`Appendix D Examples D\.1 Qwen3\-8B\-Reason\-SFT\-DAPO An example where the RL trained model successfully reflects and reasons the probable cause for one idea being better than the other and successfully predicts the correct answer\. And a second example where the same model reasons and reflects but the reasoning leads to wrong final answer prediction\. Input Output Input Output D\.2 Qwen3\-8B\-Reason\-DAPO For the same example as the above \(the second example in §\(D\.1\), the model trained with DAPO applied directly to the base model with a penalty for shorter responses, we see clear reward hacking\. The thinking starts off well and rightly acknowledges the need for evaluating likely performance on the specific task/research goal but then enters a repetitive loop of the same sentences before making the final prediction, thus failing to reason coherently or justifying the ”why”\. Yet it gets the final prediction correct unlike the Qwen3\-8B\-Reason\-SFT\-DAPO model\. Output D\.3 Qwen3\-8B\-Reason\-SFT\-DrGRPO As mentioned in §\(7\.3\), the RL trained model using Dr\. GRPO started generating very minimal number of think tokens, often just a direct statement of which idea is better and without justifying ”why” this leads to better performance\. Input Output D\.4 Qwen3\-8B\-Synthetic\-Reason\-SFT\-DAPO An example of Synthetic\-SFT\-DAPO model reasoning and debating multiple strengths and weaknesses before correctly concluding the better idea for the given research goal\. Unlike Reason\-SFT\-DrGRPO, it does not tend to minimize the reasoning traces or the responses\. Input Output`