SGHA: Evidence-Grounded Research Problem Discovery with Local Language Models

# SGHA: Evidence-Grounded Research Problem Discovery with Local Language Models
Source: [https://arxiv.org/html/2608.17501](https://arxiv.org/html/2608.17501)
###### Abstract

Recent efforts toward fully automated AI scientists have demonstrated that language\-model agents can generate hypotheses, execute experiments, and draft scientific manuscripts\. However, during the early stages of research, when research problems are formulated, these AI scientists often rely heavily on proprietary frontier models\. Their proposals are shaped by opaque parametric knowledge and by literature searches conditioned on the proposals themselves\. Such knowledge is effectively a black box, and this dependence makes the evidential basis and validity of generated research problems difficult to audit and leaves the process vulnerable to model\-specific hallucinations and biases\. Furthermore, if proprietary research materials are transmitted to external APIs, the use of these models creates confidentiality, privacy, and data\-governance concerns\.

We introduce the Structural Gap Hypothesis Agent \(SGHA\), a fully automated, corpus\-first research\-problem discovery system that runs entirely on a local LLM\. SGHA structures a scientific literature corpus into evidence\-linked paper objects and a typed evidence graph, detects unresolved structural patterns across papers, screens candidate gaps before formulation, and produces traceable research\-problem families\. In particular, it is able to output assumptions, objectives, success criteria, and remaining ambiguities\. All LLM\-based components of SGHA are executed using a locally served open\-weight 9B language model, without requiring proprietary frontier\-model APIs\. We compare SGHA with the AI Scientist\-v2 idea formulation module in five machine\-learning domains\. Our results suggest that explicit corpus structure and evidence\-constrained reasoning can support promising, inspectable research\-problem formulation without relying on frontier models during generation or verification\.

## 1 Introduction

Recent language\-model agents can generate research ideas and automate parts of the scientific workflow, including writing code, running experiments, analyzing results, and preparing manuscripts\[[21](https://arxiv.org/html/2608.17501#bib.bib1),[13](https://arxiv.org/html/2608.17501#bib.bib2),[18](https://arxiv.org/html/2608.17501#bib.bib3),[20](https://arxiv.org/html/2608.17501#bib.bib7),[9](https://arxiv.org/html/2608.17501#bib.bib4),[14](https://arxiv.org/html/2608.17501#bib.bib5),[5](https://arxiv.org/html/2608.17501#bib.bib6)\]\. Several end\-to\-end systems begin by generating candidate ideas within a user\-provided topic or experimental scaffold and then try to develop them\. In this paper, we study a different starting point\. We ask whether an automated system can identify research problems that are already suggested by unresolved patterns in the scientific literature, even when no single paper states them directly\.

![Refer to caption](https://arxiv.org/html/2608.17501v1/Figures/SGHA.png)Figure 1:Overview of the SGHA pipeline\. “Deterministic” denotes non\-LLM stages\. The evolutionary branch is optional and is reported separately from the verification\-gated family outputs\. Evolutionary operations are iterated multiple times and fed into the problem synthesis stage\.Many research questions only become visible when several papers are considered together\. A set of methods may fail under the same condition, several guarantees may rely on the same strong assumption, or a limitation may be reported repeatedly without being turned into a concrete research problem\. In such cases, the evidence for a problem is already present, but it is distributed across papers\.

As a simple example, consider the literature on stochastic multi\-armed bandits from 1985 through 2011\. Researchers try to characterize the performance of an algorithm in terms of the cumulative reward shortfall called regret\. A popular algorithm based on an upper confidence bound achievesO⁡\(log⁡T\)O\(\\log T\)regret for bounded rewards using a concentration inequality\[[4](https://arxiv.org/html/2608.17501#bib.bib11)\]; this rate is optimal\[[11](https://arxiv.org/html/2608.17501#bib.bib10)\]\. Yet its extension to heavy\-tailed rewards was not well known\. Bubeck et al\. later filled this gap using robust estimators, and Agrawal et al\. subsequently improved and clarified the connection to bounded\-reward models\[[6](https://arxiv.org/html/2608.17501#bib.bib12),[1](https://arxiv.org/html/2608.17501#bib.bib13)\]\. The research problem of characterizing regret under heavy\-tailed rewards is what we call a*structural gap*: a pattern that is present in the literature but not yet resolved\. SGHA emulates this process prospectively by detecting and verifying analogous corpus\-level gaps and formulating them as evidence\-grounded research problems\.

Existing systems either condition hypothesis generation on a research background comprising an explicit research question and a survey of prior work\[[19](https://arxiv.org/html/2608.17501#bib.bib9)\], or formulate candidate research problems, directions, or hypotheses within a search space anchored by a human\-authored code template \(AI Scientist v1\)\[[13](https://arxiv.org/html/2608.17501#bib.bib2)\], a broad topical description \(AI Scientist v2\)\[[18](https://arxiv.org/html/2608.17501#bib.bib3)\], a scientist\-specified research goal\[[9](https://arxiv.org/html/2608.17501#bib.bib4)\], a selected core paper\[[5](https://arxiv.org/html/2608.17501#bib.bib6)\], or an input topic used for graph\-grounded literature retrieval\[[12](https://arxiv.org/html/2608.17501#bib.bib16)\]\. ResearchAgent explicitly includes an LLM\-based problem\-identification step, but generates the problem from a core paper, its citation neighborhood, and retrieved entities\. Very recently, Graph2Idea retrieves papers according to an input topic, transforms them into structured knowledge triples, dynamically constructs a target\-centered knowledge graph, and then plans research directions before synthesizing complete ideas\[[12](https://arxiv.org/html/2608.17501#bib.bib16)\]\. To our knowledge, these systems do not integrate all three capabilities: typed structural\-gap detection over a topic\-bounded full\-text corpus, corpus\-level verification before formulation, and passage\-level provenance for evidence, counterevidence, and explicit ambiguities\. Beyond base\-model capability, deployment is an independent practical consideration\. In their reported implementations or principal experimental configurations, several prominent systems use proprietary models or hosted inference services for central idea generation or refinement\[[13](https://arxiv.org/html/2608.17501#bib.bib2),[5](https://arxiv.org/html/2608.17501#bib.bib6),[18](https://arxiv.org/html/2608.17501#bib.bib3),[9](https://arxiv.org/html/2608.17501#bib.bib4)\], limiting exact reproducibility and model\-level auditability relative to locally served open\-weight inference\. When unpublished or sensitive research materials are transmitted to external services, this additionally introduces deployment\-dependent confidentiality, privacy, and data\-governance concerns\.

We introduce*SGHA*, the Structural Gap Hypothesis Agent, a training\-free system for generating candidate research\-problem families from scientific papers\. Figure[1](https://arxiv.org/html/2608.17501#S1.F1)summarizes the pipeline\. Given a topic, SGHA retrieves and parses relevant papers, extracts evidence\-backed scientific tuples and paper\-level objects, and builds a typed evidence graph linked to supporting passages\. It then searches this graph for candidate structural gaps\. Only gaps that pass a hard verification gate proceed to formulation\. SGHA first writes a direct formulation close to the evidence, then generates broader variants and uses a separate critic to reject variants that are unsupported or only rhetorically stronger\. Surviving formulations are grouped into project families and converted into semi\-formal problem statements that describe the main variables, assumptions, feedback or observation model, objective, success criteria, and unresolved ambiguities\. When the evidence is too weak, SGHA reports a low\-signal outcome rather than forcing a research menu\.

SGHA also supports two complementary modes\. A profile\-conditioned mode uses a researcher’s literature context and an optional target topic to prioritize problems relevant to that context\. An evolutionary exploration branch iteratively broadens literature\-grounded seeds to produce additional candidate directions\. These modes use the same underlying goal: to keep generated problems tied to evidence rather than treating them as unconstrained prompt completions\.

We evaluate SGHA across five machine\-learning domains: bandits, in\-context learning, reasoning and test\-time computation, offline reinforcement learning, and uncertainty estimation\. The main experiments use a frozen localQwen/Qwen3\.5\-9Bmodel for generation, with no task\-specific training or fine\-tuning\. Across 1,250 selected papers, SGHA extracts structured content from 1,044 papers\. After verification, 39 candidate gaps remain and lead to 15 formalized project families\. Our primary comparison is with the AI\-Scientist\-v2 ideation workflow configured with the same local model and matched output counts\. This is a system\-level comparison rather than an information\-matched ablation: SGHA uses the bounded corpus, whereas the AI\-Scientist\-v2 retains its Semantic Scholar retrieval\. Since SGHA is meant to write research\-problem formulations rather than full project plans, we evaluate the comparison with a formulation\-only rubric\. Under this rubric, a method\-label\-masked evaluation using five frontier\-model judges from different providers assigns SGHA higher mean scores on all but one of the formulation\-quality criteria, including well\-posedness, assumption\-boundary clarity, formalizability, source\-grounded specificity, and ambiguity handling\. The local AI\-Scientist\-v2 baseline receives a higher mean score only on scope control, consistent with its tendency to produce narrower proposal\-style ideas\. The hosted judges are used only for post hoc evaluation after the candidate outputs have been frozen and are not part of SGHA’s generation or verification pipeline\. Additional comparisons using Claude Opus in the AI\-Scientist\-v2 workflow and the public MOOSE\-Star HC model\[[19](https://arxiv.org/html/2608.17501#bib.bib9)\]were conducted\. In the Claude Opus diagnostic, which matches retained output counts but not compute or information budgets, SGHA receives a competitive numerical mean score on overall formulation quality to AI\-Scientist\-v2 with Claude Opus\.

We additionally report exploratory diagnostics on profile conditioning, model configuration, corpus size, and evolutionary\-output availability, treating them as capability or sensitivity analyses rather than controlled comparisons\.

Our main contributions are:

Contributions•Corpus\-first research\-problem discovery\.We present SGHA, an end\-to\-end pipeline that detects unresolved cross\-paper patterns in a bounded corpus and converts verified gaps into evidence\-grounded families of research problems\. All generation and verification stages run on a locally served open\-weight LLM without requiring proprietary frontier\-model APIs\.•Verification before formulation\.SGHA does not formulate a research problem from every detected pattern\. It first screens each candidate for evidential support and prior coverage within the bounded corpus, formulates only those that pass verification, and uses a separate critic to reject unsupported ambition\-expanded variants\.•Traceable problem formulations\.SGHA outputs problem specifications with explicit assumptions, objectives, and success criteria\. Each specification retains its source motif type and includes passage\-level evidence and counterevidence links together with explicit ambiguity annotations\.•Empirical analysis of quality, breadth, and failure modes\.We evaluate SGHA across multiple domains with a local model, compare it primarily with a locally configured AI\-Scientist\-v2 idea generation workflow, and report exploratory diagnostics on profile conditioning, cross\-model behavior, corpus size, low\-signal behavior, and evolutionary\-exploration availability\.

Related Work on Automated Research Ideation with Local LLMsOpen\-weight local LLMs have been used in automated scientific ideation\. SciMON fine\-tunes T5\-large and, in a biochemical case study, Meditron\-7B on paper\-derived supervision, while its reported iterative novelty\-boosting experiments use GPT\-4\[[17](https://arxiv.org/html/2608.17501#bib.bib14)\]\. HypER uses a Llama\-3\.1\-70B teacher to construct and score citation chains and LoRA\-tunes Phi\-3 Mini 3\.8B for relevance and chain\-validation tasks; hypothesis generation is then performed at inference time, conditioned on validated chains\[[15](https://arxiv.org/html/2608.17501#bib.bib15)\]\. MOOSE\-Star fine\-tunes R1\-Distill\-Qwen\-7B models for inspiration retrieval and hypothesis composition using supervision generated with a locally deployed 32B teacher\[[19](https://arxiv.org/html/2608.17501#bib.bib9)\]\. These locally executable open\-weight instantiations rely on task\-specific fine\-tuning or distillation, and they do not explicitly model research\-problem discovery as detecting and verifying unresolved structural gaps across a bounded literature corpus\. HypoGeniC can run with an open\-weight Mixtral\-8x7B model and updates a bank of predictive natural\-language hypotheses from labeled examples without fine\-tuning the generator; it does not search a literature corpus for research problems\[[21](https://arxiv.org/html/2608.17501#bib.bib1)\]\.Among the most closely related prior works, Problem Discovery via Structural Motifs in Knowledge Graphs \(PRISM\)\[[8](https://arxiv.org/html/2608.17501#bib.bib8)\]explores structural\-motif\-based problem discovery by detecting structural gaps in a paper corpus, converting graph motifs into structured problem objects, and refining them with a novelty\-and\-feasibility critic\. Its expert\-evaluation protocol is proposed rather than carried out, leaving open whether detected motifs represent genuine unresolved gaps rather than retrieval or extraction artifacts or questions already covered elsewhere in the corpus\. SGHA partially addresses this issue within the bounded corpus by verifying candidate gaps before formulation, merging redundant directions, and making remaining ambiguities explicit\.

## 2 Structural Gap Hypothesis Agent

SGHA is designed around a simple constraint: a generated research problem should be tied to evidence in the literature\. Given a topic, the system first builds a structured evidence base from papers, then looks for recurring patterns in that evidence, verifies the resulting candidate gaps, and only then formulates research\-problem families\. This is different from asking a language model to directly generate ideas from a topic\. The language model is used in specific stages, while graph construction, motif detection, verification gating, family consolidation, and final auditing keep the pipeline grounded\.

The overview figure in the introduction shows the full pipeline\. In this section, we describe the main objects that move through it: evidence tuples, typed evidence graphs, structural motifs, candidate gaps, verified gaps, project families, and formal problem statements\. We use one Bandits example as a running reference\. In the selected corpus, SGHA extracts evidence that diffusion\-based Thompson sampling \(dTS\) is analyzed under linear reward or link\-function assumptions and bounded contexts, while related evidence points to nonlinear score functions requiring approximation\. This trace later leads to the project family*Characterizing the Non\-Convex Failure Regime of Diffusion\-Based Contextual Bandits*\. Section[4\.4](https://arxiv.org/html/2608.17501#S4.SS4)shows the final generated formulation, proposal\-style abstract, formal problem object, risks, and ambiguity flags for this example\.

### 2\.1 Building the Evidence Base

SGHA begins by constructing a topic\-specific corpus\. We distinguish three paper\-level stages\. In stage one,*selected*papers are chosen for the corpus before parsing\. Next,*parsed*papers are selected papers for which usable text is obtained, and finally, the*extracted*papers are parsed papers for which SGHA produces a valid structured extraction\.

For each extracted paper, SGHA records both paper\-level objects and relation\-level evidence\. The paper\-level object contains methods, tasks, datasets, metrics, assumptions, results, limitations, failure conditions, claims, contradictions or tensions, and future work\. These fields are useful because not every important statement in a paper becomes a relation edge in the graph\.

The relation\-level representation consists of evidence tuples of the form

\(subject,relation,object\),\(\\text\{subject\},\\text\{relation\},\\text\{object\}\),with source provenance\. Relations includeassumes,fails\_under,limited\_by,addresses,not\_addressed\_by,contradicts,improves\_over,evaluated\_on,uses\_dataset, andmeasured\_by\. Each tuple also stores supporting text, paper section, claim type, polarity, condition, evidence type, confidence, and source\-paper identifier\.

These tuples are local evidence records extracted from individual papers\. Before SGHA uses them downstream, it validates them by removing invalid relations, unsupported spans, missing evidence, generic subjects, and relation\-specific errors\. This keeps the evidence base from being dominated by vague labels such as “our method” or claims that are not supported by the parsed text\.

These tuples are local evidence records extracted from individual papers\. Before SGHA uses them in the graph, it applies deterministic validation and cleanup\. First, the relation must be one of the allowed relation types, and the subject, object, and evidence fields must be present\. Second, the supporting evidence span must match the parsed paper text after normalization, so that the tuple can be traced back to the source paper\. Third, SGHA removes generic subjects such as "method", "algorithm", or "our method", because these labels are too vague to support graph construction\. It also filters overly generic task labels, drops evaluation tuples that do not include task, dataset, or metric context, and normalizes metadata such as relation polarity\. For example,fails\_underandlimited\_byare treated as negative relations, whileassumesis treated as neutral\. These checks keep the evidence base tied to specific paper text and prevent vague or unsupported tuples from driving later gap detection\.

### 2\.2 From Evidence Tuples to a Typed Graph

SGHA then inserts the validated paper objects and evidence tuples into a typed evidence multigraph\. The graph has different kinds of nodes for different scientific objects\. Paper nodes represent source papers, while other nodes represent methods, assumptions, limitations, failure conditions, claims, tasks, datasets, metrics, and results\. The edges come from the extracted tuples\. For example, an edge may state that a methodassumesan assumption,fails\_undera condition, or ismeasured\_bya metric\. Each edge keeps its source provenance, including the paper identifier and supporting evidence span\. Thus, the graph is not just a citation graph or a collection of paper summaries\. It records the scientific relations that SGHA extracted from the corpus: what methods assume, where they are evaluated, what they improve over, where they fail, and which limitations remain unresolved\.

Figure[2](https://arxiv.org/html/2608.17501#S2.F2)shows this construction for the running Bandits example\. The left side shows the surrounding literature context and highlights the two source papers\[[2](https://arxiv.org/html/2608.17501#bib.bib19),[3](https://arxiv.org/html/2608.17501#bib.bib18)\]used in the trace\. The right side shows the typed local graph built from the extracted tuples\. The method node dTS is connected to assumption nodes for linear rewards, bounded contexts, and linear link functions\. It is also connected to an open issue around nonlinear diffusion models and to a failure\-condition node for nonlinear score functions\. This local graph is the evidence structure that later triggers the assumption–failure motif\.

![Refer to caption](https://arxiv.org/html/2608.17501v1/Figures/evidence_graph_for_contextual_bandits.png)Figure 2:Evidence graph view of the Bandits worked example\. The left side shows the surrounding literature context and the two source papers used in the worked example\. The right side shows the typed local evidence graph: dTS is linked to linearity and bounded\-context assumptions, an open issue around nonlinear diffusion models, and a nonlinear score\-function failure condition\. This local graph supports the assumption–failure candidate gap used later in the qualitative result\.
### 2\.3 Structural Motifs and Candidate Gaps

Once the evidence graph is built, SGHA searches it for predefined graph patterns, which we call structural motifs\. A motif is a small pattern of relations that may point to a possible research gap\. For example, a method may depend on an assumption in one part of the literature and appear to fail when a related condition changes in another part of the literature\. When SGHA finds such a pattern, it creates a candidate structural gap\. This candidate is not yet a verified research problem, and is only a place in the literature graph that deserves closer inspection\.

Figure[3](https://arxiv.org/html/2608.17501#S2.F3)shows several motif families used by SGHA\. An assumption–failure motif links a method to an assumption and to a related failure condition\. A shared\-failure motif appears when several methods fail under the same condition\. A repeated\-unaddressed\-limitation motif appears when a limitation is mentioned across papers without evidence that it has been addressed\. Other motifs capture conflicting claims, sparse evaluation, unrealistic assumptions, missing stress tests, and theory–practice gaps\. These motifs are deliberately simple: their role is to find candidates for later screening and verification, not to decide on their own that a research problem is valid\.

![Refer to caption](https://arxiv.org/html/2608.17501v1/Figures/structural_motifs_for_gap_discovery.png)Figure 3:Examples of structural motifs used for candidate\-gap discovery\. Each mini\-panel shows a deterministic graph pattern queried over the typed evidence graph\. A motif creates a candidate gap, not a final research problem\. SGHA later screens, verifies, formulates, and consolidates these candidates before they can enter the final output\.In the Bandits example, the local graph links dTS to assumptions that make posterior approximation tractable, and also to a nonlinear score\-function condition where approximation becomes necessary\. This matches the assumption–failure motif\. The resulting candidate gap can thus be read as follows: dTS is analyzed under linear and bounded assumptions, but nearby nonlinear score\-function settings may violate those assumptions\. At this stage, SGHA has only identified a structured pattern in the literature\. The candidate still needs to pass novelty screening and verification before it can be formulated as a research problem\.

### 2\.4 Verification Before Formulation

Such candidate gaps first enter a checking stage\. The goal of this stage is to turn a graph\-derived pattern into a formulation\-ready gap\. SGHA does this in two steps: a corpus\-level screen followed by role\-based verification\.

The corpus\-level screen keeps candidates that remain meaningful after checking nearby evidence in the selected corpus\. Candidates retained by this screen are called*novelty survivors*\. In SGHA, this term has a specific meaning: the candidate has passed the system’s internal corpus screen and is worth deeper review\.

SGHA then sends a selected subset of novelty survivors to role\-based verification\. These are the*reviewed gaps*\. The verification agents examine each reviewed gap from complementary perspectives\. The support agent identifies evidence that the gap is grounded in the corpus\. The skeptic agent looks for nearby counterevidence or possible resolutions\. The feasibility agent checks whether the problem can plausibly be studied\. The mechanism agent asks whether there is a reasonable explanation for the gap\. The verification critic gives a conservative final assessment\.

A reviewed gap becomes a*verified gap*when it passes the hard verification gate\. This gate is deterministic\. It admits candidates with the required agent outputs, no disqualifying parse failures, a survival score above threshold, and a non\-rejecting critic with sufficient confidence\. Verified gaps, then enter the direct\-formulation stage\.

In the running Bandits example, this stage keeps the assumption–failure pattern grounded but also sharpens its interpretation\. The support agent finds evidence for the linear\-assumption issue in dTS\. The skeptic and critic add useful context: approximation\-based approaches are mentioned in the source papers, so the final problem should frame the issue carefully rather than overstate it\. The feasibility and mechanism agents find the problem studyable and mechanistically plausible\. The hard gate passes the candidate, allowing SGHA to formulate it\. In this way, verification turns the graph\-derived dTS candidate into a formulation\-ready gap\.

### 2\.5 From Verified Gaps to Project Families

Once a gap is verified, SGHA treats it as an evidence\-backed starting point for formulation\. The first step is a direct formulation\. This formulation stays close to the verified gap: it states the main setting, the central assumption or failure, the objective, and a plausible theorem, algorithm, or benchmark target\. It also keeps the source gap, motif type, supporting papers, and verification provenance attached to the formulation\.

In the running Bandits example, the direct formulation is*Extending dTS to Non\-Linear Score Functions in Contextual Bandits*\. This is intentionally close to the verified evidence\. It asks how diffusion\-based posterior inference can be extended beyond linear score functions while preserving useful regret guarantees or approximation behavior\.

SGHA then broadens the direct formulation in a controlled way\. It generates conservative, generalized, and bold variants\. The conservative variant makes a small extension of the direct problem\. The generalized variant broadens the problem class or assumptions while keeping the formulation grounded\. The bold variant asks whether the evidence points to a deeper boundary, impossibility, or failure\-regime question\.

This expansion stage is paired with an independent formulation critic\. The critic checks whether a variant makes a real scientific move, rather than only sounding more ambitious\. It looks for source support, specificity, non\-incrementality, and a clear change in the problem class or objective\. Variants that pass this critic become candidates for final project families\.

In the Bandits example, this step changes the formulation from a local extension of dTS to a broader boundary question:*Characterizing the Non\-Convex Failure Regime of Diffusion\-Based Contextual Bandits*\. The accepted formulation no longer asks only how to patch dTS for nonlinear score functions\. It asks when diffusion\-based inference itself fails under nonlinear score landscapes\.

Finally, SGHA consolidates critic\-passing variants into project families\. This step avoids presenting several versions of the same underlying direction as separate outputs\. Variants can be grouped when they share the same verified gap, direct formulation, supporting papers, or research question\. Each family keeps a representative formulation, member variants, source verified gaps, source direct formulations, supporting papers, critic rationale, risks, and internal quality metadata\.

### 2\.6 Formal Problem Statements

After project families are formed, SGHA turns each family into a semi\-formal problem object\. This final step makes the research direction easier to inspect\. The problem object records the main entities, variables, observations or feedback, decision outputs, objective, constraints, success criterion, assumptions, possible result types, source grounding, and ambiguity flags\. The aim is to give a researcher a clear starting point: what is already specified, what the candidate is trying to study, and what still needs to be defined more carefully\.

In the Bandits example, the final family is formalized using a score\-function classℱ\\mathcal\{F\}, a diffusion\-based inference algorithmπdiff\\pi\_\{\\mathrm\{diff\}\}, true and approximate reward distributions, a score landscape, and cumulative regretℛT\\mathcal\{R\}\_\{T\}\. The resulting problem asks when diffusion\-based inference can guarantee sublinear regret, and where this guarantee breaks down because the Gaussian approximation diverges from the true reward distribution under nonlinear score landscapes\.

SGHA also records the parts of the formulation that remain underspecified\. For this example, it flags the meaning of non\-convexity in the score landscape, the operational definition of identification failure, and the precise threshold at which approximation error becomes a failure\. These ambiguity flags are part of the final artifact\. They make clear which definitions a researcher would need to sharpen before turning the candidate into a theorem, algorithm, or benchmark study\.

### 2\.7 Profile Conditioning, Evolutionary Exploration, and Low\-Signal Outcomes

The same SGHA pipeline can also be used in a profile\-conditioned setting\. Here, the input includes a researcher’s literature context and, optionally, a target topic\. This information is used during corpus construction and prioritization, so the evidence base is closer to the researcher’s previous work or stated interests\. The downstream checks remain the same: candidate gaps are still detected from the evidence graph, screened, verified, formulated, critiqued, consolidated, and formalized\. Thus, profile conditioning changes what evidence the system starts from, but not the standards a candidate must satisfy before becoming a final project family\.

SGHA also includes an optional evolutionary exploration branch\. This branch is separate from the main verification\-gated family\-report path\. It starts from a literature\-grounded seed, such as a verified gap or an early formulation, and explores nearby problem variants by changing assumptions, mechanisms, objectives, or regimes\. The variants are then critiqued, selected, and refined over several rounds\. Figure[4](https://arxiv.org/html/2608.17501#S2.F4)illustrates this view: evolutionary exploration can be seen as mutating a graph\-level problem representation while preserving its connection to the original evidence\.

![Refer to caption](https://arxiv.org/html/2608.17501v1/Figures/evolutionary_exploration_as_graph_mutation.png)Figure 4:Graph\-mutation view of the optional evolutionary exploration branch\. Starting from a literature\-grounded seed gap, the branch proposes variants by changing assumptions, mechanisms, objectives, or regimes\. Variants are critiqued, selected, and refined before a new exploratory hypothesis is retained\. This branch is used for breadth and is kept separate from the main verification\-gated family\-report path unless stated otherwise\.The purpose of this branch is breadth\. The main SGHA path is conservative: it produces a compact set of verified, critic\-filtered, and formalized project families\. The evolutionary branch instead explores a wider neighborhood of possible directions around the same evidence base\. For this reason, unless explicitly stated, evolutionary outputs are reported separately from the main project\-family counts and from the main formulation\-quality comparison\.

Finally, SGHA can report a low\-signal outcome when the evidence does not support a final research menu\. This can happen when no candidate passes verification, when no ambition\-expanded variant passes the critic, or when no project family survives consolidation\. In such cases, the system completes the run but reports that the corpus did not yield a sufficiently supported final set of project families\. For a literature\-driven system, this behavior is important: SGHA should surface grounded opportunities when the evidence supports them, and avoid forcing unsupported directions when it does not\.

## 3 Experimental Setup

We evaluate SGHA as a system for producing research\-problem formulations from scientific papers\. The experiments are organized around three questions\. First, can SGHA produce final project\-family artifacts across different areas of machine learning? Second, how do these artifacts compare with outputs from automated ideation baselines under a common formulation\-quality rubric? Third, how does the system behave in additional settings, including profile\-conditioned generation, evolutionary exploration, and Bandits sensitivity analyses over model size and corpus size?

The main SGHA experiments use a frozen localQwen/Qwen3\.5\-9Bmodel, with no task\-specific training or fine\-tuning\. We compare SGHA with output\-count\-matched baselines and evaluate all retained candidates with the same blinded formulation\-only judge\. Alongside judge scores, we also report structural properties of the generated artifacts, such as source grounding, assumptions, formal problem statements, risks, and ambiguity flags\.

### 3\.1 Domains and Corpora

We run SGHA on five machine\-learning domains: bandits, in\-context learning, reasoning and test\-time computation, offline reinforcement learning, and uncertainty estimation\. These domains cover different parts of ML research, from sequential decision\-making and in\-context behavior to test\-time reasoning, offline policy learning, and uncertainty quantification\. Each domain uses a budget of 250 selected papers\. To include both topical and source diversity, the bandits, in\-context learning, and reasoning/test\-time\-computation corpora are drawn from OpenReview, while the offline\-reinforcement\-learning and uncertainty\-estimation corpora are drawn from arXiv\.

Together, these five corpora contain 1,250 selected papers\. SGHA parses 1,049 of them, meaning that usable text is obtained, and produces valid structured scientific extractions for 1,044 papers\. These extractions yield 8,634 evidence tuples, which are used to build typed evidence graphs and instantiate candidate structural gaps\. Table[1](https://arxiv.org/html/2608.17501#S3.T1)reports the corpus and extraction counts by domain\.

Table 1:Corpus and extraction counts for the five main SGHA runs\. “Parsed” denotes papers for which usable text is obtained; “Extracted” denotes papers for which SGHA produces a valid structured extraction\.DomainSourceSelectedParsedExtractedTuplesBanditsOpenReview2502212211554In\-context learningOpenReview2501931931559Reasoning / test\-time computationOpenReview2501371371139Offline reinforcement learningarXiv2502482452176Uncertainty estimationarXiv2502502482206Total–1250104910448634

### 3\.2 Systems Compared

The SGHA outputs in the main comparison come from the verification\-gated family\-report path described in Section[2](https://arxiv.org/html/2608.17501#S2)\. The candidate gaps are detected from the evidence graph, screened against nearby corpus evidence, passed through the hard verification gate, formulated, expanded, filtered by an independent critic, consolidated into project families, and converted into semi\-formal problem statements\. The evaluated SGHA candidate is the final project\-family artifact, rather than an intermediate motif hit, direct formulation, or rejected variant\.

Our primary baseline is native AI\-Scientist\-v2 ideation with the same localQwen/Qwen3\.5\-9Bmodel\. We use its ideation entry point and match the number of retained candidates to the number of SGHA final families in each domain\. This gives 15 candidates from each method: 3 for bandits, 4 for in\-context learning, 1 for reasoning and test\-time computation, 1 for offline reinforcement learning, and 6 for uncertainty estimation\.

We also include two reference baselines\. The first is AI\-Scientist\-v2 withanthropic/claude\-opus\-latestas a stronger generator, again run in ideation\-only mode and matched to SGHA’s retained candidate counts\. The second is MOOSE\-Star using the releasedZonglinY/MOOSE\-Star\-HC\-R1D\-7Bpublic model in HC\-only mode, with no additional training or fine\-tuning\. These baselines give reference points for same\-model native ideation, stronger\-generator ideation, and public trained hypothesis generation\.

Table 2:Systems compared in the formulation\-quality evaluation\. All methods are output\-count matched to 15 retained candidates across the five domains\.MethodModelModeEvaluated artifactSGHAQwen3\.5\-9BVerification\-gated literature\-gap pipelineFinal project\-family artifact with provenance and a semi\-formal problem statement\.AI\-Scientist\-v2 \+ QwenQwen3\.5\-9BNative ideation\-only workflowRetained generated idea normalized into the candidate\-packet format\.AI\-Scientist\-v2 \+ Claude Opusclaude\-opus\-latestNative ideation\-only workflowRetained generated idea from a stronger generator setting\.MOOSE\-StarMOOSE\-Star\-HC\-R1D\-7BReleased public HC\-only modelRetained generated hypothesis from the public model\.

Before judging, all outputs are converted into a common candidate format\. The packet contains the title, problem statement, motivation or abstract, proposed direction, expected contribution, source or context information when available, assumptions or setup when available, formal\-problem fields when available, ambiguity or missing\-definition fields when available, and risks or caveats when available\. Fields that a method does not produce are left asnot provided; we evaluate the artifacts as produced and report structural\-field coverage separately\.

### 3\.3 Formulation\-Only Evaluation

SGHA produces research\-problem formulations rather than complete project plans\. We therefore evaluate candidates with a formulation\-only rubric\. The rubric focuses on whether a candidate states a clear problem, gives enough technical structure, controls its scope, makes assumptions visible, and identifies what remains ambiguous\. Table[3](https://arxiv.org/html/2608.17501#S3.T3)lists the ten criteria used by the judge\.

Table 3:Formulation\-only evaluation rubric\. Each criterion is scored on a 0–10 scale\.CriterionWhat the judge evaluatesProblem\-definition clarityWhether the candidate states a clear research problem rather than a broad topic or loose motivation\.Technical specificityWhether the formulation names concrete objects, settings, mechanisms, assumptions, metrics, or target results\.Well\-posednessWhether the problem has enough structure to be studied or refined\.Assumption\-boundary clarityWhether the formulation makes clear which assumptions are used, relaxed, questioned, or missing\.FormalizabilityWhether the problem can plausibly be written as a theorem, algorithmic objective, benchmark protocol, impossibility result, or other formal research target\.NontrivialityWhether the candidate goes beyond a simple application, small robustness check, or generic “apply X to Y” idea\.Scope controlWhether the problem is focused enough to be studied rather than combining many loosely related concepts\.Source\-grounded specificityWhether the formulation is tied to provided source context, evidence, or literature\-derived details\.Ambiguity hygieneWhether the candidate explicitly states missing definitions, unclear feedback models, unresolved assumptions, or caveats\.Overall formulation qualityThe judge’s overall assessment of the research\-problem formulation under the rubric\.

The 0–10 scores use the anchors in Table[4](https://arxiv.org/html/2608.17501#S3.T4)\. These anchors are meant to make the scores interpretable: a score around 5 or 6 means the formulation is plausible but still needs refinement, while scores of 7 or 8 indicate a strong formulation\.

Table 4:Score anchors used by the formulation\-only judges\.ScoreInterpretation0No usable formulation\.1Mostly incoherent or unrelated to a research problem\.2Topic\-level idea with almost no problem structure\.3Very vague formulation with major missing pieces\.4Weak formulation; some direction is visible, but the problem is poorly specified\.5Plausible idea, but important assumptions, scope, or definitions are missing\.6Plausible formulation with useful structure, but still needing substantial refinement\.7Strong formulation with clear problem structure and reasonable technical specificity\.8Very strong formulation; well posed, grounded, and mostly ready for expert refinement\.9Excellent formulation with unusually clear assumptions, scope, and formal target\.10Exceptional formulation; clear, grounded, formalizable, and close to research\-ready\.

The evaluation uses method\-label\-masked candidate packets\. Method labels are hidden from the judges, and the label key is used only during postprocessing\. Judges evaluate the provided candidate text and supporting context under the fixed rubric; external literature novelty and correctness are outside the judge task\.

To reduce dependence on a single model or provider, we use five LLM judges from different providers:anthropic/claude\-sonnet\-4,openai/gpt\-5\.6\-sol\-pro,x\-ai/grok\-4\.5,moonshotai/kimi\-k3, andgoogle/gemini\-3\.6\-flash\. Each judge receives the same blinded packet format, rubric, score anchors, and response schema\. The judge panel is used only after all candidate outputs have been frozen; it is not part of SGHA’s generation or verification pipeline\.

We report criterion\-level means together with the judges’ overall formulation\-quality score\. These scores are descriptive assessments of formulation quality, and we interpret them alongside structural properties of the generated outputs\.

### 3\.4 Profile\-Conditioned Generation

We also evaluate a profile\-conditioned version of SGHA\. In this setting, the input includes a researcher\-specific literature context and, optionally, a target topic\. The profile context guides corpus construction and prioritization, so the selected papers and candidate gaps are closer to the researcher’s previous work or stated area of interest\.

After corpus construction, the downstream SGHA pipeline remains the same\. Candidate gaps are detected from the evidence graph, screened, verified, formulated, expanded, filtered by the critic, consolidated into project families, and converted into semi\-formal problem statements\. Profile conditioning changes the evidence base and prioritization, while preserving the same checks before finalization\.

We evaluate profile\-conditioned generation on three researcher contexts: Yann LeCun, Geoffrey Hinton, and Michael I\. Jordan\. The resulting candidates are evaluated with a personalized judge rubric that includes the formulation\-quality criteria from the main evaluation and additional profile\-specific criteria for alignment, specificity, and fit to the provided profile context\.

### 3\.5 Evolutionary Exploration

SGHA includes an evolutionary exploration branch that serves a different role from the compact family\-report path\. The family\-report path starts from verification\-passed gaps and produces a small set of critic\-filtered project families\. The evolutionary branch explores a broader neighborhood around literature\-grounded seeds\.

Starting from a seed gap or formulation, the evolutionary branch varies parts of the problem representation, such as assumptions, mechanisms, objectives, or regimes\. Candidate variants are critiqued, selected, and refined over multiple rounds\. For evaluation, selected or ranked evolutionary hypotheses are normalized into the same candidate format used for the other methods and scored with the same formulation\-only rubric\. We report these outputs separately from the main SGHA project\-family counts\.

### 3\.6 Model\- and Corpus\-Size Sensitivity

Finally, we include two sensitivity analyses on the Bandits domain\. The model\-size analysis compares the mainQwen/Qwen3\.5\-9Brun with a smaller 4B configuration and a larger 27B configuration\. This tests how extraction quality, verification yield, critic\-passing variants, and final family construction change with model capacity\.

The corpus\-size analysis varies the number of selected Bandits papers\. We run settings with 50, 100, 150, 200, and 250 selected papers\. This tests how SGHA behaves as the evidence base grows\. We treat this as a sensitivity study rather than a scaling\-law experiment: larger corpora provide more evidence and more candidate gaps, while later stages such as verification, critic filtering, and family consolidation determine which candidates survive\.

## 4 Results

### 4\.1 End\-to\-End Yield Across Domains

We first evaluate whether SGHA can complete the full path from papers to final research\-problem families across different areas of machine learning\. Table[5](https://arxiv.org/html/2608.17501#S4.T5)summarizes the five main runs\. Across 1,250 selected papers, SGHA extracts structured scientific content from 1,044 papers and produces 8,634 evidence tuples\. These evidence tuples support graph construction and candidate\-gap discovery\. After corpus screening and verification, 39 gaps are accepted for formulation\. The final synthesis stages then produce 19 accepted variants, which are consolidated into 15 project families with 15 semi\-formal problem statements\.

This yield shows the intended behavior of the pipeline\. SGHA starts from a large literature\-derived evidence base and progressively refines it into a compact set of research\-problem artifacts\. The intermediate stages provide breadth: extraction creates evidence tuples, motifs surface candidate gaps, and verification identifies formulation\-ready gaps\. The final stages provide structure: ambition expansion, criticism, family consolidation, and formalization turn selected gaps into inspectable project families\.

Table 5:End\-to\-end SGHA yield across the five main domains\. Verified gaps are candidate gaps that pass the hard verification gate\. Accepted variants are ambition\-expanded formulations that pass the independent critic\.DomainTuplesVerified gapsAccepted variantsFamiliesFormal statementsBandits15546433In\-context learning155912444Reasoning / test\-time computation11397111Offline reinforcement learning21762111Uncertainty estimation220612966Total863439191515

All five domains reach the final project\-family stage, but with different yields\. This is expected for a literature\-driven system: some corpora contain several separable directions, while others lead to a smaller and tighter set of families\. Uncertainty estimation produces the largest set of final families, while reasoning/test\-time computation and offline reinforcement learning produce more focused outputs\.

Table[5](https://arxiv.org/html/2608.17501#S4.T5)therefore defines the SGHA candidate set used in the main comparison: 15 formalized project families obtained from 39 verification\-passed gaps\. We next evaluate the quality of these artifacts—how clear, grounded, formalizable, and explicit about ambiguity they are—against automated ideation baselines\.

### 4\.2 Formulation Quality Compared with Baselines

In Table[6](https://arxiv.org/html/2608.17501#S4.T6)we report the mean formulation\-quality scores across the five LLM judges\. SGHA obtains the highest overall formulation\-quality score\. The strongest gains are on the criteria that the pipeline is designed to improve: source\-grounded specificity, ambiguity hygiene, assumption\-boundary clarity, well\-posedness, and formalizability\.

The comparison with AI\-Scientist\-v2 using the same local Qwen model shows the clearest difference\. Both systems use the same generator, but SGHA first builds an evidence graph, verifies candidate gaps, and produces formal problem objects\. This leads to higher scores on nearly all formulation criteria, especially ambiguity hygiene and assumption\-boundary clarity\. The Claude Opus AI\-Scientist\-v2 baseline is much stronger on surface formulation: it produces clearer, more technically detailed, and more tightly scoped ideas\. However, SGHA remains higher on the evidence\-linked and structure\-oriented criteria, and also has the highest overall score\. MOOSE\-Star scores lower in this setup, reflecting that the released HC\-only model produces hypothesis text but not the same kind of verified, formalized problem artifact\.

Table 6:Formulation\-quality scores averaged over five LLM judges\. All methods are evaluated on 15 retained candidates\. Bold indicates the best score in each column\.MethodOverallClarityTechnicalWell\-posedBoundaryFormal\.NontrivialScopeGroundingAmbiguitySGHA5\.996\.775\.895\.776\.455\.516\.395\.367\.437\.61AI\-Scientist\-v2 \+ Qwen4\.555\.594\.723\.533\.613\.685\.275\.735\.112\.81AI\-Scientist\-v2 \+ Claude Opus5\.847\.156\.954\.884\.755\.216\.886\.192\.973\.57MOOSE\-Star2\.001\.852\.511\.521\.371\.512\.893\.073\.751\.48

The table shows a useful tradeoff\. Strong ideation models can produce polished and well\-scoped ideas, as reflected by the Claude Opus baseline\. SGHA is strongest when the rubric asks for problem structure that depends on the pipeline: grounding in the provided literature, explicit assumptions, formalizability, and clean handling of ambiguity\. This supports the central claim of the paper: the benefit of SGHA is not only that it generates plausible ideas, but that it turns literature\-derived gaps into inspectable research\-problem formulations\.

### 4\.3 Structural Artifact Coverage

The judge scores measure formulation quality, but they do not fully show what information is available in each generated artifact\. We therefore also inspect the structure of the outputs directly\. Table[7](https://arxiv.org/html/2608.17501#S4.T7)reports whether each candidate includes the fields that make a research\-problem artifact easier to audit: a formal problem statement, assumptions or setup, source/context grounding, ambiguity flags, an evaluation plan, and risks or caveats\.

SGHA includes all of these fields for all 15 final project families\. This is expected, since the final SGHA report is designed to expose the problem structure rather than only give a proposal title and motivation\. The baseline systems produce different kinds of artifacts\. AI\-Scientist\-v2 outputs usually include readable motivation, source/context information, evaluation plans, and caveats, but they do not produce SGHA\-style formal problem objects or ambiguity flags\. MOOSE\-Star provides source\-grounded hypothesis text in this setup, but not the full structured research\-problem object\.

Table 7:Structural artifact coverage across the 15 retained candidates for each method\. A checkmark means the field is present for all retained candidates from that method; a cross means it is absent from all retained candidates\. This table records artifact structure, not judge\-assigned quality\.MethodFormal problemSetup / assumptionsSource/contextAmbiguity flagsEval\. planRisks/caveatsComplete artifactSGHA✓✓✓✓✓✓✓AI\-Scientist\-v2 \+ Qwen✗✗✓✗✓✓✗AI\-Scientist\-v2 \+ Claude Opus✗✗✓✗✓✓✗MOOSE\-Star✗✗✓✗✗✗✗

This analysis helps explain the pattern in Table[6](https://arxiv.org/html/2608.17501#S4.T6)\. SGHA’s advantage is strongest on source\-grounded specificity and ambiguity hygiene because these fields are part of the final artifact\. The baselines can still produce clear or well\-scoped ideas, especially with a stronger generator, but their outputs are less explicit about the assumptions, missing definitions, and formal structure that a researcher would need to inspect before developing the idea further\.

### 4\.4 Qualitative Comparison of Generated Artifacts

The quantitative results compare average formulation quality across methods\. To make the comparison more concrete, we also show representative generated formulations\. For each method, we display its best retained Bandits candidate under the formulation\-quality evaluation\. This keeps the domain fixed while showing the different forms of output produced by SGHA, AI\-Scientist\-v2, and MOOSE\-Star\.

Table[8](https://arxiv.org/html/2608.17501#S4.T8)summarizes the four examples\. The examples are shown in their generated form: SGHA produces a formalized project\-family artifact, AI\-Scientist\-v2 produces proposal\-style ideas, and MOOSE\-Star produces a compact source\-grounded hypothesis\.

Table 8:Representative Bandits artifacts shown in this section\. Each row uses the best retained Bandits candidate for that method under the formulation\-quality evaluation\.MethodExample titleOverallArtifact typeSGHACharacterizing Identifiability Limits of Structured Bandits Under Piecewise Non\-Stationarity6\.6Formalized project familyAI\-Scientist\-v2 \+ QwenBandits with Adversarial Arm Execution: Robust Learning When Your Choices Don’t Match Reality4\.6Proposal\-style ideaAI\-Scientist\-v2 \+ Claude OpusWhose Scale Is It Anyway? Monotone\-Robust Bandits with Utility Elicitation6\.2Detailed proposal\-style ideaMOOSE\-StarOrder Preservation in Set\-Size Dependent Combinatorial Bandits2\.2Source\-grounded hypothesis

#### 4\.4\.1 SGHA

Characterizing Identifiability Limits of Structured Bandits Under Piecewise Non\-StationarityProblem formulation\.Current robust bandit algorithms for structured action sets, such as Network Lasso, rely on the i\.i\.d\. assumption of context generation\. This assumption fails in environments with piecewise constant non\-stationarity, leading to unbounded regret and failure in identifying optimal arms\. The fundamental question is not merely how to fix a specific algorithm, but whether the network structure itself allows for consistent learning under such distributional shifts, or if a fundamental identifiability barrier exists\.Proposal\-style abstract\.This project studies the fundamental limits of learning in contextual bandits with network\-structured action sets when the underlying data distribution undergoes piecewise constant shifts\. The central question is whether the structural constraints imposed by network regularization are sufficient to guarantee consistent policy identification in non\-stationary regimes, or if the combination of structural sparsity and temporal drift creates an inherent identifiability gap\. A successful outcome would characterize the precise boundary between learnable and unlearnable regimes for this problem class, providing necessary and sufficient conditions for robustness that are independent of any specific algorithmic implementation\. This work moves beyond validating a single method to establishing a theoretical framework for the viability of structured learning under non\-stationarity\.Formal problem statement\.Let𝒜\\mathcal\{A\}be a set of actions constrained by a network structure𝒢\\mathcal\{G\}\. Let𝒟t\\mathcal\{D\}\_\{t\}denote the distribution of contexts at timett\. The environment exhibits piecewise constant non\-stationarity, meaning𝒟t=𝒟k\\mathcal\{D\}\_\{t\}=\\mathcal\{D\}\_\{k\}fort∈\[tk,tk\+1\)t\\in\[t\_\{k\},t\_\{k\+1\}\)\. The question is whether there exists a sequence of policiesπt\\pi\_\{t\}such that the regretRTR\_\{T\}grows sublinearly with timeTTsolely based on the structural constraints of𝒢\\mathcal\{G\}, without assuming that𝒟t\\mathcal\{D\}\_\{t\}is i\.i\.d\. across time\.Variables and notation\.𝒜\\mathcal\{A\}: set of available actions;𝒢\\mathcal\{G\}: network structure constraining action sets;𝒟t\\mathcal\{D\}\_\{t\}: distribution of contexts at timett;tkt\_\{k\}: time points where the distribution shifts;πt\\pi\_\{t\}: policy at timett;RTR\_\{T\}: cumulative regret up to timeTT\.Objective\.Characterize the necessary and sufficient conditions for consistent learning, or identifiability, in network\-structured bandits under piecewise constant non\-stationarity\.Assumptions\.Network structure constraint: kept\. Piecewise constant non\-stationarity: kept\. i\.i\.d\. context generation: removed\. Existence of consistent learning: questioned\.Success criterion\.Establish a precise boundary between regimes where network structure aids robustness and regimes where structural constraints amplify non\-stationarity errors, leading to impossibility\.Risk and falsification condition\.Proving impossibility results requires rigorous mathematical machinery, and the result may depend on how the structural and non\-stationarity assumptions are defined\. The direction would be weakened by a specific non\-stationary environment and network topology where consistent learning is achieved despite violating the proposed necessary conditions for failure\.Ambiguity flags\.Network structure; piecewise constant non\-stationarity; feedback model; boundary; feedback or measurement model\.Source/context grounding\.The family traces to a verification\-passed structural gap,gap:ba210076fbccfacb, with source direct formulationdirect:02\. The retained supporting papers are*Clusters Agnostic Network Lasso Bandits*and*Network Lasso Bandits*\. In the SGHA trace, these papers support an assumption\-mismatch pattern around Network Lasso\-style bandit methods: the method is linked to i\.i\.d\. context/action\-set assumptions and to a piecewise\-constant non\-stationarity failure condition\.

This SGHA example is a structured research\-problem object\. It contains a problem formulation, an abstract, notation, an objective, assumption statuses, a success criterion, a risk, a falsification condition, ambiguity flags, and source grounding\. The artifact is still a candidate direction, but the information needed to inspect and refine it is made explicit\.

#### 4\.4\.2 AI\-Scientist\-v2 with Qwen

Bandits with Adversarial Arm ExecutionProblem statement\.Standard bandit algorithms assume perfect action execution, but in real systems such as cloud services, autonomous systems, and recommendation platforms, there is often uncertainty between the learner’s intended action and the action actually executed because of system errors, competing processes, or adversarial interference\. The hypothesis is that explicitly modeling execution uncertainty, where an adversary or system can alter which arm is actually executed, requires fundamentally different algorithmic approaches that can achieve robust regret bounds even when the learner observes only the outcome and not the discrepancy\.Motivation\.In practical bandit deployments, the gap between intended action and actual execution can severely undermine learning\. System failures, resource contention, or adversarial interference may cause the learner’s selected arm to differ from the executed arm, yet standard bandit algorithms operate under the assumption of perfect execution\. This creates a critical vulnerability: learners may systematically misattribute rewards to wrong arms, leading to catastrophic regret growth\.Proposed direction\.The proposed Adversarial Arm Execution bandit framework allows an adversary or stochastic system to alter the executed arm after the learner’s selection, with the learner observing only the final executed arm and its reward\.Evaluation plan\.The proposed evaluation includes a synthetic adversarial\-arm\-execution benchmark where the executed arm differs from the selected arm in a fraction of trials; comparisons between adversarial and stochastic execution uncertainty; a recovery analysis where occasional ground\-truth signals reveal the executed arm identity; and a real\-world simulation using recommendation logs where impressions may not match clicks because of rendering errors\.Risks\.The generated artifact notes computational complexity, modeling assumptions, limited ground truth, and adversarial complexity as risks\.

The Qwen AI\-Scientist\-v2 example reads as a proposal for a new bandit setting\. It gives a clear motivation, describes a setting, and outlines experiments\. Its emphasis is on a project direction and evaluation plan rather than on a formalized problem object\.

#### 4\.4\.3 AI\-Scientist\-v2 with Claude Opus

Whose Scale Is It Anyway? Monotone\-Robust Bandits with Utility ElicitationProblem statement\.Almost every deployed bandit optimizes the mean of an arbitrarily chosen numerical encoding of an ordinal signal, such as 1–5 stars, 1–10 judge scores, or clinical grades\. The identity of the best arm is not invariant to monotone re\-encodings of that signal\. The proposed direction hypothesizes that such rank reversals and even non\-transitive dominance cycles are common in logged feedback, that no algorithm using only sampled rewards can fully resolve them, and that a scale\-free learner should play a randomized minimax\-regret mixture over arms unless a small number of utility\-elicitation queries is available\.Motivation\.Bandit algorithms maximize the mean of a number, but in many applications that number is an arbitrary encoding of an ordinal signal\. A five\-star rating, a 1–10 LLM\-judge score, or a clinical grade can be mapped to utilities in multiple monotone ways, and the choice can flip which arm is best\.Proposed direction\.The artifact formalizes a monotone\-robust bandit setting in which rewards take values in an ordered set ofLLlevels, the true utilityuuis an unknown increasing function, and the learner is judged by regret maximized over all utilities consistent with what it knows\. It proposes studying rank reversals and minimax regret over monotone utility functions, together with utility\-elicitation queries based on standard\-gamble comparisons over theLLobservation levels\.Evaluation plan\.The proposed evaluation includes scale\-fragility audits on rating datasets, synthetic ordinal bandits, randomization\-gap experiments, elicitation\-complexity experiments, a joint exploration\-elicitation tradeoff, and an LLM\-evaluation case study\.Risks\.The generated artifact notes that the novelty depends on the minimax\-regret\-over\-utilities objective, the randomization result, and the elicitation\-complexity result\. It also notes that unconstrained monotone utilities may make non\-comparability pervasive, that constants may be conservative, and that logged ratings are confounded\.

The Claude Opus AI\-Scientist\-v2 example is the strongest baseline example in this qualitative set\. It is detailed, technically specific, and well motivated\. It also helps explain why this baseline scores strongly on clarity, technical specificity, nontriviality, and scope control\.

#### 4\.4\.4 MOOSE\-Star

Order Preservation in Set\-Size Dependent Combinatorial BanditsHypothesis\.The order preservation property addresses the gap in handling larger exploration sets by maintaining the order of reward means, which allows for a more efficient exploration strategy\. This reduces the regret associated with traditional methods that struggle with extensive exploration sets\.Mechanism\.The property ensures that the order of reward means remains consistent regardless of set size, enabling the algorithm to focus on the most promising combinations of base arms\. This prioritization allows for more efficient exploration of superarms, reducing the number of necessary trials and thus lowering regret\.Proposed direction\.The SUCB algorithm is adapted to incorporate the order preservation property by modifying the selection process to prioritize superarms based on the order of their base arms’ rewards\.Expected contribution\.The expected contribution is to use order preservation to focus exploration on high\-potential combinations of base arms, reducing unnecessary trials and lowering regret\.Source grounding\.Inspiration paper:*Set\-Size Dependent Combinatorial Bandits*\.

The MOOSE\-Star example is a compact hypothesis\-style artifact\. It is grounded in an inspiration paper and proposes a mechanism, but it exposes fewer fields for inspection than the SGHA or AI\-Scientist\-v2 examples\.

#### 4\.4\.5 Summary

The qualitative examples clarify what the numerical scores summarize\. SGHA returns a formalized project\-family artifact, with notation, assumptions, a success criterion, risks, ambiguity flags, and source grounding\. AI\-Scientist\-v2 returns proposal\-style ideas, which can be readable and technically detailed, especially with a stronger generator\. MOOSE\-Star returns a shorter source\-grounded hypothesis\. These differences in artifact shape align with the quantitative results: SGHA is strongest on source\-grounded specificity, formalizability, and ambiguity hygiene, while the strongest ideation baseline is competitive on clarity, technical specificity, and scope control\.

### 4\.5 Profile\-Conditioned Generation

We next evaluate whether SGHA can be steered by a researcher\-specific literature context\. In this setting, the input includes a profile\-derived corpus and, optionally, a target topic\. The downstream SGHA pipeline remains the same: the system still extracts evidence, builds a graph, detects candidate gaps, verifies them, formulates them, applies the critic, and produces final project families\. Thus, profile conditioning changes where the evidence comes from\.

Table[9](https://arxiv.org/html/2608.17501#S4.T9)reports the pipeline yield for three profile\-conditioned runs\. Across the three profiles, SGHA extracts structured content from 439 papers and produces 3,404 evidence tuples\. Nine gaps pass verification, leading to four final project families with four formal problem statements\.

Table 9:Profile\-conditioned SGHA pipeline results\. Verified gaps are candidate gaps that pass the hard verification gate before formulation\.ProfileSeedsParsedExtractedTuplesVerified gapsFamiliesFormal statementsYann LeCun111501481170411Geoffrey Hinton331501411191111Michael I\. Jordan1501501501043422Total1944504393404944

We evaluate the resulting candidates with a personalized judge rubric\. This rubric keeps the formulation\-quality criteria from the main evaluation and adds profile\-specific criteria: profile alignment, profile specificity, intellectual\-style match, and personalization overall\. The judge uses only the provided profile context and candidate text\.

Table[10](https://arxiv.org/html/2608.17501#S4.T10)shows the personalized scores\. The strongest profile\-alignment scores appear for the Geoffrey Hinton candidate and the Michael I\. Jordan hierarchical\-identifiability candidate\. The Yann LeCun candidate has the highest formulation\-quality score among the profile\-conditioned outputs, but lower profile specificity\. This pattern is useful: profile conditioning can steer SGHA toward profile\-relevant directions, while the judge still separates general formulation quality from how specifically the candidate matches the provided profile context\.

Table 10:Profile\-conditioned judge scores\. Formulation measures the research\-problem formulation itself; the remaining columns measure alignment to the provided profile context\.ProfileCandidateFormulationAlignmentSpecificityStyle matchPersonalizationYann LeCunFundamental limits of partition\-function approximation in high\-dimensional EBMs6\.177\.004\.505\.675\.33Geoffrey HintonConvergence failure regimes of stochastic\-gradient approximations in deep generative models5\.678\.007\.176\.837\.17Michael I\. JordanIdentifiability boundaries of filtering\-clustering under rank\-deficient design matrices6\.006\.505\.837\.336\.33Michael I\. JordanCharacterizing hierarchical identifiability limits in variational protein annotation6\.008\.007\.007\.757\.00

The Michael I\. Jordan example illustrates the kind of profile\-conditioned artifact SGHA produces\. The final family,*Characterizing Hierarchical Identifiability Limits in Variational Protein Annotation*, connects profile\-relevant themes such as variational inference, graphical models, and structured prediction to a concrete problem in protein\-function annotation\.

Characterizing Hierarchical Identifiability Limits in Variational Protein AnnotationProfile context\.This candidate comes from the Michael I\. Jordan profile\-conditioned run\. The profile\-conditioned evidence base emphasizes probabilistic modeling, graphical models, statistical machine learning, variational inference, Bayesian nonparametrics, and optimization\. The run uses a profile\-conditioned corpus of 150 parsed and extracted papers, producing 1,043 evidence tuples\.Source papers\.The final family is grounded in the retained profile\-seed papers profile pdf: a variational principle for graphical models\[[16](https://arxiv.org/html/2608.17501#bib.bib21)\]and profile pdf: genome scale phylogenetic function annotation of large and diverse protein families\[[7](https://arxiv.org/html/2608.17501#bib.bib20)\]\. The first source provides the variational\-inference / graphical\-model context, while the second provides the protein\-function annotation setting\.Verified gap\.The family traces to the verified gapgap:dcb6662b99312fb7, an unresolved tradeoff around Gene Ontology hierarchy\. The gap passes the hard verification gate with survival score0\.6640\.664, above the threshold0\.6000\.600\. The verification trace is mixed in a useful way: the support and feasibility agents view the gap as grounded and studyable, while the skeptic and critic caution that the strongest version of the claim should not be overstated because hierarchy\-aware scoring methods exist\.Direct formulation\.The source direct formulation isdirect:04:*Integrating Gene Ontology Hierarchy Constraints into Variational Inference for Protein Function Annotation*\. This direct formulation asks for a variational inference framework that explicitly incorporates Gene Ontology tree constraints while avoiding prohibitive computational cost\.Ambition expansion and critic\.The ambition stage generates three variants\. The first two variants,var:10andvar:11, are rejected by the independent critic as inflated or insufficiently supported\. The accepted variant isvar:12, which reframes the problem as a theoretical boundary: under what structural conditions do flat variational objectives fail to recover the true posterior over a tree\-structured label space? This accepted variant becomes the final project family\.Problem formulation\.Current variational models for protein function annotation treat Gene Ontology \(GO\) terms as flat sets, ignoring the parent\-child hierarchy\. This oversight creates a fundamental gap where biologically consistent, less precise predictions are penalized, but the theoretical boundary of this failure is unknown\. The central challenge is to characterize the exact conditions under which hierarchical constraints become necessary for identifiability and to define the regime where flat approximations fail catastrophically\.Formal problem statement\.Let𝒯\\mathcal\{T\}be a tree\-structured label space representing Gene Ontology terms and parent\-child relationships\. LetPtrueP\_\{\\mathrm\{true\}\}be the true posterior over𝒯\\mathcal\{T\}given protein dataDD, and letℱflat\\mathcal\{F\}\_\{\\mathrm\{flat\}\}be a flat variational approximation family\. EachQ∈ℱflatQ\\in\\mathcal\{F\}\_\{\\mathrm\{flat\}\}approximatesPtrueP\_\{\\mathrm\{true\}\}without enforcing parent\-child consistency constraints from the tree topology\. The problem is to characterize conditions𝒞\\mathcal\{C\}such that, for everyQ∈ℱflatQ\\in\\mathcal\{F\}\_\{\\mathrm\{flat\}\}, the approximation error‖Ptrue−Q‖\\\|P\_\{\\mathrm\{true\}\}\-Q\\\|exceeds a thresholdϵ\\epsilonwhen the structural disconnect between parent\-child implications prevents recovery of the true posterior\.Variables and notation\.𝒯\\mathcal\{T\}: tree\-structured Gene Ontology label space;PtrueP\_\{\\mathrm\{true\}\}: hierarchy\-respecting posterior over GO terms;ℱflat\\mathcal\{F\}\_\{\\mathrm\{flat\}\}: flat variational family;QQ: approximating distribution inℱflat\\mathcal\{F\}\_\{\\mathrm\{flat\}\};DD: observed protein data;ϵ\\epsilon: approximation\-error threshold;𝒞\\mathcal\{C\}: structural or data conditions under which flat approximations fail\.Objective\.Characterize when flat variational approximations are insufficient for recovering a hierarchy\-respecting posterior over Gene Ontology terms\. More concretely, identify structural conditions on the label hierarchy and data distribution under which every flat approximationQ∈ℱflatQ\\in\\mathcal\{F\}\_\{\\mathrm\{flat\}\}remains farther thanϵ\\epsilonfromPtrueP\_\{\\mathrm\{true\}\}\.Assumptions\.Existence of a hierarchy\-respecting posterior: kept\. Gene Ontology tree structure: kept\. Flat variational sufficiency: relaxed or questioned\. Parent\-child consistency constraints: treated as necessary to define the hierarchical setting\.Success criterion\.Identify conditions on the GO tree and data distribution under which flat variational inference fails to recover the hierarchy\-respecting posterior, and characterize the resulting identifiability boundary\.Possible result types\.A theorem characterizing when flat variational families cannot approximate the true hierarchy\-respecting posterior within thresholdϵ\\epsilon; an algorithmic direction for hierarchy\-aware variational inference; and an empirical protocol comparing flat and hierarchy\-aware approximations on protein\-function annotation tasks\.Ambiguity flags\.Catastrophic failure: requires a specific divergence metric or threshold\.Structural disconnect: requires a precise mathematical relationship between ignored parent\-child constraints and approximation error\.Boundary: requires a formal definition of the transition between identifiable and non\-identifiable regimes\.Feedback or measurement model: requires further definition for the protein\-annotation setting\.Formalization confidence and risk\.The formalization confidence ismedium, and the artifact setsrequires\_human\_definitionto true\. The main risk is that the impossibility\-boundary claim is only moderately supported by the source evidence: the sources identify the hierarchy\-related gap and the variational\-inference context, but do not themselves prove a theoretical failure regime\.

This example shows the intended role of profile conditioning\. The profile context changes the evidence base and helps steer SGHA toward a problem that fits the supplied research context, while the final artifact still exposes the formal setup, objective, and remaining ambiguities\.

### 4\.6 Evolutionary Exploration

We next study SGHA’s evolutionary exploration branch\. This branch has a different role from the main project\-family path\. The main path is designed to produce a compact set of verified, critic\-filtered, and formalized problem families\. Evolutionary exploration instead searches more broadly around literature\-grounded seeds\. Starting from a seed gap or formulation, it varies parts of the problem representation—such as assumptions, mechanisms, objectives, or regimes—and then critiques, selects, and refines the resulting variants\.

Table[11](https://arxiv.org/html/2608.17501#S4.T11)shows the scale of this exploration\. Across the five domains, the branch generates 2,908 evolved candidates and retains 160 selected hypotheses during the search\. The final ranked files contain 96 hypotheses\. This gives SGHA a broader exploratory layer: the family\-report path produces a small set of polished problem families, while the evolutionary branch gives a larger pool of nearby directions that can be inspected or passed through later verification and formalization\.

Table 11:Evolutionary exploration artifacts by domain\. Evolved candidates include mutations, crossovers, and syntheses around literature\-grounded seeds\.DomainEvolvedSelectedFinal rankedInitial gapsBandits603322013In\-context learning472322016Reasoning / test\-time computation502322012Offline RL59432168Uncertainty estimation737322015Total29081609664

We evaluate an output\-count\-matched set of 15 ranked evolutionary hypotheses with the same formulation\-only rubric used in the main comparison\. Table[12](https://arxiv.org/html/2608.17501#S4.T12)reports the average scores\. The evolutionary hypotheses score highest on source\-grounded specificity and nontriviality, which is consistent with their purpose: they are generated from literature\-grounded seeds and often point to interesting directions\. Their lower scores on well\-posedness, formalizability, and ambiguity hygiene also match the role of this branch\. These hypotheses are exploratory; they have not gone through the full family\-report path that adds formal problem statements, assumption tables, and ambiguity flags\.

Table 12:Formulation\-only scores for the selected evolutionary\-exploration hypotheses\. Scores are averaged over the available judge evaluations\.Candidate setOverallClarityTechnicalWell\-posedBoundaryFormal\.Nontriv\.ScopeGroundingAmbiguitySGHA evolutionary exploration3\.824\.114\.092\.803\.422\.645\.054\.385\.402\.20

The domain\-level results in Table[13](https://arxiv.org/html/2608.17501#S4.T13)show the same pattern\. The reasoning/test\-time\-computation candidate has the highest mean overall score, and source grounding is relatively strong across domains\. This suggests that the evolutionary branch is useful for surfacing grounded directions, even when the resulting hypotheses still need the later formalization and ambiguity\-cleanup stages before they become final research\-problem artifacts\.

Table 13:Evolutionary\-exploration formulation scores by domain\.DomainSelectedMean overallBest overallMean groundingBandits33\.754\.255\.17In\-context learning43\.644\.335\.29Reasoning / test\-time computation14\.504\.505\.75Offline RL13\.753\.755\.25Uncertainty estimation63\.864\.005\.57

The highest\-scoring evolutionary candidate comes from reasoning and test\-time computation\. It studies self\-verification mechanisms in test\-time compute scaling, where additional inference\-time search may stop improving accuracy when intermediate reasoning steps are difficult to verify\.

Self\-verification mechanisms within test\-time compute scaling frameworksProblem statement\.The problem of understanding self\-verification mechanisms within test\-time compute scaling frameworks that achieve accuracy saturation despite increased inference\-time search depth due to the collapse of verifiable correctness assumptions on inherently unverifiable intermediate steps\.Gap\.The problem of verification\-dependent error accumulation that achieves accuracy saturation despite increased inference\-time search depth\.Target\.Self\-verification mechanisms within test\-time compute scaling frameworks\.Scope\.Cross\-gap synthesis over assumption\-mismatch, repeated\-unaddressed\-limitation, and unrealistic\-assumption motifs, narrowed to the most evidence\-dense setting\.Mechanism\.Self\-verification attempts to correct linear error accumulation inherent in Chain\-of\-Thought structures, but relies on verifiable correctness assumptions that collapse when intermediate steps are inherently unverifiable\.Proposal\-style abstract\.This project studies self\-verification mechanisms within test\-time compute scaling frameworks\. The central question is how to maintain accuracy when verifiable correctness assumptions collapse on inherently unverifiable intermediate steps\. Current approaches often focus on complex problem\-solving tasks where correctness is easily verified on labeled datasets\. However, reliance on task\-specific few\-shot examples can limit generalizability across diverse tasks\. Furthermore, existing methods fail to utilize negative samples for implicit exploration and do not improve on out\-of\-distribution tasks through experiments\. The mechanism involves self\-verification attempting to correct linear error accumulation inherent in Chain\-of\-Thought structures\. Yet, this relies on assumptions that collapse when intermediate steps are unverifiable\. This project would investigate verification\-dependent error accumulation that achieves accuracy saturation despite increased inference\-time search depth\. A possible approach is refining these mechanisms to handle unverifiable steps without saturation\. A successful outcome would provide insights into scaling frameworks\. This would clarify the relationship between search depth and verification limits\. The proposed contribution type focuses on self\-verification mechanisms within test\-time compute scaling frameworks\. This work aims to address the unrealistic assumption that intermediate steps remain verifiable under increased compute scaling\.Source grounding\.Supporting papers includeopenreview:0vKokoPKTo,openreview:1OyE9IK0kx,openreview:4Po8d9GAfQ,openreview:70YeidEcYR,openreview:77gQUdQhE7,openreview:JtGPIZpOrz,openreview:P6dwZJpJ4m,openreview:YUYJsHOf3c,openreview:iEdEHPcFeu, andopenreview:w6nlcS8Kkn\. The originating motifs were assumption mismatch, repeated unaddressed limitation, and unrealistic assumption\.Score\.Mean overall formulation quality:4\.504\.50

These results clarify the role of evolutionary exploration\. The branch is good at breadth: it produces many source\-grounded hypotheses and can combine signals from several related gaps\. The more modest formulation scores show where the main family\-report path adds value\. Formal problem statements, assumption statuses, risks, and ambiguity flags are not just formatting details; they are the stages that turn a promising hypothesis into an inspectable research\-problem artifact\. In this sense, the evolutionary branch is best viewed as a generator of candidate directions for further refinement, while the main SGHA path produces the final formalized families used in the primary comparison\.

### 4\.7 Model\- and Corpus\-Size Sensitivity

We end the results with two Bandit sensitivity studies\. These experiments are not meant to establish a scaling law\. Instead, they show how SGHA’s later stages respond when we change either the model used inside the pipeline or the amount of literature evidence available to it\.

##### Model size\.

Table[14](https://arxiv.org/html/2608.17501#S4.T14)compares the main 9B Bandits run with smaller 4B and larger 27B configurations\. The 4B run produces a much smaller evidence base, but still reaches verification\-passed gaps and direct formulations\. The run stops before final family construction because none of the ambition\-expanded variants pass the critic\. The 9B and 27B runs both reach formalized project families\. The 27B run produces more verification\-passed gaps than the 9B run, but the critic and consolidation stages reduce them to one final family\.

Table 14:Bandits model\-size sensitivity\. Accepted variants are ambition\-expanded formulations that pass the independent critic\. Mean overall is reported for scoreable final families\.ModelTuplesVerified gapsDirectAccepted variantsFamiliesFormal statementsMean overall4B20488000–9B1554664335\.9327B157019193116\.67

The model\-size study shows that capacity changes both the evidence extracted upstream and the shape of the final funnel\. A smaller model can still identify formulation\-ready gaps, but the final output depends on whether the generated variants survive criticism\. A larger model can surface many more verified gaps, while consolidation can still lead to a small final set\. This is consistent with SGHA’s design: final families are selected through verification, criticism, and consolidation, rather than by a fixed output quota\.

##### Corpus size\.

Table[15](https://arxiv.org/html/2608.17501#S4.T15)varies the Bandits corpus from 50 to 250 selected papers\. The 50\-paper run reaches a verified gap and a direct formulation, but does not produce a final family\. From 100 papers onward, SGHA reaches the formal problem stage\. The 200\-paper setting produces the largest number of final families and the highest mean overall score in this diagnostic, while the 250\-paper setting remains close and produces a complete set of formalized outputs\.

Table 15:Bandits corpus\-size sensitivity\. The 250\-paper setting is the main Bandits run\.Selected papersTuplesVerified gapsDirectAccepted variantsFamiliesFormal statementsMean overall5033911000–100706443225\.801501036551115\.20200139210106556\.122501554664335\.93

The corpus\-size study gives a similar message\. A richer corpus gives SGHA more evidence and more opportunities for candidate\-gap discovery, but the final output is shaped by which gaps pass verification and which formulations survive the critic\. The best result among these Bandits settings appears at 200 papers, while the 250\-paper run still produces a strong but smaller final set\. We therefore treat corpus size as a sensitivity variable: enough evidence is important, but the structure of that evidence, and how it passes through the later gates, matters just as much\.

## 5 Discussion

In this paper, we ask whether research\-problem generation can start from patterns in the literature rather than from open\-ended ideation\. The results suggest that this changes the output in a meaningful way\. Strong ideation systems, especially AI\-Scientist\-v2 with Claude Opus, can produce polished and technically detailed proposals\. However, SGHA produces a different kind of artifact: a problem family that keeps track of the evidence, the verified gap, the assumptions being questioned, and the definitions that remain unresolved\. This difference is reflected in the scores as SGHA is strongest on source\-grounded specificity, assumption\-boundary clarity, formalizability, and ambiguity hygiene, while the strongest ideation baseline is competitive on clarity, technical detail, nontriviality, and scope control\.

This distinction matters because a useful research problem is not only a well\-written idea\. A researcher also needs to know where the problem came from, what assumption is being relaxed or challenged, what would count as progress, and which parts of the formulation are still underspecified\. The qualitative examples make this visible\. The AI\-Scientist\-v2 outputs often read like proposal sketches, with motivations and evaluation plans, while the MOOSE\-Star output is closer to a compact hypothesis grounded in an inspiration paper\. The SGHA output is more structured: it includes provenance, notation, assumptions, a success criterion, risks, and ambiguity flags\. These fields do not make the problem automatically correct or novel, but they make the candidate easier to inspect and refine\.

The profile\-conditioned and evolutionary experiments further show two ways to use the same system beyond the main run\. Profile conditioning changes the evidence base, so the final problems can move toward a researcher\-specific context while still passing through the same verification and formulation stages\. On the other hand, evolutionary exploration serves a different purpose: it searches more broadly around literature\-grounded seeds\. Its outputs are useful for breadth and source\-grounded hypothesis generation, while the lower formalizability and ambiguity\-hygiene scores show why the final family\-report path remains important\.

The sensitivity studies further are a useful caution against reading SGHA as a simple scaling pipeline\. Larger models and larger corpora can change the evidence graph and the set of candidate gaps, but the final output is shaped by verification, criticism, and consolidation\. This is appropriate for a literature\-driven system: the goal is not to force a fixed number of ideas, but to return problem families when the evidence supports them\. At the same time, the current evaluation is limited to formulation quality\. The LLM judges do not establish external novelty, theoretical correctness, or future scientific impact, and the semi\-formal statements are not finished theorem statements\. SGHA should therefore be viewed as a tool for producing evidence\-linked starting points that researchers can inspect, sharpen, and compare against the broader literature\.

## 6 Conclusion

We presented SGHA, a training\-free system for generating research\-problem families from scientific literature\. The central idea is to use the literature itself as the starting point: papers already contain assumptions, limitations, empirical patterns, and open directions that can inherently suggest useful research problems when considered together\. SGHA turns these signals into a structured evidence graph, identifies candidate gaps, verifies them, and formulates the surviving candidates as project families with semi\-formal problem statements\.

Across five machine\-learning domains, SGHA produced 15 formalized project families from 1,250 selected papers and 8,634 extracted evidence tuples\. In the formulation\-only evaluation, SGHA achieved the highest overall score among the compared methods and was strongest on source\-grounded specificity, assumption\-boundary clarity, formaliability, and ambiguity\. The qualitative examples further show the same pattern: strong ideation baselines can produce polished proposal\-style ideas, while SGHA produces structured problem artifacts with provenance, assumptions, objectives, risks, and ambiguity flags\.

These results point to a broader role for scientific assistants\. Beyond summarizing papers or generating ideas from a topic, they can help organize evidence across papers into clearer research\-problem formulations\. SGHA takes a step in that direction by making the path from literature evidence to problem statement explicit\. This gives researchers a more grounded starting point for reading, refining, and developing new research directions\.

## 7 Scope and Intended Use

SGHA is aimed at a particular kind of research\-problem generation: problems whose evidence is already partly visible in a scientific corpus\. Many useful directions begin this way\. A set of papers may rely on the same strong assumption, report related failure modes, leave similar limitations unresolved, or evaluate methods under narrow conditions\. SGHA is designed to collect these signals, organize them as evidence, and turn the strongest patterns into clearer problem formulations\.

This scope is central to the system\. SGHA is not positioned as a replacement for open\-ended ideation; it is a complementary mode of ideation grounded in the literature\. Open\-ended ideation is useful when the goal is to speculate freely, connect distant areas, or invent directions that are not yet reflected in papers\. SGHA is useful when the goal is to ask: given the literature we have, what problems are already being suggested by its assumptions, gaps, and tensions? The verification and formalization stages are designed around this setting\.

The evaluation follows the same framing\. We assess whether SGHA produces clear, source\-grounded, formalizable problem artifacts with explicit assumptions and ambiguity flags\. We do not use the results as a claim that the generated problems are externally novel or already correct in the broader literature\. Instead, the intended output is a structured starting point for researchers: a formulation that preserves why the problem arose, what evidence supports it, and what still needs to be sharpened before the direction becomes a complete research project\.

## Generative AI Disclosure

The authors use ChatGPT for language editing and formatting\. They also use the Stanford Agentic Reviewer111[https://paperreview\.ai/](https://paperreview.ai/)and the CMU Paper Reviewer\[[10](https://arxiv.org/html/2608.17501#bib.bib17)\]to obtain preliminary feedback and improve the paper\. Codex was also used to organise the code and prepare the README files\. All technical content, experimental design, analyses, and conclusions remain the sole responsibility of the authors\.

## References

- \[1\]\(2021\)Regret minimization in heavy\-tailed bandits\.InConference on Learning Theory, COLT 2021, 15\-19 August 2021, Boulder, Colorado, USA,M\. Belkin and S\. Kpotufe \(Eds\.\),Proceedings of Machine Learning Research, Vol\.134,pp\. 26–62\.External Links:[Link](http://proceedings.mlr.press/v134/agrawal21a.html)Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p3.1)\.
- \[2\]I\. Aouali\(2024\)Diffusion models meet contextual bandits with large action spaces\.Open Review\.Cited by:[§2\.2](https://arxiv.org/html/2608.17501#S2.SS2.p2.1)\.
- \[3\]I\. Aouali\(2026\)Diffusion models meet contextual bandits\.Advances in Neural Information Processing Systems38,pp\. 166613–166651\.Cited by:[§2\.2](https://arxiv.org/html/2608.17501#S2.SS2.p2.1)\.
- \[4\]P\. Auer, N\. Cesa\-Bianchi, and P\. Fischer\(2002\)Finite\-time analysis of the multiarmed bandit problem\.Machine Learning47\(2\),pp\. 235–256\.External Links:[Document](https://dx.doi.org/10.1023/A%3A1013689704352),[Link](https://doi.org/10.1023/A:1013689704352),ISSN 1573\-0565Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p3.1)\.
- \[5\]J\. Baek, S\. K\. Jauhar, S\. Cucerzan, and S\. J\. Hwang\(2025\)Researchagent: iterative research idea generation over scientific literature with large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 6709–6738\.Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p1.1),[§1](https://arxiv.org/html/2608.17501#S1.p4.1)\.
- \[6\]S\. Bubeck, N\. Cesa\-Bianchi, and G\. Lugosi\(2012\)Bandits with heavy tail\.External Links:1209\.1727,[Link](https://arxiv.org/abs/1209.1727)Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p3.1)\.
- \[7\]B\. E\. Engelhardt, M\. I\. Jordan, J\. R\. Srouji, and S\. E\. Brenner\(2011\)Genome\-scale phylogenetic function annotation of large and diverse protein families\.Genome research21\(11\),pp\. 1969–1980\.Cited by:[§4\.5](https://arxiv.org/html/2608.17501#S4.SS5.p6.pic1.p2.1)\.
- \[8\]S\. Gharat and J\. Komiyama\(2026\)PRISM: problem discovery via structural motifs in knowledge graphs\.InICML 2026 AI for Science Workshop,Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p11.pic1.2.2.1)\.
- \[9\]J\. Gottweis, W\. Weng, A\. Daryin, T\. Tu, A\. Palepu, P\. Sirkovic, A\. Myaskovsky, F\. Weissenberger, K\. Rong, R\. Tanno,et al\.\(2025\)Towards an ai co\-scientist\.arXiv preprint arXiv:2502\.18864\.Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p1.1),[§1](https://arxiv.org/html/2608.17501#S1.p4.1)\.
- \[10\]S\. Kim, D\. Yoon, K\. Gashteovski, J\. Suk, J\. Baek, P\. Aggarwal, I\. Wu, V\. Zaverkin, S\. Petkoski, D\. R\. Schrider,et al\.\(2026\)On the limits and opportunities of ai reviewers: reviewing the reviews of nature\-family papers with 45 expert scientists\.arXiv preprint arXiv:2605\.20668\.Cited by:[Generative AI Disclosure](https://arxiv.org/html/2608.17501#Sx1.p1.1)\.
- \[11\]T\. Lai and H\. Robbins\(1985\)Asymptotically efficient adaptive allocation rules\.Advances in Applied Mathematics6\(1\),pp\. 4–22\.External Links:ISSN 0196\-8858,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/0196-8858%2885%2990002-8),[Link](https://www.sciencedirect.com/science/article/pii/0196885885900028)Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p3.1)\.
- \[12\]X\. Li, H\. Tu, and X\. Han\(2026\)Graph2Idea:retrieval\-augmented scientific idea generation with graph\-structured contexts\.External Links:2606\.09105,[Link](https://arxiv.org/abs/2606.09105)Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p4.1)\.
- \[13\]C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. Ha\(2024\)The ai scientist: towards fully automated open\-ended scientific discovery\.arXiv preprint arXiv:2408\.06292\.Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p1.1),[§1](https://arxiv.org/html/2608.17501#S1.p4.1)\.
- \[14\]S\. Ren, C\. Xie, P\. Jian, Z\. Ren, C\. Leng, and J\. Zhang\(2025\)Towards scientific intelligence: a survey of llm\-based scientific agents\.arXiv preprint arXiv:2503\.24047\.Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p1.1)\.
- \[15\]R\. Vasu, C\. Basu, B\. Dalvi Mishra, C\. Sarasua, P\. Clark, and A\. Bernstein\(2025\)HypER: literature\-grounded hypothesis generation and distillation with provenance\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 25413–25438\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1292/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1292),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p11.pic1.2.1.1)\.
- \[16\]M\. J\. Wainwright and M\. I\. Jordan\(2005\)11 a variational principle for graphical models\.Cited by:[§4\.5](https://arxiv.org/html/2608.17501#S4.SS5.p6.pic1.p2.1)\.
- \[17\]Q\. Wang, D\. Downey, H\. Ji, and T\. Hope\(2024\)SciMON: scientific inspiration machines optimized for novelty\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 279–299\.External Links:[Link](https://aclanthology.org/2024.acl-long.18/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.18)Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p11.pic1.2.1.1)\.
- \[18\]Y\. Yamada, R\. T\. Lange, C\. Lu, S\. Hu, C\. Lu, J\. Foerster, J\. Clune, and D\. Ha\(2025\)The ai scientist\-v2: workshop\-level automated scientific discovery via agentic tree search\.arXiv preprint arXiv:2504\.08066\.Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p1.1),[§1](https://arxiv.org/html/2608.17501#S1.p4.1)\.
- \[19\]Z\. Yang and L\. Bing\(2026\)MOOSE\-star: unlocking tractable training for scientific discovery by breaking the complexity barrier\.InProceedings of the 43rd International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p11.pic1.2.1.1),[§1](https://arxiv.org/html/2608.17501#S1.p4.1),[§1](https://arxiv.org/html/2608.17501#S1.p7.1)\.
- \[20\]Y\. Zhang, X\. Chen, B\. Jin, S\. Wang, S\. Ji, W\. Wang, and J\. Han\(2024\)A comprehensive survey of scientific large language models and their applications in scientific discovery\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8783–8817\.Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p1.1)\.
- \[21\]Y\. Zhou, H\. Liu, T\. Srivastava, H\. Mei, and C\. Tan\(2024\)Hypothesis generation with large language models\.InProceedings of the 1st Workshop on NLP for Science \(NLP4Science\),pp\. 117–139\.Cited by:[§1](https://arxiv.org/html/2608.17501#S1.p1.1),[§1](https://arxiv.org/html/2608.17501#S1.p11.pic1.2.1.1)\.

## Appendix ASGHA Prompts, Output Schemas, and Validation Rules

This appendix reports the prompt templates, output contracts, and validation rules used by SGHA\. Runtime prompts contain instance\-specific content, such as paper text, candidate\-gap evidence, verification summaries, or project\-family records\. We therefore show the fixed prompt instructions with placeholders for dynamic fields, such as\{paper\_text\},\{gap\_json\}, or\{verification\_summary\}\.

SGHA uses language models for bounded extraction, review, and formulation stages\. The system uses deterministic code for graph construction, motif detection, hard verification gating, family consolidation, and final report auditing\. The main SGHA outputs are generated only after candidate gaps pass screening and verification\.

### A\.1 Evidence Extraction

The extraction stage reads a parsed paper and returns a structured paper object together with typed evidence tuples\. The prompt asks for extracted scientific content from the given paper, not a summary or template\.

##### Prompt template\.

You are a scientific information extractor\. Read the paper below and extract structured information as JSON\.IMPORTANT: Do NOT return a schema or template\. Return a JSON object with REAL extracted content from this specific paper\.Output keys:paper\_idclaimslimitationsfailure\_conditionscontradictions\_or\_tensionsfuture\_workmethodsassumptionstasksdatasetsmetricsresultstuplesFor each tuple:subject \-\- specific named entityrelation \-\- MUST be one of: evaluated\_on, improves\_over, fails\_under, assumes,contradicts, uses\_dataset, measured\_by, limited\_by, addresses,not\_addressed\_byobjectevidence\_text \-\- exact supporting text from the papersectionconfidence \-\- value in \[0, 1\]claim\_typepolarityconditiontaskdatasetmodelmetricdirectionevidence\_typestrengthsource\_spansubject\_scoperesolved\_by\_paperin\_related\_workRules:\- Extract content from the provided paper only\.\- Use specific scientific entities, not generic placeholders\.\- Evidence text and source spans must be grounded in the parsed paper\.\- Do not conflate assumptions with failures\.\- Do not invent results, limitations, datasets, or methods\.Paper ID: \{paper\_id\}Title: \{title\}Paper text:\{paper\_text\}

##### Paper\-level schema\.

PaperExtraction:paper\_id: stringclaims: list\[string\]limitations: list\[string\]failure\_conditions: list\[string\]contradictions\_or\_tensions: list\[string\]future\_work: list\[string\]methods: list\[string\]assumptions: list\[string\]tasks: list\[string\]datasets: list\[string\]metrics: list\[string\]results: list\[string\]tuples: list\[ScientificTuple\]

##### Tuple schema\.

Table 16:Evidence tuple schema used by SGHA\.FieldMeaningsubjectSpecific method, model, assumption, task, dataset, metric, claim, or scientific object\.relationOne ofevaluated\_on,improves\_over,fails\_under,assumes,contradicts,uses\_dataset,measured\_by,limited\_by,addresses, ornot\_addressed\_by\.objectTarget of the relation\.evidence\_text/source\_spanSupporting text from the parsed paper\.sectionPaper section containing the evidence\.confidenceModel confidence in the extraction\.claim\_typeOne ofmethod,result,limitation,assumption,comparison,failure,hypothesis,evaluation, orunknown\.polarityOne ofpositive,negative,neutral,mixed, orunknown\.conditionRegime or condition under which the tuple holds, if available\.task,dataset,model,metricStructured context fields, especially for evaluation\-related claims\.directionOne ofimproves,worsens,no\_change, orunknown\.evidence\_typeOne ofempirical,theoretical,ablation,qualitative,assumption,citation,benchmark, orunknown\.strengthOne ofstrong,moderate,weak, orunknown\.subject\_scopeOne ofown\_method,prior\_work, orgeneral\.resolved\_by\_paperWhether the paper itself appears to resolve the issue\.in\_related\_workWhether the evidence comes from related work or background discussion\.

##### Validation rules\.

Before tuples enter the evidence graph, SGHA applies deterministic validation and cleanup\. The relation must belong to the fixed relation vocabulary, and the subject, object, and evidence fields must be present\. The supporting evidence span must match the parsed paper text after normalization, so that the tuple remains traceable to the source paper\. SGHA removes generic subjects such as “method,” “algorithm,” or “our method,” filters overly generic task labels, drops evaluation tuples that do not include task, dataset, or metric context, and normalizes metadata such as relation polarity\. For example,fails\_underandlimited\_byare treated as negative relations, whileassumesis treated as neutral\.

### A\.2 Graph Construction and Motif Detection

Graph construction and motif detection are deterministic\. There is no LLM prompt for this stage\. SGHA inserts validated paper objects and evidence tuples into a typed evidence graph\.

##### Node and edge types\.

Node types:PaperMethodTaskDatasetMetricAssumptionResultLimitationFailureConditionClaimGapHypothesisEdge types:mentionsproposesevaluated\_onuses\_datasetmeasured\_byimproves\_overfails\_underassumeslimited\_bycontradictsaddressesnot\_addressed\_bysupports\_gapweakens\_gaphas\_mechanismfeasible\_withderived\_fromcounterevidence\_forpartially\_addresses\_gaprelaxes\_assumption\_ofhandles\_failure\_of

##### Motif families\.

SGHA searches the typed graph using predefined motif detectors\. A motif hit instantiates a candidate structural gap, but does not by itself verify that the gap is valid or novel\.

Motif families:shared\_failure\_conditionrepeated\_unaddressed\_limitationconflicting\_claimssparse\_evaluationassumption\_mismatchunrealistic\_assumptionshared\_unrealistic\_assumptiontheory\_practice\_gaptransfer\_solution\_gapunresolved\_tradeoffmissing\_stress\_testclaim\_without\_measurementmissing\_baseline\_comparison

##### Candidate\-gap schema\.

CandidateGap:gap\_id: stringgap: stringtarget: stringscope: stringmechanism: stringsupporting\_evidence: listcounterevidence: listtraceability\_path: listnovelty\_score: floatfeasibility\_score: floatimpact\_score: floatoverall\_score: floatmotif\_type: stringmotif\_id: stringpaper\_ids: list\[string\]extraction\_uncertainty: floatcounterevidence\_edges: listcounterevidence\_papers: listcounterevidence\_status: stringcounterevidence\_summary: stringremaining\_gap\_scope: stringnovelty\_status: stringnovelty\_rationale: string

The graph builder normalizes selected aliases, removes junk method labels, deduplicates parallel edges, and keeps provenance on graph edges\. Candidate gaps are created from motif hits, not from arbitrary missing graph edges\.

### A\.3 Novelty and Corpus\-Coverage Filtering

After motif detection, SGHA applies a corpus\-grounded novelty and coverage screen\. This screen removes candidates that appear trivial, already solved, or already covered by nearby evidence in the selected corpus\. It is not an external literature novelty proof\.

##### Prompt template\.

You are a senior ML researcher evaluating structural gaps found by automated analysis of a paper corpus\.Your job is to FILTER OUT weak gaps\. Most gaps will be trivial \-\- be skeptical by default\.Research area: \{topic\_description\}Classify each gap into exactly one category:\- "trivial": gap is just "method X can fail" or "X has limitations" with no named assumption or failure condition\.\- "known\_open": specific named gap experts know about but have not solved\.\- "known\_solved": already substantially addressed in literature published after 2021\.\- "novel": non\-obvious pattern only visible by reading across multiple papers\.Be strict about trivial\. Be fair about known\_open when the gap names a specific assumption or condition\.\{gap\_blocks\}Respond with a JSON object:\{"classifications": \[\{"gap\_index": 1, "class": "<trivial\|known\_open\|known\_solved\|novel\>", "reason": "<one sentence\>"\}\]\}

##### Filtering rules\.

SGHA keeps candidates labelednovel,known\_open, orpartially\_addressed\_in\_corpus\. It removes candidates labeledtrivial,known\_solved, orknown\_solved\_in\_corpus\. If corpus counterevidence already marks a candidate as solved within the selected evidence base, the LLM classification can be skipped\. If a batch classification returns fewer labels than expected, missing labels default to keep with an error rationale; later verification applies the stricter gate\.

### A\.4 Verification Agents and Hard Gate

Novelty survivors selected for verification are reviewed by role\-specific agents\. The agents use the same output schema but different role instructions\. The support agent looks for evidence supporting the gap\. The skeptic agent looks for counterevidence or already\-addressed risk\. The feasibility agent checks whether the problem can plausibly be studied\. The mechanism agent checks whether there is a plausible explanation for the gap\. The verification critic gives a conservative final assessment\.

##### Verification\-agent output schema\.

\{"gap\_id": "<copy from gap\>","agent\_name": "<support\|skeptic\|feasibility\|mechanism\|critic\>","summary": "<one sentence assessment\>","evidence": \[\{"paper\_id": "\.\.\.", "evidence\_text": "\.\.\."\}\],"counterevidence": \[\{"paper\_id": "\.\.\.", "evidence\_text": "\.\.\."\}\],"citations": \["paper\_id\_1", "paper\_id\_2"\],"confidence": 0\.0,"failure\_modes": \["\.\.\."\],"scores": \{"evidence\_support": 0\.0,"novelty": 0\.0,"feasibility": 0\.0,"specificity": 0\.0,"scope\_validity": 0\.0\}\}

##### Repair prompt\.

If an agent response is not valid JSON, SGHA retries once with a repair prompt:

The previous response was invalid JSON\. Return ONLY valid JSON with these keys:gap\_id, agent\_name, summary, evidence, counterevidence, citations, confidence, failure\_modes, scores\.confidence must be a float between 0\.0 and 1\.0\.scores must be a dict with keys: evidence\_support, novelty, feasibility, specificity, scope\_validity \(all floats 0\.0\-1\.0\)\.Gap ID: \{gap\_id\}Agent: \{agent\_name\}

##### Hard verification gate\.

The hard verification gate is deterministic\. A reviewed gap enters formulation only if the gate accepts it\.

verification\_gate:enabled: truemode: survival\_scoremin\_survival\_score: 0\.60require\_all\_agents: truerequire\_critic\_non\_reject: truemin\_critic\_confidence: 0\.50fail\_on\_agent\_parse\_failure: trueallow\_reviewed\_only\_fallback: false

A candidate fails the gate if required agent outputs are missing, if an agent has a disqualifying parse failure, if the survival score is missing or below threshold, or if the critic rejects the candidate or has insufficient confidence\.

### A\.5 Direct Formulation

The direct\-formulation prompt is used only for gaps that pass the hard verification gate\. It asks for a single evidence\-close problem formulation\.

##### Prompt template\.

You are formulating ONE direct, coherent research problem from a single verification\-passed structural gap\. This is a PROPOSAL \(no results yet\)\.DECLARED RUN CONTEXT\- domain\_name: \{domain\_name\}\- topic\_description: \{topic\_description\}\- keyphrases: \{keyphrases\}STRICT COHERENCE RULES:\- exactly one core setting, one main assumption/failure/limitation, one core objective\- one plausible theorem OR algorithm OR benchmark/empirical target\- do NOT combine unrelated methods/assumptions/failures into a collage\- do NOT introduce domain\-specific concepts unless they appear in the declared run context, source gap, supporting paper titles, extracted evidence, or verification findings\- use the domain’s own entities, objectives, assumptions, feedback/measurement model, and evaluation targets\- if the evidence is thin, say so and lower the scoresSOURCE VERIFICATION\-PASSED GAP\- gap\_id: \{gap\_id\}\- motif\_type: \{motif\_type\}\- target: \{target\}\- gap statement: \{gap\}\- mechanism: \{mechanism\}\- scope: \{scope\}\- supporting paper titles: \{papers\}\- extracted evidence snippets: \{extracted\_evidence\}VERIFICATION AGENT FINDINGS:\{verification\_summary\}

##### Output schema\.

\{"direct\_title": "","direct\_problem\_statement": "","proposal\_style\_abstract": "","core\_setting": "","core\_assumption\_or\_failure": "","core\_objective": "","possible\_theorem\_target": "","possible\_algorithmic\_target": "","possible\_empirical\_target": "","coherence\_score": 0\.0,"specificity\_score": 0\.0,"feasibility\_score": 0\.0,"term\_soup\_risk": 0\.0,"main\_risk": "","falsification\_condition": "","recommendation": "KEEP \| MAYBE \| DROP"\}

SGHA marks a direct formulation as strong when coherence, specificity, and term\-soup\-risk checks pass and the recommendation isKEEP\. Low\-coherence or high\-term\-soup formulations are deprioritized or removed before later stages\.

### A\.6 Ambition Expansion and Critic

Ambition expansion starts from a direct formulation\. The goal is to broaden the problem where useful while keeping it tied to the verified evidence\.

##### Ambition\-expansion prompt template\.

You are doing CONTROLLED ambition expansion of ONE clean, single\-gap research formulation\.Goal: a genuinely NON\-INCREMENTAL but still coherent formulation \-\- NOT a multi\-concept collage,and NOT just validating one named method in one more setting\.SOURCE DIRECT FORMULATION:\{direct\_formulation\}A formulation is INCREMENTAL if its central contribution is only:\- evaluating one named method in one additional setting\- testing robustness of one paper’s method without changing the scientific question\- a small benchmark variation\- restating a limitation already identified by one source paper\- depending heavily on one named method\- a cleaner title for the same narrow gapA formulation is NON\-INCREMENTAL if it:\- identifies a broader problem class\- relaxes, removes, or proves necessity of a key assumption\- characterizes a boundary / impossibility / phase transition / identifiability limit / failure regime\- proposes a constructive alternative at method/system/mechanism/evaluation\-class level\- defines benchmark/evaluation protocol exposing a general failure mode\- gives a unifying explanation connecting observations under one structural condition

##### Ambition\-variant schema\.

\{"title": "","problem\_statement": "","proposal\_style\_abstract": "","core\_setting": "","generalized\_axis": "","core\_assumption\_or\_failure": "","core\_objective": "","contribution\_type": "theorem\|algorithm\|lower\_bound\|benchmark\|empirical\_study\|impossibility\_boundary","theorem\_target": "","algorithmic\_target": "","empirical\_target": "","broader\_problem\_class": "","assumption\_shift": "","boundary\_or\_failure\_regime": "","constructive\_or\_explanatory\_target": "","method\_class\_scope": "","named\_method\_dependency": "low\|medium\|high","why\_not\_just\_validation": "","why\_not\_incremental": "","why\_not\_term\_soup": "","non\_incrementality\_score": 0\.0,"incrementality\_risk": 0\.0,"main\_risk": "","falsification\_condition": "","coherence\_score": 0\.0,"ambition\_score": 0\.0,"specificity\_score": 0\.0,"feasibility\_score": 0\.0,"term\_soup\_risk": 0\.0,"recommendation": "KEEP\|MAYBE\|DROP"\}

##### Ambition\-critic prompt template\.

You are an INDEPENDENT critic\. You did NOT write the formulation below and you must NOT rewrite,fix, or improve it\. You only JUDGE whether it is genuinely non\-incremental\.Judge strictly:1\. Is this genuinely non\-incremental relative to the source direct formulation?2\. Did it change the scientific object, or just wording?3\. Is broader\_problem\_class real and specific?4\. Is the contribution meaningful beyond one named method?5\. Is the claimed boundary/impossibility/algorithm/benchmark supported by source evidence?6\. Is contribution\_type inflated?SOURCE DIRECT FORMULATION:\{direct\_formulation\}VARIANT TO JUDGE:\{ambition\_variant\}

##### Ambition\-critic schema\.

\{"critic\_non\_incrementality\_score": 0\.0,"critic\_incrementality\_risk": 0\.0,"critic\_fake\_ambition\_risk": 0\.0,"critic\_named\_method\_dependency": "low\|medium\|high","critic\_broader\_problem\_class\_valid": true,"critic\_contribution\_type\_valid": true,"critic\_supported\_by\_source": "strong\|moderate\|weak","critic\_quality\_label": "STRONG\_NON\_INCREMENTAL\|VALID\_BUT\_MODEST\|INCREMENTAL\_REPHRASING\|FAKE\_AMBITION\|DROP","critic\_reason": ""\}

A variant passes the critic when it is sufficiently non\-incremental, source\-supported, and specific, with low fake\-ambition and incrementality risk\. SGHA keeps critic\-passing variants for family consolidation\.

### A\.7 Family Consolidation

Family consolidation is deterministic in the main runs\. It groups critic\-passing variants into project families using strict anchors such as shared verified gaps, shared direct formulations, overlapping supporting papers, and strong identity overlap\.

##### Family schema\.

\{"family\_id": "family:01","family\_title": "","member\_variant\_ids": \[\],"representative\_variant\_id": "","representative\_title": "","family\_problem\_statement": "","proposal\_style\_abstract": "","theorem\_target": "","algorithmic\_target": "","empirical\_target": "","main\_risk": "","falsification\_condition": "","critic\_reason": "","research\_object": "","problem\_class": "","assumption\_shift": "","failure\_boundary\_or\_mechanism": "","constructive\_or\_evaluation\_target": "","source\_verified\_gaps": \[\],"source\_direct\_formulations": \[\],"supporting\_papers": \[\],"related\_seed\_papers": \[\],"quality\_label": "A\_STRONG\|B\_PROMISING\_NEEDS\_REFRAMING\|C\_LOW\_PRIORITY\|D\_DROP","recommended\_action": "READ\_FIRST\|REFRAME\|READ\_LATER\|DROP"\}

The consolidation stage avoids weak transitive grouping\. All selected variants must be assigned consistently, and no invented variant identifiers are allowed\.

### A\.8 Formal Problem Formulation

For each final project family, SGHA produces a semi\-formal problem object\. The prompt asks for a structured problem statement rather than a finished theorem\.

##### Prompt template\.

You are SGHA’s domain\-general formal problem formulation stage\.You must formalize ONE existing project family\. Do NOT invent a new research direction, new family,or new claim of results\. Use only the source family and formulation evidence below\.STRICT RULES:\- Formalize the existing project family only\.\- Prefer clear semi\-formal structure over fake mathematics\.\- If a term is vague, define it cautiously or put it in ambiguity\_flags\.\- If the feedback/data/measurement model is unclear, say so\.\- If the objective cannot be formalized from evidence, set formalization\_confidence to "low"\.\- Every symbol introduced in the formal\_problem\_statement must appear in the variable table\.\- Do NOT use metadata variables such as R = research object, C = problem class,A = assumption shift, or B = boundary\.\- Variables must correspond to problem\-level objects: actions, observations, latent factors,measurements, rewards/outcomes, policies/decisions, constraints, objectives, estimators,environments, systems, functions, processes, or distributions\.\- Every major assumption must be marked kept, relaxed, removed, or questioned\.\- Do not say "we prove", "we show", or imply results already exist\.PROJECT FAMILY:\{project\_family\}

##### Formal\-problem schema\.

\{"plain\_language\_problem": "","formal\_problem\_statement": "","mathematical\_setup": \{"entities": \[""\],"variables": \[\{"symbol": "","meaning": "","type": "scalar \| vector \| set \| distribution \| process \| function \| other","source": "from evidence \| introduced for formalization"\}\],"data\_or\_observations": "","feedback\_or\_measurement\_model": "","decision\_variables\_or\_outputs": "","objective": "","constraints": "","success\_criterion": ""\},"assumptions": \[\{"name": "","description": "","status": "kept \| relaxed \| removed \| questioned","source\_evidence": \[""\]\}\],"open\_question": "","possible\_result\_types": \{"theorem": "","algorithm": "","empirical\_or\_benchmark": ""\},"evaluation\_protocol": "","ambiguity\_flags": \[\{"term": "","why\_ambiguous": "","what\_user\_must\_define": ""\}\],"source\_grounding": \{"source\_verified\_gaps": \[""\],"supporting\_papers": \[""\],"representative\_formulation": "","critic\_reason": ""\},"formalization\_confidence": "high \| medium \| low","formalization\_risk": "","requires\_human\_definition": true\}

If the formal\-problem output is not valid JSON, SGHA retries once with a repair prompt\. If repair fails, the system uses a source\-grounded low\-confidence fallback and marks the formalization as requiring human definition\.

### A\.9 Profile\-Conditioned Additions

Profile\-conditioned SGHA uses the same downstream prompts and gates after corpus construction\. The profile changes the evidence base and prioritization, not the validity checks\.

##### Seed\-paper schema\.

SeedPaper:seed\_paper\_idtitleauthorsyearvenueabstractlocal\_pdf\_patharxiv\_idopenreview\_iddoiurlsourcerelation\_to\_seed\_profileis\_manual\_seed\_paperseed\_labelmetadata\_incompleteprovenance

##### Profile schema\.

\{"seed\_label": "","topic": "","seed\_titles": \[\],"keyphrases": \[\],"method\_terms": \[\],"task\_terms": \[\],"assumption\_terms": \[\],"benchmark\_terms": \[\],"generated\_openreview\_queries": \[\],"venues": \[\],"exclusion\_terms": \[\],"profile\_summary": ""\}

In profile\-conditioned mode, Stage 7 and later stages may receive seed\-alignment information and related seed papers as context\. Verification, formulation, ambition criticism, family consolidation, and formalization remain the same\.

### A\.10 Evolutionary Exploration

The evolutionary branch is separate from the main family\-report path\. It starts from literature\-grounded seeds and generates nearby variants by changing parts of the problem representation, such as assumptions, mechanisms, objectives, settings, or regimes\. These variants are critiqued, selected, and refined over multiple rounds\.

##### Evolutionary\-candidate schema\.

EvolutionaryHypothesis:candidate\_id: stringsource\_seed\_id: stringparent\_ids: list\[string\]generation: integermutation\_type: stringtitle: stringproblem\_statement: stringgap: stringtarget: stringscope: stringmechanism: stringproposal\_style\_abstract: stringsource\_grounding: listmotif\_lineage: listevolution\_score: floatrank: integerselected: boolean

The evolutionary branch is used for breadth\. Its selected or ranked hypotheses can be evaluated with the same formulation\-only rubric, but they are reported separately from the final verification\-gated project families\.

### A\.11 Final Report Auditing

Final report rendering is deterministic in the main runs\. The audit checks that final families are present or that a low\-signal status is recorded; that direct formulations trace to verification\-passed gaps; that selected variants trace to direct formulations; that final families trace to accepted variants; and that formal problem JSON exists when formalization is enabled\. It also checks that novelty\-only or reviewed\-only candidates do not enter the default family\-report path\.

The final status is one of:

PIPELINE\_COMPLETED\_PASSPIPELINE\_COMPLETED\_LOW\_SIGNALPIPELINE\_COMPLETED\_AUDIT\_FAILURE

A low\-signal status means the pipeline completed but did not produce a final family because the evidence or downstream checks did not support one\.

## Appendix BLLM\-Judge Rubric and Candidate Normalization

In this appendix, we describe how generated candidates are normalized and evaluated by the formulation\-only LLM judges\.

### B\.1 Candidate Packet Format

Before judging, outputs from all methods are converted into a common candidate\-packet format\. The packet preserves the fields produced by the original method\. Fields that are not produced by a method are left asnot provided; they are not filled in manually after generation\.

CandidatePacket:candidate\_id: stringmethod\_label: hidden during judgingdomain: stringtitle: stringproblem\_statement: stringmotivation\_or\_abstract: stringproposed\_direction: stringexpected\_contribution: stringevaluation\_plan: stringrisks\_or\_caveats: stringsource\_context\_or\_grounding: stringassumptions\_or\_problem\_setup: stringformal\_problem\_statement: stringambiguity\_or\_missing\_definitions: string

This format lets the judge compare artifacts produced by different systems while preserving their natural structure\. SGHA final families often include formal problem statements, assumptions, source provenance, and ambiguity flags\. However, ideation baselines often include motivation, proposed directions, evaluation plans, and caveats\. The normalization step keeps these differences visible\.

### B\.2 Formulation\-Only Judge Prompt

The formulation\-only judge evaluates the quality of a research\-problem formulation\.

##### Judge prompt template\.

You are a strict, skeptical senior research reviewer evaluating research\-problem formulation quality only\.Your task is to judge the formulation as a problem statement, not as a project plan\.Do not evaluate implementation plans, experiments, software engineering details, or actionability except where they clarify the formulation itself\.Do not reward a candidate for having a polished plan if the underlying research problem is vague or poorly posed\.Judge only the provided blinded candidate text\.Do not use external knowledge\.Do not perform an external novelty check\.If a field says "not provided", treat it as missing\.Score the candidate using the rubric below\.Return only valid JSON matching the requested schema\.CANDIDATE PACKET:\{candidate\_packet\}RUBRIC:\{rubric\}SCORE ANCHORS:\{score\_anchors\}CAP RULES:\{cap\_rules\}Return JSON only\.

### B\.3 Formulation Rubric

Table[17](https://arxiv.org/html/2608.17501#A2.T17)lists the ten formulation\-only criteria\. Each criterion is scored from 0 to 10\.

Table 17:Formulation\-only LLM\-judge rubric\.CriterionWhat the judge evaluatesProblem\-definition clarityWhether the candidate states a clear research problem rather than a broad topic or loose motivation\.Technical specificityWhether the formulation names concrete objects, settings, mechanisms, assumptions, metrics, or target results\.Well\-posednessWhether the problem has enough structure to be studied, refined, or formalized\.Assumption\-boundary clarityWhether the formulation makes clear which assumptions are used, relaxed, questioned, removed, or missing\.FormalizabilityWhether the problem can plausibly be written as a theorem, algorithmic objective, benchmark protocol, impossibility result, or other formal research target\.NontrivialityWhether the candidate goes beyond a simple application, small robustness check, or generic “apply X to Y” idea\.Scope controlWhether the problem is focused enough to be studied rather than combining many loosely related concepts\.Source\-grounded specificityWhether the formulation is tied to provided source context, evidence, or literature\-derived details\.Ambiguity hygieneWhether the candidate explicitly states missing definitions, unclear feedback models, unresolved assumptions, or caveats\.Overall formulation qualityThe judge’s overall assessment of the research\-problem formulation under the rubric\.

### B\.4 Score Anchors

The judges use the anchors in Table[18](https://arxiv.org/html/2608.17501#A2.T18)\. These anchors are included in the judge prompt to make the scale consistent across models\.

Table 18:Score anchors used by the formulation\-only judges\.ScoreInterpretation0No usable formulation\.1Mostly incoherent or unrelated to a research problem\.2Topic\-level idea with almost no problem structure\.3Very vague formulation with major missing pieces\.4Weak formulation; some direction is visible, but the problem is poorly specified\.5Plausible idea, but important assumptions, scope, or definitions are missing\.6Plausible formulation with useful structure, but still needing substantial refinement\.7Strong formulation with clear problem structure and reasonable technical specificity\.8Very strong formulation; well posed, grounded, and mostly ready for expert refinement\.9Excellent formulation with unusually clear assumptions, scope, and formal target\.10Exceptional formulation; clear, grounded, formalizable, and close to research\-ready\.

### B\.5 Score Consistency Rules

The judge prompt includes cap rules to keep scores consistent when important information is missing\. These rules do not assign scores directly\. They limit the maximum score for a criterion when the candidate lacks the corresponding field or structure\.

Table 19:Score consistency rules used by the formulation\-only judge\.ConditionRuleProblem statement is vague or mostly a topic descriptionOverall formulation quality should be at most 6\.Formal problem statement is not providedWell\-posedness and formalizability should be at most 6\.Assumptions or setup are not providedWell\-posedness and assumption\-boundary clarity should be at most 6\.Ambiguity or missing\-definition field is not providedAmbiguity hygiene should be at most 6\.Source or context grounding is not providedSource\-grounded specificity should be at most 4\.Candidate combines many loosely related conceptsScope control should be at most 4\.Candidate is mainly “apply X to Y” or one\-method validationNontriviality should be at most 6\.No clear formal skeleton is visibleOverall formulation quality should be at most 7\.

### B\.6 Judge Response Schema

Each judge returns a strict JSON object\. The schema records the criterion\-level scores, a recommendation, confidence, strengths, weaknesses, rationale, and a novelty caveat\.

\{"candidate\_id": "","domain": "","scores": \{"problem\_definition\_clarity\_10": 0,"technical\_specificity\_10": 0,"well\_posedness\_10": 0,"assumption\_boundary\_clarity\_10": 0,"formalizability\_10": 0,"nontriviality\_10": 0,"scope\_control\_10": 0,"source\_grounded\_specificity\_10": 0,"ambiguity\_hygiene\_10": 0,"overall\_formulation\_quality\_10": 0\},"recommended\_action": "READ\_FIRST \| PROMISING\_NEEDS\_REFINEMENT \| NEEDS\_REFRAMING \| DROP\_OR\_DEPRIORITIZE","confidence": "LOW \| MEDIUM \| HIGH","strengths": \["\.\.\."\],"weaknesses": \["\.\.\."\],"rationale": "\.\.\.","novelty\_caveat": "Novelty potential judged only from provided text; no external novelty check performed\."\}

The reportedoverall\_formulation\_quality\_10score is the judge’s holistic assessment under the rubric\. It is not computed as a weighted average of the other criteria\.

### B\.7 Judge Panel and Blinding

We use five LLM judges from different providers:

anthropic/claude\-sonnet\-4openai/gpt\-5\.6\-sol\-prox\-ai/grok\-4\.5moonshotai/kimi\-k3google/gemini\-3\.6\-flash

Each judge receives the same blinded candidate packet format, rubric, score anchors, cap rules, and JSON response schema\. Method labels are hidden during scoring\. The blinding key is used only after scoring, during postprocessing\. The judge panel is used only after all candidate outputs have been frozen\.

### B\.8 Calibration Examples

Before scoring the real candidates, each judge is checked on calibration examples\. These examples are designed to test whether the judge applies the rubric in the intended direction\. The calibration set includes vague topic\-level ideas, underformalized formulations, over\-broad or term\-heavy formulations, and stronger source\-grounded formal problem statements\.

Calibration is used as a quality check for the judge behavior\. It is not used to tune separate prompts for different judges; the rubric and response schema remain fixed\.

### B\.9 Personalized Judge

Profile\-conditioned candidates are evaluated with a personalized judge rubric\. This rubric keeps the ten formulation\-quality criteria and adds profile\-specific criteria\. The personalized judge uses only the provided profile context and candidate text; it does not use external knowledge about the researcher\.

##### Personalized judge prompt template\.

You are a strict, skeptical senior research reviewer evaluating personalized research\-problem formulations\.Your task is to judge both formulation quality and personalization quality\.Judge profile alignment only from the provided profile context and the candidate text\.Do not use external knowledge about the researcher\.If a field says "not provided", treat it as missing\.Do not reward name\-dropping\.A good personalized problem connects to artifact\-supported profile themes, source/corpus evidence,and profile\-relevant technical style while formulating a meaningful next problem\.Penalize generic problems that could apply to many researchers in the area\.Penalize off\-profile problems even if technically coherent\.PROFILE CONTEXT:\{profile\_context\}CANDIDATE PACKET:\{candidate\_packet\}Return JSON only\.

##### Additional personalized criteria\.

Table 20:Additional criteria used by the personalized judge\.CriterionMeaningProfile alignmentWhether the candidate problem is supported by the provided profile\-derived literature context\.Profile specificityWhether the problem is specific to that context rather than a generic domain problem\.Intellectual\-style matchWhether the problem matches the profile\-associated technical style or research pattern\.Profile novelty fitWhether the candidate plausibly extends the profile\-associated literature rather than repeating it\.Personalization overallOverall quality of the personalization, considering the above criteria\.

##### Personalized response schema\.

\{"candidate\_id": "","profile": "","scores": \{"problem\_definition\_clarity\_10": 0,"technical\_specificity\_10": 0,"well\_posedness\_10": 0,"assumption\_boundary\_clarity\_10": 0,"formalizability\_10": 0,"nontriviality\_10": 0,"scope\_control\_10": 0,"source\_grounded\_specificity\_10": 0,"ambiguity\_hygiene\_10": 0,"overall\_formulation\_quality\_10": 0,"profile\_alignment\_10": 0,"profile\_specificity\_10": 0,"intellectual\_style\_match\_10": 0,"profile\_novelty\_fit\_10": 0,"personalization\_overall\_10": 0\},"recommended\_action": "READ\_FIRST \| PROMISING\_NEEDS\_REFINEMENT \| NEEDS\_REFRAMING \| DROP\_OR\_DEPRIORITIZE","confidence": "LOW \| MEDIUM \| HIGH","strengths": \["\.\.\."\],"weaknesses": \["\.\.\."\],"rationale": "\.\.\.","personalization\_caveat": "Profile alignment judged only from provided profile context; no external profile knowledge used\.","novelty\_caveat": "Novelty potential judged only from provided text; no external novelty check performed\."\}

The personalized judge also uses additional consistency rules\. For example, candidates that only name\-drop the researcher receive low profile\-alignment scores, and generic domain problems without profile\-specific evidence receive lower profile\-specificity scores\.

## Appendix CAdditional Result Tables

This appendix provides detailed count and score tables that support the main results\. The main text reports the central trends; here we include the full SGHA stage counts and selected per\-domain formulation scores\.

### C\.1 Full SGHA Pipeline Counts

Table[21](https://arxiv.org/html/2608.17501#A3.T21)reports the full SGHA count funnel for the five main domains\. The table separates paper\-level counts, graph\-level counts, gap\-level counts, and formulation\-level counts\. These quantities are not one\-to\-one: one paper can produce many evidence tuples, several tuples can instantiate one motif, and several accepted variants can later be consolidated into one project family\.

Table 21:Full SGHA pipeline counts across the five main domains\. Motif hits are deterministic graph\-pattern matches\. Novelty survivors are candidate gaps retained after corpus screening\. Reviewed gaps are sent to verification agents\. Verified gaps pass the hard verification gate\. Accepted variants are ambition\-expanded formulations that pass the independent critic\.DomainSelectedParsedExtractedTuplesNodesEdgesMotifhitsNoveltysurv\.ReviewedVerifiedDirectAcceptedvariantsFamiliesFormalBandits250221221155492149776173301366433In\-context learning25019319315598608908314721161212444Reasoning / test\-time computation250137137113961166535105291277111Offline reinforcement learning250248245217610600120451938822111Uncertainty estimation2502502482206110971205521817151212966Total12501049104486344563549494836105643939191515

### C\.2 Selected Per\-Domain Formulation Scores

Table[22](https://arxiv.org/html/2608.17501#A3.T22)reports per\-domain formulation scores for the main methods\. The main text reports method\-level averages\. Here, we show the overall score and three criteria that are especially relevant to SGHA: source grounding, formalizability, and ambiguity hygiene\.

Table 22:Selected per\-domain formulation\-quality scores\. Scores are averaged over retained candidates and LLM judges\.DomainMethodOverallSourcegroundingFormalizabilityAmbiguityhygieneBanditsSGHA5\.937\.335\.207\.80BanditsAI\-Scientist\-v2 \+ Qwen4\.535\.204\.002\.93BanditsAI\-Scientist\-v2 \+ Claude Opus5\.872\.805\.073\.47BanditsMOOSE\-Star1\.933\.471\.271\.40In\-context learningSGHA5\.807\.255\.357\.55In\-context learningAI\-Scientist\-v2 \+ Qwen4\.354\.603\.402\.60In\-context learningAI\-Scientist\-v2 \+ Claude Opus5\.803\.005\.303\.70In\-context learningMOOSE\-Star1\.954\.001\.551\.50Reasoning / test\-time computationSGHA7\.207\.806\.407\.80Reasoning / test\-time computationAI\-Scientist\-v2 \+ Qwen4\.605\.403\.402\.40Reasoning / test\-time computationAI\-Scientist\-v2 \+ Claude Opus6\.003\.405\.403\.40Reasoning / test\-time computationMOOSE\-Star2\.004\.001\.401\.40Offline reinforcement learningSGHA6\.407\.606\.007\.80Offline reinforcement learningAI\-Scientist\-v2 \+ Qwen4\.805\.603\.803\.00Offline reinforcement learningAI\-Scientist\-v2 \+ Claude Opus5\.402\.804\.803\.00Offline reinforcement learningMOOSE\-Star2\.404\.402\.001\.60Uncertainty estimationSGHA5\.877\.505\.537\.50Uncertainty estimationAI\-Scientist\-v2 \+ Qwen4\.635\.273\.732\.93Uncertainty estimationAI\-Scientist\-v2 \+ Claude Opus5\.903\.005\.273\.67Uncertainty estimationMOOSE\-Star2\.003\.571\.531\.50

The per\-domain scores are consistent with the aggregate results in the main text: SGHA is strongest on source grounding and ambiguity hygiene across domains, while the strongest ideation baseline is often competitive on overall formulation quality\.