Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews

# Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews
Source: [https://arxiv.org/html/2608.14551](https://arxiv.org/html/2608.14551)
Arya Rahgozar1,2Pouria Mortezaagha1,2 1University of Ottawa, Ottawa, Canada 2Ottawa Hospital Research Institute, Ottawa, Canada arahgoza@uottawa\.capmort101@uottawa\.ca

###### Abstract

Background\.Large language models \(LLMs\) are increasingly used for title–abstract screening in systematic reviews, but their decisions lack calibrated uncertainty, leaving reviewers without a principled way to route ambiguous records to a human\. We show that an auxiliary BERT\+GCN classifier can supply a structured uncertainty signal that meaningfully improves LLM screening efficiency, and we identify the prompt\-delivery strategy that maximises the benefit\-to\-cost ratio\.

Methods\.We evaluate five LLM prompt\-delivery conditions on eight drug\-class datasets from the Cohen \(2006\) benchmark collection using 3 seeds×\\times5\-fold stratified cross\-validation \(600 fold\-level results\)\. A hybrid BERT\+GCN model is trained per fold and classifies each test paper as INCLUDE, EXCLUDE, or MAYBE via two spectral tests \(algebraic radical and categorical paradox\)\. The five conditions are: \(i\) baseline \(title\+abstract only\), \(ii\) full spectral context \(label \+ scores on every paper\), \(iii\) decision\-only \(categorical label on every paper\), \(iv\)*MAYBE\-only*\(spectral context appended only when the BERT\+GCN flags the paper as uncertain\), and \(v\) two\-pass \(spectral context appended only when the LLM’s first response contains hedging language\)\. A pre\-registered cross\-model pilot against an earlier\-generation model \(gpt\-4\.1\-minion three datasets\) tests whether the spectral benefit transfers across LLM generations\.

Results\.Three findings shape practical guidance\.\(i\)Full\-context delivery yields statistically significant improvements in F1 \(Δ=\+0\.011\\Delta=\+0\.011, paired Wilcoxonp=0\.008p=0\.008\) and work\-saved\-at\-95%\-recall \(Δ=\+0\.050\\Delta=\+0\.050,p=0\.039p=0\.039\) over the baseline at a1\.28×1\.28\\timestoken\-cost premium, while preserving recall \(p≥0\.12p\\geq 0\.12for all conditions\)\.\(ii\)*MAYBE\-only routing is the Pareto\-optimal configuration*: it achieves the highest mean recall \(0\.920\.92, best of all conditions\) and the highest mean AUC\-ROC \(0\.540\.54\) at only1\.05×1\.05\\timesbaseline cost — one sixth of the API overhead of blanket full\-context delivery\.\(iii\)The two\-pass design escalates on22\.2%±8\.8%22\.2\\%\\pm 8\.8\\%of records yet*never*revises its initial decision \(0% flip rate across all datasets and folds\), providing decisive empirical evidence that current instruction\-tuned LLMs cannot self\-triage — a clean negative result that rules out an entire class of pipeline designs\. The cross\-model pilot shows an identical\+0\.8%\+0\.8\\%recall uplift for both LLM generations, strengthening the interpretation of the BERT\+GCN signal as a stable auxiliary classifier rather than a capability\-dependent crutch\. As a methodological by\-product, a per\-paper ablation across20,79620\{,\}796observations shows that the dual paradox test reduces empirically to a one\-line logit\-gap criterion on this benchmark, simplifying any practitioner reimplementation\.

Conclusions\.An auxiliary BERT\+GCN uncertainty classifier delivers measurable F1 and WSS@95 gains when its output is provided to an LLM screener, and*targeted*MAYBE\-only routing captures these benefits at near\-baseline API cost\. We additionally contribute a reproducible 600\-run benchmark, a clean negative result that ends the case for two\-pass LLM self\-triage, and a publicly\-released pipeline that replays in under one hour from cached LLM responses\. All code, prompts, datasets, and raw fold\-level results are publicly available\.

Keywords:systematic review, literature screening, large language models, uncertainty quantification, BERT\+GCN, work saved over sampling, targeted prompting, cost\-efficient screening, TRIPOD\-LLM, reproducible AI

## 1Introduction

Systematic reviews are the gold standard for evidence synthesis in healthcare, yet the screening phase—deciding which papers merit full\-text review based on title and abstract alone—remains a major bottleneck\. A single review may require screening thousands to tens of thousands of records, consuming hundreds of hours of expert time\(Higgins et al\.,[2019](https://arxiv.org/html/2608.14551#bib.bib8)\)\.

Large language models \(LLMs\) have recently shown promise as automated screeners\(Guo et al\.,[2024](https://arxiv.org/html/2608.14551#bib.bib7); Wang et al\.,[2024](https://arxiv.org/html/2608.14551#bib.bib23)\), achieving competitive recall on benchmark datasets\. However, LLMs produce decisions without calibrated confidence: a model that says “INCLUDE” with hedging language \(“this might be relevant”\) is indistinguishable—in terms of downstream action—from one that is certain\. This absence of structured uncertainty creates two practical problems: \(1\) reviewers cannot easily identify which LLM decisions deserve human verification, and \(2\) the LLM has no mechanism to request additional evidence when its own assessment is ambiguous\.

We address these limitations by providing the LLM with*structured uncertainty context*derived from a BERT\+GCN spectral analysis pipeline implemented specifically for this study\. The pipeline pre\-classifies each paper into INCLUDE, EXCLUDE, or MAYBE, where the MAYBE label signals classifier uncertainty detected via two complementary tests:

1. 1\.Algebraic radical test:The gap between the model’s include and exclude logits falls below a threshold \(\|P​\(Include\)−P​\(Exclude\)\|<ϵ\|P\(\\text\{Include\}\)\-P\(\\text\{Exclude\}\)\|<\\epsilon\), indicating the classifier itself is uncertain\.
2. 2\.Categorical paradox test:The cosine similarity between the BERT \(content\) and GCN \(relational\) embeddings is low, indicating that a paper’s textual content and its citation/similarity neighbourhood disagree about its relevance\.

We investigate five experimental conditions that vary the amount and type of spectral context provided to the LLM:

1. 1\.Baseline:Title and abstract only—no spectral information\.
2. 2\.Full spectral:Title, abstract, and all spectral scores \(confidence gap, model confidence, paradox type\)\.
3. 3\.Decision only:Title, abstract, and the spectral label \(INCLUDE/EXCLUDE/MAYBE\) without numerical scores\.
4. 4\.MAYBE only:Full spectral context for papers classified as MAYBE; baseline prompt for all others\.
5. 5\.Two\-pass:Baseline prompt first; if the LLM expresses uncertainty \(low confidence or hedging language\), a second pass provides spectral context\.

These five conditions create a systematic ablation along three dimensions:*information content*\(no context→\\tolabel→\\tofull scores\),*selectivity*\(all papers vs\. MAYBE only\), and*timing*\(proactive vs\. reactive two\-pass\)\.

#### Contributions\.

This paper provides a practical, reproducible evaluation of*how*to use an auxiliary BERT\+GCN uncertainty classifier as a tool for LLM\-assisted screening, with five contributions:

- •Pareto\-optimal delivery strategy\.We identify*MAYBE\-only routing*as the recommended configuration for cost\-sensitive deployments: it achieves the highest mean recall \(0\.920\.92\) and AUC\-ROC \(0\.540\.54\) of any condition at only1\.05×1\.05\\timesbaseline API cost, capturing the spectral benefit at one sixth of the overhead of full\-context delivery\.
- •Quantified benefit of full spectral context\.We report statistically significant improvements in F1 \(\+0\.011\+0\.011, paired Wilcoxonp=0\.008p=0\.008\) and WSS@95 \(\+0\.050\+0\.050,p=0\.039p=0\.039\) when full spectral scores are delivered to the LLM, at a well\-characterised1\.28×1\.28\\timescost premium\.
- •Decisive evidence against two\-pass LLM self\-triage\.A two\-pass design escalates on22%22\\%of papers yet flips0\.0%0\.0\\%of decisions across all datasets, seeds, and folds\. This clean null rules out a class of architectures and saves practitioners the cost of re\-discovering the limitation\.
- •Cross\-generation portability\.A pilot comparison finds an identical\+0\.8%\+0\.8\\%recall uplift for both an earlier\-generation and a current LLM, supporting the BERT\+GCN signal as a stable, model\-agnostic auxiliary classifier whose benefit does*not*disappear as base\-LLM capability grows\.
- •Reproducible open implementation\.We release the BERT\+GCN pipeline, all prompt templates, an experiment runner with disk\-backed LLM response caching and fold\-level resume, a deterministic analyser that emits the exact tables in this paper, and a130130\-test suite under a permissive licence\. The complete 600\-run experiment can be replayed in under one hour from the cached responses\.

## 2Related Work

### 2\.1LLMs for Systematic Review Screening

Recent work has explored GPT\-4, Claude, and open\-source LLMs for title–abstract screening\(Guo et al\.,[2024](https://arxiv.org/html/2608.14551#bib.bib7); Wang et al\.,[2024](https://arxiv.org/html/2608.14551#bib.bib23); Alshami et al\.,[2023](https://arxiv.org/html/2608.14551#bib.bib1); Tran et al\.,[2024](https://arxiv.org/html/2608.14551#bib.bib21)\)\. These studies demonstrate competitive recall but frequently report low specificity and lack principled uncertainty handling\.Khraisha et al\. \([2024](https://arxiv.org/html/2608.14551#bib.bib10)\)provide a systematic assessment of LLM screening, finding that prompt engineering substantially affects performance, andTran et al\. \([2024](https://arxiv.org/html/2608.14551#bib.bib21)\)report sensitivity/specificity ranges that overlap with our baseline \(∼\\sim0\.91 recall on Cohen\-style benchmarks\)\. Active\-learning screeners such as ASReview\(van de Schoot et al\.,[2021](https://arxiv.org/html/2608.14551#bib.bib22)\)and the extended evaluation inFerdinands et al\. \([2023](https://arxiv.org/html/2608.14551#bib.bib4)\)provide a complementary, non\-LLM baseline\. Our work differs in providing the LLM with an*external*uncertainty signal from a purpose\-built classifier and in identifying the cost\-optimal way to deliver that signal\.

### 2\.2Uncertainty Quantification in LLMs

Calibration of LLM confidence has been studied through verbalized uncertainty\(Xiong et al\.,[2024](https://arxiv.org/html/2608.14551#bib.bib25)\), token\-level probabilities\(Kadavath et al\.,[2022](https://arxiv.org/html/2608.14551#bib.bib9)\), and ensemble approaches\(Lakshminarayanan et al\.,[2017](https://arxiv.org/html/2608.14551#bib.bib12)\)\. Most methods require white\-box access to model internals\. We take a complementary approach: rather than estimating the LLM’s own uncertainty, we provide it with uncertainty estimates from a separate, purpose\-built classification model\.

### 2\.3Tool\-Augmented LLMs

Tool\-augmented LLMs\(Schick et al\.,[2023](https://arxiv.org/html/2608.14551#bib.bib19); Qin et al\.,[2023](https://arxiv.org/html/2608.14551#bib.bib18)\)leverage external tools to compensate for limitations in the language model itself\. Retrieval\-augmented generation \(RAG\) provides factual grounding\(Lewis et al\.,[2020](https://arxiv.org/html/2608.14551#bib.bib14)\), while chain\-of\-thought prompting\(Wei et al\.,[2022](https://arxiv.org/html/2608.14551#bib.bib24)\)encourages structured reasoning\. Our spectral context can be viewed as a specialised tool that provides the LLM with a structured “second opinion” from a domain\-specific model\.

### 2\.4Spectral Methods for Text Classification

Spectral graph analysis has been applied to text classification through graph neural networks\(Kipf and Welling,[2017](https://arxiv.org/html/2608.14551#bib.bib11); Yao et al\.,[2019](https://arxiv.org/html/2608.14551#bib.bib26)\), including BERT\+GCN hybrid architectures\(Lin et al\.,[2021](https://arxiv.org/html/2608.14551#bib.bib15)\)\. The use of spectral gaps to detect classification ambiguity draws on algebraic graph theory\(Chung,[1997](https://arxiv.org/html/2608.14551#bib.bib2)\)\. Our contribution in this paper is not a new spectral method but an empirical evaluation of how to*deploy*such a classifier as an auxiliary signal for an LLM screener, with an emphasis on benefit\-to\-cost trade\-offs of alternative prompt\-delivery strategies\.

## 3Background: Spectral Paradox Detection

This section summarises the BERT\+GCN spectral analysis pipeline that generates the uncertainty signals used as LLM context\. The complete implementation, including training scripts, configuration files, and the classifier ablation used in[Section˜6\.6](https://arxiv.org/html/2608.14551#S6.SS6), is released with this paper \(seeAvailability of data and materials\)\.

### 3\.1BERT\+GCN Architecture

Each paper is represented by a BERT embedding of its title and abstract \(using PubMedBERT;Gu et al\.,[2021](https://arxiv.org/html/2608.14551#bib.bib6)\) and a GCN embedding learned over akk\-nearest\-neighbour similarity graph with optional citation edges\. The architecture uses relational graph convolution \(RGCNConv;Schlichtkrull et al\.,[2018](https://arxiv.org/html/2608.14551#bib.bib20)\) to learn separate weight matrices for similarity and citation edge types\. The model is trained semi\-supervised: MAYBE and unlabelled papers participate in message passing but are excluded from the loss\.

### 3\.2Algebraic Radical Test

Let𝐳∈ℝ2\\mathbf\{z\}\\in\\mathbb\{R\}^\{2\}be the output logits for a paper, withp=softmax​\(𝐳\)p=\\text\{softmax\}\(\\mathbf\{z\}\)\. The*trace ratio*is defined asτ=\|pinclude−pexclude\|\\tau=\|p\_\{\\text\{include\}\}\-p\_\{\\text\{exclude\}\}\|\. A paper is flagged as algebraically ambiguous whenτ<ϵ\\tau<\\epsilon\(defaultϵ=0\.15\\epsilon=0\.15\), meaning the classifier cannot confidently distinguish between include and exclude\.

### 3\.3Categorical Paradox Test

Let𝐛\\mathbf\{b\}and𝐠\\mathbf\{g\}be the BERT and GCN embeddings, respectively\. The categorical paradox is detected when the cosine similaritycos⁡\(𝐛,𝐠\)<δ\\cos\(\\mathbf\{b\},\\mathbf\{g\}\)<\\delta\(defaultδ=0\.0\\delta=0\.0\), indicating that the paper’s textual content and its relational context \(neighbourhood in the similarity/citation graph\) encode contradictory signals\.

### 3\.4Three\-State Classification

The spectral engine assigns each paper to one of three states:

- •INCLUDE: High confidence in relevance \(no paradox detected\)\.
- •EXCLUDE: High confidence in irrelevance \(no paradox detected\)\.
- •MAYBE: One or both paradox tests triggered—the paper requires additional scrutiny\.

## 4Method

[Figure˜1](https://arxiv.org/html/2608.14551#S4.F1)illustrates the overall pipeline\. A BERT\+GCN model is trained per \(dataset, seed, fold\) split, producing spectral classifications\. These are then injected into LLM prompts according to one of five conditions\.

![Refer to caption](https://arxiv.org/html/2608.14551v1/x1.png)Figure 1:Pipeline overview\. Papers are encoded via frozen PubMedBERT embeddings and a 2\-layer RGCNConv over a kNN similarity graph\. The spectral engine applies two complementary uncertainty tests \(algebraic radical and categorical paradox\) to classify each paper as INCLUDE, EXCLUDE, or MAYBE\. A condition\-dependent delivery strategy determines which spectral information, if any, is injected into the LLM prompt\. The five conditions vary along three dimensions: information content \(none, label only, full scores\), selectivity \(all papers vs\. MAYBE only\), and timing \(proactive vs\. reactive two\-pass\)\. The LLM screener produces a binary decision with confidence and reasoning\.### 4\.1Experimental Conditions

We evaluate five conditions that vary the spectral context provided to the LLM screener\.[Table˜1](https://arxiv.org/html/2608.14551#S4.T1)summarises the conditions\.

Table 1:Summary of the five experimental conditions\. “Spectral context” indicates what information from the BERT\+GCN pipeline is included in the LLM prompt\. “Papers affected” indicates which papers receive spectral context\.
### 4\.2Prompt Design

All conditions share a common system prompt instructing the LLM to act as a systematic review screening assistant\. The system prompt specifies the response format \(DECISION, CONFIDENCE, REASON\) and includes dataset\-specific screening criteria derived from each benchmark’s metadata \(e\.g\., “Drug class review: ACEInhibitors \(2544 records, 41 included\)” for the Cohen ACE Inhibitors dataset\)\. This ensures the LLM receives domain\-appropriate context rather than generic instructions\.

Thebaselineuser prompt contains only the paper’s title and abstract\. Thefull spectralprompt appends a structured block containing four fields: the spectral model’s decision \(INCLUDE/EXCLUDE/MAYBE\), the confidence gap \(τ\\tau, with an explicit note that 0 = uncertain and 1 = certain\), the model’s overall confidence score, and the paradox type detected \(algebraic radical, categorical paradox, or none\)\. An explicit disclaimer states that the LLM “may agree or disagree with the model’s assessment based on your reading of the paper\.”

Thedecision onlyprompt provides only the spectral label \(“A separate BERT\+GCN classification model suggests this paper should be: \[DECISION\]”\) without numerical scores\. This ablation isolates the contribution of the categorical label from the continuous uncertainty estimates\.

TheMAYBE onlycondition selectively applies the full spectral prompt to papers classified as MAYBE by the spectral engine, while giving all other papers the baseline prompt\. This tests whether targeted context delivery \(only for ambiguous papers\) outperforms blanket augmentation\.

### 4\.3Two\-Pass Uncertainty Detection

The two\-pass condition implements a form of*LLM self\-triage*\. In pass 1, the LLM receives a baseline prompt and produces a decision with a confidence level\. The system then analyses the response for uncertainty signals using two detection mechanisms:

- •Explicit low confidence:The LLM reports CONFIDENCE as LOW or MEDIUM \(rather than HIGH\)\.
- •Hedging language:The reasoning text contains hedging patterns detected via regex matching:might,unclear,borderline,uncertain,not sure,ambiguous,possibly,could be,difficult to determine,hard to say\.

If*either*signal is detected, pass 2 re\-prompts the LLM with the paper’s title, abstract, the previous confidence level, and full spectral context, explicitly asking the model to “reconsider \[its\] decision\.” High\-confidence pass\-1 decisions with no hedging language are accepted without a second call, reducing cost\.

### 4\.4Response Parsing

The LLM’s free\-text response is parsed line by line for three structured fields:DECISION:\(mapped to INCLUDE if the token “INCLUDE” appears, EXCLUDE otherwise\),CONFIDENCE:\(HIGH, MEDIUM, or LOW\), andREASON:\(free\-text explanation\)\. The parser falls back to EXCLUDE with MEDIUM confidence if the response is unstructured, ensuring robustness against formatting variation\.

### 4\.5Metrics

We evaluate each condition using five primary metrics:

- •Recall: Fraction of true INCLUDEs correctly identified \(on the binary INCLUDE/EXCLUDE subset, excluding MAYBE predictions\)\.
- •AUC\-ROC: Area under the receiver operating characteristic curve, using the LLM’s confidence as the score \(HIGH→\\to1\.0, MEDIUM→\\to0\.5, LOW→\\to0\.1\)\.
- •F1: Harmonic mean of precision and recall on the binary INCLUDE/EXCLUDE subset\.
- •WSS@95: Work Saved over Sampling at 95% recall, defined followingCohen et al\. \([2006](https://arxiv.org/html/2608.14551#bib.bib3)\)asWSS​@​R=\(𝑇𝑁\+𝐹𝑁\)/N−\(1−R\)\\mathrm\{WSS@\}R=\(\\mathit\{TN\}\+\\mathit\{FN\}\)/N\-\(1\-R\)— the fraction of records the reviewer can skip \(relative to random screening\) while still recovering95%95\\%of true includes\. We compute it by ranking papers by model confidence \(INCLUDE→\\toMAYBE→\\toEXCLUDE\) and simulating top\-down screening until95%95\\%recall is reached; the formula used in code \(lr\_spectral/training/evaluation\.py\) is equivalent:WSS​@​95=\(1−freviewed\)−0\.05\\mathrm\{WSS@95\}=\(1\-f\_\{\\text\{reviewed\}\}\)\-0\.05, floored at zero\.
- •Cost: Total token usage \(input \+ output\) as a proxy for API cost\.

Additionally, we report three interaction metrics that characterise how the LLM engages with spectral context:

- •Override rate: Fraction of papers where the LLM’s decision disagrees with the spectral label\.
- •Second\-pass rate\(two\-pass only\): Fraction of papers triggering a second LLM call due to detected uncertainty\.
- •Second\-pass flip rate\(two\-pass only\): Fraction of escalated papers whose decision changes on the second pass\.

## 5Experimental Setup

### 5\.1Datasets

We evaluate on 8 benchmark datasets from the Cohen 2006 drug class systematic review collection\(Cohen et al\.,[2006](https://arxiv.org/html/2608.14551#bib.bib3)\), sourced via the OpenAlex API\(Priem et al\.,[2022](https://arxiv.org/html/2608.14551#bib.bib17)\)\.[Table˜2](https://arxiv.org/html/2608.14551#S5.T2)summarises dataset characteristics\.

Table 2:Benchmark dataset characteristics\. Inclusion rate is the fraction of papers labelled as INCLUDE in the original review\. All datasets are sourced from the ASReview Synergy Dataset collection via OpenAlex\.Dataset sizes range from 310 \(Antihistamines\) to 2,544 \(ACE Inhibitors\) records, with inclusion rates between 1\.6% \(ACE Inhibitors\) and 27\.0% \(Oral Hypoglycemics\)\. This diversity tests robustness across class imbalance levels and dataset scale\.

### 5\.2Cross\-Validation Protocol

For each dataset, we train the BERT\+GCN spectral model using 5\-fold stratified cross\-validation, repeated across 3 random seeds \(42, 123, 456\)\. This yields 15 runs per dataset per condition, or8×5×3×5=6008\\times 5\\times 3\\times 5=600total experimental runs\. BERT embeddings are pre\-computed once per dataset using PubMedBERT and cached to disk, so that only the GCN training varies across seeds and folds\.

Within each fold, the spectral engine classifies all test papers into INCLUDE, EXCLUDE, or MAYBE\. These spectral results are then used as context for all five LLM screening conditions on the same test set, ensuring a controlled comparison\.

### 5\.3Models

- •BERT encoder:microsoft/BiomedNLP\-PubMedBERT\-base\-uncased\-abstract\-fulltext\(Gu et al\.,[2021](https://arxiv.org/html/2608.14551#bib.bib6)\), frozen \(embeddings pre\-computed; 768\-dimensional\)\.
- •GCN:2\-layer RGCNConv\(Schlichtkrull et al\.,[2018](https://arxiv.org/html/2608.14551#bib.bib20)\)with 256 hidden dimensions, dropout 0\.3, trained for up to 100 epochs with early stopping \(patience 10\)\. The similarity graph usesk=10k=10nearest neighbours with cosine threshold 0\.5\.
- •LLMs:OpenAIgpt\-5\.4\-mini\(snapshot accessed January–April 2026\) is used as the primary screener across the full matrix; an earlier\-generationgpt\-4\.1\-mini\(snapshot accessed April 2026\) is used only for the cross\-model pilot \([Section˜6\.8](https://arxiv.org/html/2608.14551#S6.SS8)\)\. Both models run at temperature 0\.0 with at most 150 completion tokens\. Temperature 0 ensures deterministic outputs for reproducibility, and all responses are cached by a hash of \(paper id, mode, spectral decision, model name\) so that any model deprecation will not invalidate the released results\.

### 5\.4Spectral Thresholds

Default thresholds from the spectral pipeline configuration:ϵ=0\.15\\epsilon=0\.15\(trace ratio boundary for the algebraic radical test\) andδ=0\.0\\delta=0\.0\(cosine similarity threshold for the categorical paradox test\)\. These are held constant across all datasets; per\-dataset tuning is left for future work\.

### 5\.5Reporting Standards

This study evaluates an automated screening tool against gold\-standard inclusion labels on retrospective benchmark data; it is not itself a systematic review\. Where applicable, we follow PRISMA 2020\(Page et al\.,[2021](https://arxiv.org/html/2608.14551#bib.bib16)\)reporting practice for the description of the underlying systematic\-review benchmarks \(number of records, included counts, inclusion rates;[Table˜2](https://arxiv.org/html/2608.14551#S5.T2)\), and we adopt the relevant items from the TRIPOD\-LLM extension\(Gallifant et al\.,[2024](https://arxiv.org/html/2608.14551#bib.bib5)\)for reporting LLM\-based prediction studies \(model identifiers and snapshot dates, deterministic decoding configuration, prompt templates in[Appendix˜A](https://arxiv.org/html/2608.14551#A1), evaluation protocol, cost and resource use\)\. The pre\-registered analysis plan, the source code that implements each of these items, and the cached LLM responses used to compute every reported metric are all released with this paper \(seeAvailability of data and materials\)\.

### 5\.6Sample Size and Power

The eight\-dataset corpus follows the established Cohen \(2006\) benchmark\(Cohen et al\.,[2006](https://arxiv.org/html/2608.14551#bib.bib3)\): pooling additional reviews would introduce a different evaluation distribution and break direct comparability with the prior literature\. Atn=8n=8paired observations, the two\-sided paired Wilcoxon signed\-rank test has approximately80%80\\%power to detect a paired effect of standardised sized≈1\.05d\\approx 1\.05atα=0\.05\\alpha=0\.05\(Lehmann,[1975](https://arxiv.org/html/2608.14551#bib.bib13)\); this is consistent with the magnitudes we report as significant \(F1:p=0\.008p=0\.008, WSS@95:p=0\.039p=0\.039\) and conservative for the larger uplifts that drive practical recommendations\. The classifier\-ablation analysis \([Section˜6\.6](https://arxiv.org/html/2608.14551#S6.SS6)\) uses20,79620\{,\}796per\-paper observations and is therefore not power\-limited\.

### 5\.7Screening Criteria

Each dataset receives a system prompt with domain\-specific screening criteria derived from the benchmark metadata\. For example, the ACE Inhibitors dataset prompt states: “Screening criteria: Drug class review: ACEInhibitors \(2544 records, 41 included\)\.” This ensures the LLM has task\-appropriate context without introducing hand\-crafted criteria that could vary between datasets\.

### 5\.8Infrastructure and Reproducibility

The experiment runner implements three resilience mechanisms that together turn the full 600\-run study into a one\-hour replay for any reviewer or downstream user:

1. 1\.Disk\-backed LLM response cache:Each API response is cached as a JSON blob keyed by SHA\-256 of \(paper id, mode, spectral decision, model name\)\. Identical prompt–context combinations never repeat, and the released cache covers all 600 main\-matrix runs plus the 75 pilot runs — so the entire paper can be regenerated without an OpenAI key\.
2. 2\.Incremental CSV output:Results are appended to the output file after each fold, rather than batched at the end, enabling crash\-safe long runs and immediate partial inspection\.
3. 3\.Fold\-level resume:On startup, the runner scans existing results and skips \(dataset, seed, fold\) tuples that already have rows for all requested conditions, so re\-running after a partial completion is idempotent\.

The implementation is covered by a130130\-test suite \(unit and integration\), and the analysis script \(scripts/analyze\_agentic\_results\.py\) is fully deterministic and emits the LaTeX fragments used to populate every numerical table in this paper\. Re\-running the analyser against the released CSV reproduces all values reported here to within their last printed decimal\.

## 6Results

Results below are based on 600 main\-matrix runs withgpt\-5\.4\-mini\(8 datasets×\\times5 modes×\\times3 seeds×\\times5 folds\), plus 75 cross\-model pilot runs withgpt\-4\.1\-minion three of the datasets\.

### 6\.1Main Results

[Table˜3](https://arxiv.org/html/2608.14551#S6.T3)presents the aggregate results across the 8 datasets \(mean±\\pmstandard deviation over dataset\-level averages, each computed from 3 seeds×\\times5 folds = 15 runs\)\.

Table 3:Aggregate results across 8 datasets \(mean±\\pmstd over dataset\-level averages, each from 3 seeds×\\times5 folds = 15 runs\)\. Best value in each metric column isbolded\. Superscripts report two\-sided paired Wilcoxon signed\-rankpp\-values across datasets \(n=8n=8\) against baseline:∗p<0\.05p<0\.05,†p<0\.10p<0\.10; absence = not significant\.![Refer to caption](https://arxiv.org/html/2608.14551v1/x2.png)Figure 2:Aggregate recall, F1, and WSS@95 across the 8 Cohen datasets \(mean±\\pmstd over dataset\-level averages\)\. Significance annotations show paired Wilcoxonpp\-values vs\. baseline \(n=8n=8datasets\)\. Only the full\-spectral F1 and WSS@95 improvements are statistically significant\.Three observations frame these results \([Figure˜2](https://arxiv.org/html/2608.14551#S6.F2)\)\. First,full\-context delivery yields statistically significant improvementson the two metrics that matter most for screening efficiency: F1 \(Δ=\+0\.011\\Delta=\+0\.011,p=0\.008p=0\.008\) and WSS@95 \(Δ=\+0\.050\\Delta=\+0\.050,p=0\.039p=0\.039\)\. In a screening workflow processing tens of thousands of records, a five\-percentage\-point reduction in the screening burden at fixed 95% recall is operationally meaningful\. Decision\-only and two\-pass also produce significant F1 gains \(p=0\.008p=0\.008\)\. Second,recall is preserved across all conditions\(paired Wilcoxonp≥0\.12p\\geq 0\.12\): no spectral condition trades recall for precision in either direction, addressing the primary safety concern of systematic review teams\.

Third,MAYBE\-only delivery dominates the cost–quality frontier\([Figure˜6](https://arxiv.org/html/2608.14551#S6.F6)\)\. It achieves the highest mean recall \(0\.920\.92, best of any condition\) and the highest mean AUC\-ROC \(0\.540\.54\) at only1\.05×1\.05\\timesbaseline cost — one sixth of the API overhead of full\-context delivery’s1\.28×1\.28\\timespremium\. Because paired Wilcoxon tests find no metric on which MAYBE\-only differs significantly from either baseline or full\-context delivery, MAYBE\-only is the recommended configuration for cost\-sensitive deployments\. Decision\-only marginally under\-performs baseline on AUC\-ROC \(Δ=−0\.033\\Delta=\-0\.033,p=0\.055p=0\.055, marginal atp<0\.10p<0\.10\), suggesting that the categorical label without accompanying numerical context dilutes the signal; this directly motivates either delivering the full scores \(full\-spectral\) or restricting context to flagged papers \(MAYBE\-only\)\.

### 6\.2Per\-Dataset Analysis

[Table˜4](https://arxiv.org/html/2608.14551#S6.T4)and[Figure˜3](https://arxiv.org/html/2608.14551#S6.F3)break down recall for each dataset and condition, highlighting cases where spectral context helps and where it does not\.

Table 4:Per\-dataset recall / AUC\-ROC for baseline, full spectral, and two\-pass conditions \(mean over 3 seeds×\\times5 folds\)\.![Refer to caption](https://arxiv.org/html/2608.14551v1/x3.png)Figure 3:Per\-dataset mean recall across the five prompt\-delivery conditions\. Darker green indicates higher recall\. Full spectral yields its largest gains on datasets where the baseline does not already achieve perfect recall \(ADHD, Oral Hypoglycemics\), while NSAIDs \(1\.001\.00baseline\) shows no room for improvement\.Full spectral yields its largest recall improvements on ADHD \(\+0\.05\+0\.05over baseline\) and OralHypoglycemics \(\+0\.04\+0\.04\), both datasets with moderate inclusion rates where the baseline LLM does not already achieve perfect recall\. On NSAIDs—where baseline recall is already 1\.00—spectral context offers no recall improvement but does increase AUC\-ROC \(\+0\.03\+0\.03\)\. On ACEInhibitors, baseline recall is higher \(0\.98 vs\. 0\.96\), suggesting that on highly imbalanced datasets \(1\.6%1\.6\\%inclusion rate\) the spectral context may occasionally cause the LLM to override its correct initial INCLUDE decisions\.

### 6\.3Ablation: Scores vs\. Labels

Comparing*full spectral*\(scores \+ label\) to*decision only*\(label only\) isolates the contribution of numerical spectral scores\. Paired deltas across 7 datasets show that providing numerical scores alongside the categorical label yields a consistent advantage: recall \(\+0\.018±0\.034\+0\.018\\pm 0\.034\), F1 \(\+0\.004±0\.014\+0\.004\\pm 0\.014\), WSS@95 \(\+0\.063±0\.042\+0\.063\\pm 0\.042\), and AUC\-ROC \(\+0\.033±0\.029\+0\.033\\pm 0\.029\)\. Override rates are nearly identical \(55\.0% vs\. 53\.8%\), suggesting that the additional numerical context does not change*how often*the LLM disagrees with the spectral model, but rather*which papers*it overrides—and the full\-score overrides are marginally more beneficial\.

### 6\.4Override Analysis

We define an*override*as an instance where the LLM’s INCLUDE/EXCLUDE decision disagrees with the spectral model’s label\.[Table˜5](https://arxiv.org/html/2608.14551#S6.T5)reports override rates by condition\.

Table 5:LLM override behaviour by condition\. Override rate is the fraction of papers where the LLM disagrees with the spectral label\. MAYBE\-only is excluded because its spectral label for augmented papers is always MAYBE, which the LLM cannot output, making the metric non\-comparable\.Override rates are above50%50\\%in all conditions, indicating that the LLM treats spectral context as*one input among many*rather than deferring to it\. This is a desirable property: the spectral model is itself imperfect, and a screener that blindly deferred to it would inherit those errors\. Override rates vary substantially across datasets, from34\.8%34\.8\\%\(ADHD, full spectral\) to70\.8%70\.8\\%\(NSAIDs, full spectral\), and correlate with the LLM’s own confidence on the dataset — the LLM overrides more on datasets where the abstract content is sufficient for a confident decision\. This adaptive override behaviour is consistent with the auxiliary\- signal interpretation: the LLM uses spectral context as additional evidence rather than as an authoritative label, which is precisely the role we intend\.

### 6\.5Two\-Pass Analysis

The two\-pass condition provides a natural measure of LLM self\-assessed uncertainty\.

Across the 8 completed datasets,22\.2%±8\.8%22\.2\\%\\pm 8\.8\\%of papers trigger a second pass due to uncertainty detected in the LLM’s initial response\. Escalation rates vary substantially: from 6\.5% \(ADHD\) to 32\.7% \(NSAIDs\), as shown in[Table˜6](https://arxiv.org/html/2608.14551#S6.T6)and[Figure˜4](https://arxiv.org/html/2608.14551#S6.F4)\. However, the flip rate—the fraction of escalated papers whose decision changes on the second pass—is exactly0\.0%across all datasets and folds\. The LLM uniformly re\-affirms its first\-pass decision, even when presented with full spectral context\.

Table 6:Two\-pass escalation rates per dataset\. Flip rate is 0\.0% for all datasets\.The two\-pass condition’s recall \(0\.90±0\.070\.90\\pm 0\.07\) is statistically indistinguishable from the baseline \(0\.91±0\.060\.91\\pm 0\.06\), while its cost ratio \(1\.00×1\.00\\timesbaseline\) reflects the offsetting effect of saving spectral context tokens on confident first\-pass papers\. This provides a clean, decisive null on a question of immediate practical relevance: pipelines that rely on instruction\-tuned LLMs to recognise their own uncertainty and benefit from a targeted second pass do not work at the current model generation, and we therefore recommend practitioners use either MAYBE\-only routing or a human\-in\-the\-loop escalation step in place of LLM self\-triage\.

![Refer to caption](https://arxiv.org/html/2608.14551v1/x4.png)Figure 4:Two\-pass escalation rates by dataset\. Despite6\.56\.5–32\.7%32\.7\\%of papers triggering a second pass due to detected hedging language, the LLM*never*revises its initial decision \(0% flip rate across all datasets and folds\), revealing a hard anchoring limit\.
### 6\.6Classifier Ablation: What Does the MAYBE Class Actually Detect?

A natural question for any auxiliary\-classifier\-based uncertainty signal is whether the structure of the classifier matters, or whether a much simpler baseline would produce a comparable MAYBE class\. We evaluate this by computing, per test paper, four alternative MAYBE criteria from the same trained BERT\+GCN forward pass:

- •Algebraic radical\(logit gap\):\|P​\(Include\)−P​\(Exclude\)\|<ϵ\|P\(\\text\{Include\}\)\-P\(\\text\{Exclude\}\)\|<\\epsilon,ϵ=0\.15\\epsilon=0\.15\. This is one half of the dual\-test described in[Section˜3](https://arxiv.org/html/2608.14551#S3)\.
- •Categorical paradox\(cosine\):cos⁡\(𝐡BERT,𝐡GCN\)<δ\\cos\(\\mathbf\{h\}\_\{\\text\{BERT\}\},\\mathbf\{h\}\_\{\\text\{GCN\}\}\)<\\delta,δ=0\.0\\delta=0\.0\. This is the other half of the dual\-test\.
- •Dual\-test: either criterion alone fires \(the spectral engine’s default\)\.
- •Softmax entropy:−∑cpc​log⁡pc\>τ\-\\sum\_\{c\}p\_\{c\}\\log p\_\{c\}\>\\tau,τ=0\.9​ln⁡2≈0\.624\\tau=0\.9\\ln 2\\approx 0\.624\(90% of the maximum binary entropy\)\. This is a widely\-used, structure\-agnostic uncertainty baseline\.

For each criterion we report the MAYBE rate, the fraction of MAYBE papers whose ground\-truth label is INCLUDE, and the lift over the dataset’s base INCLUDE rate\.[Table˜7](https://arxiv.org/html/2608.14551#S6.T7)summarises the results across all20,79620\{,\}796per\-paper observations pooled across 8 datasets×\\times3 seeds×\\times5 folds \([Figure˜5](https://arxiv.org/html/2608.14551#S6.F5)provides a visual comparison\)\.

Table 7:Classifier ablation: four alternative MAYBE criteria computed on the same per\-fold BERT\+GCN forward passes\. “Catch rate” is the fraction of ground\-truth INCLUDE papers flagged as MAYBE; “Precision lift” isP​\(INCLUDE∣MAYBE\)/P​\(INCLUDE\)P\(\\text\{INCLUDE\}\\mid\\text\{MAYBE\}\)/P\(\\text\{INCLUDE\}\)\. Aggregated over 20,796 per\-paper observations from 8 datasets\.![Refer to caption](https://arxiv.org/html/2608.14551v1/x5.png)Figure 5:Classifier ablation: \(a\) MAYBE rate \(solid bars\) and catch rate for ground\-truth INCLUDEs \(faded bars\) for each criterion; \(b\) the rate–catch tradeoff\. The cosine test never fires \(0%0\\%MAYBE rate\), making the dual\-test identical to the logit\-gap test\. Softmax entropy catches more INCLUDEs at the cost of a higher MAYBE rate\.Two observations have direct practical consequences\.

First,the dual paradox test reduces empirically to a single logit\-gap criterion on this benchmark\. The cosine similarity between the BERT text embedding and the GCN graph embedding is positive for every labelled paper in the corpus \(min 0\.05, median 0\.91, max 1\.00\), so the categorical\-paradox test does not fire at the default threshold ofδ=0\.0\\delta=0\.0\. This is a useful*simplification finding*: practitioners reimplementing the spectral engine on this benchmark can use a one\-line logit\-gap threshold \(\|PINCL−PEXCL\|<0\.15\|P\_\{\\text\{INCL\}\}\-P\_\{\\text\{EXCL\}\}\|<0\.15\) and obtain exactly the same MAYBE class as the published dual test, with no loss in downstream performance\. A percentile\-based cosine threshold \(δ\\deltaset to, e\.g\., the1010th percentile of training\-set similarities\) would allow the categorical\-paradox test to contribute additional coverage; this adaptive variant is a natural follow\-up study\.

Second,softmax entropy is a viable alternative MAYBE criterion at a different operating point\. Atτ=0\.624\\tau=0\.624it flags42%42\\%of papers as MAYBE and catches53%53\\%of true INCLUDE papers, versus14\.5%14\.5\\%and21%21\\%for the logit\-gap criterion\. Softmax entropy has lower precision lift \(1\.26×1\.26\\timesvs\.1\.46×1\.46\\times\) but higher catch rate\. Every logit\-gap MAYBE paper is also a softmax\-entropy MAYBE paper \(strict containment\), so the two criteria sit on the same precision–recall curve at different thresholds\. Practitioners can therefore tune MAYBE coverage to their human\-reviewer capacity by sliding along this curve — a useful degree of freedom for deployment\.

These ablation findings do not change the main results \([Table˜3](https://arxiv.org/html/2608.14551#S6.T3)\), because the MAYBE\-only LLM condition uses the dual\-test MAYBE flag by construction\. They do offer practitioners two cleanly characterised alternative implementations \(single logit\-gap; tunable softmax entropy\) of the auxiliary classifier, both of which are simpler than the published dual\-test spectral engine\.

### 6\.7Cost–Benefit Analysis

[Table˜8](https://arxiv.org/html/2608.14551#S6.T8)compares the token usage across conditions, providing a cost proxy for API\-based LLM screening\.

Table 8:Token usage across conditions \(mean per paper, averaged over 8 completed datasets\)\.![Refer to caption](https://arxiv.org/html/2608.14551v1/x6.png)Figure 6:Cost–benefit frontier: each condition is plotted by its relative token cost \(xx\-axis\) and F1 improvement over baseline \(yy\-axis\)\. MAYBE\-only sits near the origin—minimal cost increase, top\-of\-table recall and AUC\-ROC—while full spectral and decision\-only pay1\.121\.12–1\.28×1\.28\\timesmore for the statistically significant F1 and WSS@95 gains documented in[Table˜3](https://arxiv.org/html/2608.14551#S6.T3)\.Full spectral incurs a 28% cost premium due to the additional spectral context appended to every prompt\.MAYBE\-only is the most cost\-efficient augmentation, adding only 5% overhead while achieving the highest mean recall and AUC\-ROC of any condition\. Two\-pass is effectively cost\-neutral \(1\.00×1\.00\\times\) because the escalated papers \(22%\) incur a second call but the non\-escalated majority saves the spectral context tokens; this cost neutrality reflects offsetting savings rather than added benefit, consistent with the 0% flip rate documented in[Section˜6\.5](https://arxiv.org/html/2608.14551#S6.SS5)\.

### 6\.8Cross\-Model Comparison

Given the spectral uplift ongpt\-5\.4\-mini\(\+0\.6%\+0\.6\\%recall on the full eight\-dataset matrix\), a natural question is whether the spectral context is*specific*to a modern LLM screener, or whether the same signal also benefits an earlier\-generation model\. To test portability across LLM generations, we repeated a pilot subset of conditions using the smallergpt\-4\.1\-minion the three datasets whose sizes allowed a full single\-seed, 5\-fold sweep within the cost budget \(Antihistamines, Triptans, UrinaryIncontinence\)\.[Table˜9](https://arxiv.org/html/2608.14551#S6.T9)and[Figure˜7](https://arxiv.org/html/2608.14551#S6.F7)report the paired results\.

Table 9:Cross\-model pilot: paired baseline vs\. full\-spectral recall for two OpenAI models on the three common pilot datasets \(Antihistamines, Triptans, Urinary Incontinence\) at a single shared seed \(seed=42\)\. Reported±\\pmvalues are*between\-dataset*standard deviations across the three pilot datasets, not within\-seed standard errors\.Δ\\Deltais the per\-model spectral uplift \(full spectral minus baseline, averaged over the three datasets\)\. Both models show an identical average uplift of\+0\.8%\+0\.8\\%recall from full spectral context, supporting the interpretation of the BERT\+GCN signal as a stable auxiliary input that transfers across LLM generations rather than a capability\-dependent booster\.![Refer to caption](https://arxiv.org/html/2608.14551v1/x7.png)Figure 7:Cross\-model pilot: mean recall \(±\\pmdataset\-level std\) for gpt\-4\.1\-mini and gpt\-5\.4\-mini across baseline, full spectral, and MAYBE\-only conditions on three shared datasets\. Both models show similar absolute performance and similar spectral uplift, consistent with the auxiliary\-signal reading\.Two recall\-uplift quantities appear in this paper and refer to different subsets, so we make the distinction explicit\. On the*full eight\-dataset main matrix*\(3 seeds, 5 folds\), thegpt\-5\.4\-minifull\-spectral uplift over baseline is\+0\.6%\+0\.6\\%recall \(0\.914→\\to0\.920\), as reported in[Table˜3](https://arxiv.org/html/2608.14551#S6.T3)\. On the*three\-dataset pilot subset*\(Antihistamines, Triptans, Urinary Incontinence\) at a single shared seed, both models show an identical average recall uplift of\+0\.8%\+0\.8\\%from full spectral context \([Table˜9](https://arxiv.org/html/2608.14551#S6.T9)\)\. The cross\-model comparison is reported on a single seed to constrain API cost, and dataset\-level standard deviations in[Table˜9](https://arxiv.org/html/2608.14551#S6.T9)are therefore between\-dataset variability on a three\-dataset sample\. With that caveat, the data are most consistent with the*model\-scale invariant*reading: paired on the same three datasets at the same seed, both models show the same\+0\.8%\+0\.8\\%uplift, well within one between\-dataset standard deviation\. The pilot therefore doesnotsupport the capability\-equalizer hypothesis: the earlier\-generation model is not meaningfully weaker than the current one on these Cohen drug\-class datasets, and adding spectral context provides a consistent uplift for both rather than a disproportionately larger uplift for the weaker model\. This is consistent with interpreting the BERT\+GCN spectral signal as an auxiliary classifier whose output is a stable external input—useful as a targeted prompt augmentation, but not as a substitute for LLM reasoning capacity\. A follow\-up full\-matrix evaluation \(3 seeds×\\times5 folds×\\times8 datasets for both models\) would be needed to test this reading with adequate statistical power\.

## 7Discussion

### 7\.1When Does Spectral Context Help?

The benefit of spectral context concentrates predictably on the datasets where it should: those where the LLM has room to improve\. Full spectral yields its largest recall improvement on ADHD \(\+0\.05\+0\.05over baseline\) and OralHypoglycemics \(\+0\.04\+0\.04\), both datasets where the spectral model’s MAYBE rate is moderate and the baseline LLM does not already achieve perfect recall\. On NSAIDs — where baseline recall is already1\.001\.00— spectral context cannot improve recall but does increase AUC\-ROC \(\+0\.03\+0\.03\), reflecting better\-calibrated confidence on borderline papers\. This dataset\-by\- dataset pattern is consistent with the auxiliary\-signal interpretation developed in[Section˜6\.8](https://arxiv.org/html/2608.14551#S6.SS8): spectral context helps where the LLM is genuinely uncertain and is harmless where the LLM is already confident\.

On ACEInhibitors — the most extreme inclusion\-rate dataset at1\.6%1\.6\\%— baseline recall \(0\.98\) marginally exceeds full\-spectral recall \(0\.95\), suggesting that adaptive thresholds tuned for severe class imbalance could improve the spectral signal further; this direction is left for future work and is the motivating use case for the percentile\-based cosine threshold proposed in[Section˜8](https://arxiv.org/html/2608.14551#S8)\.

### 7\.2Scores vs\. Labels: Information Granularity

The ablation results \([Section˜6\.3](https://arxiv.org/html/2608.14551#S6.SS3)\) show that full spectral consistently outperforms decision\-only, with a paired delta of\+0\.018\+0\.018recall and\+0\.063\+0\.063WSS@95 across the eight datasets\. This confirms that LLMs do extract additional signal from numerical uncertainty estimates beyond what the categorical label alone provides — and the WSS@95 gain in particular is operationally relevant for screening efficiency\. One mechanistic explanation is that mid\-size instruction\-tuned models have limited numerical reasoning depth: they may treat “confidence gap: 0\.03” as roughly equivalent to “confidence gap: 0\.12” despite their very different implications for the spectral model\. Larger models with stronger numerical reasoning may exploit this information channel more fully, a direction the cross\-model pilot \([Section˜6\.8](https://arxiv.org/html/2608.14551#S6.SS8)\) begins to probe\.

### 7\.3The MAYBE\-Only Efficiency Advantage

The MAYBE\-only condition achieves the highest mean recall \(0\.920\.92, best of any condition\) and the highest mean AUC\-ROC \(0\.540\.54\) at only1\.05×1\.05\\timesbaseline cost, compared to1\.28×1\.28\\timesfor full spectral\. This Pareto\-optimality has a clear mechanistic explanation: the performance benefit of spectral context is concentrated on the papers the spectral engine itself flags as ambiguous\. Providing spectral context on confident INCLUDE/EXCLUDE papers adds cost without meaningful benefit, because the LLM’s decision on those papers is already anchored by the title and abstract\. Restricting context delivery to the MAYBE subset both removes that wasted overhead and concentrates LLM attention where the auxiliary signal is most informative\.

The practical recommendation is unambiguous: a deployment using MAYBE\-only augmentation captures the spectral benefit at near\- baseline API cost and is the recommended default for cost\-sensitive screening workflows\.

### 7\.4Practical Recommendations for Systematic Review Teams

Based on our findings, we offer four practical recommendations for teams using LLM screeners with auxiliary uncertainty classifiers:

1. 1\.Use selective context delivery\.Route the auxiliary classifier’s context only to papers the classifier itself flags as uncertain \(MAYBE\-only\)\. This captures the recall benefit at roughly one sixth of the full\-context API cost\.
2. 2\.Do not rely on LLM self\-triage for re\-screening\.Two\-pass architectures in which the LLM re\-examines its own hedging responses do not yield decision revisions\. Use a human\-in\-the\-loop escalation step instead\.
3. 3\.Treat the auxiliary classifier as a signal, not a decision\.LLMs override the classifier on more than half of papers\. Pipelines that defer all classifier MAYBE papers to humans directly \(bypassing the LLM\) may be more robust than pipelines that use the LLM as a second\-opinion over the classifier\.
4. 4\.Choose your auxiliary model for cost, not ceiling\.The cross\-model pilot suggests that the uplift from auxiliary context is of similar magnitude for different LLM generations\. There is no clear reason to prefer the most expensive available LLM for the role of “spectral context consumer”\.
5. 5\.A logit\-gap threshold is sufficient to produce the MAYBE class on this benchmark\.The classifier ablation \([Section˜6\.6](https://arxiv.org/html/2608.14551#S6.SS6)\) shows that the MAYBE class used throughout this paper can be replicated exactly by a one\-line logit\-gap threshold; the BERT/GCN cosine\-paradox test does not fire at its default threshold\. Simpler implementations of the auxiliary classifier are therefore an option for deployments that do not need the full spectral pipeline\.

### 7\.5LLM Self\-Triage via Two\-Pass Screening

The two\-pass design assumes LLMs can self\-triage: recognise their own uncertainty in a first pass, then benefit from additional context on a targeted second pass\. Our results challenge this assumption\. While the escalation rate \(22% average\) suggests the hedging detector successfully identifies*some*uncertain responses, the zero flip rate reveals that the LLM does not meaningfully revise its decisions when given spectral context after expressing uncertainty\. This may reflect a limitation of instruction\-tuned models: once committed to a decision with accompanying reasoning, the model anchors on its prior output even when new evidence is introduced\.

Interestingly, the hedging\-based escalation rate does*not*correlate with the spectral model’s MAYBE rate\. The LLM’s self\-assessed uncertainty and the spectral model’s detected ambiguity appear to measure different dimensions of paper difficulty, supporting the hypothesis that hybrid human\-AI pipelines benefit from multiple independent uncertainty signals rather than relying on any single indicator\.

The escalation rate does correlate with dataset inclusion rate: NSAIDs \(10\.4% inclusion, 32\.7% escalation\) triggers far more second\-pass calls than ADHD \(2\.4% inclusion, 6\.5% escalation\)\. Datasets with more balanced classes may produce more papers whose relevance is genuinely ambiguous from the abstract alone, leading to more hedging language in the LLM’s initial response\.

### 7\.6The Spectral Context Acts as an Auxiliary Signal

The cross\-model pilot \([Section˜6\.8](https://arxiv.org/html/2608.14551#S6.SS8)\) was designed to distinguish two plausible mechanisms for the spectral uplift\. Under a*reasoning\-gap*account, the role of spectral context is to shore up weaker base reasoning, so the upliftΔ\\Deltashould be larger for the weaker model and vanish as the base model improves\. Under an*auxiliary\-signal*account, spectral labels are an independent classifier whose information is orthogonal to what the LLM can extract from the abstract alone; the uplift should then be of similar magnitude regardless of LLM capacity, because both models are integrating the same external signal into their decision\.

The data are most consistent with the auxiliary\-signal account\. Paired on the three pilot datasets at the same seed, both models receive an identical average recall uplift of\+0\.8%\+0\.8\\%from full spectral context, and the direction of the per\-dataset delta is not systematically different between the two models\. The gpt\-4\.1\-mini baseline is in fact modestly*higher*than gpt\-5\.4\-mini’s on this subset \(0\.910\.91vs\.0\.890\.89\), which is the opposite direction from what a capability\-gap account would predict\. This suggests that the spectral context functions as a stable second\-opinion signal rather than as a reasoning enhancer; it is the*classifier*doing the work, not a weakness in the LLM that the classifier is propping up\.

A practical consequence of this reading is that upgrading to a more capable LLM does not obviate the need for the auxiliary classifier, because the benefit comes from an orthogonal information channel rather than a reasoning gap\. Conversely, cheaper LLMs do not*lose*the benefit: this is reassuring for cost\-sensitive deployments considering which generation of model to use as their screener\.

### 7\.7Limitations

We note the following scope conditions on our findings:

- •LLM coverage:The main matrix evaluatesgpt\-5\.4\-miniacross 8 datasets, with a 3\-dataset single\-seed pilot forgpt\-4\.1\-mini\([Section˜6\.8](https://arxiv.org/html/2608.14551#S6.SS8)\)\. Results may differ for larger frontier models \(GPT\-4o, Claude 4 Sonnet\) or open\-source alternatives \(Llama 3, Mistral\)\. Larger models with stronger numerical reasoning may better exploit continuous spectral scores\.
- •Binary LLM output:The LLM produces INCLUDE/EXCLUDE decisions; it cannot output MAYBE, limiting the hybrid system’s ability to defer uncertain cases to human review\.
- •Fixed thresholds:The spectral thresholds \(ϵ=0\.15\\epsilon=0\.15,δ=0\.0\\delta=0\.0\) are not tuned per dataset\. Adaptive thresholds may improve the quality of the MAYBE signal\.
- •Drug class reviews:While the 8 datasets span different drug classes with varying inclusion rates and sizes, all are within the biomedical domain\. Generalisation to other systematic review domains \(social sciences, environmental science\) is untested\.
- •No human\-in\-the\-loop evaluation:We evaluate against ground\-truth labels, not against human reviewer decisions under time pressure\. The practical utility of spectral context may differ when a human is the final arbiter\.
- •Deterministic LLM:Temperature 0 produces deterministic outputs, preventing analysis of inter\-run variance in LLM decisions\. Stochastic sampling might interact differently with spectral context\.
- •Benchmark scope:All eight datasets are drug class reviews from the Cohen \(2006\) collection, covering inclusion rates from1\.6%1\.6\\%to27%27\\%and dataset sizes from310310to2,5442\{,\}544records\. This covers the main variability axes of the benchmark but not newer systematic review domains\.
- •Degenerate cosine test at the published threshold:The categorical paradox test \(cos<δ=0\.0\\cos<\\delta=0\.0\) does not fire on any paper in the 8\-dataset corpus, as the BERT/GCN cosine similarities are empirically positive throughout\. The dual\-test framing collapses to the logit\-gap test alone in practice\. A more permissive or training\-set\-adaptive threshold would be needed to make the cosine test informative\.

## 8Conclusion

We evaluated how an auxiliary BERT\+GCN uncertainty classifier should be delivered to an LLM screener to maximise the benefit\-to\-cost ratio in systematic\-review screening\. Across eight Cohen \(2006\) benchmark datasets, five prompt\-delivery conditions, and 600 fold\-level runs, together with a cross\-model pilot on three datasets, three findings have direct operational implications for systematic\-review teams\.

First,auxiliary spectral context produces statistically significant gains in screening efficiency\.Full\-context delivery yields a\+0\.050\+0\.050improvement in WSS@95 \(paired Wilcoxonp=0\.039p=0\.039\) and a\+0\.011\+0\.011improvement in F1 \(p=0\.008p=0\.008\) over the baseline, at a1\.28×1\.28\\timestoken\-cost premium\. In a workflow processing tens of thousands of records, a five\-percentage\-point reduction in the screening burden at fixed 95% recall is operationally meaningful\. Recall is preserved across all five conditions \(p≥0\.12p\\geq 0\.12\), so these efficiency gains come at no safety cost\.

Second,targeted MAYBE\-only routing is Pareto\-optimal\.Routing the auxiliary context only to papers the classifier itself flags as uncertain achieves the highest mean recall \(0\.920\.92, best of any condition\) and the highest mean AUC\-ROC \(0\.540\.54\) at only1\.05×1\.05\\timesbaseline cost — one sixth of the API overhead of blanket full\-context delivery\. For cost\-sensitive deployments, MAYBE\-only is therefore the recommended configuration\.

Third,the data provide a decisive answer to a workflow question\.Although the LLM signals uncertainty through hedging language on roughly22%22\\%of records, it never revises its decision when given the auxiliary context on a second pass \(0% flip rate across all datasets, seeds, and folds\)\. This clean null rules out two\-pass LLM self\-triage as a viable architecture for instruction\-tuned models at the current generation, and saves practitioners the cost of re\-discovering the limitation\. As a methodological by\-product, our per\-paper ablation \([Section˜6\.6](https://arxiv.org/html/2608.14551#S6.SS6)\) shows that the dual paradox test reduces empirically to a one\-line logit\-gap criterion on this benchmark, simplifying any practitioner reimplementation\.

Future work should explore: \(1\) adaptive spectral thresholds tuned per dataset, including a percentile\-based cosine threshold that would allow the categorical paradox test to fire and contribute additional MAYBE coverage; \(2\) allowing the LLM to output MAYBE directly, enabling true three\-way triage; \(3\) a multi\-seed cross\-model extension across all benchmarks to confirm the cross\-generation invariance of the spectral uplift with adequate statistical power; \(4\) evaluating larger models \(GPT\-4o, Claude 4 Sonnet\) whose stronger numerical reasoning may better leverage continuous uncertainty signals; \(5\) human\-in\-the\-loop studies measuring screener time savings in real workflows; and \(6\) extending to non\-biomedical systematic\-review domains\.

## Code and Data Availability

All source code, prompt templates, experiment configurations, and raw fold\-level results \(CSV\) generated during this study are available in the GitHub repository at[https://github\.com/rahgoar/LR\-Spectral\-BERTGCN\-Topological\-Undecidability\-in\-Clinical\-AI](https://github.com/rahgoar/LR-Spectral-BERTGCN-Topological-Undecidability-in-Clinical-AI)under a permissive open\-source licence\. The Cohen \(2006\) benchmark datasets are redistributed by the ASReview Synergy project\(van de Schoot et al\.,[2021](https://arxiv.org/html/2608.14551#bib.bib22)\)and are fetched at runtime via the OpenAlex API\(Priem et al\.,[2022](https://arxiv.org/html/2608.14551#bib.bib17)\)\. The preprocessed benchmark fetchers, LLM response cache, and analysis script \(used to generate all tables and the capability\-gap comparison\) are included in the repository\.

## Acknowledgements

We thank the ASReview team for maintaining the Synergy benchmark collection, and the OpenAlex project for providing open access to scholarly metadata\. We also thank the Ottawa Hospital Research Institute for computational infrastructure\. The authors declare no competing interests\. This work received no external funding; API costs for LLM screening calls were borne by the authors’ institutional research account\.

## References

- Alshami et al\. \[2023\]Ahmad Alshami, Moustafa Elsayed, Eslam Ali, Abdelrahman E E Eltoukhy, and Tarek Zayed\.Harnessing the power of ChatGPT for automating systematic review process: Methodology, case study, limitations, and future directions\.*Systems*, 11\(7\):351, 2023\.doi:10\.3390/systems11070351\.
- Chung \[1997\]Fan R K Chung\.*Spectral Graph Theory*\.Number 92 in CBMS Regional Conference Series in Mathematics\. American Mathematical Society, 1997\.
- Cohen et al\. \[2006\]Aaron M Cohen, William R Hersh, Kim Peterson, and Po\-Yin Yen\.Reducing workload in systematic review preparation using automated citation classification\.*Journal of the American Medical Informatics Association*, 13\(2\):206–219, 2006\.doi:10\.1197/jamia\.M1929\.
- Ferdinands et al\. \[2023\]Gerbrich Ferdinands, Raoul Schram, Jonathan de Bruin, Ayoub Bagheri, Daniel L Oberski, Lars Tummers, Jelle Jasper Teijema, and Rens van de Schoot\.Performance of active learning models for screening prioritization in systematic reviews: A simulation study into the Average Time to Discover relevant records\.*Systematic Reviews*, 12:100, 2023\.doi:10\.1186/s13643\-023\-02257\-7\.
- Gallifant et al\. \[2024\]Jack Gallifant, Majid Afshar, Saleem Ameen, Yindalon Aphinyanaphongs, Shan Chen, Giovanni Cacciamani, Dina Demner\-Fushman, Dmitriy Dligach, Roxana Daneshjou, Chrystinne Fernandes, Lasse Hyldig Hansen, Adam Landman, Lisa Soleymani Lehmann, Liam G McCoy, Timothy Miller, Amy Moreno, Nikolaj Munch, David Restrepo, Guergana Savova, Renato Umeton, Judy Wawira Gichoya, Gary S Collins, Karel G M Moons, Leo Anthony Celi, and Danielle S Bitterman\.The TRIPOD\-LLM reporting guideline for studies using large language models\.*Nature Medicine*, 2024\.doi:10\.1038/s41591\-024\-03425\-5\.arXiv:2407\.16851\.
- Gu et al\. \[2021\]Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon\.Domain\-specific language model pretraining for biomedical natural language processing\.*ACM Transactions on Computing for Healthcare*, 3\(1\):1–23, 2021\.doi:10\.1145/3458754\.
- Guo et al\. \[2024\]Eddie Guo, Mehul Gupta, Jiawen Deng, Ye\-Jean Park, Michael Paget, and Christopher Naugler\.Automated paper screening for clinical reviews using large language models: Data analysis study\.*Journal of Medical Internet Research*, 26:e48996, 2024\.doi:10\.2196/48996\.
- Higgins et al\. \[2019\]Julian P T Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J Page, and Vivian A Welch\.*Cochrane Handbook for Systematic Reviews of Interventions*\.Wiley, 2 edition, 2019\.doi:10\.1002/9781119536604\.
- Kadavath et al\. \[2022\]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield\-Dodds, Nova DasSarma, Eli Tran\-Johnson, et al\.Language models \(mostly\) know what they know\.*arXiv preprint arXiv:2207\.05221*, 2022\.
- Khraisha et al\. \[2024\]Qusai Khraisha, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield\.Can large language models replace humans in systematic reviews? evaluating GPT\-4’s efficacy in screening and extracting data from peer\-reviewed and grey literature in multiple languages\.*Research Synthesis Methods*, 15\(4\):616–626, 2024\.doi:10\.1002/jrsm\.1715\.
- Kipf and Welling \[2017\]Thomas N Kipf and Max Welling\.Semi\-supervised classification with graph convolutional networks\.In*International Conference on Learning Representations \(ICLR\)*, 2017\.
- Lakshminarayanan et al\. \[2017\]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell\.Simple and scalable predictive uncertainty estimation using deep ensembles\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2017\.
- Lehmann \[1975\]Erich L Lehmann\.*Nonparametrics: Statistical Methods Based on Ranks*\.Holden\-Day, San Francisco, 1975\.
- Lewis et al\. \[2020\]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen\-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela\.Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2020\.
- Lin et al\. \[2021\]Yuxiao Lin, Yuxian Meng, Xiaofei Sun, Qinghong Han, Kun Kuang, Jiwei Li, and Fei Wu\.BertGCN: Transductive text classification by combining GCN and BERT\.In*Findings of the Association for Computational Linguistics: ACL*, pages 1456–1462, 2021\.doi:10\.18653/v1/2021\.findings\-acl\.126\.
- Page et al\. \[2021\]Matthew J Page, Joanne E McKenzie, Patrick M Bossuyt, Isabelle Boutron, Tammy C Hoffmann, Cynthia D Mulrow, Larissa Shamseer, Jennifer M Tetzlaff, Elie A Akl, Sue E Brennan, et al\.The PRISMA 2020 statement: An updated guideline for reporting systematic reviews\.*BMJ*, 372:n71, 2021\.doi:10\.1136/bmj\.n71\.
- Priem et al\. \[2022\]Jason Priem, Heather Piwowar, and Richard Orr\.OpenAlex: A fully\-open index of scholarly works, authors, venues, institutions, and concepts\.*arXiv preprint arXiv:2205\.01833*, 2022\.
- Qin et al\. \[2023\]Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, et al\.Tool learning with foundation models\.*arXiv preprint arXiv:2304\.08354*, 2023\.
- Schick et al\. \[2023\]Timo Schick, Jane Dwivedi\-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom\.Toolformer: Language models can teach themselves to use tools\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.arXiv:2302\.04761\.
- Schlichtkrull et al\. \[2018\]Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling\.Modeling relational data with graph convolutional networks\.In*European Semantic Web Conference \(ESWC\)*, pages 593–607, 2018\.doi:10\.1007/978\-3\-319\-93417\-4\_38\.
- Tran et al\. \[2024\]Viet\-Thi Tran, Gerald Gartlehner, Sally Yaacoub, Isabelle Boutron, Lukas Schwingshackl, Julia Stadelmaier, Isolde Sommer, Farzaneh Alebouyeh, Sivem Afach, Joerg J Meerpohl, and Philippe Ravaud\.Sensitivity and specificity of using GPT\-3\.5 turbo models for title and abstract screening in systematic reviews and meta\-analyses\.*Annals of Internal Medicine*, 177\(6\):791–799, 2024\.doi:10\.7326/M23\-3389\.
- van de Schoot et al\. \[2021\]Rens van de Schoot, Jonathan de Bruin, Raoul Schram, Parisa Zahedi, Jan de Boer, Felix Weijdema, Bianca Kramer, Martijn Huijts, Maarten Hoogerwerf, Gerbrich Ferdinands, Albert Harkema, Joukje Willemsen, Yongchao Ma, Qixiang Fang, Sybren Hindriks, Lars Tummers, and Daniel L Oberski\.An open source machine learning framework for efficient and transparent systematic reviews\.*Nature Machine Intelligence*, 3:125–133, 2021\.doi:10\.1038/s42256\-020\-00287\-7\.
- Wang et al\. \[2024\]Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon\.Zero\-shot generative large language models for systematic review screening automation\.In*Advances in Information Retrieval \(ECIR 2024\)*, Lecture Notes in Computer Science\. Springer, 2024\.doi:10\.1007/978\-3\-031\-56066\-8\_32\.arXiv:2401\.06320\.
- Wei et al\. \[2022\]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H Chi, Quoc V Le, and Denny Zhou\.Chain\-of\-thought prompting elicits reasoning in large language models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- Xiong et al\. \[2024\]Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi\.Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.arXiv:2306\.13063\.
- Yao et al\. \[2019\]Liang Yao, Chengsheng Mao, and Yuan Luo\.Graph convolutional networks for text classification\.In*AAAI Conference on Artificial Intelligence*, volume 33, pages 7370–7377, 2019\.doi:10\.1609/aaai\.v33i01\.33017370\.

## Appendix APrompt Templates

### A\.1System Prompt

```
You are a systematic review screening assistant
for clinical research literature. Your task is to
decide whether a paper should be INCLUDED or EXCLUDED
from a systematic review.

Screening criteria:
[per-dataset criteria, e.g., "Drug class review:
 ACEInhibitors (2544 records, 41 included)"]

Respond in exactly this format:
DECISION: INCLUDE or EXCLUDE
CONFIDENCE: HIGH, MEDIUM, or LOW
REASON: Your 1-2 sentence explanation
```

### A\.2Baseline User Prompt \(Condition 1\)

```
Title: [paper title]

Abstract: [paper abstract]
```

### A\.3Full Spectral Context Block \(Condition 2\)

```
Title: [paper title]

Abstract: [paper abstract]

--- Spectral Analysis (from a BERT+GCN
    classification model) ---
Model decision: [INCLUDE/EXCLUDE/MAYBE]
Confidence gap: [0.000] (0=uncertain, 1=certain;
                          threshold=0.15)
Model confidence: [0.000]
Paradox detected: [algebraic_radical/
                    categorical_paradox/none]

Note: This analysis is provided as additional context.
You may agree or disagree with the model’s assessment
based on your reading of the paper.
```

### A\.4Decision Only Prompt \(Condition 3\)

```
Title: [paper title]

Abstract: [paper abstract]

--- Model Analysis ---
A separate BERT+GCN classification model suggests
this paper should be: [INCLUDE/EXCLUDE/MAYBE]
You may agree or disagree with this suggestion
based on your reading.
```

### A\.5Two\-Pass Follow\-Up Prompt \(Condition 5, Pass 2\)

```
You previously assessed this paper with [CONFIDENCE]
confidence. Here is additional analysis from a
classification model:

Title: [paper title]

Abstract: [paper abstract]

--- Spectral Analysis ---
Model decision: [INCLUDE/EXCLUDE/MAYBE]
Confidence gap: [0.000] (0=uncertain, 1=certain;
                          threshold=0.15)
Model confidence: [0.000]
Paradox detected: [algebraic_radical/
                    categorical_paradox/none]

Given this additional context, please reconsider
your decision.
DECISION: INCLUDE or EXCLUDE
CONFIDENCE: HIGH, MEDIUM, or LOW
REASON: Your 1-2 sentence explanation
```

## Appendix BHedging Language Patterns

The two\-pass uncertainty detector uses the following regex pattern to identify hedging language in LLM responses:

```
\b(might|unclear|borderline|uncertain|not sure|
ambiguous|possibly|could be|difficult to determine|
hard to say)\b
```

Both the explicit confidence check \(LOW or MEDIUM\) and the hedging pattern match trigger escalation to a second pass\.

## Appendix CFull Per\-Dataset Results

Table 10:Full per\-dataset results: mean over 3 seeds×\\times5 folds for all five conditions and primary metrics\.