A Benchmark Framework for Screening Automation in Systematic Reviews

# A Benchmark Framework for Screening Automation in Systematic Reviews
Source: [https://arxiv.org/html/2609.30298](https://arxiv.org/html/2609.30298)
Gauransh Kumar[https://orcid.org/0009-0002-0232-9427](https://orcid.org/0009-0002-0232-9427),Luciano MarchezanAffiliation:Université de Montréal,Montreal,Canadaemail:[lucianomarchp@gmail\.com](mailto:[email protected]),Guillaume GenoisAffiliation:Université de Montréal,Montreal,Canadaemail:[guillaume\.genois@umontreal\.ca](mailto:[email protected]),Kévin DelcourtAffiliation:Université de Montréal,Montreal,Canadaemail:[kevin\.delcourt@umontreal\.ca](mailto:[email protected])andEugene SyrianiAffiliation:Université de Montréal,Montreal,Canadaemail:[syriani@iro\.umontreal\.ca](mailto:[email protected])

###### Abstract\.

Systematic reviews \(SR\) are essential for evidence\-based research, but their screening phase is highly time\-consuming and labor\-intensive\. Large language models \(LLMs\) offer a promising opportunity to reduce this workload by assisting with article relevance classification\. However, existing evaluation approaches often rely on traditional metrics that may be misleading for highly imbalanced SR screening datasets\.This paper presents a benchmark dataset of45 06445\\,064labeled entries for evaluating LLM performance in SR screening across 32 curated secondary studies\. It proposes an evaluation framework that accounts for class imbalance, i\.e\., the natural prevalence of excluded articles relative to included articles in SRs\. It also introduces PromptSR, a tool designed to support prompt experimentation, experiment management, and result analysis for LLM\-based screening\. We also present a use case demonstrating the application of SRBench and PromptSR\.

###### Keywords:

systematic review screening, large language models, benchmarking and evaluation frameworks

## 1\.Introduction

Systematic reviews \(SRs\), which include systematic literature reviews\([Kitchenham, 2004](https://arxiv.org/html/2609.30298#bib.bib1)\), systematic mapping studies\([Petersen et al\., 2008](https://arxiv.org/html/2609.30298#bib.bib21)\), among others, are rigorous forms of literature review that follow structured methodologies to minimize bias and ensure reproducibility\([Zhang and Ali Babar, 2013](https://arxiv.org/html/2609.30298#bib.bib19)\)\. They are widely used in software engineering\([Kitchenham et al\., 2009](https://arxiv.org/html/2609.30298#bib.bib10)\)to synthesize existing research and identify gaps\. However, conducting an SR remains a time\-consuming and labor\-intensive process\([Borah et al\., 2017](https://arxiv.org/html/2609.30298#bib.bib2)\)\. This is because the process involves multiple steps, including defining research questions, searching for relevant studies, screening studies based on predefined inclusion and exclusion criteria, extracting data from selected studies, and synthesizing the results\.

Advances in natural language processing and machine learning, particularly through Large Language Models \(LLMs\), have shown promising results across a wide range of SR tasks\([Syriani et al\., 2024](https://arxiv.org/html/2609.30298#bib.bib12);[Wang et al\., 2023](https://arxiv.org/html/2609.30298#bib.bib13)\)\. Researchers have demonstrated strong performance in text classification, summarization, and question\-answering, often outperforming traditional machine learning approaches\. Given these capabilities, applying LLMs to automate key components of SRs presents a promising opportunity to improve both the efficiency and consistency of the review process\.

Validating LLM performance in the context of SRs, however, requires appropriate benchmarks, including datasets and evaluation protocols that reflect the specific challenges of SR screening\. Toward this goal, Huotalaet al\.\([Huotala et al\., 2025a](https://arxiv.org/html/2609.30298#bib.bib6)\)introduced SESR–Eval \(hereafter SESR\), a benchmark dataset for evaluating LLMs in title\-abstract screening for software engineering SRs, comprising25 68325\\,683labeled primary studies from 18 SRs and enabling cross\-study evaluation across multiple LLMs\. Although it presents a useful dataset, SESR covers a restrictive subset of research domains in software engineering\. In\([Syriani et al\., 2024](https://arxiv.org/html/2609.30298#bib.bib12)\), the authors have shown that LLM\-based screening performance is highly dependent on the SR topic\. Additionally, the evaluation of SESR primarily relies on traditional classification metrics, such as accuracy, precision, recall, and F1\-score\. Since SR screening datasets are typically highly imbalanced\([de la Torre\-López et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib20)\), with only a small proportion of relevant studies among thousands of candidates, these metrics may lead to misleading conclusions\. In particular, accuracy can be artificially inflated when a model predominantly predicts the majority class of irrelevant studies\([Madeyski et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib3)\)\. While SESR represents an important contribution, there remains a need for benchmark studies that explicitly account for class imbalance and the specific priorities of SR screening\. Furthermore, conducting such experiments with evolving LLMs and APIs introduces practical challenges related to experiment management, data storage, and result analysis\. A dedicated tool is therefore needed to facilitate benchmark studies and make this line of research more accessible\.

To address these limitations, we present a benchmark framework \(SRBench\) for evaluating the performance of screening automation approaches \(including LLMs\) that accounts for the characteristics of highly imbalanced datasets\. We incorporate metrics such as Matthews Correlation Coefficient \(MCC\)\([Chicco and Jurman, 2020](https://arxiv.org/html/2609.30298#bib.bib18)\)and Balanced Accuracy \(BAcc\), which better capture performance under class imbalance\. The benchmark includes a dataset comprising45 06445\\,064labeled primary studies from 32 secondary studies, extending the 18 SRs from SESR\([Huotala et al\., 2025a](https://arxiv.org/html/2609.30298#bib.bib6)\)with 14 additional SRs to enable cross\-study evaluation of LLM performance across a broader set of software engineering domains\. We also introducePromptSR, a tool designed to support prompt experimentation, experiment management, and result analysis as part of SRBench\. To validate the benchmark and its tool support, we present a use\-case application of the framework using PromptSR, providing insights into how the framework can be used to evaluate LLM performance in SR screening\.

The remainder of this paper is organized as follows\. In Section[2](https://arxiv.org/html/2609.30298#S2), we review background and related work on SRs, LLMs, and existing benchmarks\. In Section[3](https://arxiv.org/html/2609.30298#S3), we describe our benchmark dataset, including details on dataset collection and evaluation framework design\. In Section[4](https://arxiv.org/html/2609.30298#S4), we present the PromptSR tool\. In Section[5](https://arxiv.org/html/2609.30298#S5), we report our use case application and discuss insights and limitations of SRBench\. Finally, we conclude in Section[6](https://arxiv.org/html/2609.30298#S6)\.

## 2\.Background and Related Work

SRs are widely adopted in software engineering to synthesize existing evidence in a rigorous and reproducible manner, whether through in\-depth analyses using systematic literature reviews\([Kitchenham et al\., 2009](https://arxiv.org/html/2609.30298#bib.bib10)\), broad overviews using mapping studies\([Petersen et al\., 2008](https://arxiv.org/html/2609.30298#bib.bib21)\), or other secondary studies\. One critical phase of an SR is screening, in which researchers review titles and abstracts to select relevant primary studies from thousands of candidate papers retrieved through searches of digital libraries\. This phase is typically framed as a binary classification task: relevant vs\. irrelevant articles\. However, screening is time\-consuming and labor\-intensive\([Carver et al\., 2013](https://arxiv.org/html/2609.30298#bib.bib11)\), often requiring substantial manual effort from multiple reviewers\. This creates risks of delays, human error, and reduced reproducibility, motivating research into automated or semi\-automated screening approaches\. For this reason, supporting tools have been developed to assist researchers during SRs\([Marchezan et al\., 2019](https://arxiv.org/html/2609.30298#bib.bib14);[Bigendako and Syriani, 2018](https://arxiv.org/html/2609.30298#bib.bib15)\), including tools that leverage machine learning to prioritize studies for screening\([Hamel et al\., 2020](https://arxiv.org/html/2609.30298#bib.bib16);[Chappell et al\., 2023](https://arxiv.org/html/2609.30298#bib.bib17);[de la Torre\-López et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib20)\)\. Nevertheless, these tools often require substantial training data and may not generalize well across different SR topics\. Recently, LLMs have been explored as supporting tools to assist in this phase\([Scherbakov et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib4);[Wang et al\., 2023](https://arxiv.org/html/2609.30298#bib.bib13);[Syriani et al\., 2024](https://arxiv.org/html/2609.30298#bib.bib12)\)\. While empirical studies report potential efficiency gains, their results also highlight limitations in terms of reliability, consistency, and generalizability across different SR topics\([Lieberum et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib5)\)\. Consequently, systematic and rigorous evaluation frameworks are needed to assess LLM performance in SR screening using appropriate datasets and metrics\([Madeyski et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib3)\)\. This motivates the need for a dedicated benchmark to guide future research in this area\.

Existing SR tools address different aspects of the review process but leave a clear gap for structured LLM\-based screening experiments\. Process management tools such as ReLiS\([Bigendako and Syriani, 2018](https://arxiv.org/html/2609.30298#bib.bib15)\)and Covidence\([Babineau, 2014](https://arxiv.org/html/2609.30298#bib.bib22)\)support collaborative screening, protocol tracking, and data extraction, but are not designed for evaluating or comparing LLM configurations\. AI\-assisted tools such as Elicit\([Whitfield and Hofmann, 2023](https://arxiv.org/html/2609.30298#bib.bib23)\)apply language models to literature search and abstract summarization, but focus on information retrieval rather than rigorous screening decisions, and lack mechanisms for domain\-specific validation or controlled experimentation\. Rayyan\([Hamel et al\., 2020](https://arxiv.org/html/2609.30298#bib.bib16)\)employs a machine learning classifier that learns from reviewer labels to prioritize studies, but does not support LLM backends or structured prompt ablation\. None of these tools provides the infrastructure needed to systematically vary prompt configurations, execute experiments across multiple datasets, and evaluate results using metrics appropriate for imbalanced screening tasks\. This gap motivates PromptSR, a tool designed to support reproducible prompt\-engineering experiments in the context of LLM\-based SR screening\.

Several studies have investigated the role of LLMs in evidence synthesis workflows, including literature search, study selection, and data extraction\([Rathi et al\., 2024](https://arxiv.org/html/2609.30298#bib.bib8);[Lieberum et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib5);[Galli et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib9)\)\. For instance, Lieberumet al\.\([Lieberum et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib5)\)conducted a scoping review examining the application of generative models \(predominantly GPT\-based models\) across multiple stages of medical SRs\. Their findings suggest that although LLMs show promise, there is limited validation for large\-scale applications, and their performance results are mixed\. Similarly, Galliet al\.\([Galli et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib9)\)analyze methodological challenges in deploying LLMs for title and abstract screening, emphasizing the influence of prompt engineering, validation protocols, model selection, and human\-in\-the\-loop strategies on screening outcomes\. While these studies report potential time savings, they also stress the importance of careful evaluation to avoid compromising review quality\.

Considering SRs for software engineering, Huotalaet al\.\([Huotala et al\., 2025a](https://arxiv.org/html/2609.30298#bib.bib6)\)introduced SESR, a benchmark dataset designed to assess LLM performance for title\-abstract screening\. SESR includes25 68325\\,683labeled primary study entries derived from 18 secondary studies, enabling cross\-study evaluation of nine LLMs\.111Although they report2424studies comprising34,52834,528entries\([Huotala et al\., 2025a](https://arxiv.org/html/2609.30298#bib.bib6)\), their replication package does not provide the corresponding data\([Huotala et al\., 2025b](https://arxiv.org/html/2609.30298#bib.bib7)\)\.Their results indicate that LLMs are relatively inexpensive to deploy and often exhibit comparable performance across models\. However, screening effectiveness varies substantially across secondary studies, and none of the evaluated models achieve both consistently high recall and acceptable precision, suggesting that fully automated screening remains unreliable in practice\. While SESR provides a valuable large\-scale evaluation setting, its assessment is primarily based on standard classification metrics such as accuracy, precision, recall, and F1, which biases results in imbalanced classifications\.

These metrics are not fully aligned with the objectives of SR screening\. In particular, they assume symmetric error costs and do not adequately capture the highly imbalanced nature of screening datasets, where relevant studies constitute only a small fraction of the candidate pool\([Madeyski et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib3)\)\. More critically, they fail to reflect the asymmetric consequences of screening errors, where false negatives can lead to the omission of relevant studies, while false positives primarily increase manual effort\. This misalignment can lead to overly optimistic or misleading conclusions about model performance in practical review settings\. Therefore, there is a need for evaluation frameworks that move beyond standard classification metrics and explicitly incorporate class imbalance and screening\-specific error costs\. In the following section, we present a benchmark designed to address these limitations while accounting for the practical requirements of LLM\-assisted SR screening\.

## 3\.SRBench

SRBench is a benchmark designed to provide a comprehensive evaluation framework for assessing the performance of automated screening tools, including LLMs, in the context of SRs\. It comprises evaluation metrics and a dataset of labeled primary studies derived from published SRs in software engineering\. Note that our benchmark also contains baseline results from a use case application, which are reported in Section[5](https://arxiv.org/html/2609.30298#S5)\.

### 3\.1\.Evaluation Metrics

To overcome the limitations of standard metrics, we adopt a set of complementary metrics that together capture classifier behavior under class imbalance\. These metrics include:*Recall*, the ability to retrieve all relevant papers, which is critical in SR screening;*Precision*and*Negative Predictive Value*\(NPV\), which measure the reliability of positive and negative predictions respectively;*Specificity*, which captures how well the model reduces unnecessary manual screening effort\([Kusa et al\., 2023](https://arxiv.org/html/2609.30298#bib.bib24)\);BAcc, the average of Recall and specificity, treating both classes equally regardless of their sizes and serving as the main measure of screening performance across datasets; andMCCbeing the most informative metric for binary classification under class imbalance since it accounts for all four prediction outcomes proportionally to the actual class distribution\([Madeyski et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib3)\)\. By using BAcc and MCC as primary metrics, SRBench provides a more reliable and informative evaluation framework for assessing the performance of screening automation tools in highly imbalanced SR datasets\.*Accuracy*is also collected for completeness, but is not used as a primary indicator given its well\-known sensitivity to class imbalance\. With this, the benchmark provides a customizable evaluation framework that can be adapted to different research priorities and contexts, while ensuring the chosen metrics align with the specific challenges of SR screening\.

Table 1\.The SRBench dataset, including SESR and added SRs with size and class distribution\.
### 3\.2\.Dataset

Our dataset extends the one introduced by SESR\([Huotala et al\., 2025a](https://arxiv.org/html/2609.30298#bib.bib6)\), which compiled screening decisions and article metadata from 18 SRs\. By adding 14 new SRs, the dataset now comprises 32 SRs covering a total of45 06445\\,064primary studies\. All articles in the SRBench include both a title and an abstract\. The corpus size reported by Huotalaet al\.\([Huotala et al\., 2025a](https://arxiv.org/html/2609.30298#bib.bib6)\)exceeds that of their replication package\([Huotala et al\., 2025b](https://arxiv.org/html/2609.30298#bib.bib7)\)because entries lacking an abstract were excluded from the provided data\.

#### 3\.2\.1\.Dataset construction

The 14 newly added SRs were identified by reviewing high\-quality journal publications in software engineering \(Journal of Systems and Software, Information and Software Technology, and Empirical Software Engineering\), where data availability is presented as meta\-information\. From our search, we collected 21 SRs\. We contacted the authors of each paper to obtain detailed information about the screening process, including the list of excluded articles \(for each screening phase, if more than one\), the reasons for exclusion/inclusion, conflict flags, and resolutions\. We received 14 of 21 answers from the authors\.

The heterogeneity of the collected SR datasets required a curation process to ensure data consistency and screening reliability\. We established a semi\-automated procedure to curate all the screening information from these 14 SRs\. Where metadata was incomplete, we retrieved it from seven major scientific databases: IEEE Xplore, ACM Digital Library, ScienceDirect, Springer Link, Web of Science, Scopus, and PubMed Central\. Metadata was standardized across all 32 datasets using a uniform schema, capturing title, abstract, keywords, authors, venue, year, DOI, references, and publisher information\.

Construction involved a multi\-step validation pass for each dataset: comparing extracted titles and author names against the original replication packages, resolving discrepancies, and manually recovering articles that the automated retrieval failed to locate\. During this process, we identified and corrected errors in the original SESR dataset\.*Duplicate entries*were the most prevalent issue, particularly in larger SRs:CODECOMPRcontained duplicates caused by Unicode and character\-encoding normalization differences between sources, whileMODELGUIDANCEandARCHIMLexhibited the same pattern\.*Non\-article items*constituted a second recurring problem:BEHAVEandMODELINGASSISTincluded conference proceedings volumes, technical standards, and project websites, all listed alongside research articles in the original packages\.*Unfindable articles*formed a third category:ARCHIMLprovided only author last\-name fragments for some entries, making retrieval impossible\. Finally,*non\-English articles*, primarily from Chinese, German, and Portuguese sources, were found inOODP,MODELINGASSIST, andARCHIMLand excluded accordingly\. Besides this, a few edge cases happened: the article count forOODPdiffers from the original publication because the dataset was reconstructed by re\-running the search query at collection time; andESM\_2contains only articles retrieved through snowballing, as the primary search results were unavailable in the replication package\. After validation, the final dataset files were produced in a clean, standardized format\.

#### 3\.2\.2\.Data characteristics

The additional SRs broaden the topical coverage of software engineering research beyond the original SESR dataset, which primarily focused on areas such as software testing, model\-driven engineering, and program analysis\. In contrast, the newly added SRs introduce 11 categories that were previously missing, including software architecture, behavior\-driven development, low\-code/no\-code development, ethics in software engineering, and reinforcement learning for software engineering\. Overall, the resulting benchmark spans 35 ACM CCS categories, providing substantially broader coverage of the software engineering research landscape\. Table[1](https://arxiv.org/html/2609.30298#S3.T1)summarizes the composition of the SRBench dataset,222More information is available in our replication package\([Kumar et al\., 2026](https://arxiv.org/html/2609.30298#bib.bib26)\)with references to all SRsincluding both the original 18 SRs and the 14 added ones, describing their number of primary studies and inclusion rates\. These numbers range from highly imbalanced datasets, such asARCHIML\(0\.015\) andMDE4MobileRobot\(0\.031\), to highly inclusive datasets, such asSE4MVP\(0\.856\) andMLDefect\(0\.621\), in which all candidate studies were retained\. This data diversity enables a comprehensive evaluation of screening automation tools across diverse review contexts and inclusion rates\.

## 4\.The PromptSR Tool

Figure 1\.Technical Architecture of PromptSRRunning experiments on SRBench requires orchestrating prompt configurations, executing screening tasks across 32 datasets, and computing SR\-specific metrics\. PromptSR is a tool designed to support this workflow, covering experiment configuration, execution, and result analysis without requiring custom scripting\. The tool follows a modular architecture \(Figure[1](https://arxiv.org/html/2609.30298#S4.F1)\) and is implemented primarily in Python\. A data processing module handles dataset cleaning, formatting, and shot sampling for few\-shot configurations\. A template engine combines the selected feature configurations with a base prompt template to produce the final prompt submitted to the model, including role separation for multi\-role interactions\. A scheduler module manages asynchronous experiment execution, error detection, automatic retries, and incremental result storage\. An analysis and visualization engine aggregates results and exposes the full suite of metrics and plots through the web interface\. Data persistence is handled by PostgreSQL, with database access managed via Prisma, and the web interface is built with Streamlit\. The complete source code is publicly available via Zenodo\([Kumar et al\., 2026](https://arxiv.org/html/2609.30298#bib.bib26)\)\.

PromptSR supports a rich set of configurable prompt features that researchers can combine to construct and compare prompt variants\. Contextual features include aContextcomponent, which assigns a persona and situates the task within the SR topic, aDescriptionthat provides domain background, andResearch Questionsthat guide the classification decision, optionally accompanied by a short or long research question description for varying levels of domain elaboration\. Criteria features allow the prompt to carryInclusion Criteria,Exclusion Criteria, or both, expressing the rules that determine article relevance\. Few\-shot features control whether the prompt includesPositive Shots\(example included articles\),Negative Shots\(example excluded articles\), or a balanced combination of both\. Output and reasoning features shape how the model responds:Chain of Thoughtguides the model through step\-by\-step screening questions before committing to a decision,Uncertainty Outputallows borderline answers such asMAYBE\_INCLUDEorMAYBE\_EXCLUDE,Confidence Scorerequests a numeric score alongside the decision,JSON Output Formatenforces structured parseable responses, andLeniencyinstructs the model to err toward inclusion when uncertain\. While these features are grounded in established SR guidelines\([Kitchenham, 2004](https://arxiv.org/html/2609.30298#bib.bib1);[Petersen et al\., 2008](https://arxiv.org/html/2609.30298#bib.bib21)\), the set is not fixed: new features can be introduced by modifying the prompt template and providing appropriate configuration files\.

These features can be toggled individually or in combination, making PromptSR directly suited for ablation studies that isolate the contribution of each feature or feature group to screening performance\. Beyond prompt structure, experiments are parameterized with dataset\-specific contextual information, and the system supports multiple model backends, including LLMs, traditional machine learning classifiers, and random baselines for comparison\. Experiments are executed via a multithreaded scheduler, deployable locally or on HPC computing clusters, and all results, including model predictions, confidence scores, and results for all evaluation metrics, are stored in a structured database to ensure reproducibility\. The analysis module evaluates results using standard classification metrics and SR\-specific measures, with MCC for comparing prompt variants and BAcc for reporting screening performance\. In this context, Figure[2](https://arxiv.org/html/2609.30298#S4.F2)shows the visualization options, including the choice of metrics, as well as tables, radial plots, box plots, scatter plots, line plots, and pie charts to support flexible result exploration\.

![Refer to caption](https://arxiv.org/html/2609.30298v1/fig/promptslr-ui-1-crop.png)\(a\)Configuration UI
![Refer to caption](https://arxiv.org/html/2609.30298v1/fig/promptslr-ui-2-crop.png)\(b\)Analysis UI

Figure 2\.PromptSR web interface: configuration \(a\) and result analysis \(b\)\.
## 5\.Use Case Application

To demonstrate how SRBench and PromptSR work together in practice, we present a use case application in which we configure a prompt experiment, run it across all 32 SRs in the SRBench dataset, and analyze the results using PromptSR’s built\-in analysis module\.

### 5\.1\.Procedure

The prompt used in the evaluation relies on two feature groups: theContextcomponent, andSelection Criteria, which comprises only the exclusion criteria drawn directly from each dataset\. No few\-shot examples, chain\-of\-thought reasoning, or uncertainty outputs are enabled, making this a minimal but representative configuration to benchmark against SRBench\. The choice of feature groups was based on the findings of Syrianiet al\.\([Syriani et al\., 2024](https://arxiv.org/html/2609.30298#bib.bib12)\), which showed that simple prompts with exclusion criteria were effective\. The experiments are configured through PromptSR’s web interface, where each dataset is uploaded along with its ground\-truth labels and domain\-specific criteria, which are entered into the corresponding prompt fields\.Qwen3\-30B\-A3B\-Thinking\-2507model is selected as the evaluation model running via thevllmon a single Nvidia H100 GPU in a National High Performance Computing Center\. The model was chosen for its performance and efficiency at the time of the experiment, as it performed well on the general LLM benchmark333[https://arena\.ai/leaderboard/](https://arena.ai/leaderboard/), [https://huggingface\.co/Qwen/Qwen3\-30B\-A3B\-Thinking\-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507)tasks while being smaller that can fit on a single GPU, reducing inference costs\. Once configured, PromptSR’s scheduler dispatches the experiment across all 32 datasets in SRBench using a multithreaded execution pipeline\. Each article title and abstract is independently screened by the model using the rendered prompt, and predictions are stored alongside confidence scores, token usage, and response metadata\. The full execution is logged in the database\.

### 5\.2\.Results and Analysis

The results are analyzed using PromptSR’s analysis module\. The median BAcc across all datasets is 0\.69, and the median MCC is 0\.32, indicating that even this minimal prompt configuration produces useful screening decisions across a diverse range of software engineering topics\. Table[2](https://arxiv.org/html/2609.30298#S5.T2)reports an excerpt of the screening performance considering the best and worst MCC results per SR\.

Strong performance is observed on datasets with well\-defined research scopes and moderate class imbalance\.MODELAUTOCODE4WIRELESSachieves a BAcc of 0\.88 and an MCC of 0\.75, whileRL4SEreaches the highest BAcc of 0\.91 with an MCC of 0\.58\.CODESMELLDETECT\(MCC 0\.49\),OPINIONMINESD\(MCC 0\.49\), andMPM4CPS\(MCC 0\.48\) similarly show reliable discrimination\.

The worst MCC results suggest that the metrics in SRBench can reveal behavior that would likely remain hidden if only traditional metrics were used\. For example,LEARNSOFTCONFSPECachieves an apparently excellent accuracy \(0\.971\) and F1 score \(0\.985\)\. However, its MCC is 0\.00, and its BAcc is below random \(0\.485\)\. Similarly,TESTNNhas a moderate F1 score \(0\.595\) despite an MCC of 0\.00 and a BAcc of only 0\.211\. These cases show that the model collapses to predicting a single class, a behavior that would not necessarily be visible when evaluating only conventional metrics\. The expanded benchmark therefore provides insight into*when*prompt\-based screening fails, not only*how well*it performs on average\. These failure cases tend to involve either very small datasets or atypically high inclusion rates that the prompt, without few\-shot examples or uncertainty features, cannot handle\. Overall, the results support the claim that PromptSR’s richer metrics allow for a more trustworthy interpretation of screening quality and help identify classifier behaviors that standard metrics alone would overlook\.

Furthermore, Table[2](https://arxiv.org/html/2609.30298#S5.T2)also shows the results \(accuracy and F1 score\) from SERS evaluation\([Huotala et al\., 2025a](https://arxiv.org/html/2609.30298#bib.bib6)\), excluding the cases of newly added SRs\. Overall, our results demonstrate competitive, and in most cases superior, performance despite relying on an open\-source 30B\-parameter LLM\. In contrast, their study used a range of significantly larger, proprietary models, including Llama Maverick 400B, DeepSeek\-R1\-671B, and models from OpenAI and Anthropic \(with unknown parameters\)\. For example, forMODELAUTOCODE4WIRELESS, accuracy and F1 score from SERS are 0\.56 and 0\.53, while our results show an accuracy of 0\.88 and an F1 score of 0\.89\. As discussed earlier, however, these metrics are not sufficient to fully capture screening performance and serve only as a reference point for comparison with their reported results\. Note also that the SERS results in Table[2](https://arxiv.org/html/2609.30298#S5.T2)are an aggregate across all LLMs used in their study\. Even under these conditions, our results from the use case show that a simple prompt configuration can achieve performance comparable to or better than the average of multiple larger models across a diverse set of SRs\. This suggests that the choice of prompt design and evaluation framework, as facilitated by SRBench and PromptSR, can be as important as model size in determining screening performance\.

Table 2\.Screening performance across the five best and worst cases, ranked by MCC\.
### 5\.3\.Discussion and Limitations

The use case demonstrates how SRBench and PromptSR together enable systematic evaluation of LLM\-assisted screening across a diverse collection of SR datasets\. Even with a minimal prompt configuration, the results show that LLM\-based screening can achieve useful performance across several SRs while simultaneously exposing important failure cases that would otherwise remain hidden under conventional evaluation metrics\. This reinforces the importance of evaluating screening systems using metrics that explicitly account for class imbalance and asymmetric error costs\.

A key implication of SRBench is its ability to evaluate the*generalizability*of screening approaches across different software engineering domains\. The benchmark dataset spans 35 ACM CCS categories and includes SRs with highly heterogeneous inclusion rates, dataset sizes, and research topics\. The observed variability in performance across datasets suggests that prompt configurations and models that perform well in one review context may not generalize reliably to others\. Consequently, SRBench can support future studies on investigating which prompt engineering strategies, model families, or reasoning mechanisms remain robust across diverse review settings, rather than overfitting to a small number of datasets\.

The results also highlight the importance of richer evaluation metrics for SR screening\. Cases such asLEARNSOFTCONFSPECandTESTNNdemonstrate that standard metrics such as Accuracy and F1 score can provide misleadingly optimistic interpretations when datasets are highly imbalanced or fully inclusive\. In contrast, MCC and BAcc expose potential issues such as majority\-class collapse and lack of meaningful discrimination\. This suggests that future evaluations of LLM\-assisted screening should prioritize imbalance\-aware metrics to avoid overstating model effectiveness\.

Beyond evaluating prompts and models, SRBench opens several research directions\. The benchmark can support systematic ablation studies exploring the contribution of prompt features such as few\-shot examples, chain\-of\-thought reasoning, uncertainty outputs, and leniency strategies\. It also enables comparisons between proprietary and open\-source LLMs, traditional machine learning classifiers, and hybrid human\-in\-the\-loop screening workflows\. Since PromptSR stores confidence scores and prediction metadata, future work could additionally investigate calibration quality, selective prediction strategies, and adaptive reviewer assistance mechanisms\.

Despite these contributions, the experimental evaluation only considers a single model and a single baseline prompting strategy\. While this was sufficient to demonstrate the capabilities of SRBench and PromptSR, it does not constitute an exhaustive assessment of LLM\-assisted screening performance\. Different models, prompting techniques, sampling parameters, or reasoning strategies may produce substantially different results\. The goal of the use case is therefore to provide a proof of concept for the framework rather than a definitive comparison of screening methods\.

Benchmark construction demands substantial manual effort and is inherently error\-prone\. As evidence, beyond the 14 newly added SRs, we applied the same curation process described in Section[3](https://arxiv.org/html/2609.30298#S3)to five existing SESR entries and found errors requiring correction\. This analysis revealed several inconsistencies in the inclusion and exclusion labels across the original replication package published alongside the SRs analyzed\. This explains a few edge cases, such as theTESTNNdataset, which has a zero number of exclusions\. Since our data\-gathering process depends not only on the availability of replication packages but also on author responses, validation required extensive curation and correction of inconsistencies across sources\. Extending this approach to other scientific domains would therefore require considerable community effort and domain expertise\. Nevertheless, using the same methodology to add new SRs to our dataset would extend the benchmark’s generalizability beyond software engineering\. PromptSR, on the other hand, can be easily extended to new datasets from different domains like medicine\([Wang et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib27)\)by merely changing the prompt template\. This flexibility allows researchers to use our benchmarking framework as a standardized evaluation platform, which would benefit proposals focused on prompt engineering methodologies\([Syriani et al\., 2024](https://arxiv.org/html/2609.30298#bib.bib12);[Trad et al\., 2025](https://arxiv.org/html/2609.30298#bib.bib25)\)\.

Finally, the benchmark remains limited to title and abstract screening decisions derived from completed SRs\. It does not capture downstream review stages such as full\-text screening, data extraction, or quality assessment\. Future extensions of SRBench could incorporate these aspects to evaluate whether LLMs can support later phases of evidence synthesis beyond initial screening\. Similarly, PromptSR could be extended with workflows for extracting structured evidence, identifying missing information, supporting reviewer disagreement resolution, and generating traceable rationales for inclusion or exclusion decisions\. Such extensions would enable the benchmark to support end\-to\-end evaluation of AI\-assisted SR pipelines rather than only title and abstract classification\.

## 6\.Conclusion

In this paper, we presented a benchmarking framework for evaluating screening automation in software engineering SRs\. Our benchmark relies on imbalance\-aware evaluation metrics to provide a more trustworthy interpretation of screening performance and includes a dataset of45 06445\\,064labeled entries from 32 SRs\. We also introduced PromptSR, a tool for prompt experimentation, execution, and analysis\. Together, they support reproducible and controlled evaluation of screening approaches across diverse SR datasets\. In future work, we will focus on expanding SRBench with additional domains and review types, evaluating more LLMs and different prompting strategies, and improving automation in the screening pipeline, particularly for dataset curation and error handling in large\-scale experiments\. We also plan to make the benchmark publicly available through an online platform to encourage community contributions and facilitate ongoing research on LLM\-assisted screening in evidence synthesis\.

## Data Availability

A replication package, including the dataset and source code, is available at\([Kumar et al\., 2026](https://arxiv.org/html/2609.30298#bib.bib26)\)\.

## References

- Babineau \(2014\)J\. BabineauProduct review: covidence \(systematic review software\)\.Journal of the Canadian Health Libraries Association35\(2\),pp\. 68–71\.Cited by:[§2](https://arxiv.org/html/2609.30298#S2.p2.1)\.
- Bigendako and Syriani \(2018\)B\. Bigendako and E\. SyrianiModeling a tool for conducting systematic reviews iteratively\.InProceedings of the 6th International Conference on Model\-Driven Engineering and Software Development,MODELSWARD 2018,Setubal, PRT,pp\. 552–559\.External Links:ISBN 9789897582837,[Document](https://dx.doi.org/10.5220/0006664405520559)Cited by:[§2](https://arxiv.org/html/2609.30298#S2.p1.1),[§2](https://arxiv.org/html/2609.30298#S2.p2.1)\.
- Borahet al\.\(2017\)R\. Borah, A\. W\. Brown, P\. L\. Capers, and K\. A\. KaiserAnalysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the prospero registry\.BMJ open7\(2\),pp\. e012545\.Cited by:[§1](https://arxiv.org/html/2609.30298#S1.p1.1)\.
- Carveret al\.\(2013\)J\. C\. Carver, E\. Hassler, E\. Hernandes, and N\. A\. KraftIdentifying barriers to the systematic literature review process\.In2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement,Vol\.,pp\. 203–212\.External Links:[Document](https://dx.doi.org/10.1109/ESEM.2013.28)Cited by:[§2](https://arxiv.org/html/2609.30298#S2.p1.1)\.
- Chappellet al\.\(2023\)M\. Chappell, M\. Edwards, D\. Watkins, C\. Marshall, and S\. GraziadioMachine learning for accelerating screening in evidence reviews\.Cochrane Evidence Synthesis and Methods1\(5\),pp\. e12021\.Cited by:[§2](https://arxiv.org/html/2609.30298#S2.p1.1)\.
- Chicco and Jurman \(2020\)D\. Chicco and G\. JurmanThe advantages of the matthews correlation coefficient \(mcc\) over f1 score and accuracy in binary classification evaluation\.BMC genomics21\(1\),pp\. 6\.Cited by:[§1](https://arxiv.org/html/2609.30298#S1.p4.1)\.
- de la Torre\-Lópezet al\.\(2025\)J\. de la Torre\-López, A\. Ramírez, and J\. R\. RomeroAutomatic selection of primary studies in systematic reviews with evolutionary rule\-based classification\.arXiv preprint arXiv:2509\.23981\.Cited by:[§1](https://arxiv.org/html/2609.30298#S1.p3.1),[§2](https://arxiv.org/html/2609.30298#S2.p1.1)\.
- Galliet al\.\(2025\)C\. Galli, A\. V\. Gavrilova, and E\. CalciolariLarge language models in systematic review screening: opportunities, challenges, and methodological considerations\.Information16\(5\)\.External Links:ISSN 2078\-2489,[Document](https://dx.doi.org/10.3390/info16050378)Cited by:[§2](https://arxiv.org/html/2609.30298#S2.p3.1)\.
- Hamelet al\.\(2020\)C\. Hamel, S\. Kelly, K\. Thavorn, D\. Rice, G\. Wells, and B\. HuttonAn evaluation of distillersr’s machine learning\-based prioritization tool for title/abstract screening–impact on reviewer\-relevant outcomes\.BMC medical research methodology20\(1\),pp\. 256\.Cited by:[§2](https://arxiv.org/html/2609.30298#S2.p1.1),[§2](https://arxiv.org/html/2609.30298#S2.p2.1)\.
- Huotalaet al\.\(2025a\)A\. Huotala, M\. Kuutila, and M\. MäntyläSESR\-eval: dataset for evaluating llms in the title\-abstract screening of systematic reviews\.In2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement \(ESEM\),Vol\.,pp\. 01–12\.External Links:[Document](https://dx.doi.org/10.1109/ESEM64174.2025.00053)Cited by:[§1](https://arxiv.org/html/2609.30298#S1.p3.1),[§1](https://arxiv.org/html/2609.30298#S1.p4.1),[§2](https://arxiv.org/html/2609.30298#S2.p4.1),[§3\.2](https://arxiv.org/html/2609.30298#S3.SS2.p1.1),[§5\.2](https://arxiv.org/html/2609.30298#S5.SS2.p4.1),[footnote 1](https://arxiv.org/html/2609.30298#footnote1)\.
- Huotalaet al\.\(2025b\)A\. Huotala, M\. Mäntylä, and M\. KuutilaDataset for: sesr\-eval: dataset for evaluating llms in the title\-abstract screening of systematic reviews\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/ZENODO.16408882)Cited by:[§3\.2](https://arxiv.org/html/2609.30298#S3.SS2.p1.1),[footnote 1](https://arxiv.org/html/2609.30298#footnote1)\.
- Kitchenham \(2004\)B\. KitchenhamProcedures for performing systematic reviews\.Technical reportTechnical ReportTR/SE\-0401,Keele University\.Cited by:[§1](https://arxiv.org/html/2609.30298#S1.p1.1),[§4](https://arxiv.org/html/2609.30298#S4.p2.1)\.
- Kitchenhamet al\.\(2009\)B\. Kitchenham, O\. Pearl Brereton, D\. Budgen, M\. Turner, J\. Bailey, and S\. LinkmanSystematic literature reviews in software engineering \- a systematic literature review\.Information and Software Technology51\(1\),pp\. 7–15\.Note:Special Section \- Most Cited Articles in 2002 and Regular Research PapersExternal Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.infsof.2008.09.009)Cited by:[§1](https://arxiv.org/html/2609.30298#S1.p1.1),[§2](https://arxiv.org/html/2609.30298#S2.p1.1)\.
- Kumaret al\.\(2026\)G\. Kumar, L\. Marchezan, G\. Genois, K\. Delcourt, and E\. SyrianiA benchmark framework for screening automation in systematic reviews \(replication package\)\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/ZENODO.20434525),[Link](https://zenodo.org/doi/10.5281/zenodo.20434525)Cited by:[§4](https://arxiv.org/html/2609.30298#S4.p1.1),[Data Availability](https://arxiv.org/html/2609.30298#Sx1.p1.1),[footnote 2](https://arxiv.org/html/2609.30298#footnote2)\.
- Kusaet al\.\(2023\)W\. Kusa, A\. Lipani, P\. Knoth, and A\. HanburyAn analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews\.Intelligent Systems with Applications18\(200193\)\.External Links:[Document](https://dx.doi.org/10.1016/j.iswa.2023.200193)Cited by:[§3\.1](https://arxiv.org/html/2609.30298#S3.SS1.p1.1)\.
- Lieberumet al\.\(2025\)J\. Lieberum, M\. Toews, M\. Metzendorf, F\. Heilmeyer, W\. Siemens, C\. Haverkamp, D\. Böhringer, J\. J\. Meerpohl, and A\. Eisele\-MetzgerLarge language models for conducting systematic reviews: on the rise, but not yet ready for use—a scoping review\.Journal of Clinical Epidemiology181,pp\. 111746\.External Links:ISSN 0895\-4356,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.jclinepi.2025.111746)Cited by:[§2](https://arxiv.org/html/2609.30298#S2.p1.1),[§2](https://arxiv.org/html/2609.30298#S2.p3.1)\.
- Madeyskiet al\.\(2025\)L\. Madeyski, B\. Kitchenham, and M\. ShepperdLLM4SCREENLIT: recommendations on assessing the performance of large language models for screening literature in systematic reviews\.External Links:2511\.12635,[Link](https://arxiv.org/abs/2511.12635)Cited by:[§1](https://arxiv.org/html/2609.30298#S1.p3.1),[§2](https://arxiv.org/html/2609.30298#S2.p1.1),[§2](https://arxiv.org/html/2609.30298#S2.p5.1),[§3\.1](https://arxiv.org/html/2609.30298#S3.SS1.p1.1)\.
- Marchezanet al\.\(2019\)L\. Marchezan, G\. Bolfe, E\. Rodrigues, M\. Bernardino, and F\. P\. BassoThoth: a web\-based tool to support systematic reviews\.In2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement \(ESEM\),Vol\.,pp\. 1–6\.External Links:[Document](https://dx.doi.org/10.1109/ESEM.2019.8870160)Cited by:[§2](https://arxiv.org/html/2609.30298#S2.p1.1)\.
- Petersenet al\.\(2008\)K\. Petersen, R\. Feldt, S\. Mujtaba, and M\. MattssonSystematic mapping studies in software engineering\.In12th international conference on evaluation and assessment in software engineering \(EASE\),Cited by:[§1](https://arxiv.org/html/2609.30298#S1.p1.1),[§2](https://arxiv.org/html/2609.30298#S2.p1.1),[§4](https://arxiv.org/html/2609.30298#S4.p2.1)\.
- Rathiet al\.\(2024\)H\. Rathi, A\. Malik, D\. Behera, and G\. KambojMSR28 use of large language model \(llm\) for full\-text screening in systematic literature reviews: a comparative analysis\.Value in Health27\(6\),pp\. S264\.Cited by:[§2](https://arxiv.org/html/2609.30298#S2.p3.1)\.
- Scherbakovet al\.\(2025\)D\. Scherbakov, N\. Hubig, V\. Jansari, A\. Bakumenko, and L\. A\. LenertThe emergence of large language models as tools in literature reviews: a large language model\-assisted systematic review\.Journal of the American Medical Informatics Association32\(6\),pp\. 1071–1086\.External Links:[Document](https://dx.doi.org/10.1093/jamia/ocaf063)Cited by:[§2](https://arxiv.org/html/2609.30298#S2.p1.1)\.
- Syrianiet al\.\(2024\)E\. Syriani, I\. David, and G\. KumarScreening articles for systematic reviews with chatgpt\.Journal of Computer Languages80,pp\. 101287\.External Links:ISSN 2590\-1184,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.cola.2024.101287)Cited by:[§1](https://arxiv.org/html/2609.30298#S1.p2.1),[§1](https://arxiv.org/html/2609.30298#S1.p3.1),[§2](https://arxiv.org/html/2609.30298#S2.p1.1),[§5\.1](https://arxiv.org/html/2609.30298#S5.SS1.p1.1),[§5\.3](https://arxiv.org/html/2609.30298#S5.SS3.p6.1)\.
- Tradet al\.\(2025\)F\. Trad, R\. Yammine, J\. Charafeddine, M\. Chakhtoura, M\. Rahme, G\. El\-Hajj Fuleihan, and A\. ChehabStreamlining systematic reviews with large language models using prompt engineering and retrieval augmented generation\.BMC medical research methodology25\(1\),pp\. 130\.Cited by:[§5\.3](https://arxiv.org/html/2609.30298#S5.SS3.p6.1)\.
- Wanget al\.\(2023\)S\. Wang, H\. Scells, B\. Koopman, and G\. ZucconCan chatgpt write a good boolean query for systematic review literature search?\.InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval,SIGIR ’23,New York, NY, USA,pp\. 1426–1436\.External Links:ISBN 9781450394086,[Document](https://dx.doi.org/10.1145/3539618.3591703)Cited by:[§1](https://arxiv.org/html/2609.30298#S1.p2.1),[§2](https://arxiv.org/html/2609.30298#S2.p1.1)\.
- Wanget al\.\(2025\)Z\. Wang, L\. Cao, B\. Danek, Q\. Jin, Z\. Lu, and J\. SunAccelerating clinical evidence synthesis with large language models\.npj Digital Medicine8\(1\),pp\. 509\.Cited by:[§5\.3](https://arxiv.org/html/2609.30298#S5.SS3.p6.1)\.
- Whitfield and Hofmann \(2023\)S\. Whitfield and M\. A\. HofmannElicit: ai literature review research assistant\.Public Services Quarterly19\(3\),pp\. 201–207\.Cited by:[§2](https://arxiv.org/html/2609.30298#S2.p2.1)\.
- Zhang and Ali Babar \(2013\)H\. Zhang and M\. Ali BabarSystematic reviews in software engineering: an empirical investigation\.Information and Software Technology55\(7\),pp\. 1341–1354\.External Links:ISSN 0950\-5849,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.infsof.2012.09.008)Cited by:[§1](https://arxiv.org/html/2609.30298#S1.p1.1)\.