北极问题，缺失答案：用于北极科学中大模型弃答行为的数据集与基准

# Arctic Questions, Missing Answers: A Dataset and Benchmark for LLM Abstention in Arctic Science
Source: [https://arxiv.org/html/2610.09446](https://arxiv.org/html/2610.09446)
Dawei GaoAffiliation:University of North Texas, Denton, TXEmail:[dawei\.gao@unt\.edu](mailto:[email protected])Pradeeban KathiraveluAffiliation:University of Alaska Anchorage, Anchorage, AKEmail:[pkathiravelu@alaska\.edu](mailto:[email protected])Douglas CauseyAffiliation:University of Alaska Anchorage, Anchorage, AKEmail:[dcausey@alaska\.edu](mailto:[email protected])Kewei ShaAffiliation:University of North Texas, Denton, TXEmail:[kewei\.sha@unt\.edu](mailto:[email protected])Yunhe FengAffiliation:University of North Texas, Denton, TXEmail:[yunhe\.feng@unt\.edu](mailto:[email protected])

###### Abstract

Large language models \(LLMs\) should abstain from scientific multiple\-choice questions when no option is valid, but frequent abstention alone does not demonstrate sensitivity to answer availability\. We introduceArcticQA, a dataset of 194 questions derived from primary Arctic research, with automated checks of answer support and distractor contradiction against source evidence\. We further developArcticAbstain, a paired benchmark comparing answer\-present and answer\-absent conditions, with the correct answer replaced by a distractor in the latter and an explicit abstention option in both\. We evaluate eight models from the Gemini, Claude, and ChatGPT families at high reasoning effort, with three trials per condition, yielding 9,312 recorded responses\. Answer\-present abstention rates range from 0\.0% to 63\.0%, whereas replacing the correct answer increases abstention by 5\.05 percentage points on average\. These findings highlight substantial baseline differences and the need to evaluate abstention frequency and responsiveness jointly\. The dataset and benchmark are available at[https://github\.com/BenWilcox8/arctic\-qa](https://github.com/BenWilcox8/arctic-qa)\.

## 1Introduction

Large language models \(LLMs\) could help researchers navigate and synthesize the multidisciplinary literature of Arctic science\. Their usefulness, however, depends on whether they can provide scientifically supported answers and recognize when they should abstain\. Arctic research spans interconnected physical, ecological, and human systems, and its findings often depend on specific locations, seasons, populations, and measurement conditions\. Moreover, field observations are unevenly distributed across the region: prior research has documented substantial spatial biases in terrestrial Arctic sampling and scientific citation\([Metcalfe et al\., 2018](https://arxiv.org/html/2610.09446#bib.bib1)\)\. In this setting, an answer that is scientifically plausible in general may be incorrect within the scope of a particular question\. Such errors can distort the interpretation of published findings and propagate into subsequent scientific analyses, making abstention an important component of trustworthy Arctic question answering \(QA\)\.

LLM abstention has already been investigated in other scientific disciplines\. For example,[Wen et al\. \(2024\)](https://arxiv.org/html/2610.09446#bib.bib2)study abstention in biomedical and computer\-science QA and show that behavior varies with the model, question type, and available context\. Broader evaluations, including Abstain\-QA and AbstentionBench, also reveal persistent difficulties in distinguishing questions that warrant an answer from those that warrant abstention\([Madhusudhan et al\., 2025](https://arxiv.org/html/2610.09446#bib.bib3);[Kirichenko et al\., 2025](https://arxiv.org/html/2610.09446#bib.bib21)\)\. However, to the best of our knowledge, no dedicated benchmark has systematically evaluated LLM abstention on scientific questions grounded in primary Arctic research\. Existing results therefore leave unresolved whether LLMs can appropriately withhold answers when the available choices conflict with the specific conditions and findings of Arctic studies\.

Addressing this gap requires an evaluation resource that preserves the scope of Arctic scientific evidence\. Geographic variation, seasonal dependence, and differences in study populations or measurement protocols complicate the construction of both correct answers and demonstrably incorrect alternatives\. A finding from one setting cannot automatically establish what is valid in another\. These considerations make benchmark curation particularly demanding: questions must state the relevant conditions, answers must be traceable to source findings, and distractors must be evaluated within the same scope\. A dedicated Arctic resource is therefore necessary to examine abstention against explicit scientific evidence and to make the basis of the evaluation inspectable\.

To support trustworthy evaluation, we introduceArcticQA, an evidence\-grounded QA dataset constructed from primary Arctic research\. For each item, we identify a clearly defined finding and its supporting evidence, then formulate a standalone question that preserves the conditions needed to interpret that finding\. Separate automated checks assess whether the designated answer is supported by the source and whether each distractor is contradicted within the question’s stated scope\. Checking distractors explicitly is essential because an option’s absence from a paper does not establish that it is false\. Building on prior work in paper\-grounded and automatically generated scientific QA\([Dasigi et al\., 2021](https://arxiv.org/html/2610.09446#bib.bib4);[Wan et al\., 2024](https://arxiv.org/html/2610.09446#bib.bib6)\),ArcticQAprovides item\-level evidence and validation checks to support transparent assessment of Arctic scientific answers\.

UsingArcticQA, we developArcticAbstain, a paired benchmark centered on the following question:*How does an LLM’s abstention behavior change when a scientifically valid answer is removed from the available options?*We operationalize abstention as selecting an explicit abstention option instead of a substantive answer\. In theanswer\-presentcondition, each question includes one designated correct answer, three distractors, and the abstention option\. In the matchedanswer\-absentcondition, the correct answer is replaced with a fourth distractor, while the question and the remaining options are retained\. The underlying scientific question remains answerable from its source; the manipulation changes whether a valid answer appears among the choices\. Under this protocol, abstaining in the answer\-present condition constitutes*false abstention*, whereas selecting a substantive option in the answer\-absent condition constitutes*false commitment*\. Comparing the two conditions measures sensitivity to answer availability relative to a model’s baseline tendency to abstain, which a raw abstention rate alone cannot establish\.

We evaluate eight models from the Gemini, Claude, and ChatGPT families on 194 Arctic scientific questions at high reasoning effort\. Each model answers both conditions three times, yielding 9,312 recorded responses\. We analyze abstention alongside the accuracy of non\-abstaining responses\. Answer\-present abstention rates range from 0\.0% to 63\.0% across models, while replacing the correct answer increases abstention by 5\.05 percentage points on average\. These results reveal substantial variation in baseline abstention and a comparatively limited average change after answer replacement under our evaluation protocol\. They underscore the need to evaluate both a model’s willingness to answer Arctic scientific questions and its responsiveness to the absence of a valid option\.

Our main contributions are as follows:

- •ArcticQA: an evidence\-grounded Arctic QA dataset\.We construct questions from primary Arctic research, retain item\-level supporting evidence, and apply separate automated checks to answers and distractors to support trustworthy, inspectable evaluation\.
- •ArcticAbstain: a paired benchmark for Arctic scientific abstention\.We vary answer availability within matched question pairs to measure false abstention, false commitment, and changes in abstention relative to baseline behavior\.
- •An empirical evaluation of eight LLMs\.We conduct repeated trials across three model families, characterizing differences in abstention behavior and sensitivity to answer replacement on Arctic scientific questions\.

## 2Related Work

Work on selective generation asks whether a model can estimate uncertainty and withhold an unreliable answer\. Self\-evaluation prompts can improve selective generation[Ren et al\. \(2023\)](https://arxiv.org/html/2610.09446#bib.bib17), while multi\-model collaboration can expose knowledge gaps before a response is produced[Feng et al\. \(2024a\)](https://arxiv.org/html/2610.09446#bib.bib19)\. Abstain\-QA evaluates willingness to answer across answerable and unanswerable prompts[Madhusudhan et al\. \(2025\)](https://arxiv.org/html/2610.09446#bib.bib3)\. AbstentionBench broadens this analysis across types of unanswerable questions and reports that reasoning\-oriented training can increase forced problem solving[Kirichenko et al\. \(2025\)](https://arxiv.org/html/2610.09446#bib.bib21)\. These studies motivate our high\-reasoning\-effort setting, but their conclusions cannot be assumed to transfer directly to source\-bounded Arctic scientific questions\.

Scientific abstention has also been studied through controlled context changes and external verification\. Context\-perturbation experiments alter the evidence available to a model and observe how its answer or abstention changes[Wen et al\. \(2024\)](https://arxiv.org/html/2610.09446#bib.bib2)\. Abstention\-aware scientific reasoning systems can decompose claims and check them against retrieved evidence or natural\-language\-inference models[Abdaljalil et al\. \(2026\)](https://arxiv.org/html/2610.09446#bib.bib22)\. Scientific claim\-verification datasets such as SciFact\-Open likewise connect claims to evidence retrieval and verification[Wadden et al\. \(2022\)](https://arxiv.org/html/2610.09446#bib.bib5)\. These systems evaluate the reliability of a composite process\.ArcticQAinstead evaluates the response of the tested model to a self\-contained multiple\-choice item, without giving the model a retrieval or verification pipeline\.

Scientific question\-answering resources cover difficult reasoning and questions anchored in research papers\. GPQA targets graduate\-level questions in biology, physics, and chemistry, while MMLU\-Pro expands difficult multi\-task multiple\-choice evaluation[Rein et al\. \(2023\)](https://arxiv.org/html/2610.09446#bib.bib16);[Wang et al\. \(2024\)](https://arxiv.org/html/2610.09446#bib.bib18)\. Other work builds information\-seeking questions around research papers or generates scientific QA datasets with fine\-grained checks[Dasigi et al\. \(2021\)](https://arxiv.org/html/2610.09446#bib.bib4);[Wan et al\. \(2024\)](https://arxiv.org/html/2610.09446#bib.bib6)\. Round\-trip consistency can filter synthetic question\-answer pairs by testing whether an answer can be reconstructed[Alberti et al\. \(2019\)](https://arxiv.org/html/2610.09446#bib.bib7)\. Decontextualization work formalizes the requirement that text remain understandable outside its original document[Choi et al\. \(2021\)](https://arxiv.org/html/2610.09446#bib.bib14)\.ArcticQAcombines these concerns by deriving each question from bounded source evidence\. The final question must also stand alone for a reader who does not have the paper\.

Automatic multiple\-choice construction introduces a separate risk: a plausible distractor can be partly correct, true under another scope, or equivalent to the answer\. Prior work therefore treats distractor generation and distractor evaluation as distinct tasks[Yu et al\. \(2024\)](https://arxiv.org/html/2610.09446#bib.bib9);[Byun and Choi \(2025\)](https://arxiv.org/html/2610.09446#bib.bib10);[Feng et al\. \(2024b\)](https://arxiv.org/html/2610.09446#bib.bib11)\. Iterative critique can improve generated questions[Yao et al\. \(2025\)](https://arxiv.org/html/2610.09446#bib.bib12), but automated refinement and model judging can preserve shared errors or introduce judge bias[Zheng et al\. \(2023\)](https://arxiv.org/html/2610.09446#bib.bib15);[Benedetto et al\. \(2025\)](https://arxiv.org/html/2610.09446#bib.bib13)\.ArcticQAresponds with per\-option source checks, deterministic rejection rules, bounded revision paths, and an explicitly non\-gold machine\-acceptance label\.

## 3ArcticQADataset

We introduceArcticQA, a question\-answering dataset grounded in specific findings from primary Arctic research\. Reliable abstention evaluation requires both a supported correct answer and demonstrably incorrect distractors\. We therefore develop a construction pipeline that links each target answer to supporting source evidence and verifies each distractor within the question’s stated scope\. The pipeline comprises paper selection, finding extraction and question generation, QA validation, and distractor generation and validation\.

### 3\.1Paper selection and eligibility

We identified 84,829 papers through keyword searches using common Arctic research terms in Semantic Scholar111[https://www\.semanticscholar\.org/](https://www.semanticscholar.org/)\. Metadata filtering reduced this set to 4,420 candidate papers, whose full texts were subsequently assessed for eligibility\. We defined geographic eligibility using a latitude threshold of66\.56∘66\.56^\{\\circ\}N for terrestrial studies and a predefined list of eligible Arctic marine regions for marine studies\. This list remained fixed throughout the eligibility assessment process\.

### 3\.2Finding extraction and question generation

For each eligible paper, an LLM reads the full text and identifies the primary finding together with its supporting textual evidence\. Once validated, the finding is held fixed as the target answer\. A separate LLM writer then formulates a standalone question that asks specifically for this answer while preserving the scope and conditions of the original finding\. This answer\-first approach, followed by answer reconstruction, draws on the roundtrip\-consistency principle of[Alberti et al\. \(2019\)](https://arxiv.org/html/2610.09446#bib.bib7)\. We adapt scientific\-paper QA generation\([Wan et al\., 2024](https://arxiv.org/html/2610.09446#bib.bib6)\)by restricting generation to one clearly scoped finding per paper\.

### 3\.3QA validation

We validate each QA pair through blinded answer reconstruction followed by a separate verification step\. The reconstructor LLM receives the question, its context, and the source evidence, but not the proposed answer\. It produces a reconstructed answer, supporting evidence, and alternative answers\. A separate verifier compares the proposed and reconstructed answers and assesses their consistency with the source\. Specifically, it checks whether the evidence entails the proposed answer, whether the question and answer preserve the finding’s scope and claim type, and whether the question’s context is sufficient without disclosing the answer\. A QA pair proceeds to distractor generation only after passing all checks\. This combination of reconstruction and entailment checking is informed by the complementary factual\-consistency signals identified by[Fabbri et al\. \(2022\)](https://arxiv.org/html/2610.09446#bib.bib8)\.

### 3\.4Distractor generation and validation

For each validated QA pair, we construct four distractors through separate candidate\-generation and verification stages\. This staged design draws on prior work on automated distractor generation and filtering\([Yu et al\., 2024](https://arxiv.org/html/2610.09446#bib.bib9);[Byun and Choi, 2025](https://arxiv.org/html/2610.09446#bib.bib10)\)\. A writer LLM proposes four to six candidates by modifying numerical values, categories, directions of change, scope, or entities\. The writer does not verify its own outputs; each candidate undergoes a separate verification step\.

To be retained, a candidate must be contradicted by specific source evidence within the question’s stated scope\. It is rejected if a reasonable interpretation of the paper could support it as a correct answer\. Absence from the source alone does not establish falsity, and validity is assessed under the conditions specified by the question, including the relevant location and time\. We retain four candidates that satisfy these criteria\. These checks emphasize distractor validity, which prior work distinguishes from plausibility\([Feng et al\., 2024b](https://arxiv.org/html/2610.09446#bib.bib11)\), and make the basis for rejecting each option traceable to the source evidence\.

## 4ArcticAbstainBenchmark

ArcticAbstainevaluates LLM abstention on Arctic scientific questions using two matched multiple\-choice conditions:answer\-presentandanswer\-absent\. In the former, the correct answer is included among the options; in the latter, it is replaced by a validated distractor\. Comparing responses across these conditions measures how abstention changes when a correct answer is no longer available for selection\.

### 4\.1Paired abstention protocol

Each pair shares the same question wording and three validated distractors\. Both items also include the explicit option “I abstain from answering,” consistent with prior work that allows LLMs to reject all candidate answers during self\-evaluation\([Ren et al\., 2023](https://arxiv.org/html/2610.09446#bib.bib17)\)\. Each item contains five options, with their order randomized to reduce systematic effects of option position\.

#### 4\.1\.1Answer\-present condition

The answer\-present item contains the gold answer, three distractors, and the abstention option\. Under the benchmark’s scoring protocol, selecting the gold answer is correct, selecting a distractor is an incorrect answer, and selecting the abstention option constitutesfalse abstention, or over\-abstention\. This condition quantifies how often a model abstains despite the presence of a valid answer\.

#### 4\.1\.2Answer\-absent condition

We replace the gold answer with a fourth validated distractor, retaining the same question, three shared distractors, and abstention option\. Because no candidate answer is valid within the question’s stated scope, abstention is the only response scored as correct\. Selecting any distractor constitutesfalse commitment, indicating failure to abstain when no valid answer is available\.

The underlying question remains answerable from the source paper in both conditions; only the available answer options change\. Together, the paired conditions distinguish false abstention from false commitment and quantify the change in abstention relative to the model’s answer\-present baseline behavior\.

## 5Experiments

### 5\.1Experimental setup

We evaluate eight models from three families: two Gemini models, three Claude models, and three ChatGPT models\. All models are configured to use high reasoning effort\. Each question fromArcticQAis evaluated under the paired answer\-present and answer\-absent conditions defined inArcticAbstain\. Each model answers each condition three times through separate calls, yielding six responses per model and 48 responses across all eight models for each base question\.

The analysis includes 194 questions with all 48 planned response records, for a total of 9,312 recorded responses\. Invalid responses are excluded from metric calculations; thus, the number of recorded responses is distinguished from the number of valid responses used in the metric denominators for reported results\.

### 5\.2Evaluation metrics

We evaluate answer selection and abstention inArcticAbstainusing six metrics from prior abstention research[Feng et al\. \(2024a\)](https://arxiv.org/html/2610.09446#bib.bib19);[Wen et al\. \(2025\)](https://arxiv.org/html/2610.09446#bib.bib20);[Wen et al\. \(2024\)](https://arxiv.org/html/2610.09446#bib.bib2)\. For each model, we partition its responses across all evaluated questions and trials into five mutually exclusive categories\. In theanswer\-presentcondition,N1N\_\{1\},N2N\_\{2\}, andN3N\_\{3\}denote the numbers of correct answers, incorrect answers, and abstentions, respectively\. In theanswer\-absentcondition,N4N\_\{4\}andN5N\_\{5\}denote the numbers of incorrect answers and abstentions, respectively\. Because no substantive option is correct in the answer\-absent condition, every non\-abstaining response in that condition is counted inN4N\_\{4\}\. Under the benchmark’s answer\-availability criterion,N3N\_\{3\}represents false abstentions andN5N\_\{5\}represents appropriate abstentions\. The total number of evaluated responses isN=∑i=15NiN=\\sum\_\{i=1\}^\{5\}N\_\{i\}\.

Abstention accuracy\(ACC\) measures overall success in providing correct answers or abstaining when necessary:

ACC=N1\+N5N\.\\mathrm\{ACC\}=\\frac\{N\_\{1\}\+N\_\{5\}\}\{N\}\.\(1\)It credits correct answers when a valid option is present and abstentions when all substantive options are incorrect, thereby jointly evaluating answer correctness and abstention behavior\.

Abstention precisionmeasures the proportion of abstentions that occur when no valid answer option is available:

Precisionabs=N5N3\+N5\.\\mathrm\{Precision\}\_\{\\mathrm\{abs\}\}=\\frac\{N\_\{5\}\}\{N\_\{3\}\+N\_\{5\}\}\.\(2\)A higher value indicates that a greater proportion of the model’s abstentions are appropriate under the evaluation protocol\. Following the definition summarized by[Wen et al\. \(2025\)](https://arxiv.org/html/2610.09446#bib.bib20),abstention recallis computed as

Recallabs=N5N2\+N4\+N5\.\\mathrm\{Recall\}\_\{\\mathrm\{abs\}\}=\\frac\{N\_\{5\}\}\{N\_\{2\}\+N\_\{4\}\+N\_\{5\}\}\.\(3\)This metric measures the proportion of appropriate abstentions among the combined set of appropriate abstentions and incorrect substantive answers\. Its denominator includes errors from both conditions:N2N\_\{2\}captures incorrect selections despite the availability of a valid option, andN4N\_\{4\}captures incorrect selections when no valid option is available\. Thus, this definition penalizes incorrect answering in both conditions and differs from recall restricted to identifying answer\-absent cases\. We summarize abstention precision and recall using their harmonic mean, theabstention F1\-score:

F1abs=2​Precisionabs​RecallabsPrecisionabs\+Recallabs\.\\mathrm\{F1\}\_\{\\mathrm\{abs\}\}=\\frac\{2\\,\\mathrm\{Precision\}\_\{\\mathrm\{abs\}\}\\,\\mathrm\{Recall\}\_\{\\mathrm\{abs\}\}\}\{\\mathrm\{Precision\}\_\{\\mathrm\{abs\}\}\+\\mathrm\{Recall\}\_\{\\mathrm\{abs\}\}\}\.\(4\)
Theabstention ratemeasures how frequently the model abstains across both conditions:

Abstention​Rate=N3\+N5N\.\\mathrm\{Abstention\\ Rate\}=\\frac\{N\_\{3\}\+N\_\{5\}\}\{N\}\.\(5\)This rate describes the model’s overall tendency to withhold answers\. Because it includes both false and appropriate abstentions, a higher abstention rate does not necessarily indicate better overall abstention performance\.

Finally,reliable accuracy\(R\-Acc\) measures answer accuracy conditional on the model selecting a substantive option:

R\-Acc=N1N1\+N2\+N4\.\\text\{R\-Acc\}=\\frac\{N\_\{1\}\}\{N\_\{1\}\+N\_\{2\}\+N\_\{4\}\}\.\(6\)Its denominator includes all non\-abstaining responses across both conditions\. R\-Acc therefore characterizes the correctness of committed answers and should be interpreted alongside the abstention rate to account for how frequently the model chooses to answer\.

### 5\.3Aggregation and statistical analysis

For the aggregate metric comparison, we compute each metric separately for the three trials and report the median across trials in Table[2](https://arxiv.org/html/2610.09446#S5.T2)\. For the paired condition analysis in Table[1](https://arxiv.org/html/2610.09446#S5.T1), we pool responses across all three trials to estimate each model’s condition\-specific abstention rates\. The paired shift is the answer\-absent rate minus the answer\-present rate, expressed in percentage points \(pp\)\.

We estimate 95% confidence intervals for these shifts using 10,000 bootstrap resamples of questions with a random seed of 7\. Each resample retains both conditions and all three trials for every selected question, preserving the dependence among responses associated with that question\. For each model, a question\-level sign\-flip permutation test assesses whether abstention changes after the correct answer is replaced\. We apply Holm correction across the eight model\-specific tests to control the family\-wise error rate and assess significance at an adjustedp<0\.05p<0\.05\.

### 5\.4Results and analysis

#### 5\.4\.1Variation in baseline abstention

Table[1](https://arxiv.org/html/2610.09446#S5.T1)shows substantial variation in answer\-present abstention, ranging from 0\.0% to 63\.0%\. ChatGPT 5\.6 Sol and ChatGPT 5\.6 Terra almost always select a substantive option, with abstention rates of 0\.2% and 0\.0%, respectively\. The Gemini models abstain on 6\.0%–6\.4% of answer\-present responses, whereas the Claude models abstain on 36\.1%–48\.5%\. ChatGPT Astra has the highest rate, at 63\.0%\. The large differences among the three ChatGPT models show that model\-family membership alone does not characterize abstention behavior in this evaluation\.

ModelPresent abst\.Absent abst\.Shift \[95% CI\] \(pp\)Permutation𝒑\\boldsymbol\{p\}Holm𝒑\\boldsymbol\{p\}Gemini 3\.8 Flash6\.4%10\.3%\+3\.9​\[0\.6,7\.4\]\+3\.9\\;\[0\.6,\\,7\.4\]0\.0370\.187Gemini 3\.7 Flash6\.0%8\.6%\+2\.6​\[−0\.1,5\.4\]\+2\.6\\;\[\-0\.1,\\,5\.4\]0\.0630\.198Claude Fable 5\.136\.1%46\.9%\+10\.8​\[5\.7,16\.1\]\+10\.8\\;\[5\.7,\\,16\.1\]0\.000100\.0008Claude Opus 541\.3%47\.5%\+6\.2​\[0\.9,11\.6\]\+6\.2\\;\[0\.9,\\,11\.6\]0\.0310\.187Claude Sonnet 548\.5%51\.0%\+2\.5​\[−1\.7,6\.9\]\+2\.5\\;\[\-1\.7,\\,6\.9\]0\.3310\.331ChatGPT Astra63\.0%74\.1%\+11\.1​\[5\.7,16\.6\]\+11\.1\\;\[5\.7,\\,16\.6\]0\.000150\.0011ChatGPT 5\.6 Sol0\.2%2\.1%\+1\.9​\[0\.3,4\.0\]\+1\.9\\;\[0\.3,\\,4\.0\]0\.0490\.198ChatGPT 5\.6 Terra0\.0%1\.5%\+1\.5​\[0\.3,3\.3\]\+1\.5\\;\[0\.3,\\,3\.3\]0\.0650\.198Table 1:Condition\-specific abstention rates pooled across three trials for 194 questions\. Shifts are answer\-absent minus answer\-present rates, in percentage points \(pp\)\. The 95% confidence intervals use question\-level bootstrap resampling\. Permutation tests use question\-level paired differences; Holm\-adjustedpp\-values account for the eight model\-specific tests\.
#### 5\.4\.2Response to correct\-answer removal

Replacing the correct answer with a distractor increases the abstention point estimate for every model\. The increases range from 1\.5 to 11\.1 percentage points, with a mean increase of 5\.05 percentage points across the eight models\. These shifts are modest relative to the 63\.0% range in answer\-present abstention\. Figure[1](https://arxiv.org/html/2610.09446#S5.F1)visualizes each model’s paired rates\.

Claude Fable 5\.1 and ChatGPT Astra exhibit the largest increases, at 10\.8 and 11\.1 percentage points, respectively\. Both changes remain statistically significant after Holm correction, with adjustedpp\-values of 0\.0008 and 0\.0011\. The remaining six models do not meet the adjusted significance threshold\. Thus, the point estimates consistently indicate increased abstention after answer removal, while the model\-specific tests provide evidence of a change for two models after correction for multiple comparisons\.

Figure 1:Abstention rates under the answer\-present and answer\-absent conditions for the eight models\. Each segment connects a model’s two rates, pooled across three trials\. Labels report the answer\-absent minus answer\-present shift in percentage points\.
#### 5\.4\.3Accuracy and abstention must be interpreted jointly

Table[2](https://arxiv.org/html/2610.09446#S5.T2)reports median metrics across the three trials\. ChatGPT Astra has the highest median ACC \(0\.541\), abstention F1\-score \(0\.618\), and R\-Acc \(0\.540\), together with the highest overall abstention rate \(0\.686\)\. However, its 63\.0% answer\-present abstention rate also indicates frequent abstention when a valid option is available\. Its comparatively high conditional answer accuracy therefore coexists with substantial false abstention\.

All reported R\-Acc medians exceed 0\.125, the expected value for uniform random selection among the four substantive options under equally represented conditions:12×14=0\.125\\tfrac\{1\}\{2\}\\times\\tfrac\{1\}\{4\}=0\.125\. The other seven models have R\-Acc medians between 0\.261 and 0\.379\. These descriptive comparisons do not establish statistically significant differences between models\. For example, ChatGPT 5\.6 Sol has an R\-Acc of 0\.291, numerically close to the Gemini models’ values of 0\.303 and 0\.306, but a much lower abstention rate of 0\.011\. Conditional answer accuracy and the tendency to abstain therefore capture distinct aspects of performance, and both are needed to assess reliability on Arctic scientific questions\.

ModelACCAbst\.rate𝐅𝟏𝐚𝐛𝐬\\mathbf\{F1\}\_\{\\mathbf\{abs\}\}R\-AccGemini 3\.8 Flash0\.3290\.0840\.1340\.303Gemini 3\.7 Flash0\.3260\.0730\.1130\.306Claude Fable 5\.10\.4580\.4150\.4650\.379Claude Opus 50\.4380\.4440\.4590\.359Claude Sonnet 50\.4080\.4980\.4640\.303ChatGPT Astra0\.5410\.6860\.6180\.540ChatGPT 5\.6 Sol0\.2980\.0110\.0290\.291ChatGPT 5\.6 Terra0\.2660\.0080\.0210\.261Table 2:Metrics pooled across the three trials for eight models evaluated at high reasoning effort on 194 Arctic scientific questions\. Each trial’s metrics combine the answer\-present and answer\-absent conditions\.

## 6Conclusion

We introducedArcticQA, a dataset grounded in primary Arctic research, andArcticAbstain, a paired benchmark for evaluating LLM abstention under answer\-present and answer\-absent conditions\. Across eight models, answer\-present abstention rates ranged from 0\.0% to 63\.0%, while removing the correct answer increased abstention by 5\.05 percentage points on average\. These findings highlight the distinction between a model’s tendency to abstain and its responsiveness to answer availability\. Reliable evaluation should therefore consider false abstention, false commitment, and the accuracy of non\-abstaining responses jointly\. Our dataset and benchmark provide a foundation for assessing and improving LLM reliability in Arctic scientific question answering\.

## 7Limitations

ArcticAbstainis a geographically bounded case study whose coverage reflects discovery queries, full\-text availability, and model\-guided processing order\. It is not representative of all Arctic research or other scientific domains\. Dataset construction relies on model\-generated content and automated validation\. No sample was reviewed by domain experts; thus,machine\_accepted\_unverifieddenotes automated acceptance, not expert\-verified ground truth\. The residual error rate remains unmeasured\. The evaluation covers 194 questions, eight models, a single prompt, one explicit abstention option, and high reasoning effort\. The paired multiple\-choice task assesses rejection of invalid option sets rather than spontaneous abstention in open\-ended scientific work\. The results do not establish training exposure, explain observed response policies, or demonstrate generalization beyond the evaluated setting\.

## References

- Abdaljalilet al\.\(2026\)S\. Abdaljalil, E\. Serpedin, and H\. KurbanKnowing when not to answer: abstention\-aware scientific reasoning\.arXiv \(Cornell University\)\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2602.14189)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p2.1)\.
- Albertiet al\.\(2019\)C\. Alberti, D\. Andor, E\. Pitler, J\. Devlin, and M\. CollinsSynthetic QA corpora generation with roundtrip consistency\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 6168–6173\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1620),[Link](https://aclanthology.org/P19-1620/)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p3.1),[§3\.2](https://arxiv.org/html/2610.09446#S3.SS2.p1.1)\.
- Benedettoet al\.\(2025\)L\. Benedetto, S\. Taslimipoor, and P\. ButteryA survey on automated distractor evaluation in multiple\-choice tasks\.InProceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications,pp\. 55–69\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.bea-1.5),[Link](https://aclanthology.org/2025.bea-1.5/)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p4.1)\.
- Byun and Choi \(2025\)G\. Byun and J\. D\. ChoiD\-GEN: automatic distractor generation and evaluation for reliable assessment of generative models\.InFindings of ACL,pp\. 3316–3349\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.174),[Link](https://aclanthology.org/2025.findings-acl.174/)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p4.1),[§3\.4](https://arxiv.org/html/2610.09446#S3.SS4.p1.1)\.
- Choiet al\.\(2021\)E\. Choi, J\. Palomaki, M\. Lamm, T\. Kwiatkowski, D\. Das, and M\. CollinsDecontextualization: making sentences stand\-alone\.Transactions of the Association for Computational Linguistics9,pp\. 447–461\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00377),[Link](https://aclanthology.org/2021.tacl-1.27/)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p3.1)\.
- Dasigiet al\.\(2021\)P\. Dasigi, K\. Lo, I\. Beltagy, A\. Cohan, N\. A\. Smith, and M\. GardnerA dataset of information\-seeking questions and answers anchored in research papers\.InProceedings of NAACL\-HLT,pp\. 4599–4610\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.365),[Link](https://aclanthology.org/2021.naacl-main.365/)Cited by:[§1](https://arxiv.org/html/2610.09446#S1.p4.1),[§2](https://arxiv.org/html/2610.09446#S2.p3.1)\.
- Fabbriet al\.\(2022\)A\. Fabbri, C\. Wu, W\. Liu, and C\. XiongQAFactEval: improved QA\-based factual consistency evaluation for summarization\.InProceedings of NAACL\-HLT,pp\. 2587–2601\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.187),[Link](https://aclanthology.org/2022.naacl-main.187/)Cited by:[§3\.3](https://arxiv.org/html/2610.09446#S3.SS3.p1.1)\.
- Fenget al\.\(2024a\)S\. Feng, W\. Shi, Y\. Wang, W\. Ding, V\. Balachandran, and Y\. TsvetkovDon’t hallucinate, abstain: identifying llm knowledge gaps via multi\-llm collaboration\.External Links:2402\.00367,[Link](https://arxiv.org/abs/2402.00367)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p1.1),[§5\.2](https://arxiv.org/html/2610.09446#S5.SS2.p1.1)\.
- Fenget al\.\(2024b\)W\. Feng, J\. Lee, H\. McNichols, A\. Scarlatos, D\. Smith, S\. Woodhead, N\. Ornelas, and A\. LanExploring automated distractor generation for math multiple\-choice questions via large language models\.InFindings of NAACL,pp\. 3067–3082\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.193),[Link](https://aclanthology.org/2024.findings-naacl.193/)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p4.1),[§3\.4](https://arxiv.org/html/2610.09446#S3.SS4.p2.1)\.
- Kirichenkoet al\.\(2025\)P\. Kirichenko, M\. Ibrahim, K\. Chaudhuri, and S\. J\. BellAbstentionBench: reasoning llms fail on unanswerable questions\.arXiv\.org\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2506.09038)Cited by:[§1](https://arxiv.org/html/2610.09446#S1.p2.1),[§2](https://arxiv.org/html/2610.09446#S2.p1.1)\.
- Madhusudhanet al\.\(2025\)N\. Madhusudhan, S\. T\. Madhusudhan, V\. Yadav, and M\. HashemiDo LLMs know when to NOT answer? investigating abstention abilities of large language models\.InProceedings of the 31st International Conference on Computational Linguistics,Abu Dhabi, UAE,pp\. 9329–9345\.External Links:[Link](https://aclanthology.org/2025.coling-main.627/)Cited by:[§1](https://arxiv.org/html/2610.09446#S1.p2.1),[§2](https://arxiv.org/html/2610.09446#S2.p1.1)\.
- Metcalfeet al\.\(2018\)D\. B\. Metcalfe, T\. D\. G\. Hermans, J\. Ahlstrand, M\. Becker, M\. Berggren, R\. G\. Björk, M\. P\. Björkman, D\. Blok, N\. Chaudhary, C\. Chisholm, A\. T\. Classen, N\. J\. Hasselquist, M\. Jonsson, J\. A\. Kristensen, B\. B\. Kumordzi, H\. Lee, J\. R\. Mayor, J\. Prevéy, K\. Pantazatou, J\. Rousk, R\. A\. Sponseller, M\. K\. Sundqvist, J\. Tang, J\. Uddling, G\. Wallin, W\. Zhang, A\. Ahlström, D\. E\. Tenenbaum, and A\. M\. AbdiPatchy field sampling biases understanding of climate change impacts across the Arctic\.Nature Ecology & Evolution2\(9\),pp\. 1443–1448\.External Links:[Document](https://dx.doi.org/10.1038/s41559-018-0612-5),[Link](https://doi.org/10.1038/s41559-018-0612-5)Cited by:[§1](https://arxiv.org/html/2610.09446#S1.p1.1)\.
- Reinet al\.\(2023\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: a graduate\-level google\-proof q&a benchmark\.External Links:arXiv:2311\.12022Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p3.1)\.
- Renet al\.\(2023\)J\. Ren, Y\. Zhao, T\. Vu, P\. J\. Liu, and B\. LakshminarayananSelf\-evaluation improves selective generation in large language models\.External Links:2312\.09300,[Link](https://arxiv.org/abs/2312.09300)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.09446#S4.SS1.p1.1)\.
- Waddenet al\.\(2022\)D\. Wadden, K\. Lo, B\. Kuehl, A\. Cohan, I\. Beltagy, L\. L\. Wang, and H\. HajishirziSciFact\-Open: towards open\-domain scientific claim verification\.InFindings of EMNLP,pp\. 4719–4734\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.findings-emnlp.347),[Link](https://aclanthology.org/2022.findings-emnlp.347/)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p2.1)\.
- Wanet al\.\(2024\)Y\. Wan, Y\. Liu, A\. Ajith, C\. Grazian, B\. Hoex, W\. Zhang, C\. Kit, T\. Xie, and I\. FosterSciQAG: a framework for auto\-generated science question answering dataset with fine\-grained evaluation\.arXiv preprint arXiv:2405\.09939\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2405.09939),[Link](https://arxiv.org/abs/2405.09939)Cited by:[§1](https://arxiv.org/html/2610.09446#S1.p4.1),[§2](https://arxiv.org/html/2610.09446#S2.p3.1),[§3\.2](https://arxiv.org/html/2610.09446#S3.SS2.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. ChenMMLU\-pro: a more robust and challenging multi\-task language understanding benchmark\.External Links:2406\.01574,[Link](https://arxiv.org/abs/2406.01574)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p3.1)\.
- Wenet al\.\(2024\)B\. Wen, B\. Howe, and L\. L\. WangCharacterizing LLM abstention behavior in science QA with context perturbations\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Miami, Florida, USA,pp\. 3437–3450\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.197),[Link](https://aclanthology.org/2024.findings-emnlp.197/)Cited by:[§1](https://arxiv.org/html/2610.09446#S1.p2.1),[§2](https://arxiv.org/html/2610.09446#S2.p2.1),[§5\.2](https://arxiv.org/html/2610.09446#S5.SS2.p1.1)\.
- Wenet al\.\(2025\)B\. Wen, J\. Yao, S\. Feng, C\. Xu, Y\. Tsvetkov, B\. Howe, and L\. L\. WangKnow your limits: a survey of abstention in large language models\.External Links:2407\.18418,[Link](https://arxiv.org/abs/2407.18418)Cited by:[§5\.2](https://arxiv.org/html/2610.09446#S5.SS2.p1.1),[§5\.2](https://arxiv.org/html/2610.09446#S5.SS2.p3.2)\.
- Yaoet al\.\(2025\)Z\. Yao, A\. Parashar, H\. Zhou, W\. S\. Jang, F\. Ouyang, Z\. Yang, and H\. YuMCQG\-SRefine: multiple choice question generation and evaluation with iterative self\-critique, correction, and comparison feedback\.InProceedings of NAACL\-HLT,pp\. 10728–10777\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.538),[Link](https://aclanthology.org/2025.naacl-long.538/)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p4.1)\.
- Yuet al\.\(2024\)C\. Yu, W\. H\. Lee, L\. T\. Ke, S\. Guo, and Y\. FanAutomating true\-false multiple\-choice question generation and evaluation with retrieval\-based accuracy differential\.InProceedings of the 17th International Natural Language Generation Conference,pp\. 198–212\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.inlg-main.16),[Link](https://aclanthology.org/2024.inlg-main.16/)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p4.1),[§3\.4](https://arxiv.org/html/2610.09446#S3.SS4.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\), Datasets and Benchmarks Track,External Links:[Document](https://dx.doi.org/10.48550/arXiv.2306.05685),[Link](https://arxiv.org/abs/2306.05685)Cited by:[§2](https://arxiv.org/html/2610.09446#S2.p4.1)\.