SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science
Summary
SCICONVBENCH is a benchmark that evaluates LLMs on multi-turn clarification for ill-posed scientific queries across computational science domains, finding that even frontier models struggle with disambiguation and frequently make silent assumptions.
View Cached Full Text
Cached at: 05/19/26, 10:34 PM
Paper page - SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science
Source: https://huggingface.co/papers/2605.18630
Abstract
SCICONVBENCH evaluates large language models’ ability to handle ill-posed scientific queries through multi-turn dialogue, focusing on clarifying ambiguous requests and resolving inconsistent information across computational science domains.
Large Language Models(LLMs) are increasingly deployed as scientific AI as- sistants, and a growing body of benchmarks evaluates their capabilities across knowledge retrieval, reasoning, code generation, and tool use. These evaluations, however, typically assume the scientific problem is already well-posed, whereas practical scientific assistance often begins with an ill-posed user request that must be refined through dialogue before any computation, analysis, or experiment can be carried out reliably. We introduce SCICONVBENCH, a benchmark for multi- turn clarification inscientific task formulationacross fourcomputational scienceproblem domains:fluid mechanics,solid mechanics,materials science, and par- tial differential equations (PDEs). SCICONVBENCH targets two complementary capabilities: eliciting missing information (disambiguation) and detecting and correcting erroneous requests containing internally contradictory information (in- consistency resolution). Our benchmark pairs a structuredtask ontologywith arubric-based evaluationframework, enabling systematic measurement of LLM per- formance across three dimensions: clarification behavior,conversational grounding, andfinal-specification fidelity. Current frontier models perform relatively well oninconsistency resolution, but even the best model resolves only 52.7% of thedisambiguationcases influid mechanics. We further find that frontier LLMs fre- quently make silent assumptions and performimplicit specification repairsthat are not grounded in the conversation with users. SCICONVBENCH establishes a foundation for evaluating the upstream conversational reasoning that a reliablecomputational scienceassistant requires. The code and data can be found at https://github.com/csml-rpi/SciConvBench.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2605\.18630
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.18630 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.18630 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.18630 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists
Introduces SDABench, a benchmark evaluating LLMs on six scientific analysis capabilities across five domains, finding models struggle with tasks requiring assumption selection and mechanistic reasoning.
Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messages
The article introduces VSysBench, a benchmark for evaluating multimodal large language models on constraint compliance and answer correctness under system messages, finding that system messages reduce task accuracy and compliance varies between open-weight and proprietary models.
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
SciR is a new controllable benchmark for evaluating LLMs on scientific reasoning including deduction, induction, and causal abduction, with parametric control over extraction and inference difficulty. Tests show both axes degrade performance across models, with reasoning models like DeepSeek-R1 outperforming instruct models on inference.
LabyrinthBench: a local-focused, judge-free LLM benchmark that measures context recall under interference for multi-step agentic tasks.
LabyrinthBench is a local, judge-free LLM benchmark that deterministically measures context recall under interference for multi-step agentic tasks. Initial experiments show that wiping context and re-injecting gate answers helped 7 of 9 models but backfired on 2, highlighting the complexity of context management strategies.
Benchmarking LLM Competence on Logical Inference over Probability Operators
This paper introduces a benchmark of 14,320 procedurally-generated prompts for evaluating LLMs on logical inference over probability operators like 'probably', 'might', and 'must'. Testing 29 models, the authors find systematic answer biases and show that only 9 exceed random chance.