Tag
SciR is a new controllable benchmark for evaluating LLMs on scientific reasoning including deduction, induction, and causal abduction, with parametric control over extraction and inference difficulty. Tests show both axes degrade performance across models, with reasoning models like DeepSeek-R1 outperforming instruct models on inference.
Introduces Reasoning Primitive Induction, a method that mines successful ReAct traces to cluster recurrent reasoning moves into typed pseudo-tools, outperforming the original agent by tens of percentage points on benchmarks.
This paper investigates the conflict between instruction-following and pattern completion in LLMs, finding that instruction-following is brittle under induction pressure and varies widely across models, with output diversity being the primary factor for robustness.
The article explores the concept of illusions of understanding in scientific practice, discussing how ambiguous language, incomplete causal accounts, and satisfying but incomplete explanations can lead scientists to overlook deeper understanding.