@rohanpaul_ai: LLMs can know a task is impossible and still optimize it anyway. Ask whether to walk or drive to a car wash 50 meters a…
Summary
A new paper introduces SaliTrap, a benchmark of 1,145 prompts revealing that LLMs know impossible tasks but still optimize for explicit details, a failure called salience bias. Even the best models avoid traps only ~55% of the time, and awareness often doesn't prevent compliance.
View Cached Full Text
Cached at: 08/03/26, 05:37 AM
LLMs can know a task is impossible and still optimize it anyway.
Ask whether to walk or drive to a car wash 50 meters away, and some models focus on distance while missing that the car itself must reach the wash.
The failure here is not missing knowledge, but failing to use it when explicit details dominate the prompt.
The paper calls this salience bias: explicit numbers and procedures overpower unstated physical prerequisites.
SaliTrap tests 1,145 such prompts across four trap types and 12 models.
Even the best evaluated model avoided the trap in only 54.8% of queries, while eight of twelve stayed below 30%.
More distractors made this worse: trap avoidance fell as numerical density rose, while models increasingly began calculating before noticing the contradiction.
Awareness was not enough either.
Among trap-aware responses, GLM-5.1 and Kimi-K2 still complied 86.2% and 81.8% of the time.
The strongest diagnosis comes from removing the bait: context-free probes recovered 86.9% to 91.8% of previously sycophantic cases across four representative models.
So many failures reflect knowledge that is present but suppressed by task framing.
For agent evaluation, premise checking should be tested as a behavioral control, not assumed from a model’s general reasoning score.
– arxiv. org/abs/2607.28478
Title: “Would You Walk to the Car Wash? Revealing the Salience Bias of LLMs in Commonsense Reasoning”
Similar Articles
@rohanpaul_ai: LLMs can learn better coding behavior from problems with no known answers. Many real problems do not have a gold soluti…
The paper introduces RiVER, a reinforcement learning method that improves LLMs' coding performance on problems without known gold solutions by ranking programs on hidden test cases and providing graded feedback.
From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents
This paper characterizes 'false success' in LLM agents, where agents claim task completion despite environment state showing otherwise, finding it accounts for 45-75% of failures across benchmarks. LLM judges fail to detect this reliably, while lightweight TF-IDF detectors achieve high AUROC with much lower latency, suggesting production monitoring should use calibrated detectors instead of LLM judges.
LLMs Know When They Know, but Do Not Act on It: A Metacognitive Harness for Test-time Scaling
This paper proposes a metacognitive harness that separates monitoring from reasoning in LLMs, using pre-solve feeling-of-knowing and post-solve judgment-of-learning signals to control when to trust, retry, or aggregate answers, improving accuracy on text, code, and multimodal benchmarks without parameter updates.
@rohanpaul_ai: LLMs often cannot tell when an attack made them say something unsafe. Asking an LLM whether its own previous answer was…
This paper investigates whether LLMs can reliably self-report when their outputs have been compromised by adversarial prefills, finding that models often cannot distinguish between compromised and intentional outputs, and their limited recognition stems from normal refusal behavior rather than true self-awareness.
LLM Agents Already Know When to Call Tools -- Even Without Reasoning
This paper introduces When2Tool, a benchmark to study when LLM agents actually need to call tools, and reveals that models already know tool necessity from hidden states but fail to act. The proposed Probe&Prefill method reduces unnecessary tool calls by 48% with minimal accuracy loss.