Pigeonholing: Bad prompts hurt models to collapse and make mistakes
Summary
This paper introduces 'pigeonholing,' a phenomenon where bad prompts cause LLMs to collapse and repeat errors, leading to a 38-40% performance drop. Experiments across 10 tasks and 10 models show worsening with more conversation turns, and propose RLVR with synthetic errors as a mitigation.
View Cached Full Text
Cached at: 06/24/26, 07:46 AM
# Pigeonholing: Bad prompts hurt models to collapse and make mistakes Source: [https://arxiv.org/abs/2606.24267](https://arxiv.org/abs/2606.24267) [View PDF](https://arxiv.org/pdf/2606.24267) > Abstract:While in\-context learning is generally shown to be effective in Large Language Models \(LLMs\), bad contexts can cause performance degradation and mode collapse, a phenomenon we call "pigeonholing\." \*\*Unintentionally bad\*\* contexts can happen without malicious jailbreaking intents: For example, a user asks the model to justify an incorrect math theorem or fails to correct the model's buggy code\. Specifically, we investigate \`\`pigeonholing" in two scenarios: \(1\) when the user suggests a solution, and \(2\) when the conversation context includes the assistant's previous \(incorrect\) responses\. Our experiments across 10 verifiable and open\-ended tasks with 10 different models show that pigeonholing manifests in several ways: \(1\) repeating the incorrect answers from context \(leading to 38\-40% performance drop\), \(2\) converging on a narrow set of answers in coding and text generation without exploring alternatives, and \(3\) flipping stance on controversial topics to align with the user or the assistant's previous claims\. We find that pigeonholing worsens almost monotonically with the number of conversation turns \(performance drops by additional 14\+% as repeated mistakes increase from 1 to 5\), and pigeonholing\-induced mode collapse can happen even when the provided example is correct\. As a step toward mitigation, we propose RLVR with synthetic errors which improves models by 43\-60% under bad contexts compared to vanilla RLVR baselines\. ## Submission history From: Hyunji Alex Nam \[[view email](https://arxiv.org/show-email/ca0d1e76/2606.24267)\] **\[v1\]**Tue, 23 Jun 2026 07:52:22 UTC \(994 KB\)
Similar Articles
One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation
This paper empirically demonstrates that single-prompt evaluation of instruction-tuned embedding models is insufficient, as performance varies significantly with prompt phrasing and leaderboard rankings can be manipulated by prompt selection.
Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs
This paper introduces PatternEval, a diagnostic benchmark for evaluating response-pattern failures in hybrid-thinking multimodal large language models, and proposes PatternRL for aligning these patterns through reinforcement learning with specific penalties.
When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion
This paper diagnoses 'pass@k inversion' in reinforcement learning with verifiable rewards (RLVR), where training improves one-shot accuracy but reduces performance under repeated sampling, particularly on boundary prompts where correct trajectories are rare. It proposes a per-problem base anchoring (PBA) method to mitigate this issue.
When Choices Become Risks: Safety Failures of Large Language Models under Multiple-Choice Constraints
Researchers identify a systematic safety failure in LLMs where reformulating harmful requests as forced-choice multiple-choice questions (MCQs) bypasses refusal behavior, even in models that reject equivalent open-ended prompts. Evaluated across 14 proprietary and open-source models, the study reveals current safety benchmarks substantially underestimate risks in structured decision-making settings.
Mimicry without understanding: the origins of decision bias in large language models
This paper investigates how LLMs like ChatGPT-4o and Qwen develop decision biases through faulty mimicry of human behavior, even when preferences are not biased, and shows that scientific descriptions of biases can become self-fulfilling prophecies for LLM responses.