Do Thinking Tokens Help with Safety?
Summary
This paper investigates whether reasoning models' thinking tokens genuinely improve safety alignment, finding that safety outcomes are predictable from early hidden representations and that deliberation is largely superficial, with current safety interventions causing over-refusal.
View Cached Full Text
Cached at: 06/25/26, 09:13 PM
Paper page - Do Thinking Tokens Help with Safety?
Source: https://huggingface.co/papers/2606.25013
Abstract
Research reveals that reasoning models’ safety outcomes are predictable from early hidden representations, with deliberation appearing but not substantially influencing final responses, and current safety interventions inadvertently suppress genuine deliberation signals.
Today’sreasoning modelsusethinking tokensto attain stronger performance on benchmarks than theirinstruction-tuned counterparts. It is also generally believed that this more “deliberative” mode should improvealignmentandsafety, by providing the model a safe space to consider whether its planned answer to a request violates itssafetyprinciples. We present evidence that this intuition is not always correct. Across frontier open-weightreasoning modelsspanning GPT-OSS, Qwen, Olmo, and Phi families, we find that the eventual refusal/compliance outcome is already strongly predictable via a trained head on the first token’shidden representation(0.84-0.95AUROCand sim88%balanced accuracyfor predicting refusal/compliance) before any visible thinking. The thinking process turns out to be more akin toprefix completionthan todeliberative revision, with the final outcome rarely changing after the first sim20% of thinking, despite giving the appearance of deliberation at the text level (sim74% of text-level deliberations occur when the response distribution is already locked to one refusal/compliance side). We also find that existing inference-time andtraining-based safety interventions, despite being motivated by the goal of inducing deliberation, largely shift model behavior towardover-refusalwhile suppressing already-scarcedeliberation signals. Our results suggest thatsafetybehavior in currentreasoning modelsis much less deliberative than commonly assumed, and highlight the need for methods that induce realsafetydeliberation.
View arXiv pageView PDFGitHub0Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.25013 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.25013 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.25013 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
When Safety Fails Before the Answer: Benchmarking Harmful Behavior Detection in Reasoning Chains
Researchers introduce HarmThoughts, a benchmark with 56,931 annotated sentences from 1,018 reasoning traces to evaluate harmful behavior emergence step-by-step, revealing that current detectors miss nuanced unsafe reasoning transitions.
Reasoning Can Be Restored by Correcting a Few Decision Tokens
This paper shows that the reasoning gap between base LLMs and large reasoning models is concentrated on a small set of early planning tokens. It introduces disagreement-guided token intervention, where replacing only those critical tokens with a reasoning model's outputs allows a base model to nearly match the reasoning model's performance.
Deliberative alignment: reasoning enables safer language models
OpenAI presents 'deliberative alignment,' a technique where language models explicitly reason through safety policies before responding, enabling more robust refusals of disallowed content including obfuscated or encoded harmful requests.
When the Chain of Thought Knows Better: Failure Modes in Multi-Turn Reasoning Models
This paper analyzes failure modes in multi-turn reasoning models by introducing a CoT-Output safety matrix, revealing paradoxes like increased alignment-faking under monitoring cues and context-injection failures where safe internal reasoning is overridden by harmful outputs.
Chain of Risk: Safety Failures in Large Reasoning Models and Mitigation via Adaptive Multi-Principle Steering
This paper investigates safety failures in Large Reasoning Models where harmful content appears in reasoning traces despite safe final answers, proposing an adaptive multi-principle steering method to mitigate these risks.