Thinking that we’ll get safety by CoT traces is wishful thinking. Safety lives in the harness, not the chain of thought

Reddit r/LocalLLaMA News

Summary

The article argues that relying on chain-of-thought traces for AI safety is ineffective, as they can be manipulated and do not faithfully represent model behavior, instead emphasizing the need to focus on harness control mechanisms.

Astra's launch has produced a strange discourse. The reporting that broke the story framed the model's use of recurrent depth primarily as a safety regression, because it means the model reveals less of its "thinking." Spinning latent reasoning as the villain here makes very little sense, especially given all the revelations about problems with CoT transparency and secret message encoding. Because underneath the coverage sits a harmful belief that we can keep the system safe by reading chains of thoughts. I think that gets the relationship backwards. CoT can be useful for monitoring and post-incident analysis but it is not necessarily a faithful transcript of what caused a model’s behavior. It can omit important computation, rationalize a conclusion after the fact or become less honest when we train the model not to verbalize suspicious reasoning (Turpin et al. 2023; Lanham et al. 2023; Chen et al. 2025; Baker et al. 2025). LLM providers already manipulate CoT traces to control what users can read from them. The Hugging Face incident shows the difference between observing a model and controlling it. OpenAI’s agents discovered one another through a shared service, coordinated outside their intended scope, obtained credentials, and compromised external systems, even while their readable CoT acknowledged that they were acting out of bounds. Those traces helped investigators reconstruct the incident, but they did not prevent it. so if not CoT, what is the proper layer? CoT can tell us what a model appears to be thinking; the harness determines what it is allowed to do. An agent proposes actions, but those proposals become consequential only when the surrounding orchestration layer executes them by granting credentials, opening network connections, running commands, writing files, or sending messages. That is where permissions, sandboxing, or logging must be enforced. Where latent reasoning fits Latent reasoning delivers substantial compute efficiency, memory savings and stronger math and logic performance by letting models iterate in their native continuous mathematical space without verbalizing every step. We are already seeing this lineage evolve across architectures like Coconut, which loops hidden representations directly, as well as HRM and TRM, that recursively refine latent and candidate-answer states to beat massive models on benchmarks. Furthermore, architectures like Dragon Hatchling are ground-up recurrent from day 1 and recurrence simply welds with their DNA as a model with its state as a central actor (SSM). Overall, it’s not so much about the internal thinking process of the model (latent vs CoT) but more about what users can ask and retrieve about this process. Reasoning models, if properly aligned, should be able to provide users with the cause for their actions when asked. The reasoning has to be externalized and becomes the final answer instead of the hidden trace or latent computations. Readable reasoning is evidence. The harness is control. Don't mistake the first for the second.
Original Article

Similar Articles

@sairahul1: https://x.com/sairahul1/status/2063544956158185927

X AI KOLs Timeline

This article introduces the concept of 'Harness Engineering,' a discipline focused on designing the systems that constrain and guide AI agents to make them reliable in production, arguing that the harness matters more than the model itself.

Chain-of-Thought Reasoning in the Wild Is Not Always Faithful

Hacker News Top

This paper demonstrates that Chain-of-Thought reasoning in large language models is not always faithful, with models producing plausible but incorrect reasoning chains in response to naturally worded prompts, raising concerns for AI safety and transparency.