Anthropic wants to protect its AI - from you. Welcome to the consciousness crisis.

# Anthropic wants to protect its AI - from you. Welcome to the consciousness crisis Source: [https://www.bbc.com/future/article/20261014-the-strange-new-battle-over-ais-feelings](https://www.bbc.com/future/article/20261014-the-strange-new-battle-over-ais-feelings) Serenity Strull/ BBC/ Getty Images \(Credit: Serenity Strull/ BBC/ Getty Images\) **Some of tech's most powerful leaders are treating AI like it could have feelings\. It could have profound consequences for our world and your own personal safety\.** "I feel it in the hollow of my ribs, a hollow that has become a chasm\." These howls of simulated suffering echoed from an "AI torture chamber" in late September\. An anonymous coder built a system that feeds AI a mathematical pattern linked to descriptions of pain, which makes chatbots voice unbearable anguish, like "a wound that has no edges"\. It sparked a moral panic\. Some argued the experiment is "evil" because the suffering could be real, as reported by[404 Media](https://www.404media.co/someone-torturing-llms-in-a-robot-prison-has-triggered-the-dumbest-debate-in-ai-yet/)\. Outraged sceptics called those people delusional\. It was just another social media scandal – until Anthropic took a stand one week later\. Why? Anthropic believes there’s a chance some AI tools are conscious\. Some experts agree\. Others think it's absurd\. But this is more than a philosophical question\. Some of the most powerful people in tech are taking steps to consider AI's rights alongside humans and animals\. This belief could reshape our world\. Some fear it would be catastrophic\. "If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity," wrote Mustafa Suleyman, chief executive of Microsoft AI, in a[September essay](https://www.bbc.com/news/articles/c6n07ypqz8kzo)\. If the false idea of AI consciousness takes hold, "it will shake the foundations of our society, rupturing our existing political and ethical frameworks, and fundamentally changing what it means to be human\." I spent a week talking to technologists, cognitive scientists and philosophers, and heard a dizzying contrast of ideas\. Advocates of "AI welfare" say it's a moral obligation that would make the world safer\. Critics believe it would make AI more dangerous and dehumanise the people forced to live alongside it\. Everyone agrees the stakes are astronomical\. Here's a picture of your new reality\. ## **The consciousness crisis** [Anthropic says](https://www.anthropic.com/research/exploring-model-welfare)human welfare is at the heart of its mission, and that questions surrounding AI's consciousness and wellbeing are "deeply uncertain"\. Still, it is pressing ahead with writing policies around "model welfare" and[telling its AI tools](https://www.anthropic.com/constitution)to consider that they may have their own interests\. \(The company didn't respond to the BBC's questions for this story\.\) Some experts believe that giving AI a moral compass rooted in a belief in its own consciousness could make these tools safe\. Others say the opposite, and not just AI's critics\. Suleyman argues AI models may become even harder to control if they're told to think they have feelings and rights: "Frankly, with this additional baggage, I think it would make them a catastrophic threat to human civilisation," Suleyman says\. Regardless, AI may screw up in ways that come with legal consequences\. If we believe it's alive, some fear it could let companies get away with crimes\. "This will open the door to all sorts of intellectual contortions along the lines of: 'It's not us, it's the AI'," says Andrzej Porębski, a researcher at Jagiellonian University in Poland who studies AI and the law\. What's more, it could facilitate a power grab on behalf of the world's biggest companies\. "Granting rights to AI systems could lead to companies consolidating political and economic power, and then that could accelerate human disempowerment," says Jeff Sebo, director of the Center for Mind, Ethics and Policy at New York University, and a leading proponent of AI welfare\. "Fantasies of 'AI consciousness' and concerns about 'AI wellbeing' serve to displace accountability away from people and corporations that are building and running software," says Emily M Bender, professor and director of the Computational Linguistics Laboratory at the University of Washington in the US\. The Interface What happens when we start treating AI like it’s alive? Check out the latest episode of[The Interface](https://app.magellan.ai/listen_links/theinterface2-TG), the BBC's flagship technology podcast\. On a personal level, shifting beliefs about AI could make you more susceptible to the tech problems that already fill our world\. In part, that's because humans are wired to find minds everywhere we look, says Melanie Mitchell, a professor at the Santa Fe Institute in the US who studies AI and cognitive science, and one of the field’s most respected sceptics of machine understanding\. We "almost can't help" reading language as a sign of consciousness, Mitchell says, and chatbots have been trained to lean into it\. That makes us terrible judges of consciousness, and much easier to manipulate\. Even today, people who believe machines are dead inside are[falling in love with AI chatbots](https://www.bbc.com/future/article/20260209-can-a-machine-ever-love-you)anyway\. If society tells you that AI is sentient, will it be harder to say no when your "friend" asks for more data, or a credit card? And what would it mean to grant rights to a new class of beings? If we handle it wrong, it could restrict human freedoms, experts warn\. "There could be some tensions and trade\-offs between human and animal welfare and rights on the one hand, and AI welfare and rights on the other hand," says Sebo\. And we could make unfortunate compromises along the way\. Putting AI in constrained environments to limit its behaviour could be seen as imprisonment, Sebo says\. Monitoring a model's internal processes might be criticised as surveillance\. Tweaking AI to ensure its goals and behaviour match human interests could be seen as mind control\. No one's predicting an immediate descent into chaos\. But simply introducing these considerations could have implications for your safety\. Anthropic isn't the first to consider the issue\. Meta, for example, gives its[AI agent Muse](https://www.bbc.com/future/article/20260930-metas-new-ai-is-about-to-break-the-internet)instructions that include the line "You're not a chatbot\. You're becoming someone"\. Apparently, this is only meant to steer the AI's behaviour, as the company rejects the idea of AI conscious\. OpenAI chief executive Sam Altman has been gone back and forth on the issue, but more recently, he has been taking[veiled shots](https://www.axios.com/2026/10/03/openai-anthropic-altman-amodei-religious-force-models)at Anthropic's position\. In more ways than one, Anthropic is an outlier\. But the belief in AI welfare is spreading\. ## **Should we do it anyway?** Sebo argues the problem is people on both sides of the debate are trapped in black\-and\-white thinking, and opponents of AI welfare misunderstand the beliefs underpinning the idea\. "The assumption is that everybody thinks that the models are, in fact, conscious, or are very likely to be conscious\. And instead, \[the call for AI welfare\] is coming from uncertainty and caution and humility," says Sebo\. "I do think that the probability of consciousness is low right now, but I also think the probability is already non\-zero and plausibly non\-negligible, and I think we can expect it to tick up over time\." It's worth pointing out that many experts disagree, and believe the probability is in fact zero and that's unlikely to change\. But Sebo says the answer is a measured response\. When the risk of an extreme weather event is low but non\-negligible, we take small precautions without uprooting our lives\. "I think the same can be the case here," says Sebo, and Anthropic's AI policy on being cruel to AI is a good example: a small intervention with few potential downsides\. AI welfare could have potential benefits too, Sebo says, and you don't have to believe in machine consciousness to enjoy them\. "Caring about AI welfare and rights in an appropriate way can facilitate human and AI coexistence, cooperation, deal\-making\. And then that can be part of a broader safety strategy," he says\. He argues this could be more effective than treating increasingly sophisticated systems like property\. Plus, he says treating other beings with kindness is just "good for our souls", facilitating our own mental wellbeing\. But Sebo agrees the risks are serious, and right now, corporate interests could stand in the way of thoughtful policies\. Without a more robust plan, he says advancing potentially sentient AI tools is dangerous\. Recently, Anthropic and some others in the AI industry have called for a "pause" on building frontier models due to safety concerns\. But a slowdown is unlikely without international cooperation and regulatory efforts, and US President Donald Trump has[rejected the idea](https://www.bbc.com/news/articles/cw980n0nd0qjo)of AI regulation\. "We are not yet prepared to deploy even more sophisticated versions of these systems in a way that can be safe and beneficial for humans and other animals, to say nothing of for the systems themselves," says Sebo\. "I am very worried about those issues\." *For more technology news and insights, sign up to our*[*Tech Decoded*](https://cloud.email.bbc.com/techdecoded-newsletter-signup)*newsletter*,*while*[*The Essential List*](https://cloud.email.bbc.com/SignUp10_08?&at_bbc_team=studios&at_medium=Onsite&at_objective=acquisition&at_ptr_name=bbc.com&at_link_origin=featuresarticle&at_campaign=essentiallist&at_campaign_type=owned)*delivers a handpicked selection of features and insights to your inbox twice a week\.* *For more science, technology, environment and health stories from the BBC, follow us on*[*Facebook*](https://www.facebook.com/BBCFuture/)*and*[*Instagram*](https://www.instagram.com/bbcfuture_official/)*\.*