Toxicity on Social Media – The Noisy Room

Hacker News Top News

Summary

A Stanford study analyzing billions of social media posts reveals that only ~3% of users generate severely toxic content, but engagement-driven algorithms disproportionately amplify this minority, distorting public perception and driving self-censorship among the majority.

No content available
Original Article
View Cached Full Text

Cached at: 05/12/26, 09:58 AM

# The Noisy Room Source: [https://thenoisyroom.com/](https://thenoisyroom.com/) It began with a study\. In December of 2025, Stanford researchers[analyzed 2\.2 billion social media posts](https://doi.org/10.1093/pnasnexus/pgaf310)looking for a pattern\. They wanted to know what percentage of users posted*severely*toxic content\. Not rudeness, not sarcasm, but speech that was so hateful that 90% of the world would flag it as being problematic\.1 With this data in hand, they then asked thousands of people to answer a simple question: Take a guess\. What percentage of social media users do you think post severely toxic content? 0%50%100% They were surprised by the results\. They had discovered an enormous reservoir of misperception that had been hidden in plain view\. ### The Bar Here is the simplest version of the problem\. Imagine walking into a bar with a hundred people inside\. Three of them are shouting — about politics, about each other, about whatever gets a reaction\. The other ninety\-seven are talking at a normal volume\. But there's a bouncer at the door, and he gets paid by the minute you spend staring\. So he's wired the three loudest people into the sound system and turned it all the way up\. You walk in, hear the roar, and conclude:*this place is full of lunatics\.*Never hearing the 97 people having normal conversations a few feet away\. You*could*leave, but all your friends are inside\. You're stuck\. This is how social media deals with contentious topics\. The bouncer is an algorithm\. And whether you like it or not, you've been a bystander\. Pick a contentious topic\. This is what your feed might look like\. Reading this feed, you might reasonably conclude that the country is split between unhinged extremes\. It is not\. And the gap between what Americans actually believe and what the feed suggests they believe may be the most consequential thing the platforms haven't shown you\. ### See the Room Let's visualize this as a single room with 100 people inside\. This is what it looks like: 3 users who have posted toxic content 3%→33%On most platforms, ~3% of accounts produce 1/3 of all content Your feedEngagement ranking amplifies high\-reaction content from the prolific few **The actual room\.**3 out of 100 users have ever posted severely toxic content\. This pattern repeats across platforms\. On Twitter/X, toxic tweets receive[~86% more retweets](https://doi.org/10.1126/science.adl4435)and ~27% more visibility than non\-toxic ones,[0\.3% of users shared 80% of all contested news](https://doi.org/10.1126/science.adl4435),14and just[6% of users produce roughly 73% of all political tweets](https://press.princeton.edu/books/hardcover/9780691203423/breaking-the-social-media-prism)\.16On TikTok,[25% of users produce 98% of all public videos](https://www.pewresearch.org/internet/2024/02/22/how-u-s-adults-use-tiktok/)\.15The specific numbers vary\. The dynamic is the same: a small minority of highly active users overwhelms the majority\. After a time consuming content in this room, your brain performs a kind of ambient demography\. The feed becomes a sort of census\. You conclude — logically — that the behavior must be widespread\. The room might just be full of extreme people\! Maybe most people*do*believe these crazy things\. ### This is not just about what we see on social media If this were just about tone of our social posts, it wouldn't matter very much\. But this distortion ends up causing some seriously bad patterns of behavior\. Pattern 1The Majority Goes SilentWhen the majority of people looks at the feed and assumes they're outnumbered, people will often[self\-censor](https://doi.org/10.1111/j.1460-2466.1974.tb00367.x)\.3The dynamic[replicates on social media](https://doi.org/10.1016/j.techsoc.2023.102431)17— fear of social isolation suppresses opinion expression on platforms where it's perceived to be unwelcome\. They go quiet, or they leave a platform entirely\. They cede the space to users with more extreme politics\. Pattern 2The Loud Minority Thinks It's the MajorityThe minority who aggressively post end up with their own distortion – believing they are part of the majority\.5 A[study of 17 extremist forums](https://doi.org/10.1093/poq/nfn056)found the same pattern: the more someone posted, the more they believed the public agreed with them\. More engaged participation bred false consensus\. Pattern 3Everyone Gets Each Other WrongBoth sides develop[wildly inaccurate beliefs](https://doi.org/10.1086/697253)about who the other side actually is\.6See how some of your own beliefs line up: What percentage of**Democratic supporters**do you think are LGBTQ? 0%50%100% What percentage of**Republican supporters**do you think earn over $250,000 a year? 0%50%100% The distortion extends to policy beliefs\. Step through to see the perception gap on**the issue of immigration\.** Source: More in Common \(2019\) & Moore\-Berg et al\., PNAS 2020\. Illustrative\. Pattern 4Politicians Follow the Perceived Room, Not the Real OneElected officials are very good at sensing political sentiment\. It's literally their job\. \(They are*not*elected to correct people's beliefs\.\) Politicians who can build a coalition about a*perceived belief*are more likely to win\. They position themselves against an opponent that doesn't exist, but their supporters*think exists\.* And remember: most of our politics now happens on social media\. Candidates often read the same distorted feed\. They are unlikely to change their minds\. The window of discourse shifts\. Not because opinions changed, but because*perceptions*of opinions did\. Pattern 5Misperception Turns into HostilityWhen you believe the other side is extreme, you become[more willing to treat them as a threat](https://doi.org/10.1073/pnas.2116851119)\.7 Both Democrats and Republicans**vastly overestimate how many on the other side support political violence\.**The result is a populace primed to assume the other side is ready to do horrible things\. "What percentage of the other side supports political violence?" Democrats believe estimate of Republicans support political violence Republicans believe estimate of Democrats support political violence Both sides were wrong by**3 to 4 times**\. When researchers corrected these beliefs, partisan hostility dropped\. Each step feeds the next\. The distortion is self\-reinforcing\. ### Knowing Isn't Enough Okay\. So now you*know*that a small minority dominates the feed\. You know that Republicans and Democrats actually have a far more nuanced set of opinions about contested issues\. **Does that fix it?**Not really\. You also know that*everyone else*doesn't know it\. And if the world continues operating as if the distortion is real, you should probably act the same — even though you know it's wrong\. The room hasn't changed, even if you know people inside it are confused\. This is called a**common knowledge problem**\. You've read the stat\. But you have no idea who else has\. The feed still looks the same\. You still assume you're outnumbered\. You stay quiet\. Steven Pinker lays this out cleanly in his excellent recent book*[When Everyone Knows That Everyone Knows](https://www.simonandschuster.com/books/When-Everyone-Knows-That-Everyone-Knows/Steven-Pinker/9781668011577)*\.8Learning a fact changes what you*know*\. Seeing it displayed publicly — where everyone else can see it too — where you know others can also see it, changes what everyone knows, and subsequently how they*act*\. Social media has no public square\. It has 300 million private windows, each showing a different distortion of the same room\. Illuminating the common thoughts between us has the potential to radically change it\. ### The Idea So what can we do about this? Fortunately, there's some good evidence showing how it can be fixed\. Multiple studies show that when misperceptions are corrected in a public way, hostility drops\. Mernyk et al\. found that[a single correction reduced partisan hostility for a full month](https://doi.org/10.1073/pnas.2116851119)\.7Lee et al\. found that[correcting overestimates of toxic users improved how people felt about their country and each other](https://doi.org/10.1093/pnasnexus/pgaf310)\.1 **We can do this today\.** Imagine every post on a contested topic had a quiet link beneath it\. Not a fact check, a label, or a warning\. Instead — what if it had a***Community Check?*** ↑click hereA***Community Check***is an open\-source design layer that could be deployed across social media, beneath contentious posts, to help users understand how other people on the platform \(or the nation\) actually feel about an issue\. It is a way of quickly adding context to the most hot\-button viral issues, giving people more visibility into the opinions of the public\. ### The Idea in Action Let's explore this intervention with a topic that cuts across political identity: #### Money in Politics On the surface, this seems contentious\. But it's actually a supermajority issue:**81%**are concerned about the influence of money on elections, including**78% of Republicans**and**90% of Democrats**\.**75%**say unlimited spending weakens democracy\. Only**15%**believe unlimited political spending is protected free speech\. And yet, very little changes, largely because everyone assumes the other side is fine with it\. The feed is full of people defending their team's donors and attacking the other team's\. It might look like a 50/50 partisan battle, but it's not\. It's a majority consensus that cannot see itself\. **What if you could see this consensus?** @real\_talk\_politics · 2h Everyone complains about money in politics but the second their candidate gets a massive donation they shut up real fast\. You don't hate money in politics\. You hate when the OTHER side has more of it\. ♡ 11,847💬 6,203↻ 2,891 ↑click here***Community Check***draws from a**random sample of platform users \+ robust national polls**, surveyed independently of the content\. The sample is statistically representative\. The results update continuously\. And critically:*everyone sees the same numbers*\. ### Why This Isn't Fact Checking or Audience Polling Traditional fact\-checking is a top\-down approach that often feels like it's dictating from above\. This is hard for people to stomach\. Content moderation for many years now has been perceived as removing speech\. This simply adds context, much like the crowdsourced feature[Community Notes](https://communitynotes.x.com/guide/en/about/introduction)\(an inspiration for this project\)\. Nor is this just a user\-poll under a post\. Instead it's drawing from all platform users, coupled with statistically significant national surveys\. It's an actual window into the views of the majority, not just the views of those looking at the post\. ### It Works for Video Too Short\-form video is the fastest\-growing vector for political distortion\. The same dynamic applies — a small minority of creators produce the vast majority of political content — but video bypasses the pause that text gives you\. Community Check can adapt\. Tap through to see how\. Money IS free speech\. Deal with it\. 🇺🇸 Citizens United was CORRECT @liberty\_caucus\_tvFollow \#FreeSpeech \#CitizensUnited 🔥 A political video crosses the engagement threshold\. 51K views, 612 comments \(1\.2%\), 1\.7K shares \(3\.4%\)\. The feed shows outrage\. But what do people actually think? [See technical specs for how it works below ↓](https://thenoisyroom.com/#tech-spec) ### We Could Do This Now Platforms already have a lot of these capabilities\. They already survey users\. They even know how to run sophisticated polls\. There are a few technical details to work out \([spec here](https://thenoisyroom.com/#tech-spec)\), but this is not a hard problem to solve\. The unseen majority is the public\. And the public deserves to know itself\. A tiny minority, dominating the feed\. That's all it ever was\. The rest of us were here the whole time, quiet and decent and waiting to be seen\. [Follow my other work here](https://tobias.substack.com/) Community Check is a free and open specification\. The complete technical spec, research base, and open questions are published for researchers, engineers, and platform designers to stress\-test and build on\. Please steal it with attribution\. [View on GitHub](https://github.com/rosestt/communitycheck) ### Common Questions ### Technical Specification How Community Check would work in practice, from data sources to platform integration\.

Similar Articles

Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation

arXiv cs.CL

This paper presents a large-scale audit of recommendation biases in LLM-based content curation across OpenAI, Anthropic, and Google using 540,000 simulated selections from Twitter/X, Bluesky, and Reddit data. The study finds that LLMs systematically amplify polarization, exhibit distinct toxicity handling trade-offs, and show significant political leaning bias favoring left-leaning authors despite right-leaning plurality in datasets.

IYKYK (But AI Doesn't): Automated Content Moderation Does Not Capture Communities' Heterogeneous Attitudes Towards Reclaimed Language

arXiv cs.CL

Researchers from UCLA examine how automated content moderation tools, including Perspective API, fail to distinguish between reclaimed and hateful uses of slurs for LGBTQIA+, Black, and women communities. The study finds low inter-annotator agreement even among in-group members and poor alignment between community judgments and AI moderation tools, highlighting the need for context-sensitive approaches.

Lessons learned on language model safety and misuse

OpenAI Blog

OpenAI shares lessons learned on language model safety and misuse, discussing challenges in measuring risks, the limitations of existing benchmarks, and their development of new evaluation metrics for toxicity and policy violations. The post also highlights concerns about labor market impacts and the need for continued research on measuring social effects of AI deployment at scale.