LLM Agents Predict Social Media Reactions but Do Not Outperform Text Classifiers: Benchmarking Simulation Accuracy Using 120K+ Personas of 1511 Humans
Summary
Large-scale study finds LLM agents can predict individual social-media reactions with 70.7 % accuracy but still lag behind simple TF-IDF classifiers, highlighting both manipulation risks and policy-simulation utility.
View Cached Full Text
Cached at: 04/23/26, 10:02 AM
# LLM Agents Predict Social Media Reactions but Do Not Outperform Text Classifiers: Benchmarking Simulation Accuracy Using 120K+ Personas of 1511 Humans Source: [https://arxiv.org/abs/2604.19787](https://arxiv.org/abs/2604.19787) [View PDF](https://arxiv.org/pdf/2604.19787) > Abstract:Social media platforms mediate how billions form opinions and engage with public discourse\. As autonomous AI agents increasingly participate in these spaces, understanding their behavioral fidelity becomes critical for platform governance and democratic resilience\. Previous work demonstrates that LLM\-powered agents can replicate aggregate survey responses, yet few studies test whether agents can predict specific individuals' reactions to specific content\. This study benchmarks LLM\-based agents' accuracy in predicting human social media reactions \(like, dislike, comment, share, no reaction\) across 120,000\+ unique agent\-persona combinations derived from 1,511 Serbian participants and 27 large language models\. In Study 1, agents achieved 70\.7% overall accuracy, with LLM choice producing a 13 percentage\-point performance spread\. Study 2 employed binary forced\-choice \(like/dislike\) evaluation with chance\-corrected metrics\. Agents achieved Matthews Correlation Coefficient \(MCC\) of 0\.29, indicating genuine predictive signal beyond chance\. However, conventional text\-based supervised classifiers using TF\-IDF representations outperformed LLM agents \(MCC of 0\.36\), suggesting predictive gains reflect semantic access rather than uniquely agentic reasoning\. The genuine predictive validity of zero\-shot persona\-prompted agents warns against potential manipulation through easily deploying swarms of behaviorally distinct AI agents on social media, while simultaneously offering opportunities to use such agents in simulations for predicting polarization dynamics and informing AI policy\. The advantage of using zero\-shot agents is that they require no task\-specific training, making their large\-scale deployment easy across diverse contexts\. Limitations include single\-country sampling\. Future research should explore multilingual testing and fine\-tuning approaches\. ## Submission history From: Ljubisa Bojic \[[view email](https://arxiv.org/show-email/6ba6425b/2604.19787)\] **\[v1\]**Tue, 31 Mar 2026 19:27:59 UTC \(1,491 KB\)
Similar Articles
Evaluating LLMs as Human Surrogates in Controlled Experiments
This paper evaluates whether off-the-shelf LLMs can reliably simulate human responses in controlled behavioral experiments by comparing LLM-generated data with human survey responses on accuracy perception. The findings show that while LLMs capture directional effects and aggregate belief-updating patterns, they do not consistently match human-scale effect magnitudes, clarifying when synthetic LLM data can serve as behavioral proxies.
Can Multi-Agent LLMs Identify Their Peers? Stylometric Fingerprinting in Role-Constrained Political Analysis
This paper investigates whether LLMs can identify their own model family from stylometric fingerprints in role-constrained political analysis texts, even after prompt-level anonymization. The findings confirm that anonymization is insufficient and have implications for EU AI Act compliance and multi-agent system validation.
Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench
Introduces ConsumerSimBench, a benchmark for evaluating LLMs' ability to reconstruct crowd-level consumer reactions from real Chinese social media topics. Tests show frontier models cover only 47.8% of real reaction criteria, highlighting a gap between technical benchmark performance and social intuition.
Human Psychometric Questionnaires Mischaracterize LLM Behavior
This paper finds that human psychometric questionnaires fail to reliably predict LLM behavior in real-world interactions, and proposes generation-based profiling as a more accurate alternative.
Re-Centering Humans in LLM Personalization
This paper studies the gap between synthetic and human data for evaluating LLM personalization across three stages: attribute extraction, relevance matching, and response generation. Results show models perform worse on real human data, and the authors introduce lightweight training interventions to improve alignment.