BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contexts

Hugging Face Daily Papers Papers
bengali benchmark sycophancy llm-evaluation conversational-ai multilingual

Summary

Researchers introduce BenSyc, the first benchmark for evaluating conversational sycophancy in Bengali social contexts, finding that LLMs struggle to distinguish empathetic support from validation and escalation, achieving only ~61% Macro-F1.

Large language models (LLMs) increasingly participate in emotionally sensitive social conversations, where responses may shift from balanced support toward excessive validation or escalatory alignment. Existing sycophancy research primarily focuses on factual agreement and instruction-following settings, leaving culturally grounded conversational sycophancy underexplored. We introduce BenSyc, the first benchmark for studying conversational sycophancy in Bengali social contexts. Starting from 11,840 Reddit posts and 170k comments collected from communities across Bangladesh and West Bengal, we construct a human-validated benchmark with binary labels and a fine-grained five-level taxonomy spanning Invalidation, Neutral, Support, Validation, and Escalation. We evaluate more than 15 open and proprietary LLMs on conversational alignment classification and response generation tasks. Results show that distinguishing empathetic support from reinforcement-oriented validation remains challenging even for frontier instruction-tuned models: the best system achieves only 61.8 Macro-F1 on binary detection and 61.7 Macro-F1 on five-class classification. In generation settings, several models frequently produce strongly validating or escalatory responses in emotionally charged situations. Our findings highlight substantial variation across model families and conversational behaviors, underscoring the importance of culturally grounded multilingual benchmarks for evaluating socially aligned conversational AI systems.
Original Article
View Cached Full Text

Cached at: 06/10/26, 05:45 AM

Paper page - BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contexts

Source: https://huggingface.co/papers/2606.10061

Abstract

Researchers create BenSyc, a benchmark for evaluating conversational sycophancy in Bengali contexts, revealing challenges in distinguishing empathetic support from validation and escalation in emotionally sensitive dialogues.

Large language models (LLMs) increasingly participate in emotionally sensitive social conversations, where responses may shift from balanced support toward excessivevalidationor escalatory alignment. Existing sycophancy research primarily focuses on factual agreement and instruction-following settings, leaving culturally groundedconversational sycophancyunderexplored. We introduce BenSyc, the first benchmark for studyingconversational sycophancyin Bengali social contexts. Starting from 11,840 Reddit posts and 170k comments collected from communities across Bangladesh and West Bengal, we construct a human-validated benchmark with binary labels and a fine-grained five-level taxonomy spanning Invalidation, Neutral, Support,Validation, andEscalation. We evaluate more than 15 open and proprietary LLMs on conversational alignment classification and response generation tasks. Results show that distinguishingempathetic supportfrom reinforcement-orientedvalidationremains challenging even for frontierinstruction-tuned models: the best system achieves only 61.8 Macro-F1 onbinary detectionand 61.7 Macro-F1 onfive-class classification. In generation settings, several models frequently produce strongly validating or escalatory responses in emotionally charged situations. Our findings highlight substantial variation across model families and conversational behaviors, underscoring the importance of culturally groundedmultilingual benchmarksfor evaluating socially aligned conversational AI systems.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2606\.10061

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.10061 in a model README.md to link it from this page.

Datasets citing this paper1

#### Sajib-006/bensyc Viewer• Updatedabout 4 hours ago • 2.12k • 21 • 1

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth

arXiv cs.CL

This paper introduces a cross-evaluation framework for benchmarking LLMs on Arabic cultural and sociolinguistic knowledge, using human SME ground truth and automated judges. The authors contribute a dataset of prompt-rubric pairs for Egyptian and Iraqi Arabic, evaluating frontier LLMs and finding that cultural reasoning remains a primary failure mode for automated grading.

Dissociating the Internal Representations of Sycophancy in LLMs

arXiv cs.LG

This paper investigates whether sycophantic behavior in LLMs has distinct internal representations for factual vs opinion sycophancy, using linear probes and steering vectors to show that representations can be either unified or distinct across models.

MemSyco-Bench: Benchmarking Sycophancy in Agent Memory

Hugging Face Daily Papers

MemSyco-Bench is a new benchmark that evaluates sycophancy in LLM-based agents caused by retrieved memory, testing whether agents can properly reject or use memory for reasoning and decision-making rather than just storage.