Tag
The paper introduces CultureLens, a reference-grounded benchmark and mechanistic study of how LLMs use cultural information, showing that stated contradictory norms sharply reduce aligned answers and that supervised fine-tuning can largely mitigate this vulnerability.
This paper proposes using literary interpretation as a setting to evaluate cultural reasoning depth in language models, emphasizing interpretive depth and evidence-centered benchmarks over conventional metrics.
Wazobia Eval is a benchmark designed to evaluate AI models on emotion understanding, sarcasm detection, and cultural reasoning in Nigerian Pidgin, aiming to advance NLP capabilities for this language.
NARU is a benchmark for evaluating narrative evolution and cultural reasoning in Japanese extreme long-form videos, constructed through a hierarchical annotation pipeline and native-speaker verification to address gaps in current benchmarks.
ADAGE is a language-agnostic pipeline for constructing challenging, translation-free benchmarks for abstract analogical reasoning. It reveals a consistent cultural reasoning gap in models across Arabic, Amharic, and Japanese.