Tag
This paper introduces CulturalMenuBench, a benchmark with 4,870 items across 10 languages and 18 regions, to test multimodal models' ability to apply culinary knowledge beyond mere recognition. Evaluations reveal a knowledge-application gap where models drop in performance when requiring cultural attribution despite high recognition scores.
PUMA is a Polish-language benchmark with 900 tasks designed to evaluate multimodal AI models on cultural understanding and practical skills across images, audio, and documents.
Introduces SEA-NLI, a culturally grounded NLI benchmark covering eight Southeast Asian countries, revealing low performance of LLMs on culturally specific knowledge, especially in languages and science/technology. Shows that culture-aware prompting helps but chain-of-thought offers limited gains.
This paper demonstrates that LLMs are heavily biased toward English, and shows that continual pre-training does not offer cost advantages over training from scratch for adapting models to other languages, especially for cultural understanding.
OpenAI introduced IndQA, a new benchmark with 2,278 questions across 12 Indian languages and 10 cultural domains, designed to evaluate AI models' understanding of culturally nuanced and reasoning-heavy tasks that existing benchmarks fail to capture. Created with 261 domain experts, IndQA addresses the saturation of existing multilingual benchmarks like MMMLU and focuses on real-world cultural comprehension rather than translation or multiple-choice tasks.