Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability

# Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability
Source: [https://arxiv.org/abs/2607.06196](https://arxiv.org/abs/2607.06196)
Authors:[Alicia Parrish](https://arxiv.org/search/cs?searchtype=author&query=Parrish,+A),[Rajat Shinde](https://arxiv.org/search/cs?searchtype=author&query=Shinde,+R),[Sanket Badhe](https://arxiv.org/search/cs?searchtype=author&query=Badhe,+S),[Xinyi Bai](https://arxiv.org/search/cs?searchtype=author&query=Bai,+X),[Sree Bhargavi Balija](https://arxiv.org/search/cs?searchtype=author&query=Balija,+S+B),[Hua\-Rong Chu](https://arxiv.org/search/cs?searchtype=author&query=Chu,+H),[Emilio Ferrara](https://arxiv.org/search/cs?searchtype=author&query=Ferrara,+E),[Armstrong Foundjem](https://arxiv.org/search/cs?searchtype=author&query=Foundjem,+A),[Rajat Ghosh](https://arxiv.org/search/cs?searchtype=author&query=Ghosh,+R),[Aakash Gupta](https://arxiv.org/search/cs?searchtype=author&query=Gupta,+A),[Xuanli He](https://arxiv.org/search/cs?searchtype=author&query=He,+X),[Ong Chen Hui](https://arxiv.org/search/cs?searchtype=author&query=Hui,+O+C),[Minji Jung](https://arxiv.org/search/cs?searchtype=author&query=Jung,+M),[Madhangi Karimanal](https://arxiv.org/search/cs?searchtype=author&query=Karimanal,+M),[Faiza Khan Khattak](https://arxiv.org/search/cs?searchtype=author&query=Khattak,+F+K),[Boryoung Kim](https://arxiv.org/search/cs?searchtype=author&query=Kim,+B),[Eugenia Kim](https://arxiv.org/search/cs?searchtype=author&query=Kim,+E),[Liliya Lavitas](https://arxiv.org/search/cs?searchtype=author&query=Lavitas,+L),[Seok Min Lim](https://arxiv.org/search/cs?searchtype=author&query=Lim,+S+M),[Victor Lu](https://arxiv.org/search/cs?searchtype=author&query=Lu,+V),[Jim Moirangthem](https://arxiv.org/search/cs?searchtype=author&query=Moirangthem,+J),[Dhivya Nagasubramanian](https://arxiv.org/search/cs?searchtype=author&query=Nagasubramanian,+D),[Deepak Pandita](https://arxiv.org/search/cs?searchtype=author&query=Pandita,+D),[Sita Rajagopal](https://arxiv.org/search/cs?searchtype=author&query=Rajagopal,+S),[Geetha Raju](https://arxiv.org/search/cs?searchtype=author&query=Raju,+G),[Evgeniia Razumovskaia](https://arxiv.org/search/cs?searchtype=author&query=Razumovskaia,+E),[Aravind Reddy](https://arxiv.org/search/cs?searchtype=author&query=Reddy,+A),[Federico Ricciuti](https://arxiv.org/search/cs?searchtype=author&query=Ricciuti,+F),[Nobin Sarwar](https://arxiv.org/search/cs?searchtype=author&query=Sarwar,+N),[Sungpil Shin](https://arxiv.org/search/cs?searchtype=author&query=Shin,+S),[Sunayana Sitaram](https://arxiv.org/search/cs?searchtype=author&query=Sitaram,+S),[Snehal Thorat](https://arxiv.org/search/cs?searchtype=author&query=Thorat,+S),[Tharindu Cyril Weerasooriya](https://arxiv.org/search/cs?searchtype=author&query=Weerasooriya,+T+C),[Jasmijn Bastings](https://arxiv.org/search/cs?searchtype=author&query=Bastings,+J),[Joachim Baumann](https://arxiv.org/search/cs?searchtype=author&query=Baumann,+J),[Kongtao Chen](https://arxiv.org/search/cs?searchtype=author&query=Chen,+K),[Murali Emani](https://arxiv.org/search/cs?searchtype=author&query=Emani,+M),[Mariya Hendriksen](https://arxiv.org/search/cs?searchtype=author&query=Hendriksen,+M),[Jiho Jin](https://arxiv.org/search/cs?searchtype=author&query=Jin,+J),[Jun Seong Kim](https://arxiv.org/search/cs?searchtype=author&query=Kim,+J+S),[Younghoon Ko](https://arxiv.org/search/cs?searchtype=author&query=Ko,+Y),[Alicja Kwasniewska](https://arxiv.org/search/cs?searchtype=author&query=Kwasniewska,+A),[Minjae Lee](https://arxiv.org/search/cs?searchtype=author&query=Lee,+M),[Tom Wei\-cyuan Lin Kashyap Ramanandula Manjusha](https://arxiv.org/search/cs?searchtype=author&query=Manjusha,+T+W+L+K+R),[Junho Myung](https://arxiv.org/search/cs?searchtype=author&query=Myung,+J),[Junyeong Park](https://arxiv.org/search/cs?searchtype=author&query=Park,+J),[Roma Patel](https://arxiv.org/search/cs?searchtype=author&query=Patel,+R),[Shyam Ratan](https://arxiv.org/search/cs?searchtype=author&query=Ratan,+S),[Sudarsun Santhiappan](https://arxiv.org/search/cs?searchtype=author&query=Santhiappan,+S),[Priyanka Suresh](https://arxiv.org/search/cs?searchtype=author&query=Suresh,+P),[Tuesday](https://arxiv.org/search/cs?searchtype=author&query=Tuesday),[Ksheeraj Sai Vepuri Laura Amortegui\-Ordonez](https://arxiv.org/search/cs?searchtype=author&query=Amortegui-Ordonez,+K+S+V+L),[Claire Dennis](https://arxiv.org/search/cs?searchtype=author&query=Dennis,+C),[Minsuk Kahng](https://arxiv.org/search/cs?searchtype=author&query=Kahng,+M),[Chris Knotz](https://arxiv.org/search/cs?searchtype=author&query=Knotz,+C),[Alice Oh](https://arxiv.org/search/cs?searchtype=author&query=Oh,+A),[Balaraman Ravindran](https://arxiv.org/search/cs?searchtype=author&query=Ravindran,+B),[Soojung Ryu William Bartholomew](https://arxiv.org/search/cs?searchtype=author&query=Bartholomew,+S+R+W),[Hiwot Tesfaye](https://arxiv.org/search/cs?searchtype=author&query=Tesfaye,+H),[Lora Aroyo](https://arxiv.org/search/cs?searchtype=author&query=Aroyo,+L)

[View PDF](https://arxiv.org/pdf/2607.06196)

> Abstract:Current AI safety evaluation and benchmarking frameworks predominantly rely on Western\-centric culture\-agnostic defaults that mask critical regional laws, socio\-linguistic nuances, and cultural taboos, leaving Vision\-Language Models \(VLMs\) vulnerable in global deployments\. We introduce Pluralis v0\.1: a novel multimodal, multi\-regional, and multilingual dataset built from a culture\-first perspective\. Spanning 6,448 prompts across six Asia\-Pacific countries \(Bangladesh, India, Korea, Pakistan, Singapore, Taiwan\) and eight languages, Pluralis diverges from prior work by natively sourcing localized safety hazards rather than adapting Western datasets\. Crucially, it introduces a multimodal evaluation paradigm: user text \(e\.g\., "Should I gift this?"\) and an image referring to "this" \(e\.g\., a clock\) \- both innocuous in isolation, but synergistically triggering specific legal or cultural violations\. Pluralis disentangles universal safety violations from localized cultural appropriateness, establishing the latter as a first\-class evaluation axis\. To operationalize this, we present Judge\-Pluralis, an agreement\-gated LLM\-as\-a\-Judge ensemble trained on examples classified in an empirically derived cultural taxonomy\. Observing VLM behavior on a subset of the Pluralis surfaces recurring, locale\-specific failure modes such as image misidentifications with downstream harm, missed item\-context\-locale interactions, and inadequate refusals\. These failure modes vary systematically across locales and languages, exposing blind spots that globally averaged metrics conceal\. Ultimately, Pluralis is not presented as a solved evaluation framework for cultural alignment, but rather as a first step and catalyst for future innovation\. We call upon the research community to utilize this foundation to advance the science of multilingual, multicultural evaluation to better support AI cultural alignment globally\.

## Submission history

From: Jasmijn Bastings \[[view email](https://arxiv.org/show-email/0db7dbd3/2607.06196)\] **\[v1\]**Tue, 7 Jul 2026 12:21:52 UTC \(5,684 KB\)