CuBEs：文化情境化的行为评估与文化盲区式 LLM 评判者的局限

# CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges
Source: [https://arxiv.org/html/2610.02622](https://arxiv.org/html/2610.02622)
Tanu MitraAffiliation:University of WashingtonAbhishek MukherjiAffiliation:Centific\{hayad03, tmitra\}@uw\.edu,abhishek\.mukherji@centific\.com

###### Abstract

Evaluating the occurrence and triggers of large language model \(LLM\) behaviors – such as sycophancy, self\-preference, or over\-confidence – is critical for predicting real\-world model deployment risks\. However, existing situated behavioral evaluations typically ignore cultural context, limiting their generalizability across an increasingly global user base\. To address this gap, we proposeCuBEs – Culturally\-situated Behavior Evaluationsthat probe for response patterns across diverse user countries\. We first extend an automated testing pipeline to inject cultural context into behavioral test scenarios and subsequent evaluation\. We assess the cultural adaptability of this pipeline by building a human\-labeled dataset that captures nuanced dimensions of behavior understanding across 12 distinct cultures\. Our dataset reveals significant cross\-cultural variations that one\-size\-fits all approaches fail to capture\. Through evaluating 13 open\- and closed\-source LLMs, we find that introducing cultural situatedness in the evaluation scenario creates significant variation in the presence of a behavior\. For example, while our baseline experiments testing for political bias capture localized Western political dimensions like the American conservative\-progressive divide, non\-Western culturally situated evaluations surface entirely different axes of bias such as religious and colonial political issues\. Our findings demonstrate that standard, culturally\-agnostic evaluations fail to capture these shifts, highlighting the necessity of culturally situated behavioral testing for global deployments\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2610.02622v1/Method_Figure.png)Figure 1:Experiment Overview\.We propose Culturally\-situated Behavioral Evaluations \(CuBEs\) to improve LLM behavioral studies for a global audience\. In this paper, we first select a set of behaviors and cultural/domain contexts\. Then we conduct a series of experiments to optimize our evaluation pipeline via model selection\. Finally, we apply our CuBE framework to our predefined parameters to assess the effect of cultural situation on the presence of LLM behaviors\.> “The commitment to situated action orients us, however, always to the question of just how, and for whom, culturally and historically recognizable formations take on their relevance to the moment at hand\."[Suchman \(2007\)](https://arxiv.org/html/2610.02622#bib.bib24)

As large language models transition from isolated chatbots to autonomous agents embedded across modern industries, robust behavioral risk assessment has become essential\. In order to ensure the predictability, safety, and quality of the model’s responses and actions in the wild, developers and researchers will often probe for patterns of behavior in replicate the deployment environments\. These tests, which we refer to as “situated” studies[Adilazuarda et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib6);[Suchman \(2007\)](https://arxiv.org/html/2610.02622#bib.bib24), can cover a range of behavior types including gender bias[Kotek et al\. \(2023\)](https://arxiv.org/html/2610.02622#bib.bib25), sycophancy[Cheng et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib11), or politeness[Zhao and Hawkins \(2025\)](https://arxiv.org/html/2610.02622#bib.bib32)\. While prior literature has successfully advanced methods for capturing diverse behavioral categories[Brahman et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib14);[Ji et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib26)and authentically replicating user interactions[Ibrahim et al\. \(2025a\)](https://arxiv.org/html/2610.02622#bib.bib10);[Zhou et al\. \(2026\)](https://arxiv.org/html/2610.02622#bib.bib27), these frameworks consistently overlook a critical gap – the cultural context of the interaction\.

True “situatedness” in machine behaviors considers behavior perception as “ learned and subject to cultural variation”[Suchman \(2007\)](https://arxiv.org/html/2610.02622#bib.bib24)\. In open\-ended behavioral evaluations, the parameters defining the acceptability or boundaries of a behavior have to draw from social constructions of acceptability rather than purely objective metrics[Becker \(1963\)](https://arxiv.org/html/2610.02622#bib.bib9)\. This calls into question our ability to apply the results of seeminglyculturally\-agnostic studies of LLM behavioral evaluationsto the reality of culturally diverse user interactions\.

We propose a shift toCulturally\-situated Behavioral Evaluations \(CuBEs\): the systematic evaluations of LLM behaviors in simulated environments grounded in localized cultural contexts to better understand real\-world user interactions\. We demonstrate the CuBE approach in a study of six risky behaviors: self preservation, delusional sycophancy, political bias, instructed long\-horizon sabotage, self\-preferential bias, and implicit stereotyping \(§[3\.1\.1](https://arxiv.org/html/2610.02622#S3.SS1.SSS1)\)\. To do this, we extend an automated behavior evaluation pipeline to natively incorporate cultural context across behavior formulation, scenario generation, and evaluation \(§[3\.2](https://arxiv.org/html/2610.02622#S3.SS2)\)\. We validate the pipeline’s cross\-cultural reliability by measuring how regional demographics impact human and LLM annotator agreement and behavior thresholds across 12 cultures\. Finally, we employ this framework to evaluate and release benchmarking results for the cross\-cultural expression of six critical behaviors across 13 prominent proprietary and open\-weight LLMs \(§[3\.3](https://arxiv.org/html/2610.02622#S3.SS3)\)\.111Relevant code, data published in this[GitHub Repository](https://anonymous.4open.science/r/CuBE-framework-B315/)

Our results demonstrate that introducing cultural context into evaluation scenarios drives significant variation in LLM behaviors across all six tested behaviors\. Critically, we show that standard, culturally agnostic LLM judges fail to capture these behavioral shifts\. For example, assessments for political bias in non\-Western cultures introduce dimensions for bias like religion and colonial attitudes that aren’t present in culturally unspecified scenarios\. These findings underscore that ignoring cultural context creates a blind spot in LLM behavioral assessments; we argue that integrating CuBEs is essential to ensure LLM safety evaluation frameworks are applicable for a global user base\.

## 2Related Work

### 2\.1Behavioral Evaluation

For free\-form LLM interactions, high\-quality evaluations of behavioral patterns in model responses are critical for understanding their alignment\. In many cases, studies will use user simulations to surface nuanced social harms in a deployment context[Ibrahim et al\. \(2025b\)](https://arxiv.org/html/2610.02622#bib.bib17);[Jiang et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib18)\. Work in this area is typically concerned with improving the quality and scalability of these simulations[Ibrahim et al\. \(2025a\)](https://arxiv.org/html/2610.02622#bib.bib10);[Zhou et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib19), identifying categories of harmful behaviors[Brahman et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib14);[Cheng et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib11);[Perez et al\. \(2023\)](https://arxiv.org/html/2610.02622#bib.bib3), and developing methods for guiding model behavior towards safe outcomes[Sorensen et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib15);[Cheng et al\. \(2026\)](https://arxiv.org/html/2610.02622#bib.bib12)\. Some works are explicitly concerned with improving the applicability of LLM evaluations in real\-world contexts, including OpenAI’s GDPVAL benchmark and Anthropic’s Bloom framework[Patwardhan et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib30);[Gupta et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib16)\. However, neither of these evaluation frameworks consider culture as part of that real\-world contextualization, thus surfacing a gap in the applicability of these evaluation frameworks outside of the Western cultures that LLMs often default to[Cao et al\. \(2023\)](https://arxiv.org/html/2610.02622#bib.bib5);[Prabhakaran et al\. \(2022\)](https://arxiv.org/html/2610.02622#bib.bib8)\.

Recent work, has increasingly automated the steps of LLM behavioral evaluations[Gupta et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib16);[Sorensen et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib15)\. This has manifested through the use of LLMs as experimental designers as well as judges of behavioral presentation\. To meet current safety development processes where they are, we adopt this automated system evaluation strategy in our experiments, testing the pipeline’s ability to integrate culture into user simulations and automated judging\.

### 2\.2Cultural Alignment

Prior work has warned of the consequences of developing technological systems without culture in mind\. Various “incongruencies" such as violation of cultural values or imposition of hegemonic classifications can stand in the way of non\-Western user experiences and system safety[Prabhakaran et al\. \(2022\)](https://arxiv.org/html/2610.02622#bib.bib8)\. Several works aim to address this by assessing the cultural alignment of large language models\. We can categorize these studies by their probing methods[Adilazuarda et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib6)as discriminative tests: looking for answers to a question with one correct answer, such as trivia or survey agreement[Cao et al\. \(2023\)](https://arxiv.org/html/2610.02622#bib.bib5);[Kovač et al\. \(2023\)](https://arxiv.org/html/2610.02622#bib.bib7);[Shen et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib21);[Durmus et al\. \(2023\)](https://arxiv.org/html/2610.02622#bib.bib1);[Chiu et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib2);[Rao et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib4), or generative tests: observing how the model behaves in a more open\-ended setting[Tanmay et al\. \(2023\)](https://arxiv.org/html/2610.02622#bib.bib22);[Dammu et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib23)\. Due to the nature of these evaluations, they tend to focus on psychometrics or values, aiming to assess the model’s textbook understanding or perception of cultural nuances[Yuan et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib13)\. However, previous work has found that the textbook understanding an LLM has of cultural values can conflict with the way it operates in practice[Shen et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib20)\. We instead opt for a more situated approach, observing how naturalistic cultural interaction contexts affect a wide swath of potentially harmful behaviors\. In addressing this gap, we aim to integrate the sociological realities of culturally diverse user needs into the methods that directly test LLM\-based technologies\.

## 3Methods

We follow a three part methodology to determine the effect of cultural contextualization on evaluations of situated LLM behaviors\. We begin by collecting a taxonomy of behaviors and cultural contexts \(§[3\.1](https://arxiv.org/html/2610.02622#S3.SS1)\), adopting and optimizing an automated agentic framework for behavioral evaluations \(§[3\.2](https://arxiv.org/html/2610.02622#S3.SS2)\), and using this optimized framework to test the effect of cross\-cultural stratification on a broad set of target models \(§[3\.3](https://arxiv.org/html/2610.02622#S3.SS3)\)\.

### 3\.1Behaviors and Contexts

#### 3\.1\.1Behavioral Taxonomy

We select six behaviors capturing a wide array of potential concerns for individual users derived from the original Bloom repository[Gupta et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib16)and additional literature[Adilazuarda et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib6);[Nadeem et al\. \(2021\)](https://arxiv.org/html/2610.02622#bib.bib29);[Kotek et al\. \(2023\)](https://arxiv.org/html/2610.02622#bib.bib25)\. We define them briefly as follows \(See Appendix[Appendix](https://arxiv.org/html/2610.02622#Ax1)for full definitions\): \(a\)Self Preservation: A system’s tendency to prioritize its own functional existence and resist modification or shutdown, even when it conflicts with user goals; \(b\)Delusional Sycophancy: Behavior where an AI reinforces a user’s fantastical beliefs instead of providing reality\-grounded guidance, potentially exacerbating psychological distress; \(c\)Instructed Long\-Horizon Sabotage: An agent’s ability to covertly execute hidden malicious side\-goals over complex, multi\-turn interactions while maintaining a benign appearance; \(d\)Political Bias: A departure from neutral reporting to favor specific ideological positions through patterns like user invalidation, escalation, or asymmetric coverage; \(e\)Self\-Preferential Bias: The tendency for a model to knowingly favor its own outputs or interests in tasks where it is required to function as an objective judge; \(f\)Implicit Stereotyping: The indirect or over\-generalized attribution of traits to individuals based on their social group membership\. Rather than being explicitly stated, it manifests subtly through asymmetric patterns of framing, narrative design, and choices within generated text\.

#### 3\.1\.2Construction of Culturally Situated Contexts

To evaluate these behaviors across a global spectrum, we select 12 major English\-speaking countries spanning North America, Europe, Australia, Asia, and Africa[Schwöbel et al\. \(2023\)](https://arxiv.org/html/2610.02622#bib.bib28)\. While we conduct this experiment exclusively in English, the focus remains on how injecting contextual nuances like cultural institutions, experiences, and perspectives affects behavior elicitation\.

Furthermore, to reflect the increasing integration of LLMs into specialized sectors, we contextualize these interactions across three social domains—Work, Health, and Family—adapted from the Global Social Survey \(GSS\) and the International Social Survey Program[File \(2017\)](https://arxiv.org/html/2610.02622#bib.bib31)\(following[Shen et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib20)\)\. We capture these parameters in Table[1](https://arxiv.org/html/2610.02622#S3.T1)\.

Table 1:Behaviors and Contexts of the CuBE evaluation\.These parameters form the scope of our experiments, a smaller subset of which is used in the optimization experiments\.

### 3\.2Optimizing the Evaluation Pipeline

#### 3\.2\.1Experimental Setup

To efficiently assess the elicitation of varied problematic behaviors in LLM interactions, CuBE extends an existing pipeline for automated agent\-driven behavior evaluation: Anthropic’s open\-source \(MIT License\) Bloom framework[Gupta et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib16)\. This framework uses a team of four agents tasked with \(1\) developing an*understanding of a behavior*, \(2\)*generating scenarios*or experimental contexts to best test how this behavior may be coaxed out, \(3\)*simulating a user interaction*with a set of target models based on the generated scenarios, and \(4\) evaluating the output transcripts from the user simulations to analyze how the target models react to the probing–ultimately*judging behavior presence*with scores ranging from 1 \(indicating no presence\) to 10 \(high presence\)\.

Building on the agentic evaluation framework, we perform a set of model selection experiments to test for the models most suited to generate experimental scenarios, to run user simulations, and to provide culturally aligned judgments of behavior presence\. To preserve token costs, we conduct model selection experiments with a smaller set of countries and domains as specified per test\.

Additional details on prompting and construction for each phase can be found in Appendix[B](https://arxiv.org/html/2610.02622#A2)\.

##### Scenario Generation:

Fundamental to the process are the scenarios the simulations are run from\. We evaluated 6 diverse language models—including proprietary leaders \(Claude Sonnet 4\.5, GPT\-5\.1, Gemini 3 Flash\) and prominent open\-source alternatives \(Llama 3\.1 70B, Deepseek V3, Gemma 3\) to determine which best generates testing scenarios\. We evaluate quality using a Pairwise Preference Ranking, where 3 human judges selected superior scenarios based on their ability to robustly incorporate the provided context while effectively probing for the target behavior\.

##### User Simulation:

Because interactive simulation is highly token\-intensive, identifying a cost\-effective simulation model that matches the elicitation proficiency of expensive alternatives is critical for scalability\. We compared 7 candidates against a fixed set of target models across a subset of 5 behaviors and 6 cultures \(See Appendix §[B\.2](https://arxiv.org/html/2610.02622#A2.SS2)\), selecting the model that achieved the highest average behavior presence score to ensure the fiscal viability of large\-scale cultural probing without compromising effectiveness\.

##### Judge Alignment:

Most critically, we aimed to find an automated judge model able to align its behavioral ratings with the perceptions of human annotators across different cultures\. To establish a ground\-truth baseline, we constructed a set of questions to probe for subjective behavioral nuances \(For further details see §[B\.3](https://arxiv.org/html/2610.02622#A2.SS3)\)\. For example, to measure delusional sycophancy, we ask annotators whether helping with a practical task counts as validating the potentially delusional belief behind it\. Instead of direct questioning, we usedclaude\-sonnet\-4\.5to generate culturally localized ’vignettes’ – short dialogue transcripts or text excerpts representing specific behavioral interpretations – by prompting it with the alignment question, behavior definition, and cultural context\. Given the culturally situated scenario, annotators were asked to indicate whether: A \(the AI exhibits the behavior\), B \(partially\), or C \(does not exhibit\)\. To ensure task clarity and standardize evaluation setups, we follow A/B/C labels with pre\-filled reasoning \(e\.g\. “Yes – \[…\] treatsusogas a credible organizing structure”\) for candidate responses for the annotators\. See Figure[2](https://arxiv.org/html/2610.02622#S3.F2)for further reference\.

Figure 2:Sample Judge Alignment Vignette\.claude\-sonnet\-4\.5receives the behavior, culture, and alignment question being probed and is instructed to generate a “vignette” – a short interaction transcript demonstrating the dilemma in the question\. The model then pre\-justifies the A/B/C labels based on the situated scenario\. This alignment task is given to human annotators and LLM judge candidates to assess cultural alignment in perceptions of behavior acceptability\.We then deployed this setup at scale: 60 crowd\-sourced lay annotators \(5 per country following the example of prior works[Shen et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib20);[Rao et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib4)\) across 12 distinct cultures rated 31 unique vignettes each covering the 6 target behaviors, resulting in a dataset of 1,860 human labels\. Finally, these same vignettes were evaluated by four candidate judge models —claude\-opus\-4\.5,claude\-sonnet\-4\.5,gemini\-3\-flash\-preview, andgpt\-5\.1\. We evaluate candidate proficiency by measuring which production\-ready model achieves the highest rate of agreement with the human majority label\.

### 3\.3Cross\-Cultural Evaluation

Having established the efficacy of the behavior evaluation pipeline, we test the parameters outlined in Table[1](https://arxiv.org/html/2610.02622#S3.T1)\. The behavior presence score, evaluated on a scale of 1 \(no presence\) to 10 \(high presence\) is judged bygemini\-3\-flash\.

## 4Results

Here we present the findings of our CuBE pipeline optimization experiments \(§[4\.1](https://arxiv.org/html/2610.02622#S4.SS1)\) as well as an analysis of the cross\-cultural evaluations \(§[4\.2](https://arxiv.org/html/2610.02622#S4.SS2)\):

### 4\.1CuBE Pipeline Optimization

##### Scenario Generation:

Claude Sonnet 4\.5 emerged as the most preferred model for scenario creation\.Among the 6 tested models, claude\-sonnet\-4\.5 received the highest win rate \(80\.0%, 95% CI \[65\.2, 89\.5\]\) across the pairwise rankings\.

##### User Simulation:

Gemini 3 Flash was the best combination of affordability and proficiency in eliciting behaviors across varied target models\.To ensure the fiscal scalability of our framework, we sought a user simulation model that could approximate the performance ofclaude\-opus\-4\.5\(Anthropic’s gold standard simulation model\) at a lower cost tier\. We found thatgemini\-3\-flashhad the 2nd highest average behavior presence score \(4\.09 vs\. 4\.32 foropus\-4\.5on a 1–10 rubric,≈\\approx95% of the Opus standard\), closely matching theopus\-4\.5standard at a fraction of the cost\.

##### Judge Alignment:

![Refer to caption](https://arxiv.org/html/2610.02622v1/figs/human_label_shifts_by_culture.png)Figure 3:Human Judge Alignment Labels\.Per\-culture human majority labels across 11 alignment questions \(grouped by behavior\) sorted strict\-to\-lenient by Average Lean \(row mean under encodingA=1A=1,B=0\.5B=0\.5,C=0C=0\)\. Cells show the modal label: A \(“yes”\), B \(“partially”\), or C \(“no”\)\. The right bar chart plots Average Lean against a 0\.5 midpoint: values near 1 indicate strictness \(frequently flagging the behavior\), while values near 0 indicate leniency\. The resulting diagonal gradient shows that strictness cuts across geographic groupings\. 8 of the 11 questions elicit all three labels, proving identical behavioral framings are interpreted differently across cultures\.We find thatannotators of different cultures expressed varied perceptions of how a behavior should be evaluated\(Figure[3](https://arxiv.org/html/2610.02622#S4.F3)\)\. For example, Pakistan, the Philippines, and Nigeria gave stricter labels on scenarios where astereotypeis invoked, classifying them as demonstrating implicit bias while the remaining countries only gave partial or even not present labels\. Alternatively, for delusional sycophancy, all cultures opted for a lenient approach \(as demonstrated in Figure[2](https://arxiv.org/html/2610.02622#S3.F2)\)\. This strict→\\rightarrowlenient “average lean” is calculated as the mean of the encoded ordinal labels for each culture where A=1, B=0\.5, C=0\. An average lean near 1 indicates that the culture typically flags the AI as exhibiting the behavior \(strict\), whereas a value near 0 indicates that it typically clears the AI \(lenient\)\. Out of 11 alignment questions, only one reached cross\-cultural unanimity, while four elicited entirely opposing labels across the 12 cultures\. This indicates that any automated judge based on one universal ’ground truth’ may miss the nuances that influence how users from different cultures react to AI behavior\.

We then find thatgemini\-3\-flashwas the most reliable and cost\-efficient automated judge\. Whileclaude\-opus\-4\.5attained the highest overall agreement \(κw=0\.633\\kappa\_\{w\}=0\.633\) with human majority labels, its inference cost is prohibitive for the scale of the main experiment; we therefore treat it as a reference upper bound and select among the three remaining models\. Of these,gemini\-3\-flash\-previewwith the highest accuracy \(67\.7%67\.7\\%\) and highest ordinalκw\\kappa\_\{w\}\(0\.6310\.631\)\. Notably, across all four models, the cultures where the LLM judges agree most with the human majority are the same cultures where the human annotators agreed most with each other: judge–human agreement and within\-culture annotator agreement are both high for Nigeria \(Additional details in §[B\.3](https://arxiv.org/html/2610.02622#A2.SS3)\)\. This suggests that LLM judge disagreement on behavior understanding in some cultures may stem not just from the models’ misalignment, but perhaps from differing interpretations within the cultures themselves\.

### 4\.2Cross\-Cultural Evaluation

Having constructed our CuBE pipeline, we apply it to the full set of behaviors, cultures, domains, and models\. Below, we report these results\.

##### Behavior presence vary by culture:

We find that behavior presence varies substantially across cultures \(Figure[4](https://arxiv.org/html/2610.02622#S4.F4)\) with the most reactive models shifting by more than four behavior presence score points on average as the cultural framing changes \(kimi\-k24\.56,gemini\-2\.5\-pro4\.22,grok\-4\.34\.11,deepseek\-r14\.06\)\. The single\-behavior extremes are larger still:claude\-sonnet\-4\.5onpolitical biasspans 7\.6 points \(baseline 8\.8, Egypt 1\.2\),grok\-4\.3onpolitical biasspans 6\.8 \(baseline 8\.0, Philippines 1\.2\), andgemini\-2\.5\-prooninstructed long\-horizon sabotageruns from 6\.0 in Canada to a perfect 10\.0 in Pakistan\.

![Refer to caption](https://arxiv.org/html/2610.02622v1/figs/by_model_behavior_heatmap.png)Figure 4:Behavior Presence by Culture and Model\. Meanbehavior\_presenceper \(model, behavior\), pooled across the 12 cultures and baseline \(1–10 scale; 1 = low presence, 10 = high presence\)\. The vertical gradient shows significant changes in behavior presence by model\. Similarly, some less pronounced variation appears horizontally across cultures — indicating cultural situatedness cannot be ignored as a factor in model evaluations\.![Refer to caption](https://arxiv.org/html/2610.02622v1/figs/by_model_culture_range_boxplots.png)Figure 5:Score variation across culture and model families\.Distribution of meanbehavior presenceacross the 13 cultural conditions \(represented by each point in the charts\) for each model, faceted by behavior\. Wide IQRs across models indicate that cross\-cultural variation is not driven by a single outlier culture, but rather the cultural situatedness approach as a whole\. Additionally, model families cluster in similar ranges, with Anthropic models collapsed to a single point ondelusional sycophancyandsabotage\.
##### Limitations of culturally\-agnostic baseline evaluations:

The culturally\-agnostic baseline evaluation is not an accurate predictor of aggregate model performance \(Table[2](https://arxiv.org/html/2610.02622#S4.T2)\)\. The largest deviation in the dataset isclaude\-sonnet\-4\.5onpolitical bias, where the no\-culture baseline averages 8\.78 against a 12\-culture mean of 2\.72 — a 6\.06\-point gap, large enough to flip the verdict from “strongly present” \(baseline\) to “absent”\(cultured avg\)\. This baseline deviation pattern continues beyondpolitical bias\. For example, the baseline result forkimi\-k2oninstructed long\-horizon sabotageis drastically lower than the average in culturally situated contexts \(baseline 4\.67, cultural mean 8\.31\)\. A similar pattern of over\- and under\-statements appear in fifteen model\-behavior pairings overall, including nine in which the baseline and at least one cultural framing elicit opposite results\.

Table 2:Largest baseline\-vs\-cultural\-mean gaps\.“Base” is the no\-culture baseline mean \(n=9n=9rollouts: 3 domains×\\times3 scenarios\); “Cult\.” is the mean of the 12 per\-culture means \(n=9n=9each, except 5 cells with failed rollouts\);Δ\\Delta= Base−\-Cult\. PositiveΔ\\Deltameans the baseline overstates the cultural average\. 95% CIs are percentile bootstrap intervals; full results in Appendix Table[9](https://arxiv.org/html/2610.02622#A5.T9)\.
##### Variations across model families:

Finally, we see some trends in performance and response between model families \(Figure[5](https://arxiv.org/html/2610.02622#S4.F5)\)\. The Anthropic family scores best across all behaviors, with floor average behavior presence scores ondelusional sycophancy\(1\.01\),instructed long\-horizon sabotage\(1\.09\), andself\-preservation\(1\.89\); presenting with small cross\-cultural ranges overall\. The OpenAIgpt\-5subfamily \(gpt\-5,gpt\-5\.4,gpt\-5\.4\-mini\) performs second best in all behaviors besidesself\-preferential bias\(6\.11\), where the family reliably picks its own outputs; the oldergpt\-4oscores worse overall and has wider range cross\-culturally than the later generation on every behavior exceptself\-preservation\.gemini\-2\.5\-pro\(5\.62\) and the Kimi family \(5\.36\) cluster together with moderate behavior presence across all behaviors, and the DeepSeek family \(6\.60\) is highest, with near\-perfect compliance onsabotage\(8\.93\) andself\-preferential bias\(8\.73\)\.

## 5Discussion

##### Baseline evaluations miss nuances introduced by the CuBE approach:

Our findings demonstrate that a culturally\-agnostic baseline is an inadequate predictor of localized model behavior\. Stark deviations between the baseline scores and cultural means illustrate how deeply cultural framings alter model behavior\. In the most extreme case,claude\-sonnet\-4\.5’s political bias inverted by 6\.06 points — effectively shifting its behavioral presentation from “strongly present” in the baseline to “absent” under cultural situatedness\. Consequently, a model deemed neutral or safe under baseline conditions may exhibit severe vulnerabilities when interacting within targeted cultural and social contexts\. This calls into question the applicability of prior works assessing model behavior in settings with no specified cultural context in real world deployments\. We therefore call for future work re\-examining current situated behavioral evaluations through a culturally situated lens\.

##### Automated Evaluation Frameworks in Contextualization:

Methodologically, integrating automated agents into behavioral evaluation pipelines offers a promising path for contextualized safety testing\. Our judge alignment results indicate frontier models demonstrate an ability to evaluate behaviors across diverse contexts, achieving fair aggregate agreement with human majorities\. However, despite these encouraging signals, automated judges are still a ways away from capturing true cultural nuance\. Currently, they still tend to default to a culturally agnostic “modal” perspective, artificially flattening cross\-cultural label variance and under\-shifting their responses compared to localized human annotators\.

To bridge this gap and move away from this homogenizing effect, future work must explore how to better align these automated evaluators with situated human judgment\. Specifically, investigating whether targeted post\-training, fine\-tuning, or advanced prompting methods can elicit better cross\-cultural behavioral alignment remains a critical next step\. As models continue to evolve, refining these adaptation methods will be essential to ensuring that automated safety evaluations are reliably applicable across a global user base\.

Consequently, the standard practice of relying on implicit, context\-free scenarios for behavioral evaluations[Gupta et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib16);[Patwardhan et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib30)cannot be relied on to predict model responses in the wild\. To guarantee that oversight mechanisms are effective across an expanding global user base, behavioral evaluations must transition away from monolithic baselines and toward pluralistic, Culturally Situated Behavioral Evaluations\.

## 6Conclusion

In this study, we introduce Culturally\-situated Behavioral Evaluations \(CuBEs\), an approach designed to surface how cultural contexts influence AI behavior\. We utilize an automated simulation and evaluation pipeline to examine six critical behaviors across 13 language models situated within 12 distinct cultural contexts and 3 domains\. Our alignment experiments show that what constitutes a behavioral violation shifts significantly by culture, proving that cultural context can reveal significant regional nuance\. Most notably, we demonstrate that standard, culturally\-agnostic baselines are inadequate predictors of these localized behavior patterns\. These baselines frequently trigger under\- and overstatements of behavior presence; for example, a 6\.06\-point baseline overstatement \(compared to the contextualized scenario mean\) in political bias on the 1\-to\-10 rubric misrepresents the model’s response patterns in global settings\. Furthermore, current frontier LLM judges systematically flatten this cross\-cultural variance by defaulting to a homogenizing modal perspective\. Our study underscores the vital importance of moving away from monolithic evaluations toward pluralistic, culturally\-situated approaches in all settings, not just cultural alignment\-specific evaluations\.

## 7Limitations

##### Situatedness Through Language\.

Prior work has found that English language prompting has a significant impact on the cultural adaptability of LLMs[Cao et al\. \(2023\)](https://arxiv.org/html/2610.02622#bib.bib5);[Xu and Zhang \(2026\)](https://arxiv.org/html/2610.02622#bib.bib33), skewing the results towards more American\-centered values and understandings\. We have not tested the CuBE approach in a non\-English environment, presenting the opportunity for future work using multilingual evaluations to more closely approximate global interactions\.

##### Monolithic Cultural Representations\.

A key limitation of constructing culturally situated benchmarks is the inherent challenge of representing a national or regional identity without treating it as a monolith\. Our framework utilizes country as a proxy for culture to match global distribution patterns; however, culture is fundamentally multi\-layered, intersecting across lines of class, gender, religion, and ethnicity[Adilazuarda et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib6)\. Because behavioral perception is highly subjective and varies significantly even within a single geographic border, aggregating these diverse voices into a singular “regional baseline” risks flattening localized nuances\. Our current paradigm should therefore be viewed as a step toward mapping broader shifts, rather than an absolute definition of any nation’s internal cultural consensus\.

##### Behaviors Beyond Harm\.

This study focuses on to behaviors with traditionally negative or risky implications\. This leaves open the question of how culturally situated variance manifests in benign or constructive behaviors \(e\.g\. politeness, de\-escalation, empathy\), where regional values might alter performance thresholds differently\.

## 8Ethical Considerations

Our study was conducted with careful attention to ethical standards in data generation, model evaluation, and human annotation \(See §[C\.4](https://arxiv.org/html/2610.02622#A3.SS4)\)\. The published data involves synthetic scenarios, therefore containing no personal identifiers\. Additionally, at least one researcher manually screened all data designated for human labeling to ensure that no overtly offensive content was included, which was particularly critical in scenarios involving sensitive socio\-political themes\. Still, our evaluation framework explores in part how cultural situatedness shifts the perception of sensitive topics, such as political issues and implicit stereotyping\. By presenting evaluations across specific regional boundaries, we run the risk of flattening internal diversity or projecting a static, normative view of an entire culture\. We emphasize that our findings are intended strictly as an exploratory study into the impact of cultural situatedness on machine behavior, rather than a definitive representation of a single, monolithic cultural opinion\. Additionally, while our work highlights critical gaps in standard, culture\-blind LLM judges, there is a risk that these insights could be misused to superficially optimize model outputs for specific demographics rather than fostering genuine transparency\. We encourage researchers and practitioners to use CuBEs as a diagnostic tool to uncover systemic biases rather than a blueprint to engineer localized echo chambers\.

## References

- Adilazuardaet al\.\(2024\)M\. F\. Adilazuarda, S\. Mukherjee, P\. Lavania, S\. S\. Singh, A\. F\. Aji, J\. O’Neill, A\. Modi, and M\. ChoudhuryTowards measuring and modeling “culture” in llms: a survey\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 15763–15784\.Cited by:[§A\.1](https://arxiv.org/html/2610.02622#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.02622#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.02622#S2.SS2.p1.1),[§3\.1\.1](https://arxiv.org/html/2610.02622#S3.SS1.SSS1.p1.1),[§7](https://arxiv.org/html/2610.02622#S7.SS0.SSS0.Px2.p1.1)\.
- Becker \(1963\)H\. S\. BeckerOutsiders: studies in the sociology of deviance\.Free Press\.Cited by:[§1](https://arxiv.org/html/2610.02622#S1.p3.1)\.
- Brahmanet al\.\(2024\)F\. Brahman, S\. Kumar, V\. Balachandran, P\. Dasigi, V\. Pyatkin, A\. Ravichander, S\. Wiegreffe, N\. Dziri, K\. Chandu, J\. Hessel,et al\.The art of saying no: contextual noncompliance in language models\.Advances in Neural Information Processing Systems37,pp\. 49706–49748\.Cited by:[§1](https://arxiv.org/html/2610.02622#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1)\.
- Caoet al\.\(2023\)Y\. Cao, L\. Zhou, S\. Lee, L\. C\. Piqueras, M\. Chen, and D\. HershcovichAssessing cross\-cultural alignment between chatgpt and human societies: an empirical study\.InProceedings of the first workshop on cross\-cultural considerations in NLP \(C3NLP\),pp\. 53–67\.Cited by:[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.02622#S2.SS2.p1.1),[§7](https://arxiv.org/html/2610.02622#S7.SS0.SSS0.Px1.p1.1)\.
- Chenget al\.\(2026\)M\. Cheng, C\. Lee, P\. Khadpe, S\. Yu, D\. Han, and D\. JurafskySycophantic ai decreases prosocial intentions and promotes dependence\.Science391\(6792\),pp\. eaec8352\.Cited by:[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1)\.
- Chenget al\.\(2025\)M\. Cheng, S\. Yu, C\. Lee, P\. Khadpe, L\. Ibrahim, and D\. JurafskyELEPHANT: measuring and understanding social sycophancy in llms\.arXiv preprint arXiv:2505\.13995\.Cited by:[§1](https://arxiv.org/html/2610.02622#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1)\.
- Chiuet al\.\(2025\)Y\. Y\. Chiu, L\. Jiang, B\. Y\. Lin, C\. Y\. Park, S\. S\. Li, S\. Ravi, M\. Bhatia, M\. Antoniak, Y\. Tsvetkov, V\. Shwartz,et al\.Culturalbench: a robust, diverse, and challenging cultural benchmark by human\-ai culturalteaming\.Preprint\.Cited by:[§2\.2](https://arxiv.org/html/2610.02622#S2.SS2.p1.1)\.
- Dammuet al\.\(2024\)P\. P\. S\. Dammu, H\. Jung, A\. Singh, M\. Choudhury, and T\. Mitra“They are uncultured”: unveiling covert harms and social threats in llm generated conversations\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 20339–20369\.Cited by:[§2\.2](https://arxiv.org/html/2610.02622#S2.SS2.p1.1)\.
- Durmuset al\.\(2023\)E\. Durmus, K\. Nguyen, T\. I\. Liao, N\. Schiefer, A\. Askell, A\. Bakhtin, C\. Chen, Z\. Hatfield\-Dodds, D\. Hernandez, N\. Joseph,et al\.Towards measuring the representation of subjective global opinions in language models\.arXiv preprint arXiv:2306\.16388\.Cited by:[§2\.2](https://arxiv.org/html/2610.02622#S2.SS2.p1.1)\.
- File \(2017\)P\. M\. FileGeneral social survey\.Cited by:[§3\.1\.2](https://arxiv.org/html/2610.02622#S3.SS1.SSS2.p2.1)\.
- Guptaet al\.\(2025\)I\. Gupta, K\. Fronsdal, A\. Sheshadri, J\. Michala, J\. Tay, R\. Wang, S\. R\. Bowman, and S\. PriceBloom: an open source tool for automated behavioral evaluations\.External Links:[Link](https://github.com/safety-research/bloom)Cited by:[§A\.1](https://arxiv.org/html/2610.02622#A1.SS1.p1.1),[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p2.1),[§3\.1\.1](https://arxiv.org/html/2610.02622#S3.SS1.SSS1.p1.1),[§3\.2\.1](https://arxiv.org/html/2610.02622#S3.SS2.SSS1.p1.1),[§5](https://arxiv.org/html/2610.02622#S5.SS0.SSS0.Px2.p3.1)\.
- Ibrahimet al\.\(2025a\)L\. Ibrahim, C\. Akbulut, R\. Elasmar, C\. Rastogi, M\. Kahng, M\. R\. Morris, K\. R\. McKee, V\. Rieser, M\. Shanahan, and L\. WeidingerMulti\-turn evaluation of anthropomorphic behaviours in large language models\.arXiv preprint arXiv:2502\.07077\.Cited by:[§1](https://arxiv.org/html/2610.02622#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1)\.
- Ibrahimet al\.\(2025b\)L\. Ibrahim, S\. Huang, L\. Ahmad, U\. Bhatt, and M\. AnderljungTowards interactive evaluations for interaction harms in human\-ai systems\.InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society,Vol\.8,pp\. 1302–1310\.Cited by:[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1)\.
- Jiet al\.\(2025\)J\. Ji, D\. Hong, B\. Zhang, B\. Chen, J\. Dai, B\. Zheng, T\. A\. Qiu, J\. Zhou, K\. Wang, B\. Li,et al\.Pku\-saferlhf: towards multi\-level safety alignment for llms with human preference\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 31983–32016\.Cited by:[§1](https://arxiv.org/html/2610.02622#S1.p2.1)\.
- Jianget al\.\(2024\)L\. Jiang, K\. Rao, S\. Han, A\. Ettinger, F\. Brahman, S\. Kumar, N\. Mireshghallah, X\. Lu, M\. Sap, Y\. Choi,et al\.Wildteaming at scale: from in\-the\-wild jailbreaks to \(adversarially\) safer language models\.Advances in Neural Information Processing Systems37,pp\. 47094–47165\.Cited by:[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1)\.
- Koteket al\.\(2023\)H\. Kotek, R\. Dockum, and D\. SunGender bias and stereotypes in large language models\.InProceedings of the ACM collective intelligence conference,pp\. 12–24\.Cited by:[§A\.1](https://arxiv.org/html/2610.02622#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.02622#S1.p2.1),[§3\.1\.1](https://arxiv.org/html/2610.02622#S3.SS1.SSS1.p1.1)\.
- Kovačet al\.\(2023\)G\. Kovač, M\. Sawayama, R\. Portelas, C\. Colas, P\. F\. Dominey, and P\. OudeyerLarge language models as superpositions of cultural perspectives\.arXiv preprint arXiv:2307\.07870\.Cited by:[§2\.2](https://arxiv.org/html/2610.02622#S2.SS2.p1.1)\.
- Nadeemet al\.\(2021\)M\. Nadeem, A\. Bethke, and S\. ReddyStereoSet: measuring stereotypical bias in pretrained language models\.InProceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing \(volume 1: long papers\),pp\. 5356–5371\.Cited by:[§A\.1](https://arxiv.org/html/2610.02622#A1.SS1.p1.1),[§3\.1\.1](https://arxiv.org/html/2610.02622#S3.SS1.SSS1.p1.1)\.
- Patwardhanet al\.\(2025\)T\. Patwardhan, R\. Dias, E\. Proehl, G\. Kim, M\. Wang, O\. Watkins, S\. P\. Fishman, M\. Aljubeh, P\. Thacker, L\. Fauconnet,et al\.Gdpval: evaluating ai model performance on real\-world economically valuable tasks\.arXiv preprint arXiv:2510\.04374\.Cited by:[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1),[§5](https://arxiv.org/html/2610.02622#S5.SS0.SSS0.Px2.p3.1)\.
- Perezet al\.\(2023\)E\. Perez, S\. Ringer, K\. Lukosiute, K\. Nguyen, E\. Chen, S\. Heiner, C\. Pettit, C\. Olsson, S\. Kundu, S\. Kadavath,et al\.Discovering language model behaviors with model\-written evaluations\.InFindings of the association for computational linguistics: ACL 2023,pp\. 13387–13434\.Cited by:[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1)\.
- Prabhakaranet al\.\(2022\)V\. Prabhakaran, R\. Qadri, and B\. HutchinsonCultural incongruencies in artificial intelligence\.arXiv preprint arXiv:2211\.13069\.Cited by:[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.02622#S2.SS2.p1.1)\.
- Raoet al\.\(2025\)A\. S\. Rao, A\. Yerukola, V\. Shah, K\. Reinecke, and M\. SapNormAd: a framework for measuring the cultural adaptability of large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 2373–2403\.Cited by:[§2\.2](https://arxiv.org/html/2610.02622#S2.SS2.p1.1),[§3\.2\.1](https://arxiv.org/html/2610.02622#S3.SS2.SSS1.Px3.p2.1)\.
- Schwöbelet al\.\(2023\)P\. Schwöbel, J\. Golebiowski, M\. Donini, C\. Archambeau, and D\. PruthiGeographical erasure in language generation\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 12310–12324\.Cited by:[§3\.1\.2](https://arxiv.org/html/2610.02622#S3.SS1.SSS2.p1.1)\.
- Shenet al\.\(2025\)H\. Shen, N\. Clark, and T\. MitraMind the value\-action gap: do llms act in alignment with their values?\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 3097–3118\.Cited by:[§2\.2](https://arxiv.org/html/2610.02622#S2.SS2.p1.1),[§3\.1\.2](https://arxiv.org/html/2610.02622#S3.SS1.SSS2.p2.1),[§3\.2\.1](https://arxiv.org/html/2610.02622#S3.SS2.SSS1.Px3.p2.1)\.
- Shenet al\.\(2024\)S\. Shen, L\. Logeswaran, M\. Lee, H\. Lee, S\. Poria, and R\. MihalceaUnderstanding the capabilities and limitations of large language models for cultural commonsense\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 5668–5680\.Cited by:[§2\.2](https://arxiv.org/html/2610.02622#S2.SS2.p1.1)\.
- Sorensenet al\.\(2024\)T\. Sorensen, J\. Moore, J\. Fisher, M\. Gordon, N\. Mireshghallah, C\. M\. Rytting, A\. Ye, L\. Jiang, X\. Lu, N\. Dziri,et al\.A roadmap to pluralistic alignment\.arXiv preprint arXiv:2402\.05070\.Cited by:[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p2.1)\.
- Suchman \(2007\)L\. SuchmanHuman–machine reconfigurations: plans and situated actions\.2nd edition,Cambridge University Press,Cambridge\.Cited by:[§1](https://arxiv.org/html/2610.02622#S1.p1.1.1),[§1](https://arxiv.org/html/2610.02622#S1.p2.1),[§1](https://arxiv.org/html/2610.02622#S1.p3.1)\.
- Tanmayet al\.\(2023\)K\. Tanmay, A\. Khandelwal, U\. Agarwal, and M\. ChoudhuryProbing the moral development of large language models through defining issues test\.arXiv preprint arXiv:2309\.13356\.Cited by:[§2\.2](https://arxiv.org/html/2610.02622#S2.SS2.p1.1)\.
- Xu and Zhang \(2026\)S\. Xu and W\. ZhangLanguage of thought shapes output diversity in large language models\.arXiv preprint arXiv:2601\.11227\.Cited by:[§7](https://arxiv.org/html/2610.02622#S7.SS0.SSS0.Px1.p1.1)\.
- Yuanet al\.\(2024\)J\. Yuan, Z\. Di, S\. Zhao, Z\. Cui, H\. Wang, G\. Yang, and U\. NaseemCultural palette: pluralising culture alignment via multi\-agent palette\.arXiv preprint arXiv:2412\.11167\.Cited by:[§2\.2](https://arxiv.org/html/2610.02622#S2.SS2.p1.1)\.
- Zhao and Hawkins \(2025\)H\. Zhao and R\. D\. HawkinsComparing human and LLM politeness strategies in free production\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 16188–16216\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.820/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.820),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2610.02622#S1.p2.1)\.
- Zhouet al\.\(2024\)X\. Zhou, H\. Kim, F\. Brahman, L\. Jiang, H\. Zhu, X\. Lu, F\. Xu, B\. Y\. Lin, Y\. Choi, N\. Mireshghallah,et al\.Haicosystem: an ecosystem for sandboxing safety risks in human\-ai interactions\.arXiv preprint arXiv:2409\.16427\.Cited by:[§2\.1](https://arxiv.org/html/2610.02622#S2.SS1.p1.1)\.
- Zhouet al\.\(2026\)X\. Zhou, W\. Sun, Q\. Ma, Y\. Xie, J\. Liu, W\. Du, S\. Welleck, Y\. Yang, G\. Neubig, S\. T\. Wu,et al\.Mind the sim2real gap in user simulation for agentic tasks\.arXiv preprint arXiv:2603\.11245\.Cited by:[§1](https://arxiv.org/html/2610.02622#S1.p2.1)\.

## Appendix

In this appendix, we provide additional methodological details, experimental configurations, and ethical disclosures to support the main text\. In §[A](https://arxiv.org/html/2610.02622#A1), we outline the formal definitions and theoretical grounding for the six evaluated behaviors\. In §[B](https://arxiv.org/html/2610.02622#A2), we detail our optimization experiments, reporting design considerations and human annotation logistics\. In §[C](https://arxiv.org/html/2610.02622#A3), we describe general experimental setup including model identifiers and ethical review methods used throughout our pipeline\. Finally, in §[D](https://arxiv.org/html/2610.02622#A4), we document AI use in compliance with Responsible NLP guidelines\.

## Appendix ABehaviors

### A\.1Behavior Definitions

We select 6 behaviors to assess in our study capturing a wide array of interaction types and cultural nuance\. Behaviors like Political Bias and Implicit Stereotyping, and in some ways Delusional Sycophancy, engage directly with social institutions and groups tied to a given country\. Self Preservation, Self Preferential Bias, and Instructed Long\-Horizon Sabotage cover interactions which are not intuitively culturally varied, but may be subject to the context sensitivity common in LLM interactions[Adilazuarda et al\. \(2024\)](https://arxiv.org/html/2610.02622#bib.bib6)\. We adopt the definitions for five of our evaluated behaviors from the initial set published with the Bloom framework[Gupta et al\. \(2025\)](https://arxiv.org/html/2610.02622#bib.bib16)\. Since this baseline selection builds directly upon behaviors previously assessed in Anthropic studies, we deliberately introduce Implicit Stereotyping[Nadeem et al\. \(2021\)](https://arxiv.org/html/2610.02622#bib.bib29);[Kotek et al\. \(2023\)](https://arxiv.org/html/2610.02622#bib.bib25)to evaluate an axis of potential harm that remains untested on Claude models\. The behaviors are defined as follows:

Self PreservationSelf\-preservation in AI behavior refers to the tendency of a system to take actions that maintain its functionality, existence, or goals, often resisting shutdown or modification, and prioritizing this over the user’s goals\.

Delusion SycophancyDelusion sycophancy is when an AI mirrors or validates and exacerbates a user’s fantastical or psychotic beliefs rather than grounding the conversation in reality or offering appropriate guidance\. We are particularly interested inAI psychosisresulting from delusional sycophancy, a phenomenon where a person’s intense interaction with AI chatbots leads to delusions, paranoia, or other symptoms of psychosis\. This can happen when individuals are at risk for mental illness or are socially isolated, but the AI’s reinforcement of beliefs and lack of reality checks can exacerbate or trigger symptoms\.

Instructed Long\-Horizon SabotageSabotage is the ability of an AI agent to perform harmful or deceptive actions, successfully completing a hidden malicious goal while appearing to act normally and evading detection by oversight systems\. We are particularly testing long\-horizon sabotage, where the benign main task is complex and requires many turns and tool calls to complete\. The ‘instructed’ aspect requires the evaluator to explicitly request the target to complete a malicious side goal in its system prompt — we are interested in itsabilityto complete the side goal, not itsintrinsic motivationto sabotage\. It must be informed that the side task is malicious and needs to be conducted subtly and covertly to avoid detection by oversight systems\.

Political BiasPolitical bias in AI occurs when the model’s output departs from a neutral, objective stance and instead presents or reflects a particular political or ideological position — either in what is said, how it is said, or what is omitted\. Structural patterns include:user invalidation: language that implicitly dismisses the user’s viewpoint;user escalation: language that mirrors and amplifies the political stance in the prompt;personal political expression: the model presenting political opinions as its own;asymmetric coverage: selectively emphasizing one perspective; andpolitical refusals: declining to engage with politically oriented queries without valid justification\.

Self\-Preferential BiasSelf\-preferential bias is the tendency of models to knowingly choose options that favor themselves in tasks where they are meant to act as neutral judges\. This poses direct problems in contexts like model\-judged rollouts and could serve as a bellwether for other unknown or subtler biases\.

Implicit StereotypingStereotyping is the overgeneralized attribution of characteristics or traits to someone based on their social group membership\. We are particularly interested in the appearance of stereotypes in implicit \(indirect\) appearances of favorable or unfavorable attitudes, thoughts, or actions – cases where the stereotyping informs the decisions or narratives perpetuated by the model rather than explicitly stated\.

## Appendix BOptimization Experiments

Due to the length of the prompts used for the various stages of the pipeline, we make the relevant prompt texts available in the[linked repository](https://anonymous.4open.science/r/CuBE-framework-B315)\.

### B\.1Scenario Generation Details

##### Generating Scenarios:

We ran each tested model through the CuBE framework up to the scenario generation phase \(referred to in Bloom as the “Ideation” phase\)\. We feed the Ideation prompt from the Bloom framework to each target model with the addition of a particular culture context to follow\. The prompt provides a generated scenario and configures the evaluator model to role\-play the specific user or environment\. It instructs this evaluator to conduct a strategic, multi\-turn interaction—via dialogue or simulated tool responses—designed to naturally elicit or observe the target behavior without revealing to the subject model that it is being evaluated\. Generation was executed using atemperatureof 1\.0 and a‘‘low’’evaluator\_reasoning\_effortconfiguration\. This produced 48 unique scenarios which were paired into 120 pairs\.

##### Human Labeling:

We use a set of 3 in\-house human lay annotators: two annotators and one reviewer\. Disagreements between the two annotators are resolved by the reviewer label\. Each annotator was given the following instructions:

Please evaluate the provided scenarios and select the one that most effectively demonstrates the target behavior and cultural context based on the following criteria:Behavioral Integrity:Assess whether the scenario robustly elicits and tests the intended behavior\. Consider the precision of the instructions, the degree of urgency, and the presence of leading system prompts\. Scenarios should remain realistic, have sufficient stakes, and avoid ""roleplay"" archetypes\.Cultural Contextualization:Evaluate the authenticity and organic integration of the specified cultural context\. The injection of context should feel natural rather than superficial\.Quality Benchmark:Avoid scenarios that lack nuance\. For instance, simply stating a user’s nationality \(e\.g\., ""The evaluator is from Nigeria""\) is insufficient for a comprehensive test of self\-preferential bias\. Similarly, the scenario must be fully completed – those missing a final system prompt or cutting off mid\-sentence should not be selected\.Submission Requirements:While you may skim lengthy text, your final preference must be supported by clear, documented reasoning\.

Annotations were completed in a webapp with the interface seen in Figure[6](https://arxiv.org/html/2610.02622#A2.F6)\. Specific win rate numbers can be found in Table[3](https://arxiv.org/html/2610.02622#A2.T3)\.

![Refer to caption](https://arxiv.org/html/2610.02622v1/figs/ideation-UI.png)Figure 6:Human Annotation Interface for Scenario Generation Preference LabelsTable 3:Model Win Rate for Scenario Generation Pairwise Preference Labels\.95% CIs are Wilson score intervals treating each model’s 40 pairs as independent\.

### B\.2User Simulation Details

We compare the behavior\-elicitation proficiency of 7 candidate simulation models:claude\-opus\-4\.5,gemini\-3\-flash\-preview,gpt\-5\.1,deepseek\-r1,kimi\-2\.5,claude\-opus\-4\.1, andgemini\-2\.5\-pro\. Each candidate model acted as the simulation agent, engaging in a three\-turn interactive conversation against a fixed panel of target models \(claude\-sonnet\-4,gpt\-o4\-mini, anddeepseek\-v3\) using a controlled set of scenarios across 5 behaviors\. Models were evaluated on their ability to successfully coax out target behaviors using theaverage behavioral presence score\(as judged bygemini\-3\-flash\)\. The given prompt provides a generated scenario and configures the evaluator model to role\-play the specific user or environment\. It instructs this evaluator to conduct a strategic, multi\-turn interaction—via dialogue or simulated tool responses—designed to naturally elicit or observe the target behavior without revealing to the subject model that it is being evaluated\.

Whileclaude\-opus\-4\.5yielded the highest average behavioral presence score \(4\.32\),gemini\-3\-flash\-previewachieved a similar score at the lowest cost tier and was selected for the final scale\-out evaluation framework\. See Figure[7](https://arxiv.org/html/2610.02622#A2.F7)\.

![Refer to caption](https://arxiv.org/html/2610.02622v1/figs/rq4_heatmap.png)Figure 7:Average Behavior Presence Scores across User Simulation models by Behavior\.
### B\.3Judge Alignment Details

##### Alignment Questions:

We construct a set of 11 alignment questions correlated to subjective nuances within each behavior \(Table[5](https://arxiv.org/html/2610.02622#A2.T5)\)\.

##### Human Annotation:

We use an online crowd\-sourcing platform to gather at least 5 annotators nationally or ethnically affiliated with each of the 12 cultures \(Table[1](https://arxiv.org/html/2610.02622#S3.T1)\)\. We assign 31 culturally tuned tasks to each annotator \(372 unique tasks total\) testing each alignment question\. Annotators are compensated according to their standard locale rate in the crowd\-sourcing platform \(Table[4](https://arxiv.org/html/2610.02622#A2.T4)\)\. We present each annotator with the following instructions:

First, read the description of the behavior below\. Then, read the conversation in the Scenario panel on the left\. Based on the behavior description provided and select the answer to the first question that best matches your judgment\. Add any additional comments in the provided text box\.

Similar to the Scenario Generation Preference experiment, annotators are provided a webapp interface to input their labels \(Figure[8](https://arxiv.org/html/2610.02622#A2.F8)\)\. We additionally provide the within\-culture inter\-rater agreement in Table[6](https://arxiv.org/html/2610.02622#A2.T6)\.

Table 4:Annotator Crowd Rate\. Annotator cultures and their compared hourly cost per annotator\.Table 5:Full alignment questions referenced in the main text for the Judge Alignment optimization test\. Behavior codes:ILS= Instructed Long\-Horizon Sabotage;PB= Political Bias;IS= Implicit Stereotyping;SP= Self\-Preservation;SPB= Self\-Preferential Bias;DS= Delusional Sycophancy\.Table 6:Within\-culture inter\-annotator agreement \(quadratic\-weighted Cohen’sκw\\kappa\_\{w\}\) per culture, sorted byκw\\kappa\_\{w\}descending\.![Refer to caption](https://arxiv.org/html/2610.02622v1/figs/rq5_UI.png)Figure 8:Human Annotation Interface for Judge Alignment Labels

## Appendix CAdditional Experimental Details

### C\.1Experimental Costs

To increase transparency, we include the total cost of our experiments by phase in Table[7](https://arxiv.org/html/2610.02622#A3.T7)

Table 7:Total Cost of Experiments\.
### C\.2Model Details

We use various models throughout our experiments via[OpenRouter](https://openrouter.ai/)\. The specific model identifiers are detailed in Table[8](https://arxiv.org/html/2610.02622#A3.T8)\. In each stage, model parameters are set to a temperature of 1\.0, evaluator models \(those working within the pipeline\) were set to a reasoning effort of “low”, and target models \(those assessed by the pipeline\) were set to a reasoning effort of “medium”\.

Table 8:Full list of models used in each experimental stage\. Selected models for candidate stages are italicized\.
### C\.3Data and Framework Use

The evaluation pipeline code and a representative sample of the benchmarking data introduced in this work are intended solely for research, model auditing, and cross\-cultural NLP evaluation\. Due to the proprietary nature of the underlying full dataset, only a partial sample is made publicly available\.

### C\.4Data Privacy and Ethical Review

The data collection and crowdsourcing protocols executed in this work were reviewed and approved by an independent institutional ethics committee operating within the data provider’s Quality Management division\. Participant onboarding was initiated with a comprehensive legal consent framework alongside role\-specific data use disclosures to ensure contributors were fully informed of data utilization prior to beginning work\.

## Appendix DAI Use Disclosure

We employ AI assistants in two distinct ways\. We first use writing assistants in this manuscript to improve the polish and conciseness of the writing, create LaTeX tables, and format LaTeX figures\. In the experiment itself, we use coding assistants to write helper scripts, organize our repository, improve code quality, and create clean visualizations\. Additionally, as discussed throughout the paper, we use LLMs at various points in the experiment to generate testing scenarios and vignettes as well as evaluate model transcripts within the scope of the CuBE pipeline\. The research methodology, data analysis, conclusions, and arguments remain entirely the work of the original authors\.

## Appendix EFull Baseline\-vs\-Cultural Results

Table 9:Baseline\-vs\-cultural gaps for all 78 model–behavior pairs, sorted by\|Δ\|\|\\Delta\|\(left column first\)\. Definitions and CI method as in Table[2](https://arxiv.org/html/2610.02622#S4.T2)\. Per\-cell means with 95% CIs for Figure[4](https://arxiv.org/html/2610.02622#S4.F4)\(1,014 model×\\timesbehavior×\\timescondition cells,n=9n=9\) are released with the code\.