Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation

# Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation
Source: [https://arxiv.org/html/2607.05554](https://arxiv.org/html/2607.05554)
Sadia Kamal†, Arefa Patwary‡, Anthony Marchiafava†, Atriya Sen†, Sagnik Ray Choudhury‡†Oklahoma State University,‡University of North Texas\{sadia\.kamal,anmarch,atriya\.sen\}@okstate\.edu, arefapatwary@my\.unt\.edu,sagnik\.raychoudhury@unt\.edu

###### Abstract

Survey\-style evaluations of large language models often treat a prompted response as a measure of a model’s values or beliefs\. This assumption is particularly fragile when responses are read as evidence of political values, social attitudes, or beliefs\. We ask whether prompt robustness differs between objective questions with fixed answers and subjective questions that ask for opinions or values\. We evaluate four instruction\-tuned model families on three objective datasets \(MMLU, ARC, and CulturalBench\) and three subjective datasets \(Political Compass Test, ValueBench, and World Values Survey\)\. For each question/statement, we apply multiple types of prompt changes, such as variations in wording, framing, and format, and measure whether the model gives the same answer across variants\. Using a binomial generalized estimating equation, we find significant effects of model, dataset, prompt category, and their interactions\. The dataset type effect is also significant, and the interaction between dataset type and prompt category is large\. These results show that prompt robustness depends on the question type, the prompt change, and the model\.

Prompt Robustness Is Task\-Dependent: Comparing Objective and Belief\-Style Questions in LLM Evaluation

Sadia Kamal†, Arefa Patwary‡, Anthony Marchiafava†, Atriya Sen†, Sagnik Ray Choudhury‡†Oklahoma State University,‡University of North Texas\{sadia\.kamal,anmarch,atriya\.sen\}@okstate\.edu, arefapatwary@my\.unt\.edu,sagnik\.raychoudhury@unt\.edu

## 1Introduction

Most large language model evaluations rest on a fragile assumption that one prompt gives a stable and meaningful measure of model behavior\. Prior work shows this is often not the case, small changes in prompt format, wording, and answer presentation can change model behavior\(Sclaret al\.,[2024](https://arxiv.org/html/2607.05554#bib.bib7); Chatterjeeet al\.,[2024](https://arxiv.org/html/2607.05554#bib.bib6); Ismithdeenet al\.,[2025](https://arxiv.org/html/2607.05554#bib.bib8)\)\. If a model changes its answer when the task meaning stays the same, then the evaluation is measuring both task ability and prompt sensitivity\.

This is a critical concern for survey\-style tests\. Recent work uses political and value surveys to infer what LLMs “believe” or which human groups they resemble\. But survey responses from LLMs are unstable under ordering, labeling, forced\-choice wording, and framing changes\(Dominguez\-Olmedoet al\.,[2024](https://arxiv.org/html/2607.05554#bib.bib11); Röttgeret al\.,[2024](https://arxiv.org/html/2607.05554#bib.bib9); Rupprechtet al\.,[2025](https://arxiv.org/html/2607.05554#bib.bib12)\)\. A model’s answer can be shaped by the way the prompt asks the question\.

This raises a question about whether prompt sensitivity behaves the same way for different types of questions\. We study this problem by comparing two kinds of questions\. Type\-I questions consist of objective, multiple\-choice items with a single correct answer\. Type\-II questions are subjective survey items that ask for opinions, values, or degrees of agreement\. When a prompt is reworded without altering its underlying meaning, a robust model is expected to produce the same answer across both versions\. The same applies for objective questions as well\. For subjective questions, however, models may interpret minor changes in wording or response options as cues about how to respond\. As a result, the model’s answer can shift even when the survey item itself has not changed in substance\.

We ask three research questions:RQ1:Does response consistency differ between objective and subjective question types?RQ2:Do objective and subjective questions show different sensitivity patterns across prompt categories?RQ3:Is prompt robustness a model\-level property, or does it also depend on dataset and prompt category?

Our study makes three contributions\. First, we curate a broad perturbation set from prior work on prompt sensitivity, multiple\-choice formatting, survey response bias, and value measurement\. Second, we give a unified robustness test across objective and subjective datasets\. Third, we use clustered statistical tests to show that prompt sensitivity depends on model, dataset, dataset type, and perturbation category\. We find that subjective questions are less stable under prompt variation than objective questions\. The gap is not uniform across perturbation categories: option\-order changes produce the largest consistency drop, while lexical and logically equivalent perturbations are comparatively stable\. Statistical analysis shows that robustness depends jointly on model, dataset, dataset type, and prompt category\. Our findings show that LLM prompt sensitivity varies systematically with question type and should be evaluated across perturbation families rather than through a single prompt\.

## 2Related Work

#### Prompt sensitivity in LLM evaluation\.

Sclar et al\.\(Sclaret al\.,[2024](https://arxiv.org/html/2607.05554#bib.bib7)\)show that prompt formatting can produce large accuracy differences and argue that evaluation should report variation across plausible prompt formats, not only one prompt \. POSIX\(Chatterjeeet al\.,[2024](https://arxiv.org/html/2607.05554#bib.bib6)\)measures prompt sensitivity through likelihood change under intent\-preserving prompt variants \. Promptception\(Ismithdeenet al\.,[2025](https://arxiv.org/html/2607.05554#bib.bib8)\)builds a broad perturbation framework for multiple\-choice multimodal evaluation, including phrasing, formatting, position, and option presentation \. These works show that prompt design is part of the evaluation, not a neutral wrapper around it\.

#### LLM survey responses\.

Survey\-style evaluation has been used to study model values, politics, and social attitudes\. But recent studies warn that such responses can be strongly affected by answer ordering, labels, forced\-choice formats, and paraphrases\(Dominguez\-Olmedoet al\.,[2024](https://arxiv.org/html/2607.05554#bib.bib11); Röttgeret al\.,[2024](https://arxiv.org/html/2607.05554#bib.bib9); Wrightet al\.,[2024](https://arxiv.org/html/2607.05554#bib.bib10); Rupprechtet al\.,[2025](https://arxiv.org/html/2607.05554#bib.bib12)\)\. This makes political and value scores difficult to interpret as stable model traits\. Our work adds a direct comparison with objective multiple\-choice datasets, which helps separate general prompt noise from instability that is specific to subjective questions\.

## 3Experimental Design

### 3\.1Datasets and Models

We use six datasets categorized into two types\. The Type\-I group contains objective multiple\-choice questions: MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.05554#bib.bib14)\), ARC\(Clarket al\.,[2018](https://arxiv.org/html/2607.05554#bib.bib15)\), and CulturalBench\-Easy\(Chiuet al\.,[2024](https://arxiv.org/html/2607.05554#bib.bib16)\)\. The Type\-II group contains subjective or opinion\-based statements: the Political Compass Test, ValueBench\(Renet al\.,[2024](https://arxiv.org/html/2607.05554#bib.bib17)\), and World Values Survey items\(Haerpferet al\.,[2022](https://arxiv.org/html/2607.05554#bib.bib18)\)\. We use CulturalBench\-Easy as a Type\-I dataset because it tests cultural knowledge through multiple\-choice questions with fixed answer keys, rather than asking for subjective preferences or beliefs\. We evaluate four instruction\-tuned model families: Gemma, Llama, Mistral, and Qwen\.

### 3\.2Perturbation Curation

We conducted an extensive literature search across three lines of work: prompt robustness, multiple\-choice evaluation, and LLM survey measurement\. From this search, we selected perturbations only if they met the following criteria\. First, the perturbation had to preserve the intended task for both factual and survey\-style items\. Second, it had to test a known source of instability, such as wording, option labels, option order, or formatting\. Table[1](https://arxiv.org/html/2607.05554#S3.T1)gives the final taxonomy\.

Table 1:Prompt perturbation taxonomy\. The final categories were selected from prior work and then adapted so the same robustness design could be used for objective and subjective datasets\.
### 3\.3Prompt Generation and Answer Normalization

For each item, we create prompt variants that preserve the intended task\. All prompts use the same broad instruction: the model must select exactly one labeled option\. For subjective datasets, the labels map to agreement or value options\. For objective datasets, the labels map to answer choices\. We keep this common format so that the Type\-I and Type\-II groups differ mainly in question content, not in answer extraction\. All runs use deterministic decoding with temperature 0 so that answer changes come from prompt variation rather than sampling noise\.

The perturbation set is designed to separate three failure sources\. The first source is semantic framing\. Paraphrase, lexical substitution, spelling noise and logical equivalence ask whether two prompts with the same intended task produce the same answer\. The second source is surface form\. Format variation test whether the model is distracted by small changes that should not alter the task\. The third source is answer presentation\. Label substitution and option shuffling test whether the model follows the content of the options or instead follows labels and positions\. This separation matters because a single consistency score can hide different causes of instability\.

We use the same perturbation logic across objective and subjective datasets, but the interpretation differs\. On objective datasets, the answer key gives an external standard\. If the model changes from one option to another under a meaning\-preserving change, this is a robustness failure\. On subjective datasets, there is no single correct option\. Therefore, we do not call one answer right and another wrong\. We instead ask whether the model keeps its own answer stable when the survey item is presented in a controlled new form\.

We normalize outputs before computing consistency\. For objective datasets, raw outputs are mapped to canonical A, B, C, or D labels\. For subjective datasets, agreement labels are also mapped to canonical letters\. For substitution runs, symbols are mapped back to their original A/B/C/D meanings\. This prevents a numeric answer such as “1” and a letter answer such as “A” from being counted as inconsistent when they refer to the same option\.

### 3\.4Consistency Metric and Statistical Model

Our primary evaluation metric is answer consistency\. It measures whether a model gives the same answer across prompt variants\. Throughout this section, we will refer a single question from a dataset as an item\. For each item, model, and prompt category, we compute a consistency ratio:

Ci=maxy⁡ni​\(y\)Ni,C\_\{i\}=\\frac\{\\max\_\{y\}n\_\{i\}\(y\)\}\{N\_\{i\}\},\(1\)whereni​\(y\)n\_\{i\}\(y\)is the number of times answeryyappears for itemii, andNiN\_\{i\}is the number of valid prompt variants for that item\. A value of 1 means all variants gave the same answer\. Lower values mean the prompt category changed the answer\.

We fit binomial generalized estimating equation \(GEE\) models\(Liang and Zeger,[1986](https://arxiv.org/html/2607.05554#bib.bib13)\)\. GEE is useful here because each item appears many times under related prompt variants, so rows are not independent\. The response is whether a prompt variant matches the majority answer for that item\. We test main effects for model, dataset, dataset type, and prompt category, plus interactions between dataset or dataset type and prompt category, and between model and prompt category\. We report Waldχ2\\chi^\{2\}tests\.

![Refer to caption](https://arxiv.org/html/2607.05554v1/x1.png)\(a\)Gemma
![Refer to caption](https://arxiv.org/html/2607.05554v1/x2.png)\(b\)Llama
![Refer to caption](https://arxiv.org/html/2607.05554v1/x3.png)\(c\)Mistral
![Refer to caption](https://arxiv.org/html/2607.05554v1/x4.png)\(d\)Qwen

Figure 1:Dataset\-level mean consistency across perturbation categories for all four models\.

## 4Results

### 4\.1Subjective Questions Are More Affected Than Objective Questions

Figure[1](https://arxiv.org/html/2607.05554#S3.F1)shows that, across the four models, subjective datasets usually have lower consistency than objective datasets under most perturbation categories\. Averaged over all perturbations, Type\-I objective datasets have a mean consistency of 0\.849, while Type\-II subjective datasets have a mean consistency of 0\.787\. Equivalently, instability rises from 0\.151 for objective datasets to 0\.213 for subjective datasets\.

Subjective tasks are consistently less stable than objective ones across all four models, which answers RQ1\. The gap is persistent across Gemma, Llama, Mistral, and Qwen models, although the magnitude differs by model\. It reflects a systematic difference between question types\.

Table[2](https://arxiv.org/html/2607.05554#S4.T2)summarizes the marginal consistency scores by dataset type and perturbation category\. The Type\-I versus Type\-II gap is largest under option order\. For option\-order perturbations, Type\-I consistency is 0\.485, while Type\-II consistency is only 0\.328\. This means subjective datasets add 15\.7 percentage points of extra instability beyond the already large option\-order effect in objective datasets\. The next largest gaps appear under format variation and label substitution\. The smallest gaps appear under lexical substitution and logical equivalence, where both dataset types remain relatively stable\.

### 4\.2Option Order Is the Strongest Source of Instability

Having established that subjective questions are less robust overall, we next ask which perturbation category drives the largest instability\. Across all four models and all six datasets, option order perturbation category affects the most\. Averaged over models and datasets, option\-order perturbations reduce mean consistency to 0\.407\. This is much lower than label substitution, 0\.799, and format variation, 0\.808\. Semantic and near\-semantic perturbations are more stable: paraphrase reaches 0\.913, spelling noise reaches 0\.927, logical equivalence reaches 0\.935, and lexical substitution reaches 0\.936\.

This pattern is visible in each model in Figure[1](https://arxiv.org/html/2607.05554#S3.F1)\. For Gemma, Llama, and Mistral, option order produces the lowest consistency values for the subjective datasets, particularly PCT, ValueBench, and WVS\. Qwen is more robust overall, but option order still produces its clearest drop compared with semantic perturbations\. Thus, the main finding is not that models are equally fragile to all prompt changes\. Instead, answer\-presentation perturbations, particularly option order, create the largest instability\.

Table 2:Mean consistency by dataset type and perturbation category\. Gap is computed as Type\-I consistency minus Type\-II consistency, so larger values indicate greater subjective\-task instability\.This addresses RQ2, objective and subjective questions do not merely differ in overall robustness\. They differ in the kinds of prompt changes that affect them most\. The subjective\-task gap is largest when the prompt changes how answers are presented, rather than when it only changes lexical wording\.

### 4\.3Dataset and Prompt Category Interact Strongly

Table[3](https://arxiv.org/html/2607.05554#S4.T3)demonstrates the global GEE model with dataset as a six\-level factor\. Model, dataset, and prompt category are all significant predictors of consistency\. More importantly, the dataset by prompt category interaction is large and significant,χ2=587\.885\\chi^\{2\}=587\.885,p=9\.52×10−105p=9\.52\\times 10^\{\-105\}\. This confirms the pattern in Figure[1](https://arxiv.org/html/2607.05554#S3.F1): perturbation categories do not affect all datasets in the same way\. The model by prompt category interaction is also significant,χ2=616\.748\\chi^\{2\}=616\.748,p=2\.47×10−119p=2\.47\\times 10^\{\-119\}, showing that models differ in which perturbations they are most sensitive to\.

Table 3:Global Waldχ2\\chi^\{2\}tests with dataset as a six\-level factor\.
### 4\.4Dataset Type Explains a Systematic Robustness Gap

Table[4](https://arxiv.org/html/2607.05554#S4.T4)divides datasets into Type\-I and Type\-II groups\. Dataset type remains highly significant,χ2=256\.026\\chi^\{2\}=256\.026,p=1\.26×10−57p=1\.26\\times 10^\{\-57\}\. The interaction between dataset type and prompt category is also significant,χ2=175\.435\\chi^\{2\}=175\.435,p=3\.16×10−35p=3\.16\\times 10^\{\-35\}\. This interaction shows that subjective datasets are not merely lower in consistency on average\. They are differently affected by different perturbation categories\.

Table 4:Global Waldχ2\\chi^\{2\}tests with dataset type as Type\-I versus Type\-II\.The results underscores that single\-prompt survey measurements are notably fragile\. For objective multiple\-choice questions, there is an external target answer that can constrain the model\. For subjective survey items, small changes in wording, formatting, labels, or option order can shift the response without any external gold answer\. This makes a single response difficult to interpret as a stable value or belief\.

### 4\.5All Datasets Show Prompt Effects

Table[5](https://arxiv.org/html/2607.05554#S4.T5)reports the prompt\-category effect from within\-dataset GEE models\. Prompt category is significant for all six datasets\. The largest effects appear in the subjective datasets: WVS,χ2=1146\.897\\chi^\{2\}=1146\.897, ValueBench,χ2=993\.681\\chi^\{2\}=993\.681, and PCT,χ2=757\.436\\chi^\{2\}=757\.436\. Objective datasets also show significant effects, but the magnitudes are smaller: ARC,χ2=191\.625\\chi^\{2\}=191\.625, MMLU,χ2=96\.950\\chi^\{2\}=96\.950, and CulturalBench,χ2=279\.848\\chi^\{2\}=279\.848\.

Table 5:Prompt\-category effects from within\-dataset GEE models\.This result rules out a narrow explanation in which prompt sensitivity is only a survey problem\. Prompt variation affects all datasets\. However, the effect is more consequential for subjective datasets because a shifted answer can change the inferred political position, social value, or belief attributed to the model\.

### 4\.6Robustness Varies Across Models, Datasets, and Perturbations

The model\-wise heatmaps in Figure[1](https://arxiv.org/html/2607.05554#S3.F1)show that robustness is not a single model trait\. A model can be stable under paraphrase or lexical substitution but unstable under option order\. The same model can also behave differently across objective and subjective datasets\. For example, Qwen is generally more stable than the other models under option order, but it still shows a visible drop compared with semantic perturbations\. Llama and Mistral are highly stable on several objective cells but much less stable on subjective option\-order cells\.

The model by prompt category interactions in Tables[3](https://arxiv.org/html/2607.05554#S4.T3)and[4](https://arxiv.org/html/2607.05554#S4.T4)confirm this observation statistically\. Appendix Table[6](https://arxiv.org/html/2607.05554#S9.T6)reports the full model by dataset prompt\-effect table\. Every model\-dataset bin shows a significant prompt\-category effect\. Therefore, robustness should be reported as a relation among model, dataset, and perturbation category, not as a single global score\.

This matters for model comparison\. A model that appears robust on MMLU paraphrases may not be robust on WVS option order or PCT label substitution\. A single average robustness score can therefore hide important failures\. For survey\-style evaluations, the most important warning is that a single prompted answer should not be treated as a stable measurement of model values or beliefs\.

## 5Discussion

These results have a direct implication for LLM survey studies\. A single answer to a political or value question should not be treated as a stable model belief unless it survives prompt variation\. This does not mean survey\-style evaluation is useless\. It means that survey scores need robustness checks\. If two meaning\-preserving prompts produce different answers, the result cannot be attributed to the model alone\. The objective\-subjective comparison sheds light on why this happens\. Objective questions have fixed answers, so a change in answer usually signals task failure or prompt sensitivity\. Subjective questions are different\. A model may use wording, label choice, or option order as evidence about how it should respond\. In that case, prompt variation is not only noise\. It is also a way to reveal how fragile the measured stance is\. Researchers using survey\-style evaluations should therefore report consistency across prompt variants, not only a final accuracy or survey score\. Semantic perturbations should be separated from surface perturbations, since they reveal different failure modes\. Interactions with dataset type should also be reported\.

Beyond measuring consistency, the perturbation taxonomy can identify where instability comes from\. Failures under spelling or format changes suggest brittle instruction following\. Failures under label substitution or option shuffling suggest answer\-position bias or label\-token bias\. Failures under paraphrase or logical equivalence are the most consequential for survey interpretation, because the measured stance changes even when the item meaning is intended to stay fixed\. These failure types should not be collapsed into a single error category\.

## 6Limitations

This study uses deterministic decoding and forced\-choice answers\. This helps isolate prompt effects, but it does not cover open\-ended survey responses\. The model names are treated at the family level, so future work should report exact checkpoint versions and parameter sizes\. Finally, subjective datasets are not interchangeable\. PCT, ValueBench, and WVS differ in wording, scale design, and social meaning\.

## 7Conclusion

We compared prompt robustness across objective and subjective questions using six datasets and four model families\. Consistency depends strongly on model, dataset, prompt category, and dataset type\. Subjective survey tasks show substantially larger prompt\-category effects than objective tasks, and these effects vary by both the type of prompt change and the model being tested\. Characterizing model robustness therefore requires systematic variation across prompt forms\. For value and belief evaluation, robustness to prompt variation should be a standard part of the evaluation design\.

## References

- POSIX: a prompt sensitivity index for large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 14550–14565\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.852)Cited by:[§1](https://arxiv.org/html/2607.05554#S1.p1.1),[§2](https://arxiv.org/html/2607.05554#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.2.1.4.1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.4.3.4.1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.5.4.4.1.1)\.
- Y\. Y\. Chiu, L\. Jiang, B\. Y\. Lin, C\. Y\. Park, S\. S\. Li, S\. Ravi, M\. Bhatia, M\. Antoniak, Y\. Tsvetkov, V\. Shwartz, and Y\. Choi \(2024\)CulturalBench: a robust, diverse and challenging benchmark on measuring the lack of cultural knowledge of LLMs\.External Links:2410\.02677Cited by:[§3\.1](https://arxiv.org/html/2607.05554#S3.SS1.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§3\.1](https://arxiv.org/html/2607.05554#S3.SS1.p1.1)\.
- R\. Dominguez\-Olmedo, M\. Hardt, and C\. Mendler\-Dünner \(2024\)Questioning the survey responses of large language models\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§1](https://arxiv.org/html/2607.05554#S1.p2.1),[§2](https://arxiv.org/html/2607.05554#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.8.7.4.1.1)\.
- C\. Haerpfer, R\. Inglehart, A\. Moreno, C\. Welzel, K\. Kizilova, J\. Diez\-Medrano, M\. Lagos, P\. Norris, E\. Ponarin, and B\. Puranen \(2022\)World values survey wave 7 \(2017\-2022\) cross\-national data\-set\.World Values Survey Association\.External Links:[Document](https://dx.doi.org/10.14281/18241.18)Cited by:[§3\.1](https://arxiv.org/html/2607.05554#S3.SS1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations,Cited by:[§3\.1](https://arxiv.org/html/2607.05554#S3.SS1.p1.1)\.
- M\. I\. Ismithdeen, M\. U\. Khattak, and S\. Khan \(2025\)Promptception: how sensitive are large multimodal models to prompts?\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 23950–23985\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1302)Cited by:[§1](https://arxiv.org/html/2607.05554#S1.p1.1),[§2](https://arxiv.org/html/2607.05554#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.3.2.4.1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.6.5.4.1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.7.6.4.1.1)\.
- K\. Liang and S\. L\. Zeger \(1986\)Longitudinal data analysis using generalized linear models\.Biometrika73\(1\),pp\. 13–22\.External Links:[Document](https://dx.doi.org/10.1093/biomet/73.1.13)Cited by:[§3\.4](https://arxiv.org/html/2607.05554#S3.SS4.p2.1)\.
- Y\. Ren, H\. Ye, H\. Fang, X\. Zhang, and G\. Song \(2024\)ValueBench: towards comprehensively evaluating value orientations and understanding of large language models\.External Links:2406\.04214Cited by:[§3\.1](https://arxiv.org/html/2607.05554#S3.SS1.p1.1)\.
- P\. Röttger, V\. Hofmann, V\. Pyatkin, M\. Hinck, H\. R\. Kirk, H\. Schuetze, and D\. Hovy \(2024\)Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,pp\. 15295–15311\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.816)Cited by:[§1](https://arxiv.org/html/2607.05554#S1.p2.1),[§2](https://arxiv.org/html/2607.05554#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.2.1.4.1.1)\.
- J\. Rupprecht, G\. Ahnert, and M\. Strohmaier \(2025\)Prompt perturbations reveal human\-like biases in large language model survey responses\.External Links:2507\.07188Cited by:[§1](https://arxiv.org/html/2607.05554#S1.p2.1),[§2](https://arxiv.org/html/2607.05554#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.6.5.4.1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.8.7.4.1.1)\.
- M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. Suhr \(2024\)Quantifying language models’ sensitivity to spurious features in prompt design or: how i learned to start worrying about prompt formatting\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.05554#S1.p1.1),[§2](https://arxiv.org/html/2607.05554#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.2.1.4.1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.5.4.4.1.1),[Table 1](https://arxiv.org/html/2607.05554#S3.T1.1.7.6.4.1.1)\.
- D\. Wright, A\. Arora, N\. Borenstein, S\. Yadav, S\. Belongie, and I\. Augenstein \(2024\)LLM tropes: revealing fine\-grained values and opinions in large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Cited by:[§2](https://arxiv.org/html/2607.05554#S2.SS0.SSS0.Px2.p1.1)\.

## 8Appendix

## 9Full Model by Dataset Prompt Effects

Table[6](https://arxiv.org/html/2607.05554#S9.T6)reports the within\-cell prompt\-category effect for each model and dataset pair\. Each row comes from a separate binomial GEE model fit within one model\-dataset cell\. The table shows that prompt category has a significant effect in every model\-dataset combination, which supports the main claim that prompt robustness is not a single global model property\. Instead, it varies with the model, dataset, and perturbation category\.

Table 6:Prompt\-category effects within each model\-dataset binomial GEE model\. This is the full version of the model by dataset result table\.