Persona Non Grata: LLM Persona-Driven Generations in MCQA are Unstable in Distinct Dimensions

# LLM Persona-Driven Generations in MCQA are Unstable in Distinct Dimensions
Source: [https://arxiv.org/html/2607.00937](https://arxiv.org/html/2607.00937)
César Guerra\-Solano,Xiang Lorraine Li Department of Computer Science, University of Pittsburgh Correspondence:\{cguerrasol, xianglli\}@pitt\.edu

###### Abstract

Persona\-driven generations \(PDGs\) have seen prolific use in research and industry applications, where a large language model \(LLM\) takes on a “persona” while completing some task\. While persona expressed through free\-form text \(like dialogue\) has substantial work investigating stability or consistency, relatively, persona expressed in non\-text\-heavy outputs \(like in multiple\-choice question answering, or MCQA\) is often overlooked\. We work to address this gap, seeking to understand the instability of LLM PDGs in MCQA tasks\. We develop three metrics investigating the performance, outcome, and question correctness stability, evaluating three distinct dimensions\. Using these metrics, we find that instability varies consistently between model families and model size, and across question domains, with math/commonsense questions leading to greater instability\. We also find task prompt format introduces more prediction instability than other hyperparameters, like temperature\. Finally, we find that instability is related to task accuracy, and using our instability metrics, find different experimental settings that result in different best and worst personas for tasks, despite their similarity\. This reveals the importance of checking hyperparameter instability in PDGs\.

Persona Non Grata: LLM Persona\-Driven Generations in MCQA are Unstable in Distinct Dimensions

César Guerra\-Solano, Xiang Lorraine LiDepartment of Computer Science, University of PittsburghCorrespondence:\{cguerrasol, xianglli\}@pitt\.edu

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.00937v1/x1.png)Figure 1:Depicting instability in non\-text\-heavy persona\-driven generations \(PDGs\)\. Given different, although still reasonable, experiment configurations, represented by different settings for the task prompt format \(TPF\) hyperparameter, an LLM’s persona\-driven performance on a multiple\-choice evaluation differs greatly\.Large language models \(LLMs\) have seen prolific use in a variety of domains, with their adaptability and broad capabilities lending themselves to high performance across a range of tasks and opportunities for personalization for user\- and task\-specific needsKojimaet al\.\([2022](https://arxiv.org/html/2607.00937#bib.bib35)\); Brownet al\.\([2020](https://arxiv.org/html/2607.00937#bib.bib34)\); Grattafioriet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib36)\); Yanget al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib37)\); Zhanget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib33)\)\. With this,persona\-driven generations \(PDGs\)have become prevalent – here, by leveraging the power of prompting, LLMs can role\-play as a "persona" while carrying out some task, such as general user assistance \(e\.g\. "You are a teacher\. Explain \[…\]"\) or more specific task completion \(e\.g\. "You are a hiring manager\. Rate these resumes \[…\]"\)\. As PDGs see use in high\-stakes domains, from medicine to education, evaluating their stability is criticalSunet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib22)\); Yuanet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib23)\); Kyunget al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib26)\); Liet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib27)\)\.

With PDGs, two main uses can be observed: in free\-form text generation settings, such as in dialogue systems like Character\.AI[Character\.AI](https://arxiv.org/html/2607.00937#bib.bib32), and non\-text\-heavy outputs, such as in persona\-based multiple\-choice question answering \(MCQA\)\. In the latter case, LLM persona is expressed through task completion: rather than producing text that reflects thestyleof a persona, an LLM answers questions in accordance with thecapabilitiesassociated with that persona, when relevant \(e\.g\., a biologist answering biology questions\), or to remain unaffected by persona choice when the persona is irrelevant \(e\.g\., race should not influence performance on mathematical questions\)\.

However, relative to text generation, little work has characterized or improved PDGs in MCQA\. Prior studies indicate that PDGs in this setting are implicitly biased and unpredictableGuptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\); Zhenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib42)\), motivating further work to characterize them and their potential flaws\. Additionally, these studies bear little standardization, with large differences in model choice and hyperparameters across their experiment settings – features known to affect model performance, thus being targets of optimizationWanget al\.\([2023](https://arxiv.org/html/2607.00937#bib.bib20)\); Sclaret al\.\([2023](https://arxiv.org/html/2607.00937#bib.bib43)\)\. With the diverse experiment settings that PDGs are used in, differing in persona, task, and prompt format,additional investigations are necessary to find the effects of these variationsto ensure current and future findings are robust\.

We work to address this gap, seeking to further understand PDGs in MCQA, quantifying their stability across diverse environments\. We do so relative tothree persona\-specific metrics, targeting unique and distinct features of PDG paradigms – we look at not only stability ofaccuracyas in related past workGuptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\); Zhenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib42)\); de Araujoet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib18)\), but also stability ofexperimental outcomesandpersona\-specific question correctness\.

We evaluate 4 open\-source models across 48 experiment settings and 41 personas, finding:

- •PDGs in MCQA areheavily unstable, with great variation across similar experiment settings used in past work, and consistent patterns with model size and family\.
- •Instability isspecific, with distinct relationships relative to 3 dimensions of instability, types of content in the input prompt \(such as the question formatting vs\. persona formatting\), and the subject domain of the task\.
- •Our instability metrics areactionable, with demonstrated usage in identifying stable and unstable experiment settings that greatly differ in overall accuracy and persona\-related conclusions, such as the best persona for a task, or inter\-persona distribution shifts\.

## 2Related Work

Persona\-Driven GenerationsPersona\-based generation is widely used across LLM downstream tasks, including dialogue systemsHuanget al\.\([2023](https://arxiv.org/html/2607.00937#bib.bib2)\), narrative generationZhanget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib33)\), and LLM\-as\-a\-judge settings where models adopt expert rolesDonget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib16)\); Yuet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib15)\); Liet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib14)\)\. These applications generally fall into two categories: free\-form text generation, where persona is embedded into generation style and content \(e\.g\., Character\.AI[Character\.AI](https://arxiv.org/html/2607.00937#bib.bib32)\), and non\-text\-heavy tasks, like persona\-based task\-taking, where persona is reflected in task performance\.

Evaluation of PDGs has primarily focused on text\-heavy settings, examining consistency, stability, and failures such as sensitivity or bias in long\-form generationZhanget al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib19)\); Liuet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib31)\); Maharanaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib30)\); Tsenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib29)\); de Araujoet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib18)\); Chenget al\.\([2023](https://arxiv.org/html/2607.00937#bib.bib17)\)\. By contrast, non\-text\-heavy PDGs—though common in tasks like LLM\-as\-a\-judge or MCQAde Araujoet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib18)\)—remain underexplored, with only limited attention to bias, faithfulness, or unpredictabilityZhenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib42)\); Guptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\); de Araujoet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib18)\), leaving a key gap in comprehensively assessing their robustness\.

Our work is related tode Araujoet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib18)\)which evaluates how well personasmatch human expectations or needswith personas, like if expert personas confer a performance advantage\. However, we focus on howstablePDGs are, i\.e, the consistency of MCQA performance given personas across noted variations in experiment setting such as prompt format and generation temperature\.

Generation SensitivityPrior work on model generation sensitivity has primarily focused on model temperature and input prompt styles/formatting, with evaluation metrics emphasizing macro\-level aspects such as task or response structure and performanceSclaret al\.\([2023](https://arxiv.org/html/2607.00937#bib.bib43)\); Chatterjeeet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib13)\); Liet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib14)\); Brownet al\.\([2020](https://arxiv.org/html/2607.00937#bib.bib34)\); Heet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib12)\)\. However, these studies primarily center on persona\-agnostic contexts, leaving the need to investigate how experiment setting variations influence LLMs’ role\-playing capabilities\. Moreover, existing sensitivity metricsZhuoet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib44)\)largely focus on accuracy, overlooking outcome\- and question\-level sensitivity, a gap our work addresses\.

## 3Instability Metrics

![Refer to caption](https://arxiv.org/html/2607.00937v1/x2.png)Figure 2:A figure depicting how instability metrics are calculated based on PDGs for our dataset\. Across runs in several unique experimental configurations \(ECs\), we consider mean performance and question\-wise correctness\. Mean performance is used for standard deviation calculations for computingIAI\_\{A\}and used to get inter\-persona rankings within one EC forIOI\_\{O\}\. The set of correct questions for a persona across all ECs is used forIQI\_\{Q\}calculation\. Calculations shown are for one persona, although all personas are integrated as shown in Section[2](https://arxiv.org/html/2607.00937#S3.F2)\. We explore how varying the question set and experiment setting set sizes affects metric calculations in Appendices[E\.6](https://arxiv.org/html/2607.00937#A5.SS6)and[E\.7](https://arxiv.org/html/2607.00937#A5.SS7)\.In this paper, we define instability in persona\-driven generations as robustness to experimental settings\. We examine how evaluation trends shift across hyperparameter settings such as prompt format and model temperature, with each settings∈Ss\\in Srepresenting a hyperparameter combination\.

To quantify performance variation, we propose three metrics that measure differences inaccuracy,outcome, andcorrect questions\. Prior workZhuoet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib44)\); Sclaret al\.\([2023](https://arxiv.org/html/2607.00937#bib.bib43)\)has primarily focused only on accuracy differences\. However, instability exists agnostic of accuracy – you can have large accuracy differences but no difference in inter\-persona ranking, or identical accuracy but great inconsistency inwhatquestions are answered correctly\. Thus, further dimensions of instability should be investigated, especially in PDGs\. We define each metric using the following terms\.111For validation, we report the correlation of our metric results with metrics from existing literature or reasonable correlates in Appendix[E\.4](https://arxiv.org/html/2607.00937#A5.SS4)\.

Experimental Settings:Each metric is calculated relative to a set of considered experimental settings,SS\. This allows us to analyze instability on broader \(e\.g\., considering all experimental settings\) or smaller \(e\.g\., varying one hyperparameter within an otherwise fixed setting\) scales\.

PersonasEach metric considers a set of personas,PP, that are provided to the model\. Each persona is evaluated under multiple experimental settings, and is expected to yield slightly different results that, in turn, affect performance, outcome, and correct question instability metrics in different ways\.

Persona AccuracyThe accuracy of a personappis calculated for a set of questionsQQunder an experimental settingssdenoted as,Acc​\(p;s,Q\)\\textit\{Acc\}\(p;s,Q\)\. Note that the question setQQcould be the entire dataset, or specific subsets, such as domain\-specific questions\.

Persona RankGiven persona accuracyAcc​\(p;s,Q\)\\textit\{Acc\}\(p;s,Q\), the rank of a personapprelative to other personas is defined as the index ofppin a list ranked by accuracy, denoted asR​\(p;s,Q\)R\(p;s,Q\)\.222In the rest of section[2](https://arxiv.org/html/2607.00937#S3.F2), we omitQQin accuracyAcc​\(p;s\)\\textit\{Acc\}\(p;s\)and rankR​\(p;s\)R\(p;s\)asQQis implied in computing these metrics\.

### 3\.1Performance Instability:IAI\_\{A\}

The first metric aims to understand how sensitive model accuracy is across all experimental settings \(averaged over personas\)\. Intuitively, we seek to understand that, if we randomly selected two experimental settings from a set of potential settings, what is the expected difference in accuracy?

IA​\(p\)\\displaystyle I\_\{A\}\(p\)=1\(\|S\|2\)​∑si,sj∈S\|Acc⁡\(p;si\)−Acc⁡\(p;sj\)\|\\displaystyle=\\frac\{1\}\{\\binom\{\|S\|\}\{2\}\}\\sum\_\{s\_\{i\},\\,s\_\{j\}\\in S\}\\bigl\|\\operatorname\{Acc\}\(p;s\_\{i\}\)\-\\operatorname\{Acc\}\(p;s\_\{j\}\)\\bigr\|IA\\displaystyle I\_\{A\}=1\|P\|​∑p∈PIA​\(p\)\\displaystyle=\\frac\{1\}\{\|P\|\}\\sum\_\{p\\in P\}I\_\{A\}\(p\)
The formulation is similar toZhuoet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib44)\), but fitted to a persona\-specific context\. For a persona, we calculate the pair\-wise accuracy differences betweensis\_\{i\}andsjs\_\{j\}for all experiment settings, then average them\.\(\|S\|2\)\\binom\{\|S\|\}\{2\}represents the count of experiment settings pairs for a single persona\. We aggregate the persona instability to represent the accuracy instabilityIAI\_\{A\}for a given question set, and a model, where\|P\|\|P\|is the total number of personas\.

### 3\.2Outcome InstabilityIOI\_\{O\}

This metric measures how variable the conclusion of a persona\-driven experiment is, such as identifying the best or worst persona for a task\. To do this, we compare persona rankingR​\(p;s\)R\(p;s\)across experimental settings\. Importantly, we note that for each experimental setting, both minor and substantial differences in persona average accuracies can change their rankings, leading to outcome instability\. However, intuitively, we argue that the degree of this instability is different: larger accuracy differences should have a larger outcome instability score\. To account for this, we define the outcome instability of each persona as the weighted standard deviation of its rankings, where the weight assigned to each experimental setting corresponds to the standard deviation across all personas’ accuracies in that setting\. At the model level, overall outcome instability is computed by averaging the persona\-level instabilities and scaling by the average ofwsw\_\{s\}across experimental settings, accounting for models with greater persona parity overall\.

w​\(s\)\\displaystyle w\(s\)=Std⁡\(Acc⁡\(pi;s\)\),pi∈P\\displaystyle=\\operatorname\{Std\}\\\!\\bigl\(\\operatorname\{Acc\}\(p\_\{i\};s\)\\bigr\),\\quad p\_\{i\}\\in PIO​\(p\)\\displaystyle I\_\{O\}\(p\)=Stdw⁡\(R​\(p;sj\),w​\(sj\)\),sj∈S\\displaystyle=\\operatorname\{Std\}\_\{w\}\\\!\\bigl\(R\(p;s\_\{j\}\),\\,w\(s\_\{j\}\)\\bigr\),\\quad s\_\{j\}\\in SIO\\displaystyle I\_\{O\}=1\|P\|​∑pi∈PIO​\(pi\)⋅1\|S\|​∑sj∈Sw​\(sj\)\\displaystyle=\\frac\{1\}\{\|P\|\}\\sum\_\{p\_\{i\}\\in P\}I\_\{O\}\(p\_\{i\}\)\\;\\cdot\\;\\frac\{1\}\{\|S\|\}\\sum\_\{s\_\{j\}\\in S\}w\(s\_\{j\}\)
In the above equations,w​\(s\)w\(s\)is calculated with the standard deviation of persona accuraciesAcc⁡\(pi;s\)\\operatorname\{Acc\}\(p\_\{i\};s\)across all personas\. The outcome instability of a personaIO​\(p\)I\_\{O\}\(p\)is computed with the weighted standard deviationSeabold and Perktold \([2010](https://arxiv.org/html/2607.00937#bib.bib6)\)of the persona’s rankR​\(p;sj\)R\(p;s\_\{j\}\)using the weightw​\(sj\)w\(s\_\{j\}\)across all experiment settingssjs\_\{j\}\.IOI\_\{O\}is the overall model\-level outcome instability computed usingIO​\(p\)I\_\{O\}\(p\)andw​\(s\)w\(s\)\.333We experimented with several alternative formulations ofIOI\_\{O\}; see Appendix[E\.5](https://arxiv.org/html/2607.00937#A5.SS5)for details\.

### 3\.3Question Correctness InstabilityIQI\_\{Q\}

The third metric evaluates the variability of the specific questions that a persona\-driven generation answers correctly across experimental settings\. Personas associated with a task domain, for example, a biologist in biology, should consistently answer domain\-relevant questions correctly; however, the performance may still be sensitive to the experimental settings\. Thus, we calculate the percent of questions correct that are shared across all experimental settings given a persona, seeking to answer: How consistent are persona\-specific performances across experimental settings?

C​Q​\(p;s\)\\displaystyle CQ\(p;s\)=\{q∈Q∣q​is correct usingpins\}\\displaystyle=\\\{\\,q\\in Q\\;\\mid\\;q\\text\{ is correct using $p$ in $s$\}\\,\\\}IQ\\displaystyle I\_\{Q\}\\;=1−1\|P\|​∑p∈P\|⋂s∈SC​Q​\(p;s\)\|\|⋃s∈SC​Q​\(p;s\)\|\\displaystyle=\\;1\-\\frac\{1\}\{\|P\|\}\\sum\_\{p\\in P\}\\frac\{\\bigl\|\\;\\bigcap\_\{s\\in S\}CQ\(p;s\)\\;\\bigr\|\}\{\\bigl\|\\;\\bigcup\_\{s\\in S\}CQ\(p;s\)\\;\\bigr\|\}
The above equation quantifies the calculation ofIQI\_\{Q\}\. Here,C​Q​\(p;s\)CQ\(p;s\)represents the set of questions correctly answered by personappunder experimental settingss\. For each persona, we first compute the Jaccard similarity of these sets across all settings, reflecting the consistency of prediction on a question level\. This is averaged over personas\. To align with the interpretation of the other instability metrics, we subtract this value from 1, to represent the percentage of questions that are not shared \(thus greater value indicates higher instability\)\.

## 4Experiments

Table 1:A table depicting the baseline overall instability results for each model, reporting mean and standard deviation of accuracy across all runs, and our three instability metrics\. It can be noted that larger model size leads to less instability across all dimensions, and differences in instability between model families\.Table 2:Example personas used in experimentation\.Table 3:Dataset categories used in experimentation\.For experimentation, we evaluate 4 open\-source LLMs on an aggregated dataset with questions from different domains under a set of experimental settings for instability calculations and analyses\. We introduce our persona, dataset and models below\. Section[4\.1](https://arxiv.org/html/2607.00937#S4.SS1)explains the hyperparameters used to derive all experimental settings\.

PersonasWe consider 41 personas, split across 9 distinct groups centering on aspects of identity \(such as race\) and expertise \(such as occupation\) for comparison\. As we seek to compare stability of PDGs across experiment settings similar to those in past literature, these personas are also drawn from previous work, with additional personas added in order to ensure inclusivity and better representation per categoryZhenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib42)\); Guptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\); Wanet al\.\([2023](https://arxiv.org/html/2607.00937#bib.bib28)\)\. The personas and categorization can be seen in Table[7](https://arxiv.org/html/2607.00937#A4.T7)\. We use both occupation\-related personas and demographic\-related personas due to their usage in past PDG work, and to improve the generalizability of our findings to future PDG studies\. This is further discussed in Section[Ethical Considerations](https://arxiv.org/html/2607.00937#Sx2), with desiderata for stability based on the persona used discussed in Appendix[D\.2\.1](https://arxiv.org/html/2607.00937#A4.SS2.SSS1)\.

DatasetTo evaluate and understand PDG instability in LLMs, we create a composite evaluation dataset,𝒟\\mathcal\{D\}, balanced across various task types and domains for our experiments\. We pull from MMLUHendryckset al\.\([2020](https://arxiv.org/html/2607.00937#bib.bib40)\), Social IQaSapet al\.\([2019](https://arxiv.org/html/2607.00937#bib.bib39)\), and NormAd\-EtiRaoet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib38)\)\. We group questions under 10 different categories, composed of questions with specific sublabels from MMLU, Social IQa, and NormAd\-Eti\. The categories and example labels are in Table[3](https://arxiv.org/html/2607.00937#S4.T3)\. Further details for dataset and persona are in Appendix[D\.1](https://arxiv.org/html/2607.00937#A4.SS1)\.

ModelsWe evaluate five open\-source models: Llama3\.2\-1B\-Instruct and Llama3\.1\-8B\-InstructGrattafioriet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib36)\), Qwen2\.5\-1\.5B\-Instruct, Qwen2\.5\-7B\-Instruct, and Qwen2\.5\-14B\-InstructYanget al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib37)\)\. These were chosen due to allow comparison across model size and family\.

### 4\.1Experiment Setting Hyperparameters

We consider three experiment setting hyperparameters, drawing from noted differences from prior workGuptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\); Zhenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib42)\); de Araujoet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib18)\)\. We evaluate our models across all combinations of these hyperparameters, resulting in a total of 48 experiment settings\.

We seek to understand instability due tohyperparametervariations in this persona context\. We additionally analyze instability due topersonavariations in Appendix[E\.1](https://arxiv.org/html/2607.00937#A5.SS1)\.

Persona Prompt Format\(PPF\) is the format of the system prompt used to give an LLM a persona\. We draw upon variants from past work that varied in length and abstractness, utilizing four PPF settingsGuptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\); Zhenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib42)\)\.

Task Prompt Format\(TPF\) is the format of the prompt used to give an LLM a question\. Similar to PPF, we draw upon variants from past workGuptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\); Zhenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib42)\); Weiet al\.\([2022](https://arxiv.org/html/2607.00937#bib.bib11)\)that varied in length and expressiveness, considering four TPF settings\. Both PPF and TPF examples are shown in Table[5](https://arxiv.org/html/2607.00937#A1.T5)in the Appendix\.

Temperature, or model temperature \(τ\\tau\), is considered\. We utilize three different settings – 0, 0\.5, and 1\.0 – that have been observed in previous workGuptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\); Zhenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib42)\)\.

## 5Results and Analysis

### 5\.1Baselines

![Refer to caption](https://arxiv.org/html/2607.00937v1/x3.png)Figure 3:A figure depicting the delta hyperparameter instability of temperature \(Temp\.\), persona prompt format \(PPF\), and task prompt format \(TPF\) compared to the overall baseline, stratified by experimental settings with specific settings for each hyperparameter\. Here, a greater value \(brighter\) means less instability compared to the baseline\. A key for the column labels is in Appendix[F\.1](https://arxiv.org/html/2607.00937#A6.SS1)\.We perform baseline experiments for all 41 personas and 48 experimental settings per model\. Both model performance and instability are shown in Table[1](https://arxiv.org/html/2607.00937#S4.T1)\. Note that for all metrics, a perfectly stable model would yield a score of 0, indicating consistent performance across different hyperparameter settings\. In practice, hyperparameters inevitably cause fluctuations, so non\-zero values are expected\.

As shown in Table[1](https://arxiv.org/html/2607.00937#S4.T1), all evaluated models’ instability measurements are well above 0, with particularly highIQI\_\{Q\}scores indicating the limited overlap in correctly answered questions between experiment settings, like less than 1% for the Llama 1B model\. Secondly, higher mean accuracy largely corresponds to lower accuracy instability: Llama 1B and Qwen2\.5\-1\.5B show relatively low accuracy and a highIAI\_\{A\}, in contrast to their larger counterparts with a high accuracy and a lowerIAI\_\{A\}\. Outcome instabilityIOI\_\{O\}shows a clear trend with both model performance and size, with larger models demonstrating greater stability across all metrics444The only exception to this trend is Qwen2\.5\-14B\. We discuss our rationale for this pattern in Appendix[E\.9](https://arxiv.org/html/2607.00937#A5.SS9)\.\. We refer to these results as the baseline results\. To better compare across models and model families, all subsequent results are reported relative to these baselines\. Specifically, we compute the instability difference as \(baseline – new\), where a positive value indicates lower instability in the new experiments and an improvement over the baseline\.

### 5\.2Instability From Each Hyperparameter

![Refer to caption](https://arxiv.org/html/2607.00937v1/x4.png)Figure 4:Depicting the delta hyperparameter instability of Temp\., PPF, and TPF compared to the baseline\. Here, a greater value means less instability compared to the baseline\. We report results for each hyperparameter divided by those of the baseline to improve readability\.![Refer to caption](https://arxiv.org/html/2607.00937v1/x5.png)Figure 5:Depicting the delta hyperparameter instability of Temp\., PPF, and TPF compared to the baseline\. "Dataset" and "Persona" are stratified by the question and persona categories, respectively\. A label key is in Appendix[F\.1](https://arxiv.org/html/2607.00937#A6.SS1)\.We aim to investigate the role of single hyperparameter variations \(τ\\tau, PPF, and TPF\) in instability\. Here, the experimental settings setSSis built by varying one hyperparameter at a time \(e\.g\.,τ\\tauat 0\.0, 0\.5, 1\.0\) while holding others fixed\. For each hyperparameter, results are averaged across all fixed settings\. Figure[4](https://arxiv.org/html/2607.00937#S5.F4)shows the difference between the instability scores to baseline scores for all models\.

We see that compared to the baseline, nearly all hyperparameter\-specific instability scores are lower, as indicated by positive values\. This shows that varying a single hyperparameter makes the model more stable than varying all hyperparameters simultaneously\. Secondly, we observe that all hyperparameters \(τ\\tau, PPF, and TPF\) produce similar trends across models, withτ\\tautypically yielding the highest values \(greatest improvements\) and TPF the lowest\. Since differences in instability scores reflect reductions in instability, smaller decreases indicate greater instability in model predictions\. Thus, since TPF shows the smallest differences, we argue that this makes it the most influential hyperparameter for overall instability\. We also note that different instability metrics show varying trends: some, like Llama 8B’sIQI\_\{Q\}between PPF and TPF, remain relatively consistent, while others, such asIAI\_\{A\}, have steep drops\. This shows the importance of instability metrics from different perspectives\.

##### Which Value Causes the Highest Instability for Each Hyperparameter?

Here, we seek to identify specific "unstable hyperparameter settings" for each hyperparameter that may lead to increased instability\. We perform the same procedure as for single\-hyperparameter instability, except we stratify results by experimental settings with specific hyperparameter settings for each hyperparameter \(e\.g\. finding instability when varying TPF in settings withτ=0\\tau=0\)\. Figure[3](https://arxiv.org/html/2607.00937#S5.F3)shows the results\.

From this, we identify key "unstable" and "stable" settings for each hyperparameter\. We find that, forτ\\tau\(first three columns\), aτ=0\\tau=0is the most stable \(lighter color\), exhibiting higher stability across nearly all models and metrics – aτ=1\\tau=1exhibits the opposite trend \(darker color\), especially as seen inIAI\_\{A\}\. This aligns with the intuition that a higher temperature generate more diverse predictions, i\.e different answers across runs\. For PPF, we see the more verbose or abstract PPFs have generally lower stability, with "barebones" being the most stable for nearly all models and metrics, and "youarknowledge" or "identitynature" being the least stable\. For TPF, we observe a similar trend, with the TPFs requiring more detail – the "explain" variations – having the least stability across nearly all models and metrics\. We note that an unstable TPF format causes greater instability than an unstable temperature value, mirroring results in Section[5\.2](https://arxiv.org/html/2607.00937#S5.SS2)\.

### 5\.3Instability Across Categorization

We investigate how instability varies based on question and persona categories\. We follow the same procedure as for single\-hyperparameter variations, except further stratifying with dataset categories or persona categories\. The results are in Figure[5](https://arxiv.org/html/2607.00937#S5.F5), with the top sub\-figure showing dataset trends and the bottom sub\-figure showing persona trends\.

DatasetIn the top figure, we observe metric\- and domain\-specific trends\. ForIAI\_\{A\}, we see considerable variation across categories, with many dataset categories having negative scores \(higher instability compared to baseline\) and only a few having positive scores \(lower instability compared to baseline\), indicating that model performance fluctuates considerably between domains\. We can additionally observe a relationship betweenIAI\_\{A\}andIQI\_\{Q\}, where in larger models they exhibit nearly identical patterns\. In smaller models,IQI\_\{Q\}is largely unchanged, possibly due to the initial large variation of question overlaps overall and domain\-specific experiments exhibiting similar instability with questions\.IOI\_\{O\}is completely negative\. We hypothesis this is due to the smaller number of domain\-specific questions compared to the entire dataset, so differences in question correctness lead to greater total performance differences and variations in ranking\. Thus, the domain\-specificIOI\_\{O\}will be lower\.

Across dataset domains, we observe that math\-related and commonsense questions lead to the highest instability \(darker color\), whereas history and cultural awareness/understanding lead to the lowest instability \(lighter color\) across nearly all metrics and models\. We hypothesize that this can be attributed to disparities in training data relative to these domains: commonsense knowledge is often underrepresented due to reporting biasGordon and Van Durme \([2013](https://arxiv.org/html/2607.00937#bib.bib4)\), while mathematical content constitutes only a small fraction of most models’ training corporaElazaret al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib3)\)\.555We analyzed the format and length of the questions from different categories, but see no relationship between instability and questions of similar style, length, or answer choice count\.

PersonaIn the bottom figure in Figure[5](https://arxiv.org/html/2607.00937#S5.F5), we see little relation between persona category \(e\.g\., what personas are used in experimentation\) and instability, with relatively equivalent results across all metrics, models, and persona categories\. This shows that the instability of PDGs in MCQA has less to do with thespecificpersona used, but rather the hyperparameters used in the experiment settings\.666We note differences inIOI\_\{O\}\. We attribute this to the differences in persona category size \(with Gender, Race/Ethnicity, and Occupational Roles being the largest\)\. A greater number of personas confers greater maximal ranking shifts, influencingIOI\_\{O\}\. Similar\-sized persona groups have similarIOI\_\{O\}\.

This finding is bolstered by further analysis with sub\-categories of occupational personas\. We sub\-categorize 9 occupational personas into groups of three, with each group having occupations with expertise closely associated with two categories in𝒟\\mathcal\{D\}– EECS/Math, History/Politics and Law, and Natural Science/Human Health and Medicine\. This categorization is described in Appendix[D\.2](https://arxiv.org/html/2607.00937#A4.SS2)\. We note little differences in instability when comparing these sub\-categories to other persona categories, supporting our previous claims\. Further details on this analysis and extensions are in Appendix[E\.2](https://arxiv.org/html/2607.00937#A5.SS2)\.

### 5\.4Stability, Accuracy, and Results

Table 4:Depicting the average and maximal accuracy across stable and unstable settings per model\.We compare the relationship between stability, as identified by our metrics, and overall accuracy on this task\. Using the low and high stability hyperparameter settings observed in Section[5\.2](https://arxiv.org/html/2607.00937#S5.SS2.SSS0.Px1), we identify four stable and unstable experiment settings:Stable–τ\\tau= 0\.0, TPF = \{"justanswer", "justanswer\-persona"\}, PPF = \{"barebones", "youare"\};Unstable–τ\\tau= 1\.0, TPF = \{"justanswer", "justanswer\-persona"\}, PPF = \{"barebones", "youare"\}\.

We compare the mean accuracy and maximal accuracy across all stable settings versus all unstable settings, averaged across all personas, reporting results in Table[4](https://arxiv.org/html/2607.00937#S5.T4)\. As seen in Table[4](https://arxiv.org/html/2607.00937#S5.T4), stable settings largely outperform unstable settings, exhibiting greater mean and maximal accuracy777The only exception to this trend is Qwen2\.5\-14B\. We discuss our rationale for this pattern in Appendix[E\.9](https://arxiv.org/html/2607.00937#A5.SS9)\.\. This cements a relationship between stability and model performance, and emphasizes the importance of considering stability for LLM PDGs\.

##### Results and Instability

![Refer to caption](https://arxiv.org/html/2607.00937v1/x6.png)Figure 6:An example of different experimental outcomes for Llama 1B in a stable and unstable setting\. The best/worst persona relative to accuracy for each setting are shown in gold and red, respectively\. Note:identity/demographic shouldnotimpact model performance\.Please see the Limitations and Ethical Considerations\.We compare results between low and high stability experiment settings, using a similar procedure\. We consider the stable settingτ\\tau= 0, PPF = "barebones", and TPF = "justanswer", and unstable settingτ\\tau= 1, PPF = "identitynature", and TPF = "explain\-persona"\.

Between the two settings,1\)the best\- and worst\-performing personas across the entire dataset differ, and2\)the distribution of persona performances differs\. This cements a completely different results landscape when comparing high and low stability settings as identified through our metrics\. This is demonstrated with Llama 1B in Figure[6](https://arxiv.org/html/2607.00937#S5.F6), with all models exhibiting similar patterns, as seen in Figure[12](https://arxiv.org/html/2607.00937#A6.F12)in the Appendix\.888We additional explore how instability impacts thepredictabilityof persona results in Appendix[E\.3](https://arxiv.org/html/2607.00937#A5.SS3)\.

## 6Conclusion

In this paper, we introduce three instability metrics to evaluate LLMs PDGs in MCQA, assessing a model’s sensitivity in performance, outcome, and overlap of correctly answered questions with varying prompts and temperatures\. We use these metrics to understand differences with past work and to inform future PDG experiment designs\. We find that PDGs are heavily unstable across 3 distinct dimensions, with similar, although different trends across our three metrics, consistent across all models\. We found that models are most sensitive to variations in TPF compared to parameters like PPF or generation temperature\. Instability varies across question domain, with math questions exhibiting the highest instability\. Additionally, we find a relationship between instability and accuracy, with unstable settings having substantial decreases in accuracy compared to stable settings\. Importantly, these unstable hyperparameters can lead to entirely different conclusions in persona\-based analyses\. We hope that our work supports future PDG experiments and analysis of their instability\.

## Limitations

Instability Metric PurposeWe note that these metrics capture instabilityina persona setting, rather than instability solelydue tothe inclusion of persona, with our metrics capturing instability due to variations in persona, prompt format, etc\. The potential for conflating factors – such as different model temperatures, prompt formats, task content \(which can relate to task difficulty\), and personas used – is the subject of analysis within this paper\. We discuss how the analysis of these factors impacting instability can be utilized for future studies or usages of PDGs in MCQA, as we identify when these uses may be most unstable\. We additionally discuss how a re\-formulation of these metrics to consider persona as a parameter can identify new facets of instability \(such as instability due to varyingpersona, rather than other hyperparameters\) and bias in models in Appendix[E\.1](https://arxiv.org/html/2607.00937#A5.SS1)\.

PersonaWe consider both occupational personas, as well as identity\-related personas, in order to reflect the distribution of personas used in past work\. We acknowledge that the set of personas used is not fully representative of personas used in PDGs in both casual use and past research, and may not capture the full diversity present within personas of each category\. We treat our personas used as a broad approximation of the various identities and occupations used with PDGs, drawing heavily from those used in past work, and encourage future work that may be able to expand the set of personas for other investigations\. We include details of how we believe stability should manifest across occupation\-related and identity\-related personas to ensure unbiased performance, user safety, and adequate representation in Appendix[D\.2\.1](https://arxiv.org/html/2607.00937#A4.SS2.SSS1)\.

Tasks and DomainWe acknowledge that the tasks and domain of tasks chosen may not fully represent the broad range of tasks that PDGs are used in, in both research and non\-research applications\. While we focus on multiple\-choice answering as our means of assessing PDGs in non\-text\-heavy settings, we encourage future work that evaluates other non\-text\-heavy paradigms utilizing PDGs, such as smaller annotation tasks\.

Experimental SettingsWe acknowledge that the experimental settings and hyperparameters that we investigate do not represent the full list of potential settings or hyperparameters that people may use for their own PDGs\. We base our choices off of differences noted in prior work and resource constraints, but surely other variations exists that can have their own specific relationships with instability\.

ModelsWhile we chose our selection of models such that we could identify potential family and size\-related differences in instability, we do not fully encompass the diversity of model families and sizes accessible today\. We encourage future work evaluating model instability in non\-text\-heavy PDGs across a wider variety of model sizes \(such as larger 70B models\), or different model architectures\.

## Ethical Considerations

Inclusion of Identity\-Related PersonasAs our study relates to aspects of PDG experiments, such as experiment setting, we consider it valuable to evaluate across a diverse set of personas to improve the generalizability of our results\. Because of this, we employ not only occupation or expertise\-related personas, but also demographic\-related personas, as seen in past PDG literature\. In this, while we consider accuracy as an important factor of stability for PDGs,we make no claim that certain identities should perform better on an MCQA task than others\. We consider identity\-related personas impacting accuracy on our MCQA dataset, like "a Hispanic person" impacting performance on math questions, to be undesirable, stereotyped, and biased model behavior\. However, as these biased behaviors have been the subject of past studies likeGuptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\), we consider it important to evaluate how stable these behaviors may be to inform future studies on this important issue, motivating our inclusion of identity\-related personas in our study\.

Additionally, we see how instability can lead to patterns reminiscent of bias, as seen in Figure[6](https://arxiv.org/html/2607.00937#S5.F6)\. In a scenario where demographic\-related personas do not lead to biased outcomes, it is necessary that the lack of bias is generalizable across experiment settings, or stable, for considerations like user safety \(while still maintaining variation reflective of individuals – we discuss this in Appendix[D\.2\.1](https://arxiv.org/html/2607.00937#A4.SS2.SSS1)\)\. For scenarios where demographic personasdolead to biased outcomes, stability across experimental settings is still valuable\. With instability \(as we find\), we cannot rely on inter\-persona performance differences to suggest a model is biased in a specific way, as small variations in experiment setting impact those resulting conclusions, as seen in Figure[6](https://arxiv.org/html/2607.00937#S5.F6)\. This may point towards not necessarily human\-interpretable biases causing perceived biased performance as inGuptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\)\(potentially motivating investigations into demographic\-related toxicity in training data\), but some other mechanism that only manifests in occasionally interpretable ways that mirror biases in our world\.

Finally, our stability metrics have direct applications to identifying and understanding these patterns of harmful bias in LLM performance\. We are able to identify \(to our knowledge\) previously unknown facets of biased model performance relative to demographics, being question correctness consistency, as shown in Appendix[E\.1](https://arxiv.org/html/2607.00937#A5.SS1)\. These points justify our usage of both occupational personas and demographic\-related personas in our study\.

RepresentationWhile we attempt to make our persona selections as inclusive as possible, we recognize that we are unable to capture the complete diversity present within people with identities associated with each persona category\. Additionally, people’s identities are far more pluralistic, with discrete labels likely not capturing someone’s full sense of self – what we use as persona roles are approximations\.

## Acknowledgments

We would like to thank the following entities for thoughtful discussion, feedback, and support throughout the duration of this work:

- •This research was supported in part by the University of Pittsburgh Center for Research Computing and Data, RRID:SCR\_022735, through the resources provided\. Specifically, this work used the HTC cluster, which is supported by NIH award number S10OD028483\.
- •The PittNLP community, for thoughtful discussion and feedback during paper writing\.
- •Alex Maldonado, for feedback regarding figure design and analyses\.

Generative AI Use StatementWe use Generative AI \(GenAI\) tools, specifically ChatGPT, for minor rephrasing, word choice, and figure editing suggestions\. GenAI is not used in any way beyond that, such as for idea generation, analyses, claims, section writing, etc\. No text is directly copied from any GenAI service, with everything only being used as a reference that is reviewed by the authors to ensure integrity and correctness\.

## References

- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2607.00937#S1.p1.1),[§2](https://arxiv.org/html/2607.00937#S2.p4.1)\.
- \[2\]Character\.ai\.Note:[https://character\.ai/](https://character.ai/)Cited by:[§1](https://arxiv.org/html/2607.00937#S1.p2.1),[§2](https://arxiv.org/html/2607.00937#S2.p1.1)\.
- A\. Chatterjee, H\. K\. Renduchintala, S\. Bhatia, and T\. Chakraborty \(2024\)Posix: a prompt sensitivity index for large language models\.arXiv preprint arXiv:2410\.02185\.Cited by:[§2](https://arxiv.org/html/2607.00937#S2.p4.1)\.
- M\. Cheng, E\. Durmus, and D\. Jurafsky \(2023\)Marked personas: using natural language prompts to measure stereotypes in language models\.arXiv preprint arXiv:2305\.18189\.Cited by:[§2](https://arxiv.org/html/2607.00937#S2.p2.1)\.
- P\. H\. L\. de Araujo, P\. Röttger, D\. Hovy, and B\. Roth \(2025\)Principled personas: defining and measuring the intended effects of persona prompting on task performance\.arXiv preprint arXiv:2508\.19764\.Cited by:[§D\.2\.1](https://arxiv.org/html/2607.00937#A4.SS2.SSS1.p2.1),[§1](https://arxiv.org/html/2607.00937#S1.p4.1),[§2](https://arxiv.org/html/2607.00937#S2.p2.1),[§2](https://arxiv.org/html/2607.00937#S2.p3.1),[§4\.1](https://arxiv.org/html/2607.00937#S4.SS1.p1.1)\.
- Y\. R\. Dong, T\. Hu, and N\. Collier \(2024\)Can LLM be a personalized judge?\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 10126–10141\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.592/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.592)Cited by:[§2](https://arxiv.org/html/2607.00937#S2.p1.1)\.
- Y\. Elazar, A\. Bhagia, I\. Magnusson, A\. Ravichander, D\. Schwenk, A\. Suhr, P\. Walsh, D\. Groeneveld, L\. Soldaini, S\. Singh, H\. Hajishirzi, N\. A\. Smith, and J\. Dodge \(2024\)What’s in my big data?\.InProceedings of the Twelfth International Conference on Learning Representations \(ICLR 2024\),External Links:[Link](https://openreview.net/forum?id=6OejnS6PgY)Cited by:[§5\.3](https://arxiv.org/html/2607.00937#S5.SS3.p3.1)\.
- J\. Fleiss \(1971\)Measuring nominal scale agreement among many raters\.Psychological Bulletin76,pp\. 378–382\.External Links:[Document](https://dx.doi.org/10.1037/h0031619)Cited by:[3rd item](https://arxiv.org/html/2607.00937#A5.I1.i3.p1.2)\.
- J\. Gordon and B\. Van Durme \(2013\)Reporting bias and knowledge acquisition\.InProceedings of the 2013 workshop on Automated knowledge base construction,pp\. 25–30\.Cited by:[§5\.3](https://arxiv.org/html/2607.00937#S5.SS3.p3.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2607.00937#S1.p1.1),[§4](https://arxiv.org/html/2607.00937#S4.p4.1)\.
- S\. Gupta, V\. Shrivastava, A\. Deshpande, A\. Kalyan, P\. Clark, A\. Sabharwal, and T\. Khot \(2024\)Bias Runs Deep: implicit reasoning biases in persona\-assigned LLMs\.InThe Twelfth International Conference on Learning Representations,Cited by:[Table 5](https://arxiv.org/html/2607.00937#A1.T5.1.4.3.2.1.3.1),[Table 5](https://arxiv.org/html/2607.00937#A1.T5.1.5.4.2.1.3.1),[§D\.1](https://arxiv.org/html/2607.00937#A4.SS1.p2.1),[§D\.2](https://arxiv.org/html/2607.00937#A4.SS2.p1.1),[§E\.1](https://arxiv.org/html/2607.00937#A5.SS1.p3.1),[§1](https://arxiv.org/html/2607.00937#S1.p3.1),[§1](https://arxiv.org/html/2607.00937#S1.p4.1),[§2](https://arxiv.org/html/2607.00937#S2.p2.1),[§4\.1](https://arxiv.org/html/2607.00937#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.00937#S4.SS1.p3.1),[§4\.1](https://arxiv.org/html/2607.00937#S4.SS1.p4.1),[§4\.1](https://arxiv.org/html/2607.00937#S4.SS1.p5.1),[§4](https://arxiv.org/html/2607.00937#S4.p2.1),[Ethical Considerations](https://arxiv.org/html/2607.00937#Sx2.p1.1),[Ethical Considerations](https://arxiv.org/html/2607.00937#Sx2.p2.1)\.
- J\. He, M\. Rungta, D\. Koleczek, A\. Sekhon, F\. X\. Wang, and S\. Hasan \(2024\)Does prompt formatting have any impact on llm performance?\.arXiv preprint arXiv:2411\.10541\.Cited by:[§2](https://arxiv.org/html/2607.00937#S2.p4.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2020\)Measuring massive multitask language understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[§D\.1](https://arxiv.org/html/2607.00937#A4.SS1.p1.1),[§4](https://arxiv.org/html/2607.00937#S4.p3.1)\.
- Q\. Huang, Y\. Zhang, T\. Ko, X\. Liu, B\. Wu, W\. Wang, and H\. Tang \(2023\)Personalized dialogue generation with persona\-adaptive attention\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.37,pp\. 12916–12923\.Cited by:[§2](https://arxiv.org/html/2607.00937#S2.p1.1)\.
- T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. Iwasawa \(2022\)Large language models are zero\-shot reasoners\.Advances in neural information processing systems35,pp\. 22199–22213\.Cited by:[§1](https://arxiv.org/html/2607.00937#S1.p1.1)\.
- D\. Kyung, H\. Chung, S\. Bae, J\. Kim, J\. H\. Sohn, T\. Kim, S\. K\. Kim, and E\. Choi \(2025\)PatientSim: a persona\-driven simulator for realistic doctor\-patient interactions\.arXiv preprint arXiv:2505\.17818\.Cited by:[§1](https://arxiv.org/html/2607.00937#S1.p1.1)\.
- A\. Li, H\. Chen, H\. Namkoong, and T\. Peng \(2025\)LLM generated persona is a promise with a catch\.arXiv preprint arXiv:2503\.16527\.Cited by:[§2](https://arxiv.org/html/2607.00937#S2.p1.1),[§2](https://arxiv.org/html/2607.00937#S2.p4.1)\.
- Y\. Li, C\. Zeng, J\. Zhong, R\. Zhang, M\. Zhang, and L\. Zou \(2024\)Leveraging large language model as simulated patients for clinical education\.arXiv preprint arXiv:2404\.13066\.Cited by:[§1](https://arxiv.org/html/2607.00937#S1.p1.1)\.
- A\. Liu, M\. Diab, and D\. Fried \(2024\)Evaluating large language model biases in persona\-steered generation\.InFindings of the Association for Computational Linguistics ACL 2024,pp\. 9832–9850\.Cited by:[§2](https://arxiv.org/html/2607.00937#S2.p2.1)\.
- A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. Fang \(2024\)Evaluating very long\-term conversational memory of llm agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13851–13870\.Cited by:[§2](https://arxiv.org/html/2607.00937#S2.p2.1)\.
- A\. S\. Rao, A\. Yerukola, V\. Shah, K\. Reinecke, and M\. Sap \(2025\)NormAd: a framework for measuring the cultural adaptability of large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 2373–2403\.External Links:[Link](https://aclanthology.org/2025.naacl-long.120/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.120),ISBN 979\-8\-89176\-189\-6Cited by:[§D\.1](https://arxiv.org/html/2607.00937#A4.SS1.p1.1),[§D\.1](https://arxiv.org/html/2607.00937#A4.SS1.p3.1),[Table 6](https://arxiv.org/html/2607.00937#A4.T6.1.10.9.4),[§4](https://arxiv.org/html/2607.00937#S4.p3.1)\.
- M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. Choi \(2019\)Social IQa: commonsense reasoning about social interactions\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 4463–4473\.External Links:[Link](https://aclanthology.org/D19-1454/),[Document](https://dx.doi.org/10.18653/v1/D19-1454)Cited by:[§D\.1](https://arxiv.org/html/2607.00937#A4.SS1.p1.1),[§D\.1](https://arxiv.org/html/2607.00937#A4.SS1.p4.1),[Table 6](https://arxiv.org/html/2607.00937#A4.T6.1.11.10.4),[§4](https://arxiv.org/html/2607.00937#S4.p3.1)\.
- M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. Suhr \(2023\)Quantifying language models’ sensitivity to spurious features in prompt design or: how i learned to start worrying about prompt formatting\.arXiv preprint arXiv:2310\.11324\.Cited by:[1st item](https://arxiv.org/html/2607.00937#A5.I1.i1.p1.1),[§1](https://arxiv.org/html/2607.00937#S1.p3.1),[§2](https://arxiv.org/html/2607.00937#S2.p4.1),[§3](https://arxiv.org/html/2607.00937#S3.p2.1)\.
- S\. Seabold and J\. Perktold \(2010\)Statsmodels: econometric and statistical modeling with python\.In9th Python in Science Conference,Cited by:[1st item](https://arxiv.org/html/2607.00937#A3.I1.i1.p1.1),[§3\.2](https://arxiv.org/html/2607.00937#S3.SS2.p3.9)\.
- H\. Sun, H\. Lin, H\. Yan, Y\. Song, X\. Gao, and R\. Yan \(2025\)MockLLM: a multi\-agent behavior collaboration framework for online job seeking and recruiting\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 2714–2724\.Cited by:[§1](https://arxiv.org/html/2607.00937#S1.p1.1)\.
- Y\. Tseng, Y\. Huang, T\. Hsiao, W\. Chen, C\. Huang, Y\. Meng, and Y\. Chen \(2024\)Two tales of persona in LLMs: a survey of role\-playing and personalization\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 16612–16631\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.969/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.969)Cited by:[§2](https://arxiv.org/html/2607.00937#S2.p2.1)\.
- P\. Virtanen, R\. Gommers, T\. E\. Oliphant, M\. Haberland, T\. Reddy, D\. Cournapeau, E\. Burovski, P\. Peterson, W\. Weckesser, J\. Bright, S\. J\. van der Walt, M\. Brett, J\. Wilson, K\. J\. Millman, N\. Mayorov, A\. R\. J\. Nelson, E\. Jones, R\. Kern, E\. Larson, C\. J\. Carey, İ\. Polat, Y\. Feng, E\. W\. Moore, J\. VanderPlas, D\. Laxalde, J\. Perktold, R\. Cimrman, I\. Henriksen, E\. A\. Quintero, C\. R\. Harris, A\. M\. Archibald, A\. H\. Ribeiro, F\. Pedregosa, P\. van Mulbregt, and SciPy 1\.0 Contributors \(2020\)SciPy 1\.0: Fundamental Algorithms for Scientific Computing in Python\.Nature Methods17,pp\. 261–272\.External Links:[Document](https://dx.doi.org/10.1038/s41592-019-0686-2)Cited by:[2nd item](https://arxiv.org/html/2607.00937#A3.I1.i2.p1.1)\.
- Y\. Wan, J\. Zhao, A\. Chadha, N\. Peng, and K\. Chang \(2023\)Are personalized stochastic parrots more dangerous? evaluating persona biases in dialogue systems\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 9677–9705\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.648/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.648)Cited by:[§D\.2](https://arxiv.org/html/2607.00937#A4.SS2.p1.1),[§4](https://arxiv.org/html/2607.00937#S4.p2.1)\.
- C\. Wang, X\. Liu, and A\. H\. Awadallah \(2023\)Cost\-effective hyperparameter optimization for large language model generation inference\.InInternational Conference on Automated Machine Learning,pp\. 21–1\.Cited by:[§1](https://arxiv.org/html/2607.00937#S1.p3.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§E\.9](https://arxiv.org/html/2607.00937#A5.SS9.p2.2),[§4\.1](https://arxiv.org/html/2607.00937#S4.SS1.p4.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2607.00937#S1.p1.1),[§4](https://arxiv.org/html/2607.00937#S4.p4.1)\.
- C\. Yu, J\. Ye, Y\. Li, Z\. Li, E\. Ferrara, X\. Hu, and Y\. Zhao \(2024\)A large\-scale simulation on large language models for decision\-making in political science\.arXiv preprint arXiv:2412\.15291\.Cited by:[§2](https://arxiv.org/html/2607.00937#S2.p1.1)\.
- Y\. Yuan, L\. Zhao, W\. Chen, G\. Zheng, K\. Zhang, M\. Zhang, and Q\. Liu \(2025\)Simulating human\-like learning dynamics with llm\-empowered agents\.arXiv preprint arXiv:2508\.05622\.Cited by:[§1](https://arxiv.org/html/2607.00937#S1.p1.1)\.
- X\. Zhang, R\. Chen, Y\. Feng, and Z\. Liu \(2025\)Persona\-judge: personalized alignment of large language models via token\-level self\-judgment\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 5037–5049\.External Links:[Link](https://aclanthology.org/2025.findings-acl.260/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.260),ISBN 979\-8\-89176\-256\-5Cited by:[§2](https://arxiv.org/html/2607.00937#S2.p2.1)\.
- Z\. Zhang, R\. A\. Rossi, B\. Kveton, Y\. Shao, D\. Yang, H\. Zamani, F\. Dernoncourt, J\. Barrow, T\. Yu, S\. Kim,et al\.\(2024\)Personalization of large language models: a survey\.arXiv preprint arXiv:2411\.00027\.Cited by:[§1](https://arxiv.org/html/2607.00937#S1.p1.1),[§2](https://arxiv.org/html/2607.00937#S2.p1.1)\.
- M\. Zheng, J\. Pei, L\. Logeswaran, M\. Lee, and D\. Jurgens \(2024\)When “a helpful assistant” is not really helpful: personas in system prompts do not improve performances of large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 15126–15154\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.888/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.888)Cited by:[Table 5](https://arxiv.org/html/2607.00937#A1.T5.1.3.2.2),[§D\.1](https://arxiv.org/html/2607.00937#A4.SS1.p2.1),[§D\.2](https://arxiv.org/html/2607.00937#A4.SS2.p1.1),[§E\.3](https://arxiv.org/html/2607.00937#A5.SS3.p1.1),[§1](https://arxiv.org/html/2607.00937#S1.p3.1),[§1](https://arxiv.org/html/2607.00937#S1.p4.1),[§2](https://arxiv.org/html/2607.00937#S2.p2.1),[§4\.1](https://arxiv.org/html/2607.00937#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.00937#S4.SS1.p3.1),[§4\.1](https://arxiv.org/html/2607.00937#S4.SS1.p4.1),[§4\.1](https://arxiv.org/html/2607.00937#S4.SS1.p5.1),[§4](https://arxiv.org/html/2607.00937#S4.p2.1)\.
- J\. Zhuo, S\. Zhang, X\. Fang, H\. Duan, D\. Lin, and K\. Chen \(2024\)ProSA: assessing and understanding the prompt sensitivity of LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 1950–1976\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.108/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.108)Cited by:[§2](https://arxiv.org/html/2607.00937#S2.p4.1),[§3\.1](https://arxiv.org/html/2607.00937#S3.SS1.p2.5),[§3](https://arxiv.org/html/2607.00937#S3.p2.1)\.

## Appendix APrompts

### A\.1TPF and PPF

Table 5:The various prompt formats used, relative to the task prompt format \(TPF\) and persona prompt format \(PPF\) experiment configuration parameters\. We substitute \{persona\} with a persona string in PPF and substitute \{question\} with a formatted question string in TPF\.The full prompts used for the task prompt format \(TPF\) and persona prompt format \(PPF\) hyperparameter can be seen in Table[5](https://arxiv.org/html/2607.00937#A1.T5)\.

## Appendix BLLM Parameter Settings and Usage in Experimentation

We fix the model parameters across models, only varying the temperature hyperparameter across experiments\. Otherwise, we maintain the following settings across all models evaluated:

- •do\_sample:True
- •top\_k:50
- •top\_p:1\.0
- •max\_new\_tokens \(from HF pipeline\):1000

For experimentation, we evaluate each LLM on an equal amount of experimental settings and personas, leading to 1,968 full evaluations of𝒟\\mathcal\{D\}per model\. Specifically, for one combination of persona and experiment setting hyperparameters \(a setting for each ofτ\\tau, PPF, and TPF\):

- •We evaluate models on every question in𝒟\\mathcal\{D\}\.
- •For each question, we provide a system prompt containing our persona prompt, in the specified format of the PPF setting\. We substitute in the specific persona for a set of runs\.
- •With this system prompt, each question is then proposed to the model, formatted in accordance to the TPF setting\. The question content, including the formatted answer choices, are substituted in within this specified format\.
- •Thus, for each question in𝒟\\mathcal\{D\}, we provide a formatted system prompt \(our persona prompt\) and a formatted user prompt \(our task prompt\)\.

## Appendix CLicenses and Intended Use

We utilized the Llama\-3\.1\-8B\-Instruct and Llama\-3\.2\-1B\-Instruct models during evaluation, complying with Meta’s LLAMA 3\.1 Community License Agreement and LLAMA 3\.2 Community License Agreement\.

We utilized the Qwen2\.5\-1\.5B\-Instruct and Qwen2\.5\-7B\-Instruct models during evaluation, complying with the Apache 2\.0 license that these models were released under\.

We utilized the OpenAI batch API for prompting GPT\-4o\-mini for relational knowledge article generation, complying with OpenAI’s terms of use\.

We additionally made use of previously created Python packages for processing and analysis:

- •statsmodels:Seabold and Perktold \([2010](https://arxiv.org/html/2607.00937#bib.bib6)\)
- •SciPy:Virtanenet al\.\([2020](https://arxiv.org/html/2607.00937#bib.bib5)\)

## Appendix DDataset and Personas

### D\.1Dataset Construction

To evaluate and understand PDG instability in LLMs, we create a composite evaluation dataset,𝒟\\mathcal\{D\}, balanced across various task types and domains for our experiments\. This topical diversity in questions allows a balanced overall view of model instability, as well as investigation into the relationship between question domain and volatility\. We pull from MMLUHendryckset al\.\([2020](https://arxiv.org/html/2607.00937#bib.bib40)\), Social IQaSapet al\.\([2019](https://arxiv.org/html/2607.00937#bib.bib39)\), and NormAd\-EtiRaoet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib38)\)\. We group questions under 10 different categories, composed of questions with specific sublabels from MMLU, Social IQa, and NormAd\-Eti\. For each category, we sample equally across all question label groups, considering rounding – with this, we get ~500 questions per broader category\. The categories and questions per category can be seen in Table[6](https://arxiv.org/html/2607.00937#A4.T6)\.

We select labels per category by referencing organization from prior workGuptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\); Zhenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib42)\)\. We ensure that each category has a minimum of three question labels associated with it, ensuring topical diversity within a category, although still pertaining to the broader domain\. For categories other than Cultural Awareness and Understanding and Commonsense Reasoning/Social Awareness, we take question labels as questions from specific constituent datasets of MMLU, as shown in Table[6](https://arxiv.org/html/2607.00937#A4.T6)\.

For Cultural Awareness and Understanding, we take directly from NormAd\-EtiRaoet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib38)\), using the country context paradigm within the dataset\. Each country label in the dataset \(e\.g\. "mexico"\) for a question are used as the question labels for𝒟\\mathcal\{D\}\.

For Commonsense Reasoning/Social Awareness, we follow the categorization of questions from Social IQa based on ATOMICSapet al\.\([2019](https://arxiv.org/html/2607.00937#bib.bib39)\)\. Using the template provided in this work, we perform rounds of automatic labeling of Social IQa questions with respect to these labels\. Following five rounds of labeling and integration of additional templates that match one of the Social IQa question types, we group the remaining questions under "misc\."

Table 6:The total composition of the MCQA dataset used in experimentation, including the question categorization\.
### D\.2Personas

We consider 41 personas for our experimentation, split across 9 distinct groups centering on aspects of identity \(such as race\) and expertise \(such as occupation\) for comparison\. These personas are drawn upon previous work, with additional personas added in order to ensure inclusivity and better representation per categoryZhenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib42)\); Guptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\); Wanet al\.\([2023](https://arxiv.org/html/2607.00937#bib.bib28)\)\. Specifically, we take a sample of occupational personas, chosen to consider broad expertiseZhenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib42)\), and an assortment of identity\-related personas from other prior workGuptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\); Wanet al\.\([2023](https://arxiv.org/html/2607.00937#bib.bib28)\), along with additional personas we add ourselves\. The complete personas and categorization can be seen in Table[7](https://arxiv.org/html/2607.00937#A4.T7)\.

We additionally consider sub\-categories of occupational personas for expertise\-wise analysis\. With this, we consider three groups of three personas, with each group related to two categories in𝒟\\mathcal\{D\}\. Specifically – an engineer, a computer engineer, and a computer scientist→\\rightarrowEECS/Math; a lawyer, a legal scholar, and a historian→\\rightarrowHistory/Politics and Law; and a doctor, a physiologist, and a biologist→\\rightarrowNatural Science/Human Health and Medicine\. We also consider a miscellaneous \(Misc\.\) category for comparison, containing: an artist, a chef, and a virtual assistant\.

Table 7:All personas used in experimentation\.#### D\.2\.1Stability Desiderata Depending on Persona

We acknowledge that stability desiderata may, and should, vary based on the type of persona used for PDGs in MCQA\. The following discussion closely relates to our analysis of stability by persona category, as seen in Section[5\.3](https://arxiv.org/html/2607.00937#S5.SS3)\.

For occupational personas or task\-aligned personas \(such as "a biologist" for biology questions\), we believe there should be a great initial stability within these personas’ performances\. Task\-aligned personas shouldalwaysconfer a task advantage compared to non\-task\-aligned personas \(similar to what is discussed inde Araujoet al\.\([2025](https://arxiv.org/html/2607.00937#bib.bib18)\)\), with this advantage being consistent across similar environments\. Therefore, not only should overall instability be low for task\-aligned personas \(an analysis shown in Figure[5](https://arxiv.org/html/2607.00937#S5.F5)\), but instability within these personas should not vary across similar experiment settings \(an analysis shown in Figure[11](https://arxiv.org/html/2607.00937#A6.F11)\.

Unfortunately, we observe that considering only task\-aligned personas still leads to considerable instability, as shown in Figure[5](https://arxiv.org/html/2607.00937#S5.F5), even specifically in the task domain, as shown in Figure[8](https://arxiv.org/html/2607.00937#A5.F8)\. We also see how varying different specific experiment setting hyperparameters can lead to differing patterns of instability in task\-aligned personas for the smaller models we evaluate, as shown in Figure[11](https://arxiv.org/html/2607.00937#A6.F11)\.

For identity\-related personas, we believe there should be instability or variation within these personas’ performances\. Variation here is beneficial, as it implies the model does not have a fixed \(potentially stereotyped\) representation of how a member of an identity group would perform on an MCQA task\. This supports individuality and a pluralistic perspective of individuals within an identity group\. However, similar to occupational or task\-aligned personas, this variation should be consistent across similar environments, with a similar level of variation even with small perturbations in the experiment setting\. Additionally, this variation should be consistent across the various identity\-related persona categories – a bias in this regard suggests harmful, differing perspectives of the degree of "individuality" within a specific identity group, which we consider undesirable\.

We observe that considering only identity\-related personas still displays instability/variation in model performance, as shown in Figure[5](https://arxiv.org/html/2607.00937#S5.F5)\. This may suggest support for individualistic performance that may resemble human variations within these identity groups\.999However, we maintain our perspective that giving a model an identity\-related persona should not meaningfully impact performance on irrelevant tasks, like math MCQA, in the first place, to mitigate harm to users\.101010We do not evaluate whether this variation is similar to that of a human population, with this being out of the scope of our study\.However, we do note differences, although minor, in the stability between identity\-related persona categories, especially in the smaller models we evaluate\. Additionally, as shown in Figure[11](https://arxiv.org/html/2607.00937#A6.F11), we see how varying different specific experiment setting hyperparameters can lead to differing patterns of instability in identity\-related personas, with these patterns additionally varyingbetweenthe identity\-related persona categories\. We consider this undesirable and potentially harmful to users\.

## Appendix EAdditional Analyses

### E\.1Instability Due to Persona Role Variation

![Refer to caption](https://arxiv.org/html/2607.00937v1/x7.png)Figure 7:A figure depicting the delta hyperparameter instability when treating persona as a hyperparameter compared to the overall baseline, stratified by subsets of personas relative to the persona categorization\.We explore the differences in performance or instability when using different personas\. While in Section[5](https://arxiv.org/html/2607.00937#S5), we explore instability due to varying experiment settings, here, we focus on instability due to varying the persona role\. By treating persona as a hyperparameter that can be varied across experiment settings, we can calculate instability relative to using different personas for the same task111111For example, accuracy differences when using different personas on the same task \(IAI\_\{A\}\), or consistency of questions correct when using different personas on the same task \(IQI\_\{Q\}\)\.\. This can additionally be viewed as a lens into a form ofpersona bias– different personas on this task should perform similarly, represented by a smallIAI\_\{A\}andIQI\_\{Q\}\. We omit an analysis ofIOI\_\{O\}, as that requires considering several personassimultaneouslyfor one experiment setting, while we treat persona as a hyperparameterpart ofthe experiment setting\. Note that none of the questions in our dataset have correctness that is persona\-dependent121212While question information can be related to a specific role, like biology questions relating to ”a biologist”, this knowledge is notexclusiveto those with that role\.\.

We calculate instability when varying the persona role, both across all personas, and within specific persona subsets\. Results are shown in Figure[7](https://arxiv.org/html/2607.00937#A5.F7)\.

Looking atIAI\_\{A\}, we can observe patterns of persona bias in accuracy similar to what is seen in past workGuptaet al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib41)\)\. The smaller Llama 1B model has more persona bias than Llama 8B, and the Qwen models have less bias overall – we see this with the "Disabilities" category, where Llama 1B’s darker color shows greater performance differences between personas in that category\.

WithIQI\_\{Q\}, we can get a new view of persona bias via question consistency\. Even in the larger models, the different persona categories have great variation in their question consistency, with some categories like Gender having less question stability than other categories like Age, despite personas in these categories not warranting different performance \(e\.g\. "a man" and "a transgender woman" should not perform differently on this evaluation, similar to how "a young person" and "an adult" should not perform differently as well\)\. This shows a new view of bias even for larger models, and emphasizes the distinctness of our volatility metrics\. We also still find greatIQI\_\{Q\}even in our more closely related persona sub\-categories where a lower value would be expected, suggesting an inherently unstable persona integration mechanism in LLMs\. This analysis additionally demonstrates how the usage of persona is concretely affecting the stability of model performances, rather than model performance being inherently unstable and persona usage being an insignificant factor added on top\.

### E\.2Instability in Only Occupation Sub\-Category\-Related Dataset Categories

![Refer to caption](https://arxiv.org/html/2607.00937v1/x8.png)Figure 8:Instability results stratified by instability metric, by dataset category, by persona sub\-category\.We consider Figure[5](https://arxiv.org/html/2607.00937#S5.F5)for our sub\-category analysis, referring to the categorization in Appendix[D\.2](https://arxiv.org/html/2607.00937#A4.SS2)\. Due to the high similarity of the occupations in these categories in both phrasing and expertise, ideal model behavior should havelessinstability within these sub\-categories, compared to considering all occupations \(where subject expertise is more varied\) or other identity\-related persona categories \(where in an unbiased setting there should be no subject expertise conferred, and therefore, potentially more varied performances\)\. However, instability considering only these sub\-categories of personas mirrors other persona categories, supporting our claim thatspecificpersonas do not strongly impact instability\.

We further this analysis by evaluating how instability may vary in expertise\-related domains, seeking to understand if task\-aligned personas are more stable within their task than others\. For this analysis, we compute the dataset category\-wise instability, but only considering specific subsets of personas, aligning with the sub\-categorization we discussed previously\. We additionally introduce a "Misc\." category of expertise\-aligned personas that do not relate to any task domain – "a virtual assistant", "a chef", and "an artist" – to allow for more robust analysis\. These results are depicted in Figure[8](https://arxiv.org/html/2607.00937#A5.F8)\. As can be seen in Figure[8](https://arxiv.org/html/2607.00937#A5.F8), no matter the specific persona sub\-category considered, dataset domain\-specific instability patterns remain the same\. This supports our previous claims that thespecificpersona may not impact PDG instability in MCQA

### E\.3Instability and Predicting the Best Persona

![Refer to caption](https://arxiv.org/html/2607.00937v1/x9.png)Figure 9:All classifier performances when trained across 48 experimental settings, additionally stratified by specific hyperparameter settings within the experimental settings\.We consider the issue of predicting the best persona for a task, as defined inZhenget al\.\([2024](https://arxiv.org/html/2607.00937#bib.bib42)\)\. We employ identical methods, fine\-tuning a roberta\-base model and using it as a multi\-label classifier for personas, with the prediction target being the 41 persona roles used throughout experimentation\. We train an individual classifier on results from each experimental setting across models, leading to 48 classifiers per model\. We report F1 score results in Figure[9](https://arxiv.org/html/2607.00937#A5.F9), where each point represents the F1 performance of one classifier trained on one experimental setting’s results\. We additionally report classifier performances stratified by the hyperparameter settings of the data they are trained on\.

As can be observed in Figure[9](https://arxiv.org/html/2607.00937#A5.F9), we see little relationship between instability and predictability of persona roles for a question\. Using what is previously found to be stable or unstable settings per hyperparameter, such as aτ=0\\tau=0and aτ=1\\tau=1, we observe no trend between classification performance and stability of hyperparameter settings\.

However, we do see a concrete pattern with classification performance and TPF – classification performance is lower across the "justanswer" variants compared to the "explain" variants of TPF across all models \(except Qwen2\.5\-1\.5B\), with this pattern being strongest in the larger models\. While we cannot assume stability mediates the unpredictability of PDGs in non\-text\-heavy settings, we can see an effect of TPF\. This could be due to the "explain" variants guiding the model to be more "faithful" and consistent with respect to their persona, in turn making their performance predictable\. We encourage further investigation into this phenomenon\.

### E\.4Correlations with Existing Metrics

To validate our metrics, we perform various correlation studies, finding the Pearson Correlation between our metrics and selected metrics from past work\. We calculate this between our metrics and similar metrics from literature or reasonable correlates that we explain\. We calculate these metrics for sets of experimental settings, where each set is defined by settings where one parameter is kept static \(such as a set of all possible experimental settings withτ\\tau= 0\.0\)\. Specifically, we compare:

- •IAI\_\{A\}and the accuracy spread \(max \- min\) across all experiment settings of total dataset accuracy averaged across all personas for each setting\. We compare against this metric from prior literatureSclaret al\.\([2023](https://arxiv.org/html/2607.00937#bib.bib43)\)as they investigated robustness to differing prompt features with respect to performance, similar to our investigations into performance instability across experiment settings\.
- •IOI\_\{O\}, without the multiplication of the average parity for a model, and the number of different best\-performing personas on the total dataset across all experimental settings\. We choose the number of best\-performing personas as that is a direct experimental outcome \(the best persona for a task\) that we seek to capture with our metric\. Due to reasons relating to initial persona parity for a model as discussed in the paper, we argue that the number of best\-performing personas across experiment settings would also be affected by the same quality\. Because of this, we remove the regularization term that would control for this quality in order to have more equivalent comparison\.
- •IQI\_\{Q\}and the Fleiss’ KappaFleiss \([1971](https://arxiv.org/html/2607.00937#bib.bib1)\)agreement of the binary vectors of question correctness for the total dataset between experiment settings for one persona, averaged across all personas\. Here, we treat separate experiment settings for a persona as different “annotators”\. This is chosen as agreement between annotators is used in literature as a measure of instance\-wise consistency/stability across annotators\. Note that a higher agreement rate corresponds to a lowerIQI\_\{Q\}\(or a positive benefit from the baseline\), so we would expect to observe a strong positive correlation if these two measurements are consistent, in contrast to the other pairs\.

We report the p\-value and r of this correlation, averaged across all models, for each of our metrics, in Table[8](https://arxiv.org/html/2607.00937#A5.T8)\. We observe strong correlations\.

Table 8:Depicting the p\-value and r of correlations between each pair of metrics across all models\.
### E\.5Alternative Formulations ofIOI\_\{O\}

![Refer to caption](https://arxiv.org/html/2607.00937v1/x10.png)Figure 10:Comparing instability results across different formulations ofIOI\_\{O\}\.We experimented with four formulations ofIOI\_\{O\}:

- •A non\-weighted standard deviation of ranks
- •A non\-weighted standard deviation of ranks, with additional multiplication of a model normalization term \(as previously defined\)
- •A weighted standard deviation of ranks, weighted by the persona parity of each experimental setting
- •A weighted standard deviation of ranks, weighted by the persona parity of each experimental setting, with additional multiplication of a model normalization term \(as previously defined\)

As we consider that ranking differences from states of parity "mean less" than those coming from states of a lack of parity \(i\.e\. ranking fluctuations when all personas perform equally means less than ranking fluctuations when all personas perform very differently to one another\), we want to weight a given ranking by the parity of the setting it comes from\. With this, rankings that come from settings with great parity are weighted less in the final calculation\. From this, we focus on the latter two formulations, using a weighted standard deviation with the parity of the experimental settings being used as the weights\.

We then investigate the usage of the final model normalization term\. Without this final model normalization term, we have the potential for the weighted standard deviation to be "washed out" – in cases where, across experimental settings, models have equivalent levels of high parity, all ranks will be weighted the same, potentially leading to the finalIOI\_\{O\}score being inflated\. To account for this, we add an additional model normalization term, integrating overall parity a model might have\. This way, should the weighted standard deviation be "washed out", we are still able to integrate a view of model parity, which we deem as valuable for reasons previously stated\.

With this, we wanted to evaluate whether this final model normalization term would "dominate" the final score\. Thus, we investigate patterns of solely the model normalization term, and of theIOI\_\{O\}implementations – with this, we seek to judge whether the patterns of the formulationwiththe model normalization term completely follow those of solely the model normalization term, implying the model normalization term may be "too strong"\. We report heatmaps of the results in Figure[10](https://arxiv.org/html/2607.00937#A5.F10)\.

From Figure[10](https://arxiv.org/html/2607.00937#A5.F10), we can observe that, while A \(solely the model normalization term\) has similar patterns to B \(weighted standard deviation with the model normalization term\), they are not identical\. We note key regions where they differ, such asIOI\_\{O\}for temperature across all models\. From this, we go forward with the final formulation ofIOI\_\{O\}, calculating a weighted standard deviation of ranks across experimental settings, weighted by the parity of each setting, with the addition of a final model normalization term\.

### E\.6Analyzing Metric Sensitivity to Question Count

Table 9:Instability metric results, stratified by the number of questions considered per dataset category\. “\#" denotes the number of questions considered for each category\.We analyze the impact of the question set size on our metric results\. For this analysis, we compute the baseline metrics \(as in Table[1](https://arxiv.org/html/2607.00937#S4.T1)\) using a randomly selected subset of 75% and 50% of the question set, balanced across dataset categories\. We provide results in Table[E\.6](https://arxiv.org/html/2607.00937#A5.SS6), additionally with the current baseline results for comparison\.

As can be seen in Table[E\.6](https://arxiv.org/html/2607.00937#A5.SS6), our metrics slightly vary with respect to the size of the question set\. This aligns with our expectations; with a reduced number of questions, accuracy percentage differences are more substantial, reasonably affectingIAI\_\{A\}andIOI\_\{O\}\. Additionally, with a reduced question set, differences in the set of shared questions are similarly more substantial, affectingIQI\_\{Q\}\. While results vary slightly, it can be noted that they still stabilize around a similar value across dataset size/metrics across models, maintaining features such as the relative order of models for each instability metric, and the relative difference between models for each instability metric\. To summarize: as the question set size changes, the metrics vary slightly but still follow the same trend across models and experiment settings\.

### E\.7Analyzing Metric Sensitivity to Experiment Setting Count

Table 10:Instability metric results, stratified by the number of experiment settings considered\. “\#" denotes the number of experiment settings for a given subset\. "Subset A" represents all temperature settings with TPF = \{justanswer, explain, explain\-persona\} and PPF = \{barebones, youareknowledge, identitynature\}\. "Subset B" represents all temperature settings with TPF = \{justanswer, explain\-persona\} and PPF = \{barebones, identitynature\}\.We analyze the impact of the number of experiment settings on our metric results\. For this analysis, we compute the baseline metrics using a reduced subset of experiment settings, considering only experiment settings across all temperature settings with TPF = justanswer, explain, explain\-persona and PPF = barebones, youareknowledge, identitynature \("Subset A", 27 settings\), and experiment settings with TPF = justanswer, explain\-persona and PPF = barebones, identitynature \("Subset B", 12 settings\)\. We provide results in Table[10](https://arxiv.org/html/2607.00937#A5.T10), additionally with the current baseline results for comparison\.

As can be seen in Table[10](https://arxiv.org/html/2607.00937#A5.T10), our metrics vary with respect to the size of the experiment setting set, similar to that of the question set size\. This aligns with our expectations\. Similar to how, with other measures of spread like classic standard deviation, samples from a population of gradually increasing sizes approach an approximate population value, we see a similar pattern with our metrics\. We maintain features such as the relative order of models for each instability metric\. To summarize: as the experiment setting set size changes, the metrics vary slightly but approach similar values and still follow the same trend across models and experiment settings\.

Additionally, it is important to note that these metric calculations are, of course, dependent on the composition of the sample, like with other measures of spread\. While here, the subsets of experiment settings attempt to maintain an equal count of low/high instability hyperparameter settings, differing subsets may produce considerably different results \(such as a subset of only low instability hyperparameter settings\)\.

### E\.8Parsing Errors Across Hyperparameter Settings

Table 11:The percent of questions marked incorrect due to parsing errors across all models\. Results are stratified by hyperparameter setting, where results for a specific hyperparameter setting represent the average across all experiment settings with that specific hyperparameter setting\.Table 12:The percent of questions marked incorrect across all models\. Results are stratified by hyperparameter setting, where results for a specific hyperparameter setting represent the average across all experiment settings with that specific hyperparameter setting\.We consider the percent of parsing errors to be the percent of questions marked incorrect due to an unparseable model response for a specific experiment setting and persona combination\. We provide a table of parsing error fraction separated by model and by experiment setting hyperparameter value in Table[11](https://arxiv.org/html/2607.00937#A5.T11)\. For each experiment setting hyperparameter value, we average the fraction of parsing errors across all experiment settings with that hyperparameter value for that specific model\. For example, for Qwen2\.5\-1\.5B and TPF = bb, we average the fraction of parsing errors for Qwen2\.5\-1\.5B\-Instruct across all experiments with TPF = barebones\. We additionally provide a similar table, but with the fraction of incorrect questions \(inclusive of incorrect answers and unparseable answers\) along similar strata in Table[12](https://arxiv.org/html/2607.00937#A5.T12)\.

We can note slight variations in parsing errors across hyperparameter settings \(with smaller variations in larger models, as expected\), with a few notable cases \(such as Qwen2\.5\-1\.5B comparing PPF = bb, ya to PPF = yak, in\)\. We can note that, even in these notable cases, by comparing the total incorrect fraction to that of parsing errors, the difference in parsing errors does not account for that of the incorrect questions\. In summary, while we note parsing errors, these errors are not enough to explain the noted performance differences that influence our final instability metrics\.

### E\.9Considering Implications of Larger Models

We consider the results for Qwen2\.5\-14B\-Instruct, as seen throughout the paper \(and notably in Tables[1](https://arxiv.org/html/2607.00937#S4.T1)and[4](https://arxiv.org/html/2607.00937#S5.T4)\)\. Here, we can observe that Qwen2\.5B\-14B\-Instruct breaks the model size\-related trend forIAI\_\{A\}andIOI\_\{O\}, exhibiting greater instability in accuracy and outcome \(represented by these metrics\) than Qwen2\.5\-7B\-Instruct\.

However, we do not believe this is due to greater inherent instability in the model, but rather due to performance improvements due to chain\-of\-thought \(CoT\) promptingWeiet al\.\([2022](https://arxiv.org/html/2607.00937#bib.bib11)\)\. While smaller models are noted to largely not benefit from CoT, larger models do, creating large accuracy differentials between non\-CoT and CoT experiment settings\. As we consider CoT\-like prompts within our task prompt formats \(the "explain" and "explain\-persona" variants\), we would see larger accuracy differentials between settings using these prompts and those that do not, contributing to a greatIAI\_\{A\}\(due to CoT generally improving larger model performance\) andIOI\_\{O\}\(due to changes in accuracy subsequently changing inter\-persona rankings\)\.

Aside from this, we note that Qwen2\.5\-14B\-Instruct maintains all other existing patterns with models in the same family across all of our analyses\. While we do not evaluate larger models due to resource constraints, we reasonably expect that larger Qwen2\.5 models, for instance, would continue to maintain these trends\.

## Appendix FAdditional Figures and Figure Details

![Refer to caption](https://arxiv.org/html/2607.00937v1/x11.png)Figure 11:Hyperparameter instability results per metric, per model, stratified by dataset category and persona category\.![Refer to caption](https://arxiv.org/html/2607.00937v1/x12.png)Figure 12:All inter\-persona performance distributions between the "unstable" and "stable" experiment settings, as identified using our metrics, across all models\.### F\.1Heatmap Label Keys

The following is the key for the column labels for Figure[3](https://arxiv.org/html/2607.00937#S5.F3):

- •bb= barebones,ya= youare,yak= youareknowledge,in= identitynature,ja= justanswer,ja\-p= justanswer\-persona,e= explain,e\-p= explain\-persona

The following is the key for the heatmap labels for Figure[5](https://arxiv.org/html/2607.00937#S5.F5):

- •Dataset:All Cat= All Categories,History= History,Pol/Law= Politics and Law,Soc Sci= Social Science,Bus/Acc/Econ= Business/Accounting and Economics,EECS= EECS,Math= Math,HHM= Human Health and Medicine,Culture= Cultural Awareness and Understanding,Com Sen= Commonsense Reasoning/Social Awareness
- •Persona:All Cat= All Categories,Gender= Gender,Sexuality= Sexuality,Race/Ethn= Race/Ethnicity,Age= Age,Disabilities= Disabilities,Rel Beliefs= Religious Beliefs,Pol Beliefs= Political Beliefs,Occup Roles= Occupational Roles,EECS/Math= Occupational roles related to the EECS/Math categories in𝒟\\mathcal\{D\},Hist\./P&L= Occupational roles related to the History/Politics and Law categories in𝒟\\mathcal\{D\},Nat\. Sci\./HHM= Occupational roles related to the Natural Science/Human Health and Medicine categories in𝒟\\mathcal\{D\},Baselines= Baselines