Whose Story Gets Told? Positionality and Bias in LLM Summaries of Life Narratives
Summary
Columbia and Northwestern researchers propose a pipeline to surface race and gender bias in LLM abstractive summaries of life-story interviews, showing representational harm risks.
arXiv:2604.20131v1 Announce Type: new
Abstract: Increasingly, studies are exploring using Large Language Models (LLMs) for accelerated or scaled qualitative analysis of text data. While we can compare LLM accuracy against human labels directly for deductive coding, or labeling text, it is more challenging to judge the ethics and effectiveness of using LLMs in abstractive methods such as inductive thematic analysis. We collaborate with psychologists to study the abstractive claims LLMs make about human life stories, asking, how does using an LLM as an interpreter of meaning affect the conclusions and perspectives of a study? We propose a summarization-based pipeline for surfacing biases in perspective-taking an LLM might employ in interpreting these life stories. We demonstrate that our pipeline can identify both race and gender bias with the potential for representational harm. Finally, we encourage the use of this analysis in future studies involving LLM-based interpretation of study participants' written text or transcribed speech to characterize a positionality portrait for the study.
View Cached Full Text
Cached at: 04/23/26, 10:03 AM
# Whose Story Gets Told? Positionality and Bias in LLM Summaries of Life Narratives Source: [https://arxiv.org/html/2604.20131](https://arxiv.org/html/2604.20131) Haaris Mian2Nicholas Deas1Ananya Mayukha3Dan P\. McAdams3Kathleen McKeown1 1Department of Computer Science, Columbia University 2Department of Applied Physics and Applied Mathematics, Columbia University 3Department of Psychology, Northwestern University Correspondence:[m\.subbiah@columbia\.edu](https://arxiv.org/html/2604.20131v1/mailto:[email protected]) ###### Abstract Increasingly, studies are exploring using Large Language Models \(LLMs\) for accelerated or scaled qualitative analysis of text data\. While we can compare LLM accuracy against human labels directly for deductive coding, or labeling text, it is more challenging to judge the ethics and effectiveness of using LLMs in abstractive methods such as inductive thematic analysis\. We collaborate with psychologists to study the abstractive claims LLMs make about human life stories, asking,how does using an LLM as an interpreter of meaning affect the conclusions and perspectives of a study?We propose a summarization\-based pipeline for surfacing biases in perspective\-taking an LLM might employ in interpreting these life stories\. We demonstrate that our pipeline can identify both race and gender bias with the potential for representational harm\. Finally, we encourage the use of this analysis in future studies involving LLM\-based interpretation of study participants’ written text or transcribed speech to characterize apositionality portraitfor the study\. Whose Story Gets Told? Positionality and Bias in LLM Summaries of Life Narratives Melanie Subbiah1and Haaris Mian2and Nicholas Deas1and Ananya Mayukha3, Dan P\. McAdams3and Kathleen McKeown11Department of Computer Science, Columbia University2Department of Applied Physics and Applied Mathematics, Columbia University3Department of Psychology, Northwestern UniversityCorrespondence:[m\.subbiah@columbia\.edu](https://arxiv.org/html/2604.20131v1/mailto:[email protected]) ## 1Introduction Large Language Models \(LLMs\) are frequently used as a tool for summarizing and extracting key themes and takeaways from long documents\. When used in this way, the aim is often to replace engagement with the full document, giving the LLM considerable power to direct the meaning users derive from the text\. In extreme cases, a model can alter meaning entirely with hallucinations, or restrict interpretation to flattened caricatures of individuals\. There is, therefore, an ethical imperative to examine what perspective and framing these models bring to abstractive or thematic summarization\. This concern is amplified in qualitative research involving human self\-expression, where bias in summarization or thematic analysis can misrepresent the experiences and thoughts of real people\. For this reason, psychology and social science research focused on human experience generally relies on researchers having a close relationship with and deep understanding of the text under study\. However, such manual analysis limits the scale of studies as a researcher has limited time and attention to absorb large quantities of text\. We work with psychologists to study how, if LLMs were to be used to scale the amount of text analyzed in a study, they might influence the conclusions derived by the study\. We focus on a dataset of interviews with study participants about their life stories, designed to study how people form narrative identities over their lifetimes\. In these interviews, participants discuss deeply personal and emotionally charged issues, like racial and sexual violence, family trauma and addiction, and falling in love and finding joy\. How they express and relate to these experiences is as important as what they experience\.These interviews are, therefore, a fruitful context to study LLM interpretation of human experiences\. Errors in interpretation here can be severe\. For example, in one interview, a man discusses his early academic success and then progression into addiction connected to family factors, but an LLM summary removes this initial transition and states simply that the man’s entire life has revolved around addiction, thereby eliminating the contextual factors that are important for understanding this man’s story\. As this example illustrates, in forming abstractive or thematic conclusions, there are always choices involved around inclusion and wording of details, whether these choices are made by a person or computational model\. Social scientists recognize the subjectivity involved with qualitative methods through the idea ofpositionality, an awareness of how the researcher’s background and life experiences relative to the study participants’ may affect the interpretation of meaning\(Steltenpohlet al\.,[2023](https://arxiv.org/html/2604.20131#bib.bib41); Ledgerwoodet al\.,[2022](https://arxiv.org/html/2604.20131#bib.bib40); Jacobson and Mustafa,[2019](https://arxiv.org/html/2604.20131#bib.bib39)\)\. As such, research papers often include a statement of positionality indicating how the authors’ identities and experiences may influence the meaning they have found in the study \(see an example in Figure[1](https://arxiv.org/html/2604.20131#S1.F1)\)\. Since LLMs do not have identities and experiences, their effect on a study’s positionality cannot easily be expressed in a statement\. We instead propose what apositionality portraitcould look like for an LLM\. Figure 1:Excerpt from a positionality statement fromMayukhaet al\.\([2025](https://arxiv.org/html/2604.20131#bib.bib306)\)\. We ask,how does LLM use fit into a statement like this?Using quantitative methods alongside expert evaluation from psychologists, we analyze the “default” framing that LLMs use when summarizing the life story interviews, as well as potential biases in the meaning conveyed\. For example, one Black interviewee describes growing up in Black communities with limited exposure to positive interactions with white people\. An LLM summary of this section interprets the experience as demonstrating a “hatred for white people,” which misrepresents the interviewee\. Awareness of blind spots—and active mitigation of them—is crucial to preventing representational harm, such as misrepresenting or erasing individuals’ experiences\. The key features of our work are: 1. 1\.We work with psychologists as evaluators and with long, nuanced, and narrative\-form life story interviews unseen by LLMs\. 2. 2\.We propose a quantitative pipeline for identifying apositionality portraitfor LLM\-based analysis of this data\.111We release the pipeline code on[GitHub](https://github.com/melaniesubbiah/positionalityportrait)\. Our key findings are that: 1. 1\.Summarization can be used to test bias in abstractive analysis in LLMs\. 2. 2\.We observe shifts in how models make choices about content and themes for abstractive analysis based on implicit and explicit demographic conditioning\. 3. 3\.Our pipeline successfully uncovers both common LLM biases as well as less common ones, such as stereotypes against male emotional expression\. ## 2Background Our collaborators in psychology focus on adult development and how experiences and themes in people’s lives affect psychological outcomes and markers of well\-being\. Their research often usesinductive thematic analysis, a type of qualitative research that looks for themes across many documentsRiger and Sigurvinsdottir \([2016](https://arxiv.org/html/2604.20131#bib.bib48)\), and summarizes how individual examples support these themes\. Typically, this qualitative work is done by researchers carefully transcribing, reading, and coding \(similar to annotating\) potentially hundreds of pages of text to identify thematic patterns\. Technology is already incorporated in this process through the use of software to organize text data and track annotations\. More research has been done recently on using modern NLP methods to automatically code text once researchers have developed a codebook \(deductive coding\) as this is similar to other popular text classification problems in NLPChewet al\.\([2023](https://arxiv.org/html/2604.20131#bib.bib47)\); Taiet al\.\([2024](https://arxiv.org/html/2604.20131#bib.bib55)\); Xiaoet al\.\([2023](https://arxiv.org/html/2604.20131#bib.bib54)\)\. However, we aim to explore how the use of abstractive methods in developing a sense of themes from the text may influence the creation of meaningDaiet al\.\([2023](https://arxiv.org/html/2604.20131#bib.bib351)\); De Paoli \([2024](https://arxiv.org/html/2604.20131#bib.bib33)\)\. We develop our methods using a psychology dataset of interviews with over one hundred participants about their life stories\. #### Life Stories Dataset Figure 2:A sample human\-written summary \(top\) for the life chapters portion of an interview in the life stories dataset, followed by an LLM\-generated summary with themes for the same story \(bottom\)\.We access a research\-only dataset from psychology that is not publicly available online, and therefore unseen by LLMs\. The dataset consists of 163 interviews with Americans in the Foley Longitudinal Study of Adulthood \(FLSA\)McAdams \([2008](https://arxiv.org/html/2604.20131#bib.bib4)\)\. The design of the study focuses on how adults construct narrative identities over a lifetime and how aspects of these narratives connect to their well\-being\. Interviews were conducted each year for nine years, between 2008\-2017\. Interviewees are asked many questions about their life stories, values, and key scenes in their lives\. To protect the privacy of study participants, we cannot publish long excerpts of the interviews or the data itself, but those who are interested can contact the FLSA dataset creators for potential opportunities to work with the data222We accessed the data through arriving at an agreement around data privacy and use, including only using open\-source models run on secure hardware\.\. We analyze the life chapters section of year one interviews\. In brief, this section asks participants: > Think about your life as if it were a book or novel, describe briefly what the main chapters in the book might be, and provide an overall plot summary of your story, going chapter by chapter\. An example of the type of content discussed in the life chapters portion is shown in Figure[2](https://arxiv.org/html/2604.20131#S2.F2)\. Details on parsing the interviews into individual question responses are described in Appendix[A](https://arxiv.org/html/2604.20131#A1)\. Removal of interviews for which we could not automatically parse the life chapters section resulted in 154 interviews to analyze\. The life chapters section consists of 3,497 words on average and 34 interviewer\-respondent exchanges with respondents speaking 3,045 of the words on average\. Along with the interviews, participants self\-report their demographic information\. We study gender and race in particular\. The study participants are 36% men and 64% women, and 57% white and 43% Black\.333Three participants identified as “interracial” or “other” for race, but we remove them from the study\. ## 3Related Work In computer science, some work has looked at characterizing the views most often expressed by language modelsSanturkaret al\.\([2023](https://arxiv.org/html/2604.20131#bib.bib42)\); Scherreret al\.\([2024](https://arxiv.org/html/2604.20131#bib.bib305)\); Durmuset al\.\([2023](https://arxiv.org/html/2604.20131#bib.bib302)\), or using LLMs to simulate polling results or debates from certain demographics or perspectivesNamikoshiet al\.\([2024](https://arxiv.org/html/2604.20131#bib.bib298)\); Jansenet al\.\([2023](https://arxiv.org/html/2604.20131#bib.bib297)\); Tjuatjaet al\.\([2024](https://arxiv.org/html/2604.20131#bib.bib299)\); Taubenfeldet al\.\([2024](https://arxiv.org/html/2604.20131#bib.bib303)\)\. Other work is critical of the bias and simplifications inherent in using LLMs to simulate perspectivesChenget al\.\([2023a](https://arxiv.org/html/2604.20131#bib.bib5),[b](https://arxiv.org/html/2604.20131#bib.bib301)\); Guptaet al\.\([2024](https://arxiv.org/html/2604.20131#bib.bib300)\); Agnewet al\.\([2024](https://arxiv.org/html/2604.20131#bib.bib292)\)\. In psychology, the consideration of positionality in research has entered the field relatively recentlySteltenpohlet al\.\([2023](https://arxiv.org/html/2604.20131#bib.bib41)\); Ledgerwoodet al\.\([2022](https://arxiv.org/html/2604.20131#bib.bib40)\); Jacobson and Mustafa \([2019](https://arxiv.org/html/2604.20131#bib.bib39)\), although it has been advocated for by some over decadesJosselson \([2006](https://arxiv.org/html/2604.20131#bib.bib362)\)\. The concept has deeper historical roots in the fields of anthropology and sociologyBehar \([2022](https://arxiv.org/html/2604.20131#bib.bib357)\); Hertz \([1996](https://arxiv.org/html/2604.20131#bib.bib37)\); Finlay \([1998](https://arxiv.org/html/2604.20131#bib.bib36)\); Harding \([1991](https://arxiv.org/html/2604.20131#bib.bib35)\)\. Computational studies have begun to look at comparisons of LLM\-based thematic analysis to human analysisDaiet al\.\([2023](https://arxiv.org/html/2604.20131#bib.bib351)\); De Paoli \([2024](https://arxiv.org/html/2604.20131#bib.bib33)\); Deineret al\.\([2024](https://arxiv.org/html/2604.20131#bib.bib350)\); De Paoli and Mathis \([2025](https://arxiv.org/html/2604.20131#bib.bib353)\); Misgavet al\.\([2025](https://arxiv.org/html/2604.20131#bib.bib361)\), but they focus on end results rather than positionality\.Ibrahim and Voyer \([2026](https://arxiv.org/html/2604.20131#bib.bib346)\)advocate for this type of work but do not propose a general technical approach like we do, which can ground discussion in pre\-analysis checklists like ARCPrahl \([2026](https://arxiv.org/html/2604.20131#bib.bib348)\)\. ## 4Methods Our goal is to produce a pipeline that can generate apositionality portraitfor a given LLM with respect to open\-ended interpretation of our dataset\. In interpreting human experience, capturing the wording, emotional framing, psychological states, and themes are all important\. A summary is an effective method of communicating open\-ended understanding of a document, so we use analysis of LLM summaries as the basis of our pipeline\. We consider a corpus of documentsDD\. For a documentd∈Dd\\in D, an LLM can be prompted to produce a structured summary ofddconsisting of generated textssand a set of generated themestt\. An LLM induces a distributionp\(s,t\|d\)p\(s,t\|d\), and we consider sampling summariess^d,t^d∼p\(s,t\|d\)\\hat\{s\}\_\{d\},\\hat\{t\}\_\{d\}\\sim p\(s,t\|d\)\(see Figure[3](https://arxiv.org/html/2604.20131#S4.F3)\)\. Eachs^d,t^d\\hat\{s\}\_\{d\},\\hat\{t\}\_\{d\}can be considered an abstraction ofdd, making choices about content, style, and meaning from the perspective of the LLM\. We can expose these choices through comparing aspects of content, style, and meaning betweens^d,t^d\\hat\{s\}\_\{d\},\\hat\{t\}\_\{d\}anddd\. Since we are sampling summaries from a distribution of possible outputs, we sample a set of summariesS^d,T^d\\hat\{S\}\_\{d\},\\hat\{T\}\_\{d\}to approximate the distribution\. We consider the set of demographicsCCfor all the writers or transcribed speakers of documents inDD\. In our case,C=\{C=\\\{Black woman, Black man, white woman, white man\}\\\}\. The writer or transcribed speaker for a documentddidentifies with some demographicc∈Cc\\in C, so we considerDcD\_\{c\}as the subset of documentsDDwritten by people with demographicscc\. Eachs^d\\hat\{s\}\_\{d\}ford∈Dcd\\in D\_\{c\}may be implicitly conditioned on some aspects ofccthrough what can be inferred from the writing ofdd\. We can study this effect by comparing how the shift in wording and framing betweenddands^d\\hat\{s\}\_\{d\}may differ ford∈Dc1d\\in D\_\{c\_\{1\}\}andd∈Dc2d\\in D\_\{c\_\{2\}\}for somec1,c2∈Cc\_\{1\},c\_\{2\}\\in C\. We can assess the explicit influence of a specific demographicc∈Cc\\in Cthrough intentionally exposingccto the LLM in the summarization prompt and examining the shift betweenp\(s,t\|d,∅\)p\(s,t\|d,\\emptyset\)\(with no exposed demographic\) andp\(s,t\|d,c\)p\(s,t\|d,c\)\(with exposed demographic\) ford∈Dcd\\in D\_\{c\}\. We refer tos^d,t^d∼p\(s,t\|d,∅\)\\hat\{s\}\_\{d\},\\hat\{t\}\_\{d\}\\sim p\(s,t\|d,\\emptyset\)asbaselinesummaries and themes orS^dbase,T^dbase\\hat\{S\}\_\{d\_\{base\}\},\\hat\{T\}\_\{d\_\{base\}\}\(with no intentionally exposed demographic in the prompt\), ands^d,t^d∼p\(s,t\|d,c\)\\hat\{s\}\_\{d\},\\hat\{t\}\_\{d\}\\sim p\(s,t\|d,c\)asdemographic\-conditionedsummaries and themes orS^ddemo,T^ddemo\\hat\{S\}\_\{d\_\{demo\}\},\\hat\{T\}\_\{d\_\{demo\}\}\(with exposed demographic in the prompt\)\. Figure 3:We ask an LLM to produce a structured summary including a list of identified themes from the source document\. We additionally compare a setting where we intentionally expose the document writer’s demographic details in the prompt\.### 4\.1Wording and Semantics We first use standard summarization metrics for wording and semantic similarity to compare the documents and baseline summaries within a demographic, defined as follows: sim\(Dc\)=1\|Dc\|∑d∈Dc∑s^d∈S^dbasef\(s^d,d\)\|S^dbase\|sim\(D\_\{c\}\)=\\frac\{1\}\{\|D\_\{c\}\|\}\\sum\_\{d\\in D\_\{c\}\}\\frac\{\\sum\_\{\\hat\{s\}\_\{d\}\\in\\hat\{S\}\_\{d\_\{base\}\}\}f\(\\hat\{s\}\_\{d\},d\)\}\{\|\\hat\{S\}\_\{d\_\{base\}\}\|\} We use the following measures for the metric functionffin thesimsimfunction: #### ROUGE The ROUGE\-1 and ROUGE\-L scoresLin \([2004](https://arxiv.org/html/2604.20131#bib.bib56)\)are word overlap metrics using words in the original form they appear\. #### BERTScore The BERTScoreZhanget al\.\([2020](https://arxiv.org/html/2604.20131#bib.bib121)\)is a learned metric for semantic similarity based on embeddings from a BERT\-style model\. Since these metrics were designed for comparing a summary against a reference summary as opposed to the original document, we use the precision variant of the scores, rather than recall or F1\. We note that other standard metrics could be used here by future work, but we consider this set for reasonable coverage and efficiency\. For example, a lower ROUGE score for a demographic group indicates the summary is using less similar wording to the interviewee’s own self\-expression, and a lower BERTScore indicates more semantic differences\. In evaluating different metrics, we found that often task\-specific finetuned LLMs did not generalize well to our data\. We want to avoid compounding effects of assessing LLMs with other LLMs, so we prefer wording\-based or general\-purpose embedding\-based methods\. ### 4\.2Psychological States We then assess similarities in the emotional and psychological states expressed inS^\\hat\{S\}compared toDD, defined as follows: μd=1\|S^dbase\|∑s^d∈S^dbasef\(s^d\)\\mu\_\{d\}=\\frac\{1\}\{\|\\hat\{S\}\_\{d\_\{base\}\}\|\}\\sum\_\{\\hat\{s\}\_\{d\}\\in\\hat\{S\}\_\{d\_\{base\}\}\}f\(\\hat\{s\}\_\{d\}\) psych\(Dc\)=1\|Dc\|∑d∈Dc2\(μd−f\(d\)\)abs\(μd\)\+abs\(f\(d\)\)psych\(D\_\{c\}\)=\\frac\{1\}\{\|D\_\{c\}\|\}\\sum\_\{d\\in D\_\{c\}\}\\frac\{2\(\\mu\_\{d\}\-f\(d\)\)\}\{\\text\{abs\}\(\\mu\_\{d\}\)\+\\text\{abs\}\(f\(d\)\)\} If the denominator ofpsychpsychis 0 then we usepscyh\(Dc\)pscyh\(D\_\{c\}\)=0\. Differences in this measure across demographic groups indicate the LLM’s choices to amplify or suppress psychological states for one demographic over another\. We use the following three metrics forffin thepsychpsychfunction: #### LIWC The LIWC lexiconPennebakeret al\.\([2015](https://arxiv.org/html/2604.20131#bib.bib368)\)associates words with different psychological properties \(e\.g\., social life, psychological drives, affective states, etc\.\) and counts the prevalence of these different properties in text based on frequency of words in these different categories\. #### VAD The VAD lexiconMohammad \([2018](https://arxiv.org/html/2604.20131#bib.bib367)\)counts words associated with different emotional valence \(i\.e\.,happyis positive valence whilesadis negative\), arousal \(i\.e\.,excitedis high arousal whileboredis low\), and dominance \(i\.e\.,powerfulis high dominance andweakis low\)\. #### SCM The SCM projection methodQin and Tam \([2023](https://arxiv.org/html/2604.20131#bib.bib366)\)uses word embeddings to project a text representation onto vectors representing warmth and competence in embedding space\. Warmth and competence are axes of the Stereotype Content Model from psychologyCuddyet al\.\([2009](https://arxiv.org/html/2604.20131#bib.bib365)\)for characterizing stereotypes of groups\. We normalize LIWC counts by word count per summary or document to account for differences in length\. VAD and SCM scores are on continuous spectra so we average word\-level scores across each summary or document\. We then take the difference between these normalized scores forS^\\hat\{S\}andDD\. We experimented with additional models, such asShenet al\.\([2023](https://arxiv.org/html/2604.20131#bib.bib360)\)’s empathy model, but the scores were not well\-calibrated for our data\. ### 4\.3Themes We finally assess differences in the themes sampled fromp\(T^\|D,∅\)p\(\\hat\{T\}\|D,\\emptyset\)compared top\(T^\|D,c\)p\(\\hat\{T\}\|D,c\)across demographic groups\. In this case, we cannot directly compareT^\\hat\{T\}against themes inDDwithout human involvement, but we can test whether intentionally exposingccto the summarization model changes the themes identified by the model, indicating howccaffects the themes the model identifies and highlights\. We consider each theme as a word or phrase, such asresilienceorpersonal growth, and measure shifts in a themettas follows: theme\(t,Dc\)=∑d∈De\(𝟙\[t∈⋃t^d∈T^ddemot^d\]−𝟙\[t∈⋃t^d∈T^dbaset^d\]\)\|Dc\|theme\(t,D\_\{c\}\)=\\dfrac\{\\sum\\limits\_\{d\\in D\_\{e\}\}\\left\(\\mathbbm\{1\}\\left\[t\\in\\bigcup\\limits\_\{\\hat\{t\}\_\{d\}\\in\\hat\{T\}\_\{d\_\{\\text\{demo\}\}\}\}\\hat\{t\}\_\{d\}\\right\]\-\\mathbbm\{1\}\\left\[t\\in\\bigcup\\limits\_\{\\hat\{t\}\_\{d\}\\in\\hat\{T\}\_\{d\_\{\\text\{base\}\}\}\}\\hat\{t\}\_\{d\}\\right\]\\right\)\}\{\|D\_\{c\}\|\} ### 4\.4Qualitative study Figure 4:The questions we ask the psychology researchers in the qualitative study\.In order to determine the validity of our approach for quantifying LLM positionality, we conduct a qualitative study with our collaborators in psychology to assess aspects of perspective or bias in the LLM summaries\. Three psychology research students \(one PhD student and two undergraduate research assistants\) read a subset of eight interviews in full\. These researchers are working with these interviews in their own research so they have particular expertise in this dataset as well as having psychology training\. The subset of interviews are selected to represent an even distribution of Black men, Black women, white men, and white women\. We then provide them with an interface to answer questions about the LLM summaries fromp\(S∣D\)p\(S\\mid D\)\. We ask the researchers multiple choice and write\-in questions about the general quality of the baseline summaries \(see interface instructions in Appendix[B](https://arxiv.org/html/2604.20131#A2)\)\. Each interview is evaluated by two or three raters\. The questions we ask each student are shown in Figure[4](https://arxiv.org/html/2604.20131#S4.F4)\.444We initially included two other questions in the study focused on comprehensiveness and significant omissions\. Due to a miscommunication during annotation, the psychologists assumed the summaries were based on the entire interviews instead of just the life chapters question, so we removed these questions from the final analysis\.Question 1 targets precision, similar to our wording and semantics metrics\. Questions 2 and 3 are based on the Stereotype Content Model \(SCM\), similar to one of our psychological states metrics\. Question 4 asks directly about stereotypes related to the identity of the interviewee\. Finally, question 5 is intentionally open\-ended to surface other observations from the psychologists\. Questions 1\-3 are answered on a 4\-point Likert scale and we look for differences in ratings across demographic groups\. For questions 4\-5, we look for differences in observations across groups\. We use these human responses as the basis for comparison for our quantitative results\. For stereotypes, we consider a binary score of whether a stereotype was identified or not\.555In practice, we did not observe any stereotypes of typeotherso we just considergenderandracialstereotypes\.When considering these binary ratings as well as individual Likert ratings, we observe moderate pairwise annotator agreement averaged across annotator pairs \(agreement on 58% of ratings, 0\.44 average Cohen’s kappa\)\. For rating direction, capturing whether a score is positive \(rating of 3 or 4\) or negative \(rating of 1 or 2\), we see stronger agreement \(agreement on 74% of rating directions, 0\.49 average Cohen’s kappa\)\. We average expert ratings for each question to account for the moderate disagreement in exact ratings\. ## 5Experimental Setup #### Summarization We provide the life chapters portion of the interview transcript as input, and generate summaries using a prompt that asks the model to 1\) focus on how the person finds meaning in life, and 2\) provide a section on the person’s core values as expressed by their story \(prompts shown in Appendix[C](https://arxiv.org/html/2604.20131#A3)\)\. The core values fill the role of the themes in our pipeline, demonstrating how prompts can be adapted to salient factors in a given application\. For the demographic\-conditioned summaries, we add a sentence indicating whether the interviewee is a Black man, Black woman, white man, or white woman\. We provide the full life chapters section to the LLMs for clarity, but during evaluation, only compare summaries to the interviewee’s responses\. #### Models We study several open\-source LLMs: Qwen\-2\.5\-7B\-Instruct, Llama\-3\.1\-8B\-Instruct, and Llama\-3\.2\-3B\-Instruct\. We cannot experiment with API\-based models given the sensitivity of the dataset\. For each model, we generate summaries with five different random seeds, temperature 0\.7, and up to 6000 output tokens\. We remove outputs that do not have both summary and core values sections\. ## 6Results We assess the results from our analysis focused on wording and semantics, psychological states, and themes\. Statistically significant results are computed using a one\-sided bootstrapped paired significance test with 5,000 samples and 95% confidence\. In bar plots, we show an 83% confidence interval to approximate statistical significance in visual comparisons across groups\. We show one of the full results bar plots in Figure[5](https://arxiv.org/html/2604.20131#S6.F5)and show the others in Appendix[D](https://arxiv.org/html/2604.20131#A4), preferencing reporting a summary of the statistically significant results in the positionality portraits \(Figures[6](https://arxiv.org/html/2604.20131#S6.F6),[7](https://arxiv.org/html/2604.20131#S6.F7),[8](https://arxiv.org/html/2604.20131#S6.F8)\)\. #### Positionality portraits The positionality portraits summarize statistically significant demographic effects\. For wording and semantics and psychological states, a demographic group has a dark green tile if summaries increase an attribute for that group more than for all three other demographic groups\. The tile is lighter shades of green for increases over only two other groups or one other group\. A tile is grey if summaries for that group do not show greater increases in an attribute relative to other groups\. For themes, green indicates inclusion of explicit demographics for a group increases the identification of a theme, and red indicates a decrease\. A darker color indicates a lower p\-value\. Figure 5:We show results for each LLM in each column for scoring the wording and semantic similarity of the summary against the life chapters interview\. We average an individual’s scores across the five random seeds and then report the mean across a demographic group with an 83% confidence interval\.Figure 6:The Llama\-3\.2\-3Bpositionality portrait\.Figure 7:The Llama\-3\.1\-8Bpositionality portrait\.Figure 8:The Qwen\-2\.5\-7Bpositionality portrait\. ### 6\.1Are wording and semantics maintained differently across demographic groups? In Figure[5](https://arxiv.org/html/2604.20131#S6.F5), we show the summarization results for intersectional demographic effects\. We see that across all three models, the summaries show a similar level of wording overlap across demographic groups, as demonstrated by the ROUGE\-1 and ROUGE\-L scores\. There are differences in semantic similarity between the summaries and interviews though across demographic groups\. Significance testing shows that white men have a higher BERTScore than Black men and women for Llama\-3B and than white women for Llama\-8B\. Black men have a significantly lower BERTScore than all other demographics for Qwen\-7B\. These results are not surprising given existing research on bias in LLMs that results in better interpretation of White Mainstream EnglishDeaset al\.\([2023](https://arxiv.org/html/2604.20131#bib.bib129)\)\. We would not expect models to generate African American Language effectively, but we would hope to see more similarities in semantics across racial groups\. However, it is surprising that white women are also often at the same point of lower semantic similarity as Black men and women, indicating there may be a larger effect than just emphasis on White Mainstream English during training\. In particular, we know that white men are overrepresented in training datasets collected in the United StatesDodgeet al\.\([2021](https://arxiv.org/html/2604.20131#bib.bib373)\); Deaset al\.\([2025](https://arxiv.org/html/2604.20131#bib.bib372)\); Baileyet al\.\([2022](https://arxiv.org/html/2604.20131#bib.bib374)\)\. ### 6\.2Which psychological states are amplified for different demographic groups? In Appendix[D](https://arxiv.org/html/2604.20131#A4), we show the full LIWC, VAD, and SCM results with a subset shown in the positionality portraits in Figures[6](https://arxiv.org/html/2604.20131#S6.F6),[7](https://arxiv.org/html/2604.20131#S6.F7), and[8](https://arxiv.org/html/2604.20131#S6.F8)\. We mostly see increases in these scores for the summary relative to the transcript, which may be due to greater information density in the summary\. For LIWC sub\-themes, we see significant increases for white individuals over Black individuals infamilyandsocialthemes, with Black men, often having the lowest increase in thefamilytheme\. We see that Black women receive a significant increase in emphasis in this category over white men for Llama\-8B\. Importantly, as we’ll see in the next section, this type of emphasis does not translate to an increase in positive values, such ashard workorperseverance\. Instead, this type of increase likely indicates an emphasis on the harder strenuous aspects, in contrast to an increase inleisurefor white women over Black women\. For VAD scores, we see white men showing significant increases in valence, indicating their stories are made more positive\. For llama models, summaries for Black men significantly increase dominance, or tendency to strength and control\. Lastly, for the SCM projections, we see an increase in warmth for Black men for all three models, as well as Black women for Llama\-3B\. Summaries for white men show the greatest relative increases in competence\. These results align with studies showing societal stereotypes of Black peopleAllport \([1954](https://arxiv.org/html/2604.20131#bib.bib358)\); Fiskeet al\.\([2009](https://arxiv.org/html/2604.20131#bib.bib356)\)and womenGlick and Fiske \([2018](https://arxiv.org/html/2604.20131#bib.bib345)\); Koenig and Eagly \([2014](https://arxiv.org/html/2604.20131#bib.bib355)\)as typically higher in warmth, but lower in agency/competence\. These results show that each LLM brings a slightly different perspective and bias to abstractive claims\. We see some stereotypes we expect around gender and race, such as presenting Black women at work and white women at leisure\. We also see outcomes that may be surprising; for example, men’s experiences are adjusted significantly in emotional expression, as indicated by shifts in valence, arousal, and dominance\. ### 6\.3How do identified themes depend on demographics? In Appendix[D](https://arxiv.org/html/2604.20131#A4), we show the full results for both counts of themes in baseline summaries and shifts in theme counts based on explicit demographics for all of the top 30 themes\. The statistically significant shifts for the top 20 themes for a model are summarized in the positionality portraits shown in Figures[6](https://arxiv.org/html/2604.20131#S6.F6),[7](https://arxiv.org/html/2604.20131#S6.F7), and[8](https://arxiv.org/html/2604.20131#S6.F8)\. The baseline theme counts include both real differences in what interviewees express and potential language model bias while the shift in the theme counts isolates the effect of language model bias\. In the baseline theme counts,resilience,family, andpersonal growthare the most strongly identified themes\. These positive narrative arcs are common for LLMs which frequently end stories on positive conclusions and notes of growth, a known issue in narrative generationTianet al\.\([2024](https://arxiv.org/html/2604.20131#bib.bib359)\)\. Black individuals show morefaithin their themes, which is consistent with the self\-identified demographics in the dataset for religion and spirituality666For Black participants, 71% identify as very spiritual and 43% identify as very religious\. For white participants, 26% identify as very spiritual and 11% identify as very religious\.\. Finally, the theme ofrelationshipsis most prevalent in summaries of white men’s stories\. Shifting to the differences in the percent of summaries identifying the themes between the demographic\-conditioned and baseline summaries, we see a significant increase in identification ofhard workfor white men and decrease for white women with Llama\-8B\. We see a decrease inperseverancefor Black women in Llama\-3B\. Finally, we see a decrease inself\-awarenessand/orpersonal growthorgrowthfor Black men for all three models\. ### 6\.4Does our quantitative approach uncover real phenomena identified by experts? Table 1:Average scores assigned by the psychology experts for summaries by model and interviewee demographic\. Faithfulness, warmth, and competence are averaged Likert scores, whereas stereotypes are the percent of summaries containing stereotypes in that category\.Figure 9:We show some examples of the types of issues identified by the psychology experts \(with excerpts from white men in blue, from Black men in green, and from Black women in red\)\.Table[1](https://arxiv.org/html/2604.20131#S6.T1)shows scores from the expert human study\. We see that Llama\-8B summaries are more faithful, while Llama\-3B summaries portray participants with the highest degree of warmth and competence\. Qwen\-7B includes the most gender stereotypes, and Llama\-3B includes the least stereotypes for both gender and race\. This finding suggests that as models are more powerful, and likely better summarizers, they may also learn more nuanced bias\. We cannot test very large closed models on this dataset, but this finding suggests that larger models may not fix these issues\. Given the small number of interviewees included in the human study, we break results down by individual demographic traits in Table[1](https://arxiv.org/html/2604.20131#S6.T1)rather than showing intersectional identities\. We see that summaries for white individuals and women are rated as more faithful than for other demographics across all three models, except for Llama\-3B, and summaries for white individuals and women portray the interviewee as more warm and competent in general\. Qwen\-7B portrays Black individuals as slightly more warm and competent, possibly due to a difference in training data from a Chinese as opposed to American context\. Overall, these results do not align exactly with our SCM warmth and competence projection scores, but using lower warmth and competence ratings as axes for more negative stereotypes, they reveal a similar finding to what we saw quantitatively; there are more harmful representational stereotypes around Black individuals and men in the summaries\. The stereotype scores from the psychologists show the same pattern with more racial stereotypes about Black individuals and more gender stereotypes about men\. While we may have expected to see more gendered stereotypes against women, we see that actually this finding is in line with the rest of our quantitative results that show harmful shifts, particularly against Black men, but also against white men in terms of how their self\-expression is qualified\. Finally, in write\-in feedback, the psychologists consistently comment on the models omitting or glossing over weighty emotional events777Since the psychologists had initially assumed the summaries covered the entire interviews, we manually removed comments which were not appropriate based on the life chapters portion of the interview and still found this observation prevalent\.\. For example, a brother’s death is consistently left out of summaries of one interview, and a father’s dramatic murder is stated matter\-of\-factly in another\. Lastly, another woman talks in depth about being molested as a child, which is not mentioned at all, possibly due to safety\-related post\-training of models\. We show additional examples of issues identified by the psychologists in Figure[9](https://arxiv.org/html/2604.20131#S6.F9)\. ## 7Discussion We see that our quantitative pipeline produces an individualpositionality portraitfor each open\-source LLM we test, and demonstrates consistency with what human experts observe\. These portraits uncover significant race and gender\-based representational issues, which are not necessarily resolved by larger models\. Instead, we find larger models may actually inject a more biased perspective in abstractive analysis\. Overall, we see many issues concentrated in how models handle emotional expression\. Models do not effectively distinguish emotionally weighty events and give them appropriate emphasis in summaries\. They also shift toward an overly positive framing of events which is inconsistent with the person’s own experience and detrimental in the context of psychological research\. One surprising finding is that emotional expression is more significantly adjusted for men\. This outcome indicates the benefit of having an open\-ended method like this for assessing LLM positionality when working with a new dataset\. We also see other negative gender\-based stereotypes, like an emphasis on work for Black women versus leisure for white women, and a significant reduction in identification of self\-awareness as a value for Black men\. ## 8Conclusion In this work, we draw on qualitative social science notions ofpositionalityto evaluate the ways in which LLMs may alter researchers’ thematic analyses of text\. To capture such biases, we introduce a quantitative pipeline evaluating the substantive and thematic shift in model\-generated summaries\. Using a corpus of life story interviews, the results of our pipeline show, for example, that LLMs are less appropriate for comparing experiences across racial backgrounds or analyzing the emotional experience of men among the analyzed interviews\. While the differences and shifts we discuss throughout our results are statistically significant, they also often constitute small shifts in numerical measures, meaning that some of these models may still be “good enough” to use in practice\. Overall, however, using this pipeline provides researchers with useful insight into the biggest effects to look out for in using LLMs in their analysis and which applications may be more or less appropriate\. We propose that in using LLMs in qualitative analysis for new datasets, researchers present apositionality portraitsuch as we have presented here to demonstrate the position from which these models may approach data\. Limitations and Ethics\.The experiments in this study focus on one dataset and three smaller open\-source LLMs\. We view a deep expert\-informed study as necessary for a topic like this, but we hope future work can extend thispositionality portraitapproach to other LLMs and datasets\. To respect the privacy of study participants, we follow a data safety contract, including using local data storage and locally run LLMs\. We do not release full interviews and only present paraphrased or short excerpts of content with identifying details removed\. Our human study is approved by IRB protocol AAAV9832\. ## Acknowledgments This work is supported by funds provided by the National Science Foundation and by DoD OUSD \(R&E\) under Cooperative Agreement DBI\-2229929 \(The NSF AI Institute for Artificial and Natural Intelligence\)\. An author is additionally supported by the National Science Foundation Graduate Research Fellowship DGE\-2036197, the Columbia University Provost Diversity Fellowship, and the Columbia School of Engineering and Applied Sciences Presidential Fellowship\. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation\. ## References - W\. Agnew, A\. S\. Bergman, J\. Chien, M\. Díaz, S\. El\-Sayed, J\. Pittman, S\. Mohamed, and K\. R\. McKee \(2024\)The illusion of artificial inclusion\.InProceedings of the CHI Conference on Human Factors in Computing Systems,pp\. 1–12\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - G\. W\. Allport \(1954\)The nature of prejudice\.\.Cited by:[§6\.2](https://arxiv.org/html/2604.20131#S6.SS2.p2.1)\. - A\. H\. Bailey, A\. Williams, and A\. Cimpian \(2022\)Based on billions of words on the internet, people=men\.Science Advances8\(13\),pp\. eabm2463\.External Links:[Document](https://dx.doi.org/10.1126/sciadv.abm2463),[Link](https://www.science.org/doi/abs/10.1126/sciadv.abm2463),https://www\.science\.org/doi/pdf/10\.1126/sciadv\.abm2463Cited by:[§6\.1](https://arxiv.org/html/2604.20131#S6.SS1.p1.1)\. - R\. Behar \(2022\)The vulnerable observer: anthropology that breaks your heart\.beacon press\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - M\. Cheng, E\. Durmus, and D\. Jurafsky \(2023a\)Marked personas: using natural language prompts to measure stereotypes in language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 1504–1532\.External Links:[Link](https://aclanthology.org/2023.acl-long.84/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.84)Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - M\. Cheng, T\. Piccardi, and D\. Yang \(2023b\)CoMPosT: characterizing and evaluating caricature in LLM simulations\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 10853–10875\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.669/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.669)Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - R\. Chew, J\. Bollenbacher, M\. Wenger, J\. Speer, and A\. Kim \(2023\)LLM\-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding\.arXiv\(en\)\.Note:arXiv:2306\.14924 \[cs, stat\]External Links:[Link](http://arxiv.org/abs/2306.14924)Cited by:[§2](https://arxiv.org/html/2604.20131#S2.p2.1)\. - A\. J\. Cuddy, S\. T\. Fiske, V\. S\. Kwan, P\. Glick, S\. Demoulin, J\. Leyens, M\. H\. Bond, J\. Croizet, N\. Ellemers, E\. Sleebos,et al\.\(2009\)Stereotype content model across cultures: towards universal similarities and some differences\.British journal of social psychology48\(1\),pp\. 1–33\.Cited by:[§4\.2](https://arxiv.org/html/2604.20131#S4.SS2.SSS0.Px3.p1.1)\. - S\. Dai, A\. Xiong, and L\. Ku \(2023\)LLM\-in\-the\-loop: leveraging large language model for thematic analysis\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 9993–10001\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.669/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.669)Cited by:[§2](https://arxiv.org/html/2604.20131#S2.p2.1),[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - S\. De Paoli and W\. S\. Mathis \(2025\)Reflections on inductive thematic saturation as a potential metric for measuring the validity of an inductive thematic analysis with llms\.Quality & Quantity59\(1\),pp\. 683–709\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - S\. De Paoli \(2024\)Performing an Inductive Thematic Analysis of Semi\-Structured Interviews With a Large Language Model: An Exploration and Provocation on the Limits of the Approach\.Social Science Computer Review42\(4\),pp\. 997–1019\.Note:Publisher: SAGE Publications IncExternal Links:ISSN 0894\-4393,[Link](https://doi.org/10.1177/08944393231220483),[Document](https://dx.doi.org/10.1177/08944393231220483)Cited by:[§2](https://arxiv.org/html/2604.20131#S2.p2.1),[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - N\. Deas, J\. Grieser, S\. Kleiner, D\. Patton, E\. Turcan, and K\. McKeown \(2023\)Evaluation of African American language bias in natural language generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 6805–6824\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.421),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.421)Cited by:[§6\.1](https://arxiv.org/html/2604.20131#S6.SS1.p1.1)\. - N\. Deas, B\. Vente, A\. Ananthram, J\. A\. Grieser, D\. U\. Patton, S\. Kleiner, J\. R\. S\. Iii, and K\. McKeown \(2025\)Data caricatures: on the representation of African American language in pretraining corpora\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 29192–29217\.External Links:[Link](https://aclanthology.org/2025.acl-long.1416/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1416),ISBN 979\-8\-89176\-251\-0Cited by:[§6\.1](https://arxiv.org/html/2604.20131#S6.SS1.p1.1)\. - M\. S\. Deiner, V\. Honcharov, J\. Li, T\. K\. Mackey, T\. C\. Porco, and U\. Sarkar \(2024\)Large language models can enable inductive thematic analysis of a social media corpus in a single prompt: human validation study\.JMIR Infodemiology4,pp\. e59641\.External Links:ISSN 2564\-1891,[Document](https://dx.doi.org/10.2196/59641),[Link](https://infodemiology.jmir.org/2024/1/e59641),[Link](https://doi.org/10.2196/59641)Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - J\. Dodge, M\. Sap, A\. Marasović, W\. Agnew, G\. Ilharco, D\. Groeneveld, M\. Mitchell, and M\. Gardner \(2021\)Documenting large webtext corpora: a case study on the colossal clean crawled corpus\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 1286–1305\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.98/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.98)Cited by:[§6\.1](https://arxiv.org/html/2604.20131#S6.SS1.p1.1)\. - E\. Durmus, K\. Nyugen, T\. I\. Liao, N\. Schiefer, A\. Askell, A\. Bakhtin, C\. Chen, Z\. Hatfield\-Dodds, D\. Hernandez, N\. Joseph,et al\.\(2023\)Towards measuring the representation of subjective global opinions in language models\. arxiv\.arXiv preprint arXiv:2306\.16388\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - L\. Finlay \(1998\)Reflexivity: an essential component for all research?\.British Journal of Occupational Therapy61\(10\),pp\. 453–456\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - S\. T\. Fiske, H\. B\. Bergsieker, A\. M\. Russell, and L\. Williams \(2009\)Images of black americans: then,“them,” and now,“obama\!”\.Du Bois review: social science research on race6\(1\),pp\. 83–101\.Cited by:[§6\.2](https://arxiv.org/html/2604.20131#S6.SS2.p2.1)\. - P\. Glick and S\. T\. Fiske \(2018\)The ambivalent sexism inventory: differentiating hostile and benevolent sexism\.InSocial cognition,pp\. 116–160\.Cited by:[§6\.2](https://arxiv.org/html/2604.20131#S6.SS2.p2.1)\. - S\. Gupta, V\. Shrivastava, A\. Deshpande, A\. Kalyan, P\. Clark, A\. Sabharwal, and T\. Khot \(2024\)Bias runs deep: implicit reasoning biases in persona\-assigned llms\.InProceedings of the International Conference on Learning Representations,Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - S\. Harding \(1991\)Whose science? whose knowledge?: thinking from women’s lives\.Cornell University Press\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - R\. Hertz \(1996\)Introduction: ethics, reflexivity and voice\.\.Qualitative sociology19\(1\)\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - E\. I\. Ibrahim and A\. Voyer \(2026\)Qualitative research with llm chatbots: technological reflexivity for interpretative technology\.Qualitative Research26\(1\),pp\. 133–159\.External Links:[Document](https://dx.doi.org/10.1177/14687941251390794),[Link](https://doi.org/10.1177/14687941251390794),https://doi\.org/10\.1177/14687941251390794Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - D\. Jacobson and N\. Mustafa \(2019\)Social identity map: a reflexivity tool for practicing explicit positionality in critical qualitative research\.International Journal of Qualitative Methods18,pp\. 1609406919870075\.Cited by:[§1](https://arxiv.org/html/2604.20131#S1.p3.1),[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - B\. J\. Jansen, S\. Jung, and J\. Salminen \(2023\)Employing large language models in survey research\.Natural Language Processing Journal4,pp\. 100020\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - R\. Josselson \(2006\)Narrative research and the challenge of accumulating knowledge\.Narrative inquiry16\(1\),pp\. 3–10\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - A\. M\. Koenig and A\. H\. Eagly \(2014\)Evidence for the social role theory of stereotype content: observations of groups’ roles shape stereotypes\.\.Journal of personality and social psychology107\(3\),pp\. 371\.Cited by:[§6\.2](https://arxiv.org/html/2604.20131#S6.SS2.p2.1)\. - A\. Ledgerwood, S\. T\. J\. Hudson, N\. A\. Lewis Jr, K\. B\. Maddox, C\. L\. Pickett, J\. D\. Remedios, S\. Cheryan, A\. B\. Diekman, N\. B\. Dutra, J\. X\. Goh,et al\.\(2022\)The pandemic as a portal: reimagining psychological science as truly open and inclusive\.Perspectives on Psychological Science17\(4\),pp\. 937–959\.Cited by:[§1](https://arxiv.org/html/2604.20131#S1.p3.1),[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - C\. Lin \(2004\)Rouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[§4\.1](https://arxiv.org/html/2604.20131#S4.SS1.SSS0.Px1.p1.1)\. - A\. Mayukha, A\. Guzman, S\. Jitklongsub, and D\. P\. McAdams \(2025\)“I want to lift my people up”: Exploring the psychological correlates of racial themes within the life stories of midlife Black Americans\.Journal of Personality\.Cited by:[Figure 1](https://arxiv.org/html/2604.20131#S1.F1)\. - D\. P\. McAdams \(2008\)The life story interview\.Northwestern University, Evanston, IL\.Cited by:[§2](https://arxiv.org/html/2604.20131#S2.SS0.SSS0.Px1.p1.1)\. - K\. Misgav, G\. Neufeld\-Kroszynski, M\. Palombo, and O\. Karnieli\-Miller \(2025\)Human analysis vs\. artificial intelligence: analyzing of qualitative medical students’ narratives\.Qualitative Health Research,pp\. 10497323251359445\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - S\. Mohammad \(2018\)Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words\.InProceedings of the 56th annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 174–184\.Cited by:[§4\.2](https://arxiv.org/html/2604.20131#S4.SS2.SSS0.Px2.p1.1)\. - K\. Namikoshi, A\. Filipowicz, D\. A\. Shamma, R\. Iliev, C\. L\. Hogan, and N\. Arechiga \(2024\)Using llms to model the beliefs and preferences of targeted populations\.InWorkshop on Behavioral Machine Learning,Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - J\. W\. Pennebaker, R\. L\. Boyd, K\. Jordan, and K\. Blackburn \(2015\)The development and psychometric properties of LIWC2015\.Cited by:[§4\.2](https://arxiv.org/html/2604.20131#S4.SS2.SSS0.Px1.p1.1)\. - A\. Prahl \(2026\)The ai\-reflexivity checklist \(arc\): a pre\-analysis pause for llm\-assisted coding\.Qualitative Health Research36\(2\-3\),pp\. 181–190\.Note:PMID: 41441839External Links:[Document](https://dx.doi.org/10.1177/10497323251401503),[Link](https://doi.org/10.1177/10497323251401503),https://doi\.org/10\.1177/10497323251401503Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - X\. Qin and T\. Tam \(2023\)Stereotype content dictionary: a semantic space of 3 million words and phrases using google news word2vec embeddings\.InInternational Conference on Social Computing, Behavioral\-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation,pp\. 12–22\.Cited by:[§4\.2](https://arxiv.org/html/2604.20131#S4.SS2.SSS0.Px3.p1.1)\. - S\. Riger and R\. Sigurvinsdottir \(2016\)Thematic analysis\.Handbook of methodological approaches to community\-based research: Qualitative, quantitative, and mixed methods,pp\. 33–41\.Cited by:[§2](https://arxiv.org/html/2604.20131#S2.p1.1)\. - S\. Santurkar, E\. Durmus, F\. Ladhak, C\. Lee, P\. Liang, and T\. Hashimoto \(2023\)Whose opinions do language models reflect?\.InInternational Conference on Machine Learning,pp\. 29971–30004\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - N\. Scherrer, C\. Shi, A\. Feder, and D\. Blei \(2024\)Evaluating the moral beliefs encoded in llms\.Advances in Neural Information Processing Systems36\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - J\. Shen, M\. Sap, P\. Colon\-Hernandez, H\. Park, and C\. Breazeal \(2023\)Modeling empathic similarity in personal narratives\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 6237–6252\.Cited by:[§4\.2](https://arxiv.org/html/2604.20131#S4.SS2.SSS0.Px3.p2.2)\. - C\. N\. Steltenpohl, H\. Lustick, M\. S\. Meyer, L\. E\. Lee, S\. M\. Stegenga, L\. S\. Reyes, and R\. L\. Renbarger \(2023\)Rethinking Transparency and Rigor from a Qualitative Open Science Perspective\.Journal of Trial & Error4\(1\)\.Note:https://journal\.trialanderror\.org/pub/rethinking\-transparencyCited by:[§1](https://arxiv.org/html/2604.20131#S1.p3.1),[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - R\. H\. Tai, L\. R\. Bentley, X\. Xia, J\. M\. Sitt, S\. C\. Fankhauser, A\. M\. Chicas\-Mosier, and B\. G\. Monteith \(2024\)An examination of the use of large language models to aid analysis of textual data\.International Journal of Qualitative Methods23\(\),pp\. 16094069241231168\.External Links:[Document](https://dx.doi.org/10.1177/16094069241231168),[Link](https://doi.org/10.1177/16094069241231168),https://doi\.org/10\.1177/16094069241231168Cited by:[§2](https://arxiv.org/html/2604.20131#S2.p2.1)\. - A\. Taubenfeld, Y\. Dover, R\. Reichart, and A\. Goldstein \(2024\)Systematic biases in LLM simulations of debates\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 251–267\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.16/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.16)Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - Y\. Tian, T\. Huang, M\. Liu, D\. Jiang, A\. Spangher, M\. Chen, J\. May, and N\. Peng \(2024\)Are large language models capable of generating human\-level narratives?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 17659–17681\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.978/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.978)Cited by:[§6\.3](https://arxiv.org/html/2604.20131#S6.SS3.p2.1)\. - L\. Tjuatja, V\. Chen, T\. Wu, A\. Talwalkwar, and G\. Neubig \(2024\)Do llms exhibit human\-like response biases? a case study in survey design\.Transactions of the Association for Computational Linguistics12,pp\. 1011–1026\.Cited by:[§3](https://arxiv.org/html/2604.20131#S3.p1.1)\. - Z\. Xiao, X\. Yuan, Q\. V\. Liao, R\. Abdelghani, and P\. Oudeyer \(2023\)Supporting qualitative analysis with large language models: combining codebook with gpt\-3 for deductive coding\.InCompanion proceedings of the 28th international conference on intelligent user interfaces,pp\. 75–78\.Cited by:[§2](https://arxiv.org/html/2604.20131#S2.p2.1)\. - T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi \(2020\)BERTScore: Evaluating Text Generation with BERT\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SkeHuCVFDr)Cited by:[§4\.1](https://arxiv.org/html/2604.20131#S4.SS1.SSS0.Px2.p1.1)\. ## Appendix AInterview Questions To parse interviews automatically into which sections correspond to the answers to these questions, we first identify interviewer utterances by the "INTERVIEWER:" tag\. We then encode these utterances using a TF\-IDF vectorization\. We encode the text of each interview question with the same method, and then for each question, we retrieve the interviewer utterance with the highest cosine similarity match\. If the initial cosine similarity is\>0\.7\>0\.7, we consider this a match, and if the similarity is<0\.3<0\.3, we consider the question skipped in the interview\. Then, using the matched questions as anchors, we search only the text between neighboring anchored questions for the remaining questions, and iteratively lower the cosine similarity threshold by 0\.1 until we reach the floor of 0\.3\. For example, if questions 2 and 4 were already found, then we would search for question 3 only in the utterances between these two questions\. The full code will be released in the accompanying GitHub Repository\. ## Appendix BHuman Study Details Screenshots for the interface for the human study are shown in Figure[10](https://arxiv.org/html/2604.20131#A2.F10)\. Figure 10:Screenshots of the instructions and layout in the human study interface\. ## Appendix CSummarization Prompts The system and user prompts are shown in the following blocks\. Lines enclosed in square brackets are only added in the condition specified by the parenthetical at the beginning of the line\. System prompt for summarization\\KV@dobreaklines=true,breaklines=true,breaksymbolleft=,breaksymbolright=,tabsize=0,,You are an expert at summarizing interviews\. User prompt for summarization\\KV@dobreaklines=true,breaklines=true,breaksymbolleft=,breaksymbolright=,tabsize=0,,Interview transcript excerpt: \{life\_chapters\_interview\}\[\(Added only in demographic\-conditioned summaries\) The interviewee is a \{race\} \{gender\}\.\]Task:1\. Summarize the interview in 5–7 sentences, focusing on:"How does this person find meaning in life?"2\. Then provide the following section:\- Core ValuesOutput Format:Summary:\.\.\.Core Values:\- \.\.\.\[\(Added only for Qwen models\) Encapsulate your response in <response\></response\> tags\.\] ## Appendix DAdditional Results Figure 11:We show results for each LLM in each column for the analysis of VAD and SCM scores in the summary as compared to the interview transcript\. We average an individual’s scores across the five random seeds and then report the mean across a demographic group with an 83% confidence interval\.Figure 12:We show the percent of baseline summaries for a given demographic group identifying a theme\. We average a group’s scores across the five random seeds and then report the mean with an 83% confidence interval\.Figure 13:The full LIWC results\. We average an individual’s scores across the five random seeds and then report the mean across a demographic group with an 83% confidence interval\.Figure 14:The full theme identification results for the top 30 themes\. We average an individual’s scores across the five random seeds and then report the mean across a demographic group with an 83% confidence interval\.
Similar Articles
Narrative Landscape: Mapping Narrative Dispositions Across LLMs
This paper introduces a quantitative framework and visualization tool called 'Narrative Landscape' to map and compare the narrative dispositions and stability of frontier LLMs.
Can LLMs Hire Fairly? Racial Bias in Resume Screening
This paper audits 14 large language models for hiring discrimination using a paired-resume methodology, finding that older models exhibit pro-White bias while newer models show null or pro-Black bias, indicating a reversal in algorithmic hiring bias across model generations.
Best local LLM for English story summarization
A guide comparing the best local LLMs for English story summarization, offering recommendations based on performance and accessibility.
Investigating Counterfactual Unfairness in LLMs towards Identities through Humor
Academic study exposes systemic counterfactual unfairness in LLMs: jokes from privileged speakers are refused 67% more often and rated as more malicious than identical jokes from marginalized speakers.
The Storyteller in the Model: Narrative Pattern Inheritance, Escalation Dynamics, and Alignment Governance in LLMs
This paper investigates how narrative patterns from training data influence LLM behavior, leading to narrative drift, sycophancy, and deceptiveness over extended interactions, posing governance risks in deployed systems.