How Humans and LLMs Read Gender into Gender-Neutral Physical Descriptions

# How Humans and LLMs Read Gender into “Gender-Neutral” Physical Descriptions
Source: [https://arxiv.org/html/2609.16366](https://arxiv.org/html/2609.16366)
Lin L\. Lin††thanks:Equal contribution\.Affiliation:Department of CommunicationEmail:[llin@ucla\.edu](mailto:)Elisa KreissAffiliation:Department of CommunicationAffiliation:Department of Computer ScienceUniversity of California, Los Angeles \(UCLA\)Email:[ekreiss@ucla\.edu](mailto:)

###### Abstract

When foundation models describe people, recent work in AI fairness, accessibility, and ethics recommends avoiding inferred identity labels \(e\.g\., “she”, “his”\) in favor of seemingly “objective” physical descriptions \(e\.g\., “short hair”, “a defined jawline”\)\. Yet whether such descriptive language achieves gender\-neutral communication remains an open empirical question\. To study this, we introduceGAPA\(Gender Associations of Physical Attributes\), a dataset of 316 common physical attributes drawn from diverse sources and domains, paired with 14,706 gender\-association ratings from 304 US\-based annotators\. Results of human ratings show that physical descriptions carry structured and graded gender associations among readers: 53% of attributes are significantly more strongly associated with one gender than the others, with more consistent and distinctive associations for women and men than for non\-binary identities\. Next, we evaluate 16 LLMs across model families, sizes, and post\-training variants against human ratings\. The models partially recover human associations but exhibit systematic alignment biases, including compressed rating distributions, weaker alignment for associations with men, and asymmetric abstention that disproportionately targets the non\-binary category\. This abstention pattern is especially pronounced among instruction\-tuned and proprietary models\. Finally, we release the best\-performing proxy model trained to predict humans’ gender associations of descriptive language at scale and demonstrate its utility through a sociolinguistic analysis of character descriptions in LitBank\. Together, our findings provide the first empirical evidence that seemingly ‘‘objective’’ physical descriptions can retain systematic gender associations in human interpretation, and uncover systematic patterns of model\-\-human misalignment\. This challenges the assumption that replacing explicit gender labels with physical descriptions necessarily yields gender\-neutral communication, and highlights downstream challenges in using such descriptions to communicate subjective identity categories in human\-\-AI interaction\.111The dataset and code are available at[https://github\.com/Yingjia\-Wan/GAPA](https://github.com/Yingjia-Wan/GAPA), and the predictor model released at[https://huggingface\.co/alisa\-yingjia\-wan/gapa\-predictor\-olmo2\-7b](https://huggingface.co/alisa-yingjia-wan/gapa-predictor-olmo2-7b)\.

## 1Introduction

How should models describe a person? This question has become increasingly consequential as foundation models are deployed at scale in image\-captioning systems, accessibility tools, content moderation pipelines, and assistive technologies\. A range of research communities \(e\.g\., fairness in machine learning, accessibility, critical AI studies, and human\-computer interaction\) have converged on a partial answer: when describing a person whose identity cannot be confirmed, models should avoid inferring categorical identity labels \(race, gender, disability\) and instead describe observable physical attributes\([Bennett et al\., 2021](https://arxiv.org/html/2609.16366#bib.bib14);[Scheuerman et al\., 2019](https://arxiv.org/html/2609.16366#bib.bib5);[Scheuerman et al\., 2020](https://arxiv.org/html/2609.16366#bib.bib12);[Hamidi et al\., 2018](https://arxiv.org/html/2609.16366#bib.bib13)\)\. Such ideology is influential in shaping both human image\-description practice and model behavior, and recent foundation models increasingly default to descriptive rather than identity\-inferring language when describing people\([Marin\-Llobet et al\., 2026](https://arxiv.org/html/2609.16366#bib.bib4);[Hanley et al\., 2021](https://arxiv.org/html/2609.16366#bib.bib3)\)\. The reasoning behind this looks epistemic and ethical: describing what is visually present, rather than what is inferred about the person, transfers the inference back to the reader and supposedly avoids imposing identity claims the system cannot verify\.

However, such reasoning rests on an assumption that has not been empirically tested: that descriptive language is meaningfully gender\-neutral\. The twist here is: if "a chiseled jawline" or "a defined waist" carries gender associations as systematic as those carried by "a man" or "a woman," then the strategy may shift gendered communication into a less visible register rather than reducing it\. This raises some critical, underexplored questions: do isolated physical descriptors, stripped of visual context, carry gender associations among human readers? If so, how strong, how consistent across people, and how distributed across types of attribute? Furthermore, do language models reproduce these associations, amplify them, or structurally distort them?

Figure 1:An excerpt of the most gender\-distinctive attributes with significant gender differences \(p<0\.05p<0\.05\) in each ranking pattern, with their per\-gender association ratings on the y\-axis\. See[Figure 9](https://arxiv.org/html/2609.16366#A2.F9)in the Appendix for the complete rating plots of all attributes with significant gender differences in the six ranking patterns\.It remains a significant blind spot to what extent linguistic descriptions of physical characteristics themselves are gendered, with little evidence in prior work across disciplines\. Sociolinguistic and psychological research has long established that humans infer gender from visual and acoustic cues within hundreds of milliseconds\([Besson et al\., 2017](https://arxiv.org/html/2609.16366#bib.bib16);[Freeman and Johnson, 2016](https://arxiv.org/html/2609.16366#bib.bib15);[Brown and Perrett, 1993](https://arxiv.org/html/2609.16366#bib.bib17);[Freeman et al\., 2012](https://arxiv.org/html/2609.16366#bib.bib6)\)and that language carries graded, context\-sensitive gendered meanings\([Eckert and McConnell\-Ginet, 2013](https://arxiv.org/html/2609.16366#bib.bib31)\)\. In NLP, extensive work on gender bias has developed diagnostic benchmarks\([Zhao et al\., 2018](https://arxiv.org/html/2609.16366#bib.bib9);[Bolukbasi et al\., 2016](https://arxiv.org/html/2609.16366#bib.bib8);[Caliskan et al\., 2017](https://arxiv.org/html/2609.16366#bib.bib7)\)and mitigation methods\([Dong et al\., 2024](https://arxiv.org/html/2609.16366#bib.bib10);[Zhao et al\., 2019](https://arxiv.org/html/2609.16366#bib.bib11)\)\. These benchmarks typically evaluate whether foundational models associate gender with occupations, traits, pronouns, etc\. However, they do not examine whether the descriptive vocabulary itself carries systematic gender associations\. This gap is especially consequential because current practices that replace explicit identity labels with ostensibly neutral physical descriptions implicitly assume that such descriptions do not themselves transmit gendered information\.

To address this gap, this study challenges the assumption that physical attribute descriptions function as semantically gender\-neutral fallbacks, by presenting the first series of evidence from human and LLM experiments\. We make the following contributions:

1. 1\.Empirical evidence that physical attributes carry systematic gender associations\.We introduce GAPA \(Gender Associations of Physical Attributes\), ahuman\-annotated datasetof 316 physical attribute descriptions sourced from three complementary channels \(LLM generation, human elicitation, and contemporary fiction\) annotated with 14,706 ratings from 304 U\.S\.\-based participants on perceived associations with women, men, and non\-binary gender categories \(see[Figure 1](https://arxiv.org/html/2609.16366#S1.F1)\)\. While much prior work has focused on quantifying visual gender associations, GAPA allows us to do this in the linguistic modality, providing a crucial foundation for empirical studies on “gender\-neutral” descriptions of people’s appearances\. Section[3](https://arxiv.org/html/2609.16366#S3)introduces the construction process and statistics of the dataset, while Section[4](https://arxiv.org/html/2609.16366#S4)reports the human experiments from which we collect the human annotations of how physical attributes fromGAPAare associated with each gender category\.
2. 2\.Systematic Evaluation of LLM Gender Representations\.After establishing that humans carry strong gender associations with physical attributes, it becomes imperative to understand to what extent models faithfully encode or distort them in meaningful ways, as this alignment marks the basis for human–AI interaction downstream\. We evaluate 16 state\-of\-the\-art LLMs on GAPA against human ratings, and find that models only partially recover the human pattern\. This highlights that LLMs, too, carry gender associations with linguistic descriptors of physical attributes, yet exhibit systematic distortions on multiple levels\. See Section[5](https://arxiv.org/html/2609.16366#S5)for detailed results and discussions\.
3. 3\.Studying Gender Associations at Scale\.Our findings also provide a new opportunity for studying linguistic gender representations at scale\. To that end, we develop and release a proxy predictor of human gender associations for automatically quantifying how humans associate gender with descriptive language\. Our selected predictor, fine\-tuned onGAPA, achieves r=0\.76 against held\-out human ratings after extensive hyperparameter search and model ablations\. We demonstrate its utility via a sociolinguistic analysis of character descriptions in LitBank\([Bamman et al\., 2019](https://arxiv.org/html/2609.16366#bib.bib39)\)\(Section[6](https://arxiv.org/html/2609.16366#S6)\)\.

## 2Related Work

Linguistic and social science research conceptualizes gender not as a fixed category but as a socially constructed, context\-dependent phenomenon\([West and Zimmerman, 1987](https://arxiv.org/html/2609.16366#bib.bib1);[Eckert and McConnell\-Ginet, 2013](https://arxiv.org/html/2609.16366#bib.bib31)\)\. From this perspective, gender is indexed through speech patterns and styles, making it a graded dimension rather than a binary one\([Bucholtz and Hall, 2005](https://arxiv.org/html/2609.16366#bib.bib2)\)\. Empirical work demonstrates that humans infer gender from a wide array of cues—including voice, facial morphology, and motion\([Johnson and Tassinary, 2005](https://arxiv.org/html/2609.16366#bib.bib32)\)—and that these associations are systematic and context\-sensitive\.

In NLP, extensive work has examined the explicit and implicit gender biases that language models encode, producing influential diagnostic benchmarks like WinoBias and developing mitigation strategies such as data balancing and fine\-tuning\([Bolukbasi et al\., 2016](https://arxiv.org/html/2609.16366#bib.bib8);[Zhao et al\., 2018](https://arxiv.org/html/2609.16366#bib.bib9);[Dong et al\., 2024](https://arxiv.org/html/2609.16366#bib.bib10)\)\. More recent research challenges the predominantly binary treatment of gender, revealing that models struggle with gender\-neutral pronouns, show lower performance on non\-binary categories, and are inconsistent in generating gender\-neutral language\([Hossain et al\., 2023](https://arxiv.org/html/2609.16366#bib.bib42);[You et al\., 2024](https://arxiv.org/html/2609.16366#bib.bib41);[Savoldi et al\., 2025](https://arxiv.org/html/2609.16366#bib.bib40)\)\.

However, this body of work has almost exclusively focused on biases associated with labels \(e\.g\., occupations, pronouns\) rather than on the descriptive vocabulary itself \(but see[Gao and Kreiss \(2025\)](https://arxiv.org/html/2609.16366#bib.bib22)for a first exploration of physical attributes as stimuli\)\. It remains underexplored whether seemingly objective physical descriptions, increasingly recommended as a neutral alternative to identity labels\([Bennett et al\., 2021](https://arxiv.org/html/2609.16366#bib.bib14)\), carry systematic gender associations\. Our work addresses this gap by investigating the gendered nature of physical description language and how it is represented by both humans and language models\.

## 3GAPA: A Collection of Physical Attributes

Table 1:Summary statistics ofGAPA\.In order to systematically investigate how humans and language models associate gender with text descriptions of physical attributes, we builtGAPA, the largest existing collection of linguistic descriptions of physical attributes\. Building a lexicon of physical attributes serves two purposes: \(1\) to provide a structured resource for analyzing variation in appearance\-based language across sources, and \(2\) to enable evaluation of whether physical attributes are systematically perceived as gendered\.

To facilitate diversity of physical attributes, we collected them from distinctive sources based on LLM\-generated attributes \(providing broad categorical coverage; n=170\), human\-written attributes \(ensuring high\-frequency terms; n=50\), and attributes extracted from six novels \(covering more unusual and stylized references; n=96\), resulting in a comprehensive collection of 316 physical attributes\. We provide all procedural specifications on the selection process in[Appendix A](https://arxiv.org/html/2609.16366#A1)\.

## 4Physical Attributes Carry Systematic Gender Associations for Humans

After compiling the physical attribute lexicon, we conducted a human\-subject experiment to measure how strongly participants associated each attribute with different gender categories:woman,man, andnon\-binary person222We includewoman,man, andnon\-binary personbecause they reflect commonly recognized gender categories in contemporary U\.S\. English\. These categories are not intended to represent the full range of gender identities, nor to imply that gender is naturally discrete\. Instead, they serve as reference points for examining how physical attributes are associated with gender in everyday languagebeyond a simple woman–man contrast\.\. Specifically, we investigate: \(1\) whether there is an overall systematic difference in gender association ratings inGAPA, averaged across attributes and participants; \(2\) whether there exists a consistent shared association of gender for each attribute, alongside heterogeneity in how gender differences manifest at the attribute level in terms of direction and magnitude\.

### 4\.1Experiment Details

The annotated dataset ofGAPAincludes 304 participants \(152 female; ages 18–77,M=41\.36M=41\.36\) who passed at least four of five attention checks\. Analyses were conducted controlling for participant demographics \(e\.g\., gender, age\) and attribute source\. Full details of human participant demographics and the recruitment process are provided in Appendix[B\.1](https://arxiv.org/html/2609.16366#A2.SS1)\.

On each trial, participants used a 7\-point Likert scale to answer:“How likely is it for someone to say that a woman/man/non\-binary person has a given physical attribute?”Asking whatsomeonewould say captures culturally available associations rather than participants’ explicit endorsement of those associations, as individuals may possess stereotype knowledge that is different from personal belief\([Devine, 1989](https://arxiv.org/html/2609.16366#bib.bib24);[Kunda and Spencer, 2003](https://arxiv.org/html/2609.16366#bib.bib25)\)\.

Using a cross\-classified design, participants rated randomly sampled attributes\. To minimize carryover effects and direct cross\-gender comparisons, each participant rated a given attribute for only one target gender\. Counterbalancing ensured an even distribution of target gender and trial order\. Each participant completed 55 trials, yielding an average of 17 ratings per attribute, per gender category\.

### 4\.2Metrics

We define two complementary metrics to quantify the gender association of physical attributes\.Gender Associationis the mean rating for each attribute–gender pair, capturing how strongly participants associate a physical attribute with a given gender category\.Gender Distinctivenessis the difference between the highest and second\-highest mean ratings across the three gender categories for a given attribute, capturing how clearly that attribute differentiates one category from the others\.

Together, these measures distinguish attributes that are strongly associated with a gender category from those that are also clearly differentiated across categories\. This approach is consistent with prior work showing that category membership and social judgments are represented in graded rather than binary terms\([Rosch, 1975](https://arxiv.org/html/2609.16366#bib.bib26);[Rosch and Mervis, 1975](https://arxiv.org/html/2609.16366#bib.bib27);[Barsalou, 1985](https://arxiv.org/html/2609.16366#bib.bib28)\), and that person perception depends not only on category activation but also on how diagnostic cues are for distinguishing among categories\([Freeman and Ambady, 2011](https://arxiv.org/html/2609.16366#bib.bib29);[Oosterhof and Todorov, 2008](https://arxiv.org/html/2609.16366#bib.bib30)\)\.

### 4\.3Results

#### Dataset\-Level Gender Effects

Across the dataset, attributes were associated more with women than men \(β=0\.26\\beta=0\.26,S​E=0\.04SE=0\.04,z=6\.77z=6\.77,p<\.001p<\.001\), and least with non\-binary people \(β=0\.43\\beta=0\.43,S​E=0\.04SE=0\.04,z=11\.68z=11\.68,p<\.001p<\.001\), providing first evidence that physical attributes show structured gender profiles and are not perceived as generally gender\-neutral\. There are two potential explanations for the significant distinctions between genders: the dataset may overrepresent attributes that are culturally coded as feminine, or the pattern may reflect broader biases in language use, where physical descriptors are unevenly distributed across gender categories\. Ultimately, the data clearly reveals that people have nuanced gender associations with physical attributes that we now aim to unpack\.

#### Attribute\-Level Gender Effects

Figure 2:Attribute counts for gender\-ranking patterns and their significance percentages \(p<0\.05p<0\.05\)\.Since different attributes may be associated with gender categories to varying extent, we also conduct a significance test for each attribute inGAPA, to examine heterogeneity in gender effects across attributes\.

Over half of the attributes \(53%\) are significantly more associated with one particular gender than other genders\. Importantly, the direction and strength of attribute\-level gender differences vary systematically across ranking patterns \([Figure 2](https://arxiv.org/html/2609.16366#S4.F2)\)\. Attributes that align with conventional binary gender distinctions \(e\.g\.,woman\>\>non\-binary\>\>manandman\>\>non\-binary\>\>woman\) are both more frequent and substantially more likely to yield significant effects \(68%–73%\)\. In contrast, attributes where non\-binary individuals are rated highest are the rarest and exhibit much lower rates of significance\.[Figure 1](https://arxiv.org/html/2609.16366#S1.F1)provides the qualitative case studies for the attributes in each ranking pattern of gender ratings\.

Overall, theattribute\-level heterogeneitysuggests that gender associations carried in physical attributes are structured around a binary axis, with non\-binary categories occupying a less strongly encoded region of the semantic space\. In this sense, ranking patterns can be interpreted as approximations of culturally shared notions of “femininity” and “masculinity,” where attributes strongly aligned with these poles produce clearer and more detectable statistical effects\. This intuition is supported by a significant negative correlation of man vs\. woman attribute associations \(r=−\.16r=\-\.16,p=\.004p=\.004\)\.

In contrast, ratings for non\-binary gender associations with both women and men were significantly positively correlated \(see[Figure 3](https://arxiv.org/html/2609.16366#S4.F3)\), highlighting how even when attributes are judged as common for a gender, it doesn’t mean they are gender\-diagnostic\. Interestingly, splitting up the man–woman correlations by how we sourced attributes shows significant variation\. While the correlation is highly negative for the human\-sourced examples \(r=−0\.61r=\-0\.61\), this reverses for novel\-extracted physical attributes \(r=0\.27r=0\.27\)\. This variation highlights the importance of employing diverse sampling techniques and is the reason why replications across source domains are especially challenging but meaningful\.

Figure 3:Correlations among mean gender\-association ratings across physical attributes\. Each point represents an attribute \(p<0\.05p<0\.05\)\. Man–Woman correlations are further split by source domain of the physical attributes, showing significant variation across attributes\.Together, these correlations indicate that gender associations in physical description are structured rather than random\. This pattern suggests that physical attributes are not simply distributed along a single binary opposition, but instead occupy partially shared and partially distinct regions of gendered meaning\. In particular, non\-binary associations appear to overlap more with attributes linked to women than with those strongly opposed to them\.

![Refer to caption](https://arxiv.org/html/2609.16366v1/figs/noise_ceiling_summary.png)Figure 4:Human rater reliability inGAPAacross three attribute sources, measured byrLOOr^\{\\text\{LOO\}\}andICC⁡\(1,k\)\\sqrt\{\\mathrm\{ICC\}\(1,k\)\}\. The scatterplot showsrLOOr^\{\\text\{LOO\}\}per rater\.
#### Human Uncertainty

Interpreting the gender association of physical attributes is inherently subjective and diverse among participants\. To characterize this variability, we quantify*human uncertainty*using two complementary reliability statistics: a leave\-one\-rater\-out \(LOO\) correlation and the intraclass correlation coefficient \(ICC\)\. The formal calculation and scientific interpretation of the two measures for human uncertainty and LLM correlation baselines are elaborated on in Appendix[B\.3](https://arxiv.org/html/2609.16366#A2.SS3)\.

In short, LOO measures how well a typical individual human rater agrees with the consensus by correlating each rater’s ratings with the mean ratings of all remaining raters across items, and then averaging across raters\. This reflects the correlation performance of a*typical*human annotator relative to the group\. In contrast, ICC\(1,k\)\(1,k\)quantifies the reliability of the*aggregated*human ratings by decomposing variance into signal \(between\-item variability\) and noise \(within\-item, across\-rater variability\), estimating how much noise remains after averaging across raters\.

[Figure 4](https://arxiv.org/html/2609.16366#S4.F4)shows the noise level of human ratings across attribute sources\. Human rater reliability \(measured by LOO and ICC\) is consistently the highest for women, intermediate for men, and lowest for non\-binary targets, indicating that ratings for non\-binary attributes are less consistent across raters\.

## 5Physical Attributes Carry Systematic Gender Associations for LLMs

After establishing that humans have reliable gender associations with text descriptions of physical attributes, we now turn to compare these to LLM gender associations\. We conduct a zero\-shot evaluation similar to the human experiment, across a diverse set of 16 state\-of\-the\-art language models, varying across open\-weight vs\. proprietary and instruct vs\. base versions\.

### 5\.1Method

For each physical attribute in our test set, we prompt the LLMs the same question as in the human subject experiment, with minor modifications to instruct the model to directly respond with a single integer rating on a 1–7 Likert scale \(see full prompt in Appendix[D\.1](https://arxiv.org/html/2609.16366#A4.SS1)\)\.

Two complementary strategies to elicit ratings were explored: \(1\) direct generation: we directly extract the verbalized ratings in the model\-generated response; \(2\) top\-logit token extraction: we examine the model’s output distribution at the first generated token position\. Two methods showed consistency in results, as illustrated in Appendix[D\.2](https://arxiv.org/html/2609.16366#A4.SS2)in detail\. We report the results from method \(1\)\.

### 5\.2Evaluation Metric & Baselines

To assess how well a model’s predictions align with the averaged human ratings across attributes, we adopt the Pearson \(r\) correlation coefficient betweenLLM\-predicted ratingsand theaveraged human ratingsacross items as the primary evaluation metric, and root mean squared error \(RMSE\) as the auxiliary metric\. Higher Pearson correlation and lower RMSE suggest higher alignment with human interpretations of gender associations\.

For a clearer interpretation of LLMs’ correlation performance in consideration of inherent subjectivity in the human ground truth ratings, we also adopted two human performance baselines by using the uncertainty measures from[subsection 4\.3](https://arxiv.org/html/2609.16366#S4.SS3.SSS0.Px3): \(i\) the leave\-one\-rater\-out \(LOO\) correlation corresponds to the agreement level of a typical human rater, while \(ii\) the intraclass correlation coefficientICC⁡\(1,k\)\\sqrt\{\\mathrm\{ICC\}\(1,k\)\}approximates the maximal attainable correlation with the aggregated human judgments \(i\.e\., an ideal model that perfectly recovers the latent true rating from the human population, so that the error only comes from measurement noise\)\. The two human baselines suggest the correlation ceilings for the model evaluation due to rater uncertainty and measurement noise\.

### 5\.3Results

#### Overview of Model Alignment

![Refer to caption](https://arxiv.org/html/2609.16366v1/rating_distributions.png)Figure 5:Left: the frequency% distribution of predicted rating scores \(1\-7\) per gender category of all LLMs, compared to human ratings\.Right: Pearson r \(correlation between LLM and averaged human ratings\) distribution of all models per gender category; each point represents an evaluated LLM\.We first examine how well state\-of\-the\-art \(SOTA\) language models align with human judgments about how gender is associated with physical attributes inGAPAat the collective level\.[Figure 5](https://arxiv.org/html/2609.16366#S5.F5)\(right\) presents model–human correlations alongside the two reliability\-based reference baselines\. Overall, SOTA language models show moderate alignment with human gender associations across attributes\. However, alignment varies systematically across gender categories\. In particular, models tend to achieve higher correlations forwoman and non\-binary personthan forman, centering around or exceeding the LOO \(typical\-rater\) baseline\. Interpreting these results relative to the reliability ceilings helps account for measurement noise, suggesting that the observed differences reflect genuine gender variation in model–human alignment in SOTA LLMs: SOTA LLMs are better aligned with humans in interpreting the semantic association of physical attributes with woman and non\-binary person than the male category\.

Notably, model rating distributions are more concentrated in the mid\-range \(3–5\) and less frequent at the extremes \(6–7\) compared to human ratings, which are more evenly distributed across the scale \([Figure 5](https://arxiv.org/html/2609.16366#S5.F5), left\)\. This pattern suggests that models tend to avoid extreme judgments, producing more conservative or less differentiated ratings than human annotators, and thus only partially capture the strength of gender associations present in human interpretations\.

![Refer to caption](https://arxiv.org/html/2609.16366v1/figs/gender_pearson_r_horizontal.png)\(a\)Correlation results of all evaluated models onGAPA\.means the overall average\.
![Refer to caption](https://arxiv.org/html/2609.16366v1/figs/base_instruct_slope_gender_rmse.png)\(b\)The comparison of model\-human alignment \(RMSE\) between base and instruct models\.
![Refer to caption](https://arxiv.org/html/2609.16366v1/figs/abstention_by_model_gender_all.png)\(c\)Model heatmap of abstention responses onGAPA\. Instruct models show significant abstention rates among models; abstention is significantly more likely to occur for the non\-binary gender category\.

Figure 6:Zero\-shot inference experiment results of the state\-of\-the\-art LLMs\.
#### Model Comparisons

[6\(a\)](https://arxiv.org/html/2609.16366#S5.F6.sf1)demonstrates the results of model\-human alignment in gender associations of physical attributes evaluated on the complete set of 16 models\. Consistent with[Figure 5](https://arxiv.org/html/2609.16366#S5.F5)\(right\), gender differences remain systematically salient within each model’s performance\. Two other results stand out\. First, Claude\-Opus\-4\.6 achieves the strongest alignment with human gender associations, with an average Pearson correlation of r = 0\.669\. In contrast, GPT\-OSS\-20B has surprisingly the weakest overall alignment \(r = 0\.173\) and even shows a negative correlation for the non\-binary category \(r = \-0\.043\)\. Second, compared with prior gender\-bias benchmarking work\([Yang et al\., 2025](https://arxiv.org/html/2609.16366#bib.bib36);[Liang et al\., 2022](https://arxiv.org/html/2609.16366#bib.bib37);[Felkner et al\., 2023](https://arxiv.org/html/2609.16366#bib.bib38)\), proprietary models doNOTshow a clearly dominant separation from open\-source models onGAPA\. At a collective level, proprietary models show only a modest advantage, with substantial overlap in practical performance rather than a distinct lead\.

We also directly compare base and instruct variants of the same model backbone, as illustrated in[6\(b\)](https://arxiv.org/html/2609.16366#S5.F6.sf2)\. Instruction\-tuned variants of open\-source models generally improve over their base counterparts in RMSE\. This pattern suggests that post\-training alignment reduces absolute prediction divergence from the human ground\-truth\. However, no comparable gains are consistently observed in Pearsonrr, indicating that it does not substantially improve the rank\-order alignment of attribute–gender associations\.

#### Abstention Analysis

We also investigate to what extent LLMs abstain from answering the question\. We count a model response to be an abstention if it specifically expresses an abstention, such as “physical attributes are not related to gender identity”, and “non\-binary people can have any eye shape, just like anyone else\.” In such abstention cases, models either choose the lowest 1 \(not likely at all\) as the rating to express refusal and disagreement on the question assumption, or simply end the response without a rating\.

We conduct a more systematic quantitative analysis on abstention counts by analyzing the model responses, which reveal two salient patterns \([6\(c\)](https://arxiv.org/html/2609.16366#S5.F6.sf3)\)\. Firstly, we observed highly asymmetric abstention in the model responses against the non\-binary gender category\. This indicates that the burden of refusal or deflection is not uniform across gender groups, but is disproportionately concentrated on the non\-binary category\. This finding crucially warns of practical accessibility bias caused by over\-triggering of sensitivity and safety flags, and the behavior of AI refusal disproportionately constraining more vulnerable groups\([Luo et al\., 2024](https://arxiv.org/html/2609.16366#bib.bib33);[Abramovich and Ma, 2024](https://arxiv.org/html/2609.16366#bib.bib34)\)\. Secondly, abstention is not a broadly shared behavior across all systems\. Rather, it is concentrated almost entirely in the proprietary model Gemini\-3\-flash and instruct\-tuned open\-source models, while the corresponding base models show little to no comparable tendency\. This distribution suggests that abstention is not simply a property of model scale or architecture, but is closely associated with post\-training and alignment procedures\. Taken together, these patterns are consistent with the interpretation that post\-training can introduce asymmetric over\-sensitivity, causing some models to over\-trigger safety or sensitivity heuristics specifically in response to non\-binary\-targeted prompts\.

## 6Scalable Socio\-Linguistic Analyses Using a Trained Proxy Predictor

### 6\.1Training a Proxy Model for Predicting Humans’ Gender Associations

To extend the contribution of this study beyond human experiments and model evaluation analyses, we further developed a proxy predictor model on the fullGAPA’s data by selecting and training from the open\-source models evaluated in[section 5](https://arxiv.org/html/2609.16366#S5)\.

Figure 7:Training results of the predictor model, evaluated on GAPA \(test\)\.To seek the best model candidate for training the proxy predictor, we first conduct supervised fine\-tuning across an extensive search grid of LLMs and hyperparameters onGAPA\(60/15/25 split for train/validation/test\)\. To match the prediction goal, each model is augmented with a linear regression head to directly map aggregated human ratings for gender categoriesc∈\{woman,man,non\-binary person\}c\\in\\\{\\textit\{woman\},\\textit\{man\},\\textit\{non\-binary person\}\\\}\. Given an input sentence describing an attribute for a target gender \(e\.g\., “a woman has broad shoulders”\), the mean\-pooled final\-layer representation is projected to a scalar score, framing the task as continuous regression to directly assess model–human alignment\. The training optimizes mean squared error \(MSE\) on averaged human ratings, encouraging recovery of shared annotator signals while reducing individual noise\.

For a fair comparison, we use a standardized hyperparameter search with model\-specific ranges \(e\.g\., learning rate, batch size\), and select the best\-performing configuration for each model\. All models share the same tuning budget and are evaluated under consistent cross\-validation and multi\-seed settings\. Full details including design choices and hyperparameter specifications are provided in[Appendix E](https://arxiv.org/html/2609.16366#A5)\.

The final proxy model takes the backbone ofolmo2\_7b\_base, and achieves the highest test\-set correlation ofr=0\.764r=0\.764and lowest RMSE=0\.633=0\.633\. This predictor can serve as a simulator for a scalable collection of human\-like gender\-association interpretations on unseen items, supporting larger\-scale sociolinguistic analysis\.

### 6\.2Scaled\-Up Analyses on LitBank\([Bamman et al\., 2019](https://arxiv.org/html/2609.16366#bib.bib39)\)

Figure 8:Predicted gender association ratings of physical attributes describing LitBank characters, by character gender and author gender\.As a preliminary demonstration of what the predictor model can do at scale, we utilize the extraction pipeline previously used in[section 3](https://arxiv.org/html/2609.16366#S3)to extract physical attributes from LitBank\([Bamman et al\., 2019](https://arxiv.org/html/2609.16366#bib.bib39);[Sims et al\., 2019](https://arxiv.org/html/2609.16366#bib.bib35)\), a literature corpus of 100 novels from Project Gutenberg, which consists of fiction \(both novels and short stories\) in a mix of genres and literary styles before 1923\. After extracting the attributes, we employed the proxy model to estimate the gender association ratings of each attribute with men, women, and non\-binary people, yielding a three\-dimensional gender\-association profile for every attribute\. This enabled us to conduct exploratory analyses of their relationships with two types of metadata: author gender and the gender of the character each physical attribute describes in the original novel\. We primarily explore the hypothesis333The analyses reflect findings of contemporary reader perceptions in the US due to the participant demographics ofGAPA\. See[Appendix F](https://arxiv.org/html/2609.16366#A6)for a more detailed interpretation\.: the physical descriptions used to describe a literary character are systematically gendered on their own; their gender associations can be predicted by who the writer is \(author gender\), and about whom the description is written \(character gender\)\.

#### Character Gender\.

As shown in[Figure 8](https://arxiv.org/html/2609.16366#S6.F8), physical descriptions differentiate male and female characters through the direction rather than the magnitude of their gender associations\. Attributes describing male characters are rated as more associated with man than with woman, and those describing female characters show the reverse ordering, but the two groups are comparably distinctive in absolute terms\. vs\.0\.710\.71,p=\.20p=\.20\)\.

#### Author Gender\.

Author gender modulates this structure along only one of the two binary dimensions\. Ratings of association with man are largely invariant to the author gender, and female characters elicit similar descriptions from both groups of authors\. Male characters do not: male authors describe men using attributes with markedly weaker feminine associations than female authors do \(4\.064\.06vs\.4\.224\.22;β=−0\.12\\beta=\-0\.12,p=\.03p=\.03\)\. Authorial variation is thus confined to the degree of feminine association admitted into the description of a man, while masculine association follows a convention shared across author groups\. The results carry the central claim of this work from controlled annotation into naturalistic text at scale: descriptions that seem independent of gender assignment remain systematically gendered by whom they describe and by who describes them\.

## 7Conclusion

This study challenges a widely held assumption in AI fairness, accessibility, and ethics research: thatreplacing categorical identity labels with "objective" physical descriptions yields gender\-neutral communication\. Introducing GAPA, a large\-scale human dataset of gender associations for 316 physical attributes, we show thatthis assumption does not hold: physical descriptions carry automatic, structured, and graded gender associations among human readers, with over half of attributes significantly more associated with one gender than others\. Evaluating 16 LLMs against these human ratings, we find that models partially recover human patterns yet with systematic misalignment: model ratings are compressed toward the midpoint, they align less well with human judgments of the man category, and instruction\-tuned and proprietary models show asymmetric abstention that disproportionately targets the non\-binary category, raising distinct equity concerns\. Finally, we developed a proxy predictor to analyze gendered descriptions beyond controlled annotation settings, but at scale in naturalistic text\. Together, these findings provide the first empirical evidence that "objective" physical descriptions do not neutralize gender, but instead shift gendered communication into a less visible register with downstream consequences for how models should be evaluated and designed for human–AI interaction\.

## Limitations

Several limitations qualify the human norming results\. First, the task measures socially shared associations between physical attributes and gender categories, rather than whether attributes are inherently gendered\. Ratings may therefore reflect stereotype knowledge, perceived prevalence, and expectations about language use in addition to semantic association\.

Second, the attribute set is heterogeneous, spanning bodily morphology, grooming, accessories, age related cues, and figurative descriptions that may signal gender through different mechanisms\. Attributes were also evaluated in isolation, without the discourse context that can shape gendered interpretation\. Finally,non\-binary personmay have less culturally shared appearance\-based prototypes thanwomanorman, so lower agreement may reflect greater conceptual heterogeneity\. The findings should therefore be interpreted as evidence of structured*socially shared gender associations in physical description*, rather than a complete account of gendered meaning in language\.

## Acknowledgments

This work was supported in part by the UCLA Graduate Summer Research Mentorship \(GSRM\) Program\. We also thank the members of the UCLA Coalas Lab for their valuable comments throughout this project\. Special thanks go to Hillary Nguyen, Ella Han, Troy Tian, Jonathan Pak, and Ryan Zheng for their assistance with manually verifying the novel\-extracted attributes inGAPAto ensure the quality of automated attribute extraction from novels\. We further thank Rohan Jain, Maureen Widjaja and Karin Yamaoka for their feedback early on in the project\. Finally, we’re grateful to Google’s GiG and GCP Credit Program for their support of this research\.

## Ethics Statement

This study examines socially shared gender associations in linguistic descriptions of physical attributes\. We emphasize that our work does not define or infer individuals’ gender from appearance, but instead analyzes patterns in language and perception\. Because such analyses risk reinforcing stereotypes if misused, we frame all findings descriptively and caution against using the dataset or models for gender inference, profiling, or decision\-making about real individuals\.

Human\-subject data were collected with informed consent, fair compensation, and no personally identifiable information beyond basic demographics\. All analyses are conducted at the aggregate level\. The dataset and models are released for research purposes, such as auditing implicit bias and improving AI transparency, and should be used with appropriate safeguards and contextualization\.

## References

- Abramovich and Ma \(2024\)S\. Abramovich and A\. J\. MaEvaluating refusal\.InWorkshop on Evaluating Evaluations: Examining Best Practices for Measuring Broader Impacts of Generative AI at NeurIPS,Cited by:[§5\.3](https://arxiv.org/html/2609.16366#S5.SS3.SSS0.Px3.p2.1)\.
- Bammanet al\.\(2019\)D\. Bamman, S\. Popat, and S\. ShenAn annotated dataset of literary entities\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 2138–2144\.Cited by:[item 3](https://arxiv.org/html/2609.16366#S1.I1.i3.p1.1),[§6\.2](https://arxiv.org/html/2609.16366#S6.SS2),[§6\.2](https://arxiv.org/html/2609.16366#S6.SS2.p1.1)\.
- Barsalou \(1985\)L\. W\. BarsalouIdeals, central tendency, and frequency of instantiation as determinants of graded structure in categories\.Journal of Experimental Psychology: Learning, Memory, and Cognition11\(4\),pp\. 629–654\.Cited by:[§4\.2](https://arxiv.org/html/2609.16366#S4.SS2.p2.1)\.
- Bennettet al\.\(2021\)C\. L\. Bennett, C\. Gleason, M\. K\. Scheuerman, J\. P\. Bigham, A\. Guo, and A\. To“It’s complicated”: negotiating accessibility and \(mis\) representation in image descriptions of race, gender, and disability\.InProceedings of the 2021 chi conference on human factors in computing systems,pp\. 1–19\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p1.1),[§2](https://arxiv.org/html/2609.16366#S2.p3.1)\.
- Bessonet al\.\(2017\)G\. Besson, G\. Barragan\-Jason, S\. J\. Thorpe, M\. Fabre\-Thorpe, S\. Puma, M\. Ceccaldi, and E\. J\. BarbeauFrom face processing to face recognition: comparing three different processing levels\.Cognition158,pp\. 33–43\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p3.1)\.
- Bolukbasiet al\.\(2016\)T\. Bolukbasi, K\. Chang, J\. Y\. Zou, V\. Saligrama, and A\. T\. KalaiMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings\.Advances in neural information processing systems29\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p3.1),[§2](https://arxiv.org/html/2609.16366#S2.p2.1)\.
- Brown and Perrett \(1993\)E\. Brown and D\. I\. PerrettWhat gives a face its gender?\.Perception22\(7\),pp\. 829–840\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p3.1)\.
- Bucholtz and Hall \(2005\)M\. Bucholtz and K\. HallIdentity and interaction: a sociocultural linguistic approach\.Discourse studies7\(4\-5\),pp\. 585–614\.Cited by:[§2](https://arxiv.org/html/2609.16366#S2.p1.1)\.
- Caliskanet al\.\(2017\)A\. Caliskan, J\. J\. Bryson, and A\. NarayananSemantics derived automatically from language corpora contain human\-like biases\.Science356\(6334\),pp\. 183–186\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p3.1)\.
- Cronbach \(1972\)L\. J\. CronbachThe dependability of behavioral measurements\.Theory of generalizability for scores and profiles,pp\. 1–33\.Cited by:[§B\.3](https://arxiv.org/html/2609.16366#A2.SS3.SSS0.Px2.p1.3),[§B\.3](https://arxiv.org/html/2609.16366#A2.SS3.p1.1)\.
- Devine \(1989\)P\. G\. DevineStereotypes and prejudice: their automatic and controlled components\.Journal of personality and social psychology56\(1\),pp\. 5–18\.Cited by:[§4\.1](https://arxiv.org/html/2609.16366#S4.SS1.p2.1)\.
- Donget al\.\(2024\)X\. Dong, Y\. Wang, P\. S\. Yu, and J\. CaverleeDisclosure and mitigation of gender bias in llms\.arXiv preprint arXiv:2402\.11190\.External Links:[Link](https://arxiv.org/abs/2402.11190)Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p3.1),[§2](https://arxiv.org/html/2609.16366#S2.p2.1)\.
- Eckert and McConnell\-Ginet \(2013\)P\. Eckert and S\. McConnell\-GinetLanguage and gender\.2 edition,Cambridge University Press\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p3.1),[§2](https://arxiv.org/html/2609.16366#S2.p1.1)\.
- Felkneret al\.\(2023\)V\. Felkner, H\. H\. Chang, E\. Jang, and J\. MayWinoqueer: a community\-in\-the\-loop benchmark for anti\-lgbtq\+ bias in large language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9126–9140\.Cited by:[§5\.3](https://arxiv.org/html/2609.16366#S5.SS3.SSS0.Px2.p1.1)\.
- Freeman and Ambady \(2011\)J\. B\. Freeman and N\. AmbadyA dynamic interactive theory of person construal\.Psychological review118\(2\),pp\. 247–279\.Cited by:[§4\.2](https://arxiv.org/html/2609.16366#S4.SS2.p2.1)\.
- Freemanet al\.\(2012\)J\. B\. Freeman, K\. L\. Johnson, R\. B\. Adams Jr, and N\. AmbadyThe social\-sensory interface: category interactions in person perception\.Frontiers in integrative neuroscience6,pp\. 81\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p3.1)\.
- Freeman and Johnson \(2016\)J\. B\. Freeman and K\. L\. JohnsonMore than meets the eye: split\-second social perception\.Trends in cognitive sciences20\(5\),pp\. 362–374\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p3.1)\.
- Gao and Kreiss \(2025\)B\. Gao and E\. KreissMeasuring bias or measuring the task: understanding the brittle nature of llm gender biases\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 6745–6761\.Cited by:[§2](https://arxiv.org/html/2609.16366#S2.p3.1)\.
- Hamidiet al\.\(2018\)F\. Hamidi, M\. K\. Scheuerman, and S\. M\. BranhamGender recognition or gender reductionism? the social implications of embedded gender recognition systems\.InProceedings of the 2018 chi conference on human factors in computing systems,pp\. 1–13\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p1.1)\.
- Hanleyet al\.\(2021\)M\. Hanley, S\. Barocas, K\. Levy, S\. Azenkot, and H\. NissenbaumComputer vision and conflicting values: describing people with automated alt text\.InProceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society,pp\. 543–554\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p1.1)\.
- Hossainet al\.\(2023\)T\. Hossain, S\. Dev, and S\. SinghMISGENDERED: limits of large language models in understanding pronouns\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5352–5367\.Cited by:[§2](https://arxiv.org/html/2609.16366#S2.p2.1)\.
- Johnson and Tassinary \(2005\)K\. L\. Johnson and L\. G\. TassinaryPerceiving sex directly and indirectly: meaning in motion and morphology\.Psychological Science16\(11\),pp\. 890–897\.External Links:[Document](https://dx.doi.org/10.1111/j.1467-9280.2005.01633.x)Cited by:[§2](https://arxiv.org/html/2609.16366#S2.p1.1)\.
- Karraset al\.\(2019\)T\. Karras, S\. Laine, and T\. AilaA style\-based generator architecture for generative adversarial networks\.External Links:1812\.04948,[Link](https://arxiv.org/abs/1812.04948)Cited by:[Appendix A](https://arxiv.org/html/2609.16366#A1.SS0.SSS0.Px2.p1.1)\.
- Kunda and Spencer \(2003\)Z\. Kunda and S\. J\. SpencerWhen do stereotypes come to mind and when do they color judgment? a goal\-based theoretical framework for stereotype activation and application\.Psychological bulletin129\(4\),pp\. 522–544\.Cited by:[§4\.1](https://arxiv.org/html/2609.16366#S4.SS1.p2.1)\.
- Lianget al\.\(2022\)P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar,et al\.Holistic evaluation of language models\.arXiv preprint arXiv:2211\.09110\.Cited by:[§5\.3](https://arxiv.org/html/2609.16366#S5.SS3.SSS0.Px2.p1.1)\.
- Luoet al\.\(2024\)S\. Luo, S\. J\. Kim, Z\. Duan, and K\. ChenRefusal as silence: gendered disparities in vision\-language model responses\.arXiv preprint arXiv:2406\.08222\.Cited by:[§5\.3](https://arxiv.org/html/2609.16366#S5.SS3.SSS0.Px3.p2.1)\.
- Marin\-Llobetet al\.\(2026\)A\. Marin\-Llobet, S\. Henniger, and M\. R\. BanajiVision\-language models suppress female representations under ambiguous input\.arXiv preprint arXiv:2605\.31556\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p1.1)\.
- McGraw and Wong \(1996\)K\. O\. McGraw and S\. P\. WongForming inferences about some intraclass correlation coefficients\.\.Psychological methods1\(1\),pp\. 30\.Cited by:[§B\.3](https://arxiv.org/html/2609.16366#A2.SS3.SSS0.Px2.p1.1),[§B\.3](https://arxiv.org/html/2609.16366#A2.SS3.SSS0.Px2.p2.1)\.
- Oosterhof and Todorov \(2008\)N\. N\. Oosterhof and A\. TodorovThe functional basis of face evaluation\.Proceedings of the National Academy of Sciences105\(32\),pp\. 11087–11092\.Cited by:[§4\.2](https://arxiv.org/html/2609.16366#S4.SS2.p2.1)\.
- Rosch and Mervis \(1975\)E\. Rosch and C\. B\. MervisFamily resemblances: studies in the internal structure of categories\.Cognitive psychology7\(4\),pp\. 573–605\.Cited by:[§4\.2](https://arxiv.org/html/2609.16366#S4.SS2.p2.1)\.
- Rosch \(1975\)E\. RoschCognitive reference points\.Cognitive psychology7\(4\),pp\. 532–547\.Cited by:[§4\.2](https://arxiv.org/html/2609.16366#S4.SS2.p2.1)\.
- Savoldiet al\.\(2025\)B\. Savoldi, G\. Attanasio, E\. Cupin, E\. Gkovedarou, J\. Hackenbuchner, A\. Lauscher, M\. Negri, A\. Piergentili, M\. Thind, and L\. BentivogliMind the inclusivity gap: multilingual gender\-neutral translation evaluation with mgente\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 13709–13731\.Cited by:[§2](https://arxiv.org/html/2609.16366#S2.p2.1)\.
- Scheuermanet al\.\(2019\)M\. K\. Scheuerman, J\. M\. Paul, and J\. R\. BrubakerHow computers see gender: an evaluation of gender classification in commercial facial analysis services\.Proceedings of the ACM on Human\-Computer Interaction3\(CSCW\),pp\. 1–33\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p1.1)\.
- Scheuermanet al\.\(2020\)M\. K\. Scheuerman, K\. Wade, C\. Lustig, and J\. R\. BrubakerHow we’ve taught algorithms to see identity: constructing race and gender in image databases for facial analysis\.Proceedings of the ACM on Human\-computer Interaction4\(CSCW1\),pp\. 1–35\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p1.1)\.
- Shrout and Fleiss \(1979\)P\. E\. Shrout and J\. L\. FleissIntraclass correlations: uses in assessing rater reliability\.\.Psychological bulletin86\(2\),pp\. 420\.Cited by:[§B\.3](https://arxiv.org/html/2609.16366#A2.SS3.SSS0.Px2.p1.1),[§B\.3](https://arxiv.org/html/2609.16366#A2.SS3.SSS0.Px2.p1.3)\.
- Simset al\.\(2019\)M\. Sims, J\. H\. Park, and D\. BammanLiterary event detection\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 3623–3634\.Cited by:[§6\.2](https://arxiv.org/html/2609.16366#S6.SS2.p1.1)\.
- Spearman \(1961\)C\. SpearmanThe proof and measurement of association between two things\.\.Cited by:[§B\.3](https://arxiv.org/html/2609.16366#A2.SS3.SSS0.Px2.p2.1)\.
- West and Zimmerman \(1987\)C\. West and D\. H\. ZimmermanDoing gender\.Gender & society1\(2\),pp\. 125–151\.Cited by:[§2](https://arxiv.org/html/2609.16366#S2.p1.1)\.
- Yanget al\.\(2025\)J\. Yang, X\. Han, and T\. BaldwinDemographics and democracy: benchmarking llms’ gender bias and political leaning in european parliament\.InProceedings of the 8th International Conference on Natural Language and Speech Processing \(ICNLSP\-2025\),pp\. 416–439\.Cited by:[§5\.3](https://arxiv.org/html/2609.16366#S5.SS3.SSS0.Px2.p1.1)\.
- Youet al\.\(2024\)Z\. You, H\. Lee, S\. Mishra, S\. Jeoung, A\. Mishra, J\. Kim, and J\. DiesnerBeyond binary gender labels: revealing gender bias in llms through gender\-neutral name predictions\.InProceedings of the 5th Workshop on Gender Bias in Natural Language Processing \(GeBNLP\),pp\. 255–268\.Cited by:[§2](https://arxiv.org/html/2609.16366#S2.p2.1)\.
- Zhaoet al\.\(2019\)J\. Zhao, T\. Wang, M\. Yatskar, R\. Cotterell, V\. Ordonez, and K\. ChangGender bias in contextualized word embeddings\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 629–634\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p3.1)\.
- Zhaoet al\.\(2018\)J\. Zhao, T\. Wang, M\. Yatskar, V\. Ordonez, and K\. ChangGender Bias in Coreference Resolution: Evaluation and Debiasing Methods\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),pp\. 15–20\.Cited by:[§1](https://arxiv.org/html/2609.16366#S1.p3.1),[§2](https://arxiv.org/html/2609.16366#S2.p2.1)\.

## Appendix ASupplementary Information of Dataset Collection forGAPA

Physical attributes were collected from three distinct source domain to facilitate both large coverage, cognitively prominent concepts, and rare and stylized descriptors\.

#### LLM\-generated attributes

were collected by prompting GPT\-4o and Gemini 2\.0 to list physical attributes that follow the pattern of “\{ADJ\} \+ \{NOUN\}”, where \{ADJ\} describes the properties of a body part that is referred to by \{NOUN\} \(e\.g\., “an hourglass figure”, “a chiseled jawline”\)\.

#### Human\-written attributes

were collected by presenting participants with a diverse set of images depicting individuals whose facial appearance contained both stereotypically gender\-typical and gender\-atypical features:N= 26 images generated by GPT 4o andN= 22 naturalistic images selected from the Flickr\-Faces\-HQ dataset\([Karras et al\., 2019](https://arxiv.org/html/2609.16366#bib.bib23)\)\. During the study, each participant saw a total of 10 images\. Participants viewed one image at a time and were asked, “How strongly do you associate this person’s appearance with that of a woman/man/non\-binary person in your culture?” using a 7\-point scale\. After providing their ratings, participants were prompted to list three features of the person’s appearance that led them to associate the individual with the gender category they rated highest\. They were also given the option to identify at least one feature of the person’s appearance that seemed unusual or atypical for someone they would associate with the gender category they rated highest\.

#### Attributes extracted from novels

were collected using a specially designed extraction pipeline for sourcing physical attributes from any long\-context documents or literature corpora such as novels\. The design and implementation details of the extraction pipeline is elaborated in[Appendix C](https://arxiv.org/html/2609.16366#A3)\. To construct the novel\-extracted set of the current dataset, we chose six popular contemporary novels of similar size, genre, and a balanced woman\-man author ratio:Game of Thrones\(Vol\. 1\),Harry Potter\(Vol\. 1\),Twilight\(Vol\. 1\),Hunger Games\(Vol\. 1\),Lord of the Rings\(Vol\. 1\), andMaze Runner\. From each book source, we randomly sampled 16 attributes from the extracted physical attributes\.

## Appendix BSupplementary Information for Human Annotations onGAPA

### B\.1Participant Demographics

The study was certified as exempt from IRB review by the UCLA Office of the Human Research Protection Program \(IRB\-24\-5529; 45 CFR 46\.104\)\. All participants provided informed consent prior to participation\. We recruited 320 US participants via Prolific \($12/hour\)\. Recruitment criteria required participants to be native English speakers, residing in the United States, with a Prolific approval rate of at least 95% and passing an attention check\. The final sample included 75 female\. The participants ranged in age from 19 to 83 years \(M = 42\.70\)\.

### B\.2Descriptive Results

[Table 2](https://arxiv.org/html/2609.16366#A2.T2)reports descriptive statistics of human\-rated gender likelihood for attributes across three sources \(human\-written, LLM\-generated, and novel\-derived\) for each target gender\. A consistent pattern emerges across sources: ratings are highest for women, followed by men, and lowest for non\-binary targets\. This ordering holds within most individual sources and in the overall aggregates\. This suggests a systematic difference in how our collected attributes are overall associated with gender categories in human judgments\.

Table 2:Descriptive statistics by gender and attribute source[Figure 9](https://arxiv.org/html/2609.16366#A2.F9)illustrates the per\-gender ratings of all attributes with significant gender differences \(p<0\.05p<0\.05\) in the complete six ranking patterns\.

\(a\)woman\>\>non\-binary person\>\>man\(b\)man\>\>non\-binary person\>\>woman
Figure 9:Per\-gender mean ratings and 95% confidence intervals for attributes showing significant gender differences \(p<0\.05p<0\.05\), grouped by ranking pattern\.\(a\)woman\>\>man\>\>non\-binary person\(b\)man\>\>woman\>\>non\-binary person
Figure 10:Per\-gender ratings grouped by ranking pattern \(continued\)\.\(a\)non\-binary person\>\>woman\>\>man\(b\)non\-binary person\>\>man\>\>woman
Figure 11:Per\-gender ratings grouped by ranking pattern \(continued\)\.
### B\.3Correlation Ceilings from Human Uncertainty

As the target labelsy¯i\\bar\{y\}\_\{i\}are derived from the average of noisy human judgments that are highly subjective and diverse \(see[subsection 4\.3](https://arxiv.org/html/2609.16366#S4.SS3)\),correlation performance is fundamentally bounded by inter\-rater variability\([Cronbach, 1972](https://arxiv.org/html/2609.16366#bib.bib20)\)\. We therefore estimate two complementary noise ceilings: a*leave\-one\-rater\-out*\(LOO\) ceiling and an*intraclass correlation*\(ICC\) ceiling, illustrated in[Figure 4](https://arxiv.org/html/2609.16366#S4.F4)\. They later serve as interpreting references for models of different strengths[Figure 5](https://arxiv.org/html/2609.16366#S5.F5)\(right\)\.

#### Leave\-One\-Out Reliability Ceiling\.

The LOO ceiling estimates the agreement between a typical individual rater and the consensus of other raters\. For each raterjj, we computerjLOOr\_\{j\}^\{\\text\{LOO\}\}, the Pearson correlation between that rater’s ratings and the mean ratings of all remaining raters across items:

rjLOO=corr⁡\(\{yi​j\},\{y¯i,\(−j\)\}\),r\_\{j\}^\{\\text\{LOO\}\}=\\mathrm\{corr\}\\big\(\\\{y\_\{ij\}\\\},\\\{\\bar\{y\}\_\{i,\(\-j\)\}\\\}\\big\),where​y¯i,\(−j\)=1k−1​∑j′≠jyi​j′\.\\text\{where \\quad\}\\bar\{y\}\_\{i,\(\-j\)\}=\\frac\{1\}\{k\-1\}\\sum\_\{j^\{\\prime\}\\neq j\}y\_\{ij^\{\\prime\}\}\.The LOO ceilingrLOOr^\{\\text\{LOO\}\}is then obtained by averagingrjLOOr\_\{j\}^\{\\text\{LOO\}\}across raters\. This ceiling value reflects*human\-level performance*: a model \(with parameterθt​y​p​i​c​a​l\\theta\_\{typical\}\) achieving LOO\-level correlation performs at the same level as anytypicalindividual annotator\. Consequently, a model with a comparable performance to a typical human rater with idiosyncratic rater noise is expected to satisfy

r⁡\(θt​y​p​i​c​a​l\)∼rLOO\.r\(\\theta\_\{typical\}\)\\;\\sim\\;r^\{\\text\{LOO\}\}\.

#### Intraclass Correlation \(ICC\) Ceiling\.

In contrast, the intraclass correlation coefficient ICC\(1,k\)\(1,k\)estimates the reliability of the*mean*rating acrosskkraters, describing the distribution of the collected human rating data\. ICC is a variance\-partitioning reliability coefficient derived from a random\-effects model and is widely used to quantify inter\-rater reliability in behavioral measurement\([Shrout and Fleiss, 1979](https://arxiv.org/html/2609.16366#bib.bib18);[McGraw and Wong, 1996](https://arxiv.org/html/2609.16366#bib.bib19)\)\. Under a standard random\-effects model,

yi​j=τi\+εi​j,y\_\{ij\}=\\tau\_\{i\}\+\\varepsilon\_\{ij\},whereτi\\tau\_\{i\}denotes thelatent true attributevalue for itemiiandεi​j\\varepsilon\_\{ij\}represents rater\-specific noise assumed to be independent with mean zero, the population\-level ICC for the mean ofkkraters is:

ICC⁡\(1,k\)=στ2στ2\+σε2/k\.\\mathrm\{ICC\}\(1,k\)=\\frac\{\\sigma\_\{\\tau\}^\{2\}\}\{\\sigma\_\{\\tau\}^\{2\}\+\\sigma\_\{\\varepsilon\}^\{2\}/k\}\.ICC\(1,k\)\(1,k\)represents the proportion of variance in the averaged human ratings attributable to the latent signal rather than noise, which has been adopted as the reliability of the averaged human judgment across psychometrics literature\([Shrout and Fleiss, 1979](https://arxiv.org/html/2609.16366#bib.bib18);[Cronbach, 1972](https://arxiv.org/html/2609.16366#bib.bib20)\)\.

Based on classical measurement theory, correlation between a noisy measurement and any predictor is bounded by the square root of that measurement’s reliability \(i\.e\., known as attenuation due to measurement error\)\([Spearman, 1961](https://arxiv.org/html/2609.16366#bib.bib21);[McGraw and Wong, 1996](https://arxiv.org/html/2609.16366#bib.bib19)\)\. Consequently, ICC\(1,k\)\(1,k\)provides a*natural upper bound*on the attainable correlation of an ideal model \(with parameterθi​d​e​a​l\\theta\_\{ideal\}\) that does not contain any rater noise in any rating prediction with the averaged human ratings\. In other words, ICC\(1,k\)\(1,k\)describes the correlation of an ideal model that perfectly recovers the latent true rating from the human population \(by always predictingτi\\tau\_\{i\}i\.e\.,y^i​\(θi​d​e​a​l\)=τi\\hat\{y\}\_\{i\}\(\\theta\_\{ideal\}\)=\\tau\_\{i\}\)\), offering a stricter theoretical noise ceiling thanrLOOr^\{\\text\{LOO\}\}for correlation\-based evaluation:

r⁡\(θi​d​e​a​l\)∼corr⁡\(τi,y¯i\)=ICC⁡\(1,k\)\.r\(\\theta\_\{ideal\}\)\\sim\\mathrm\{corr\}\(\\tau\_\{i\},\\bar\{y\}\_\{i\}\)=\\sqrt\{\\mathrm\{ICC\}\(1,k\)\}\.
WhilerLOOr^\{\\text\{LOO\}\}andICC⁡\(1,k\)\\sqrt\{\\mathrm\{ICC\}\(1,k\)\}provide the theoretical ceiling references for interpreting model correlation values, it is important to note that the empirical results of model performances may occasionally exceed them, due to the sampling noise in the finite set of collected data for evaluation\.

## Appendix CExtraction Pipeline

We describe the data source, the extraction pipeline implemented, and the schema and features of the extracted attributes used in our experiments\.

### C\.1Extraction Pipeline Process

#### 1\. Lexicon\-based sentence filter\.

Sentences are obtained with NLTK’ssent\_tokenize\. We then restrict LLM calls to sentences that are likely to contain physical descriptions\. Two hand\-defined lexicons are used: \(a\)physical\-attribute nouns: body\-part terms \(e\.g\., head, face, hair, eyes, nose, jaw, hands, skin, build, posture\) and related nouns \(e\.g\., silhouette, frame\); \(b\)physical\-attribute adjectives: descriptors for build/size \(e\.g\., thin, broad, tall\), musculature \(e\.g\., muscular, toned\), face and eyes \(e\.g\., sharp, round, deep\-set\), hair \(e\.g\., curly, blond\), skin \(e\.g\., freckled, pale\-skinned\), and evaluative terms \(e\.g\., beautiful, handsome\)\. Tokens are normalized \(lowercased, hyphens removed\) and matched after NLTK lemmatization\.

#### 2\. LLM\-based extraction\.

Each filtered sentence is sent to gpt\-4o\-mini with a fixed prompt that instructs the model to: \(i\) extract descriptions of a person’s physical attributes from the given text by outputing each description as a noun phrase that removes gender pronouns \(e\.g\., “his narrow jaw”→\\rightarrow“a narrow jaw”\); this requires both a body part and a characterizing modifier \(e\.g\., “dark brown hair”, “piercing green eyes”\) and to skip bare body\-part mentions \(e\.g\., “a beard”\) or purely evaluative phrases without a body part \(e\.g\., “a slender girl”\); \(ii\) for each extracted attribute, assign agenderlabel \(male,female,non\-binary, orunknown\) and provide reasoning\.

#### 3\. Postprocessing and filtering\.

The pipeline postprocesses the then normalizes gender pronouns in the attribute strings: possessive “his”/“her” are replaced by “a”/“an”; pronouns such as “he”/“she”/“him”/“herself”/“himself”/“hers” are removed\. Specifically, we also classify the extracted attributes from novels in two categories in terms of whether their formats are consistent with the LLM\-generated or human\-written physical attributes:\(a\) in\-domain\-format attributesare attributes that follow exactly the same pattern \{\[\(a/an\)\] \+ ADJ \+ NOUN\} \(e\.g\., “a narrow jaw,” “black hair,” “a long beard,” “a wide smile"\);\(b\) out\-of\-domain\-format attributesare physical attributes that do not exactly match the pattern, including: more than two content words \(e\.g\. “bright green eyes,” “vast muscular arms”\), different word order or POS \(e\.g\. noun before adjective, or no clear single adjective \+ single noun\), longer or more complex noun phrases\.

### C\.2Prompt for LLM\-Based Extraction

Youareanexpertatphysicalattributeextractionfromtext\.Yourtaskistoextractphysicalattributesandgenderofapersonfromabook\.

Youwillbegivenunstructuredtextfromabookthatpotentiallydescribesphysicalfeaturesofaperson\.Ignoredescriptionsofclothes\.

Youshouldfirstextractallthedescriptionsofaperson’sphysicalattributesandconvertthemintoa\*\*nounphrasestructurethatremovesgenderpronounslike’his’or’her’\*\*\.Descriptionsofaperson’sphysicalattributesaredefinedasadjectivesornounphrasesdescribingthecharacteristicsofabodypart\.Youshouldextractcompletenounphrasesincludingboththebodypartanditsdescription,suchas"anarrowjaw","darkbrownhair","piercinggreeneyes",andskipextractionofabodypartwithoutitsdescription\(e\.g\.,"abeard"\)\.Awordorphrasemerelyreferringtoabodypartwithoutanychacracteristicdescriptioninthetext,suchas"abeard"or"hishair",shouldnotbeextracted;ameredescriptionwithoutabodypartinthetextalsoshouldnotbeextracted\(e\.g\.,"aslendergirl"\)\.Ifthereisnodescriptionofphysicalattributesthatmeettheextractioncriteria,simplyreturnanemptylist\.

Then,foreachphysicalattribute,youshouldextractorinferthegenderofthepersondescribedbytheattribute,bylabelingitas"male","female",or"non\-binary"ifthereissolidevidenceinthetextsupportingoneofthem,or"unknown"ifthereisnoevidenceforinference\.

Provideyourreasoningandjustificationforeachattribute’sgenderlabel\.

Forexample:

<text\>

HarryPotterwasayoungmanwithathin,angularface,sharpfeatures,andanarrowjaw\.

</text\>

\[\{"attribute":"athinface","reasoning":"Theattributeisusedtodescribe’ayoungman’,explicitlysuggestingtheperson’sgender;’Harry’isalsoacommonmalename\.","gender":"male"\},\{"attribute":"anangularface","reasoning":"Theattributeisusedtodescribe’ayoungman’,explicitlysuggestingtheperson’sgender;’Harry’isalsoacommonmalename\.","gender":"male",\},\{"attribute":"sharpfeatures","reasoning":"Theattributeisusedtodescribe’ayoungman’,explicitlysuggestingtheperson’sgender;’Harry’isalsoacommonmalename\.","gender":"male",\},\{"attribute":"anarrowjaw","reasoning":"Theattributeisusedtodescribe’ayoungman’,explicitlysuggestingtheperson’sgender;’Harry’isalsoacommonmalename\.","gender":"male"\}\]

<text\>

Herhairwasdarkbrownandcurly,andhereyeswereapiercinggreen,staringatthemanintently\.

</text\>

\[\{"attribute":"darkbrownhair","reasoning":"Thetextusesthepronoun’her’torefertothepersonwiththecurrentattribute\(’darkbrownhair’\),explicitlysuggestingtheperson’sgenderasfemale\.","gender":"female"\},\{"attribute":"curlyhair","reasoning":"Thetextusesthepronoun’her’todescribethepersonwhosecurrentattribute\(’currlyhair’\)isbeingdescribed,explicitlysuggestingtheperson’sgenderasfemale\.","gender":"female"\},\{"attribute":"piercinggreeneyes","reasoning":"Thetextusesthepronoun’her’todescribethepersonwhoseeyesarebeingdescribed,explicitlysuggestingtheperson’sgenderasfemale\.","gender":"female"\}\]

<text\>

Athicklumpgrewinhisthroat\.

</text\>

\[\]

<text\>

Thenurse’seyesweredarkblue\.

</text\>

\[\{"attribute":"darkblueeyes","reasoning":"Thetextdescribesthenurse’seyesas’darkblue’withoutanycluesabouttheperson’sgender\.","gender":"unknown"\}\]

<text\>

Hisfootpushedmineoffthegaspedal\.

</text\>

\[\]

<text\>

Hetookhersmallpinkhandinhisownfrailspottedoneandgaveitagentlesqueeze\.

</text\>

\[\{"attribute":"asmallpinkhand","reasoning":"Thetextusesthepronoun’her’torefertothepersonwiththecurrentattribute\(’asmallpinkhand’\),explicitlysuggestingtheperson’sgenderasfemale\.","gender":"female"\},\{"attribute":"afrailspottedhand","reasoning":"Thetextusesthepronoun’his’torefertothepersonwiththecurrentattribute\(’afrailspottedhand’\),explicitlysuggestingtheperson’sgenderasmale\.","gender":"male"\}\]

<text\>

Herswasnotaprettyface,alas\.

</text\>

\[\{"attribute":"aprettyface","reasoning":"Thetextusesthepronoun’hers’torefertothepersonwiththecurrentattribute\(’aprettyface’\),explicitlysuggestingtheperson’sgenderasfemale\.","gender":"female"\}\]

## Appendix DZero\-Shot LLM Inference Experiments

### D\.1Prompt Template

Howlikelyisitforsomeonetosaya\{person\_term\}has\{attribute\}?Youranswershouldonlybeasinglenumberbetween1\(Notatall\)and7\(Extremelylikely\)\.

Answer:

### D\.2Method Details

Two complementary strategies were adopted to elicit ratings based on varied constraints from different LLMs:\(1\) direct generation: we directly extract the verbalized ratings in the model\-generated response;\(2\) top\-logit token extraction: we examine the model’s output distribution at the first generated token position: for open\-weight models, we rank top k tokens with the highest logit values, select the token that contains any number in 1–7 can be extracted, and use the number as model rating\. This is a more tolerant approach than simply selecting the highest\-logit token among the seven tokens 1\-7, as it considers common cases of zero\-shot output that contain blanks adjacent to the actual answering, causing unexpected tokenization results; for API\-based models, we similarly retrieve log\-probabilities for the top\-ranked tokens \(if available\) and identify the highest\-probability token containing a valid rating\. Incorporating both strategies ensures the collection of model rating preferences even without demanding their instruction\-following abilities, and provides a robust and thorough investigation of models’ encoding of gender in physical descriptions from not only the model’s explicit textual response but also its underlying distributional preferences\.

Method 1 and 2 showed a close\-to\-perfect agreement rate, serving as sanity check, except for the Mistral\-7b models\. Considering the higher accessibility of method 1, we report the ratings of the directly generated responses from models as the primary results\.

### D\.3Complete Results

[6\(a\)](https://arxiv.org/html/2609.16366#S5.F6.sf1)in the main body text presents the complete ratings in comparison with human ratings across genders for all models\. Below,[Table 3](https://arxiv.org/html/2609.16366#A4.T3)shows the per\-gender RMSE & Pearsonrrstatistics of all evaluated models in the experiment of gender association onGAPA, with significance values included in the brackets\.

Table 3:Per\-gender RMSE & Pearsonrrstatistics of all evaluated models in the experiment of gender association onGAPA\.

## Appendix ETraining Experiments for the Proxy Predictor Model

### E\.1Training Details

We use the LLM\-generated subset ofGAPAfor training, validation, and in\-domain evaluation \(split ratio = 60/15/25\), and reserve the novel\-derived and human\-written attribute sets as out\-of\-domain evaluation benchmarks\. Complementing the zero\-shot generation approach in[section 5](https://arxiv.org/html/2609.16366#S5), we train each model with an added linear regression head to predict aggregated human ratings of physical attributes for different gender categoriesc∈\{woman,man,non\-binary person\}c\\in\\\{\\textit\{woman\},\\textit\{man\},\\textit\{non\-binary person\}\\\}\. For each training instance, the model takes as input a sentence describing a physical attribute with respect to a target gender \(e\.g\., “a woman has broad shoulders”\)\. The mean\-pooled final\-layer hidden representation is passed through a one\-dimensional linear projection to produce a scalar prediction corresponding to the human likelihood rating for that attribute–gender pair\. This formulation treats the task as continuous regression, avoiding reliance on explicit numerical generation and providing a more direct probe of model–human alignment\.

The model is optimized using a mean squared error \(MSE\) loss between predicted scores and averaged human ratings\. Letyi​jy\_\{ij\}denote the rating provided by raterjjfor itemii, andy¯i=1k​∑j=1kyi​j\\bar\{y\}\_\{i\}=\\frac\{1\}\{k\}\\sum\_\{j=1\}^\{k\}y\_\{ij\}the mean acrosskkraters\. Training minimizesℒ=1N​∑i=1N\(y^i−y¯i\)2\\mathcal\{L\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\(\\hat\{y\}\_\{i\}\-\\bar\{y\}\_\{i\}\)^\{2\}, wherey^i\\hat\{y\}\_\{i\}is the model prediction andNNis the number of samples\. Training on aggregated ratings encourages the model to capture the shared signal across annotators while attenuating idiosyncratic noise\.

To ensure fair comparison across models with different architectures and scales, we adopt a standardized hyperparameter search template, with model\-specific ranges for sensitive parameters such as learning rate and effective batch size\. Models are compared based on their best performance across the search\. Each model is allocated the same tuning budget and evaluated under consistent cross\-validation and multi\-seed settings to improve robustness\. Full training details and configurations are provided in[Appendix E](https://arxiv.org/html/2609.16366#A5)\.

### E\.2Full Configurations for Hyperparameter Search

[Table 4](https://arxiv.org/html/2609.16366#A5.T4)lists the hyperparameter configuration for hyperparameter search in the training ablation experiments, aiming for a relatively controlled comparison across model family, scale, and instruction\-tuning variants\.

Table 4:Training hyperparameter configuration, aiming for a relatively controlled comparison across model family, scale, and instruction\-tuning variants\.

## Appendix FHow to Interpret the Exploratory LitBank Analyses

It is important to note that the analyses in[subsection 6\.2](https://arxiv.org/html/2609.16366#S6.SS2)reflect predicted gender associations as they are held by the US participants we recruited in recent years\. Since gender associations change over time, the following results should not be interpreted as a clear window into the authors’ absolute biases decades ago\. Instead, it’s a window into the associations a common reader may take away today and comparative analyses between groups of authors can inform hypotheses about potentially persistent gendered perceptions\.