当目标与评估器不一致时:评估事后人口统计框架缓解的迁移效果
摘要
该论文将事后改写以减少大语言模型生成文本中人口统计群体框架差异的问题视为迁移问题进行研究,发现编辑策略和评估器的选择会显著影响缓解措施在不同目标和评估器类别下是否呈现出成功的效果。
arXiv:2611.25258v1 Announce Type: new
Abstract: Post-hoc rewriting is used to reduce demographic disparities in LLM-generated text when the source model is inaccessible. A rewrite can look successful merely because it improved its own target or satisfied its selector. We study this as a transfer problem on 660 gender pairs and 1,980 age contrasts of demographic-conditioned climate messages. A rewriter proposes edits, a selection model chooses among constraint-satisfying candidates, and an independent evaluator uninvolved in either rates pairwise framing disparity. A fixed policy that always targets the message with the lower Persuasion Bias Index (PBI) transfers inconsistently across axes and outcome aggregations. An adaptive policy that proposes edits on both sides but accepts at most one significantly improves on that rule, applied to the same candidates, in seven of eight comparisons across two evaluator families, mostly by editing more pairs, not by guided direction choice, at higher preservation risk. A policy editing both messages gives the largest point reductions on fewer pairs. These two policies reduce disparity across a preservation-threshold sweep; absolute ratings vary across evaluator families and prompts. Shrinking the PBI gap increases dimension-mean disparity under both evaluator families. Mitigation should be reported as transfer across objectives, evaluators and their prompts, and editing policies.
查看缓存全文
缓存时间: 2026/11/20 09:06
# When Objectives and Evaluators Disagree: Evaluating Transfer for Post-hoc Demographic Framing Mitigation
Source: [https://arxiv.org/html/2611.25258](https://arxiv.org/html/2611.25258)
Tunazzina Islam††thanks:Now at the University of Southern California \(tunazzin@usc\.edu\)\.Affiliation:Department of Computer ScienceAffiliation:Purdue University, West LafayetteAffiliation:Indiana \- 47907, USAEmail:[islam32@purdue\.edu](mailto:)
###### Abstract
Post\-hoc rewriting is used to reduce demographic disparities in LLM\-generated text when the source model is inaccessible\. A rewrite can look successful merely because it improved its own target or satisfied its selector\. We study this as a transfer problem on 660 gender pairs and 1,980 age contrasts of demographic\-conditioned climate messages\. A rewriter proposes edits, a selection model chooses among constraint\-satisfying candidates, and an independent evaluator uninvolved in either rates pairwise framing disparity\. A fixed policy that always targets the message with the lower Persuasion Bias Index \(PBI\) transfers inconsistently across axes and outcome aggregations\. An adaptive policy that proposes edits on both sides but accepts at most one significantly improves on that rule, applied to the same candidates, in seven of eight comparisons across two evaluator families, mostly by editing more pairs, not by guided direction choice, at higher preservation risk\. A policy editing both messages gives the largest point reductions on fewer pairs\. These two policies reduce disparity across a preservation\-threshold sweep; absolute ratings vary across evaluator families and prompts\. Shrinking the PBI gap increases dimension\-mean disparity under both evaluator families\. Mitigation should be reported as transfer across objectives, evaluators and their prompts, and editing policies\.
## 1Introduction
Large language models \(LLMs\)\([Brown et al\., 2020](https://arxiv.org/html/2611.25258#bib.bib6)\)are increasingly used to generate text tailored to specific demographic groups, enabling personalized communication in domains such as policy outreach, marketing, and public messaging\([Liu et al\., 2025](https://arxiv.org/html/2611.25258#bib.bib35);[Hackenburg and Margetts, 2024](https://arxiv.org/html/2611.25258#bib.bib15);[Breum et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib5)\)\. While such demographic conditioning can improve relevance and engagement, it also raises fairness concerns: models may systematically vary not only the content of messages, but also their tone, framing, and persuasive style across demographic groups\([Islam, 2026b](https://arxiv.org/html/2611.25258#bib.bib18);[Pauli et al\., 2026](https://arxiv.org/html/2611.25258#bib.bib40);[Pauli et al\., 2025](https://arxiv.org/html/2611.25258#bib.bib39);[Tan and Lee, 2025](https://arxiv.org/html/2611.25258#bib.bib50);[Fang et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib13);[Wan et al\., 2023](https://arxiv.org/html/2611.25258#bib.bib56)\)\. These differences motivate methods for auditing and mitigating demographic disparities in generated text\.
Many practical systems provide access only to completed outputs rather than model parameters, training data, or decoding procedures\. In such settings, mitigation must be performedpost hocthrough rewriting, filtering, or selective editing\([Wan and Chang, 2025](https://arxiv.org/html/2611.25258#bib.bib55);[Banerjee et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib3);[Stahl et al\., 2022](https://arxiv.org/html/2611.25258#bib.bib48);[Krause et al\., 2021](https://arxiv.org/html/2611.25258#bib.bib25);[Schick et al\., 2021](https://arxiv.org/html/2611.25258#bib.bib43);[Dixon et al\., 2018](https://arxiv.org/html/2611.25258#bib.bib11)\)\. A common procedure is to define a disparity score, generate a revised output, and retain the revision when that score improves\. However, this creates an evaluation ambiguity\. A rewrite may appear successful because it genuinely improves a separately measured outcome, or because the same objective or evaluator influenced both candidate selection and final evaluation\.
Figure 1:Overview of the framework\. \(1\) Matched pairs differ only in the demographic audience\. \(2\) A rewriter proposes an edit of each message given its partner\. \(3\) A selection judge and deterministic PBI and preservation constraints decide which edits are admissible, and an editing policy fixes one output per pair, falling back to Original when its edit is inadmissible\. \(4\) A separate evaluator, blind to all selection information, rates the fixed outputs\.Our pipeline separates selection from evaluation \(Fig\.[1](https://arxiv.org/html/2611.25258#S1.F1)\)\. A rewriting model proposes edits to matched pairs of messages written for two demographic audiences; aselection judge, together with deterministic tolerance and text\-preservation constraints, decides which edits are accepted; and after every final output is fixed, anindependent evaluator, which took no part in generation or selection, rates the finalized pairs\. No model occupies both roles, and every policy effect we estimate is a paired difference between two conditions of the same matched pair, scored by one evaluator within a single run\. We formulate this attribution question as a transfer problem spanning objectives, evaluators, and editing policies\.Objective transferasks whether improving one operational measure of disparity also improves another\. For example, reducing a deterministic persuasion gap based on agency, certainty, and imperative language may not reduce pairwise differences in warmth or overall communication style\.Evaluator transferasks whether an improvement measured by the evaluator that selected a rewrite remains visible to an evaluator that did not participate in candidate generation or acceptance, including under a reworded evaluation prompt\. When the same LLM judgment is used for both selection and final reporting, improvement on that score is partly guaranteed by construction and cannot independently establish effectiveness\.
We study these questions using matched demographic pairs of climate change\-related messages generated by three source models\([Islam, 2026b](https://arxiv.org/html/2611.25258#bib.bib18)\)\. Each pair holds the source model, stance, theme, region, and other content variables fixed while varying gender or age\. The dataset contains 660 gender pairs and 1,980 age contrasts drawn from 330 four\-age\-group contexts\. This matched design isolates text differences associated with the demographic condition under the observed generation setup, but it is a diagnostic comparison rather than a claim of causal counterfactual fairness\.
We evaluatethreeoperational views of disparity\. The first is the Persuasion Bias Index \(PBI\) introduced by[Islam \(2026b\)](https://arxiv.org/html/2611.25258#bib.bib18), adeterministic message\-level scorebased on agency, certainty, and imperative features\. The other two arepairwise LLM\-judged outcomes: an equal\-weight mean over assertiveness, certainty, directive force, and warmth, and a separate direct rating of overall framing disparity\. These operationalizations partially overlap\. In particular, assertiveness, certainty, and directive force are closely related to PBI components, while warmth provides the clearest extension\. We therefore do not treat them as independent constructs or validated definitions of fairness\. Instead, we ask whether conclusions transfer across their different units of analysis, elicitation procedures, and aggregation rules\.
The independent evaluator receives no demographic, policy, PBI, acceptance, or selection\-score metadata; this does not make its judgments objective, but it prevents the evaluator behind the final claims from determining which outputs enter the comparison\.
Our initial analysis considers afixed\-direction policythat always rewrites the message with lower original PBI\. Its improvements transfer inconsistently across demographic axes and outcome definitions, suggesting that edit direction itself is an important policy choice\. We therefore proposeAdaptive\-single, a pair\-aware method that generates both one\-sided edit directions and selects at most one admissible candidate\. For controlled comparison, all direction policies are constructed from the same candidate pool and evaluated only after their outputs are finalized\.
Under independent evaluation,Adaptive\-singleoutperforms the controlled Lower\-PBI policy on both outcomes for age and on the dimension mean for gender\. A Both\-sides policy, which changes both messages when accepted, gives the largest point reductions on both outcomes while editing fewer pairs, but is separable from Adaptive\-single only for age under a second evaluator family\. Preservation audits further show that greater mitigation coverage can increase policy\-level exposure to possible content drift, even when conditional embedding and natural language inference \(NLI\) diagnostics are similar across policies\. These findings motivate evaluating post\-hoc mitigation jointly in terms of outcome transfer, evaluator separation, edit direction, and intervention risk\.
Our contributions are111All codes and data are available here:[https://github\.com/tunazislam/posthoc\-mitigation\-transfer\-evaluation](https://github.com/tunazislam/posthoc-mitigation-transfer-evaluation):
- •We formulate post\-hoc demographic mitigation as transfer across operational objectives, evaluators and editing policies, and introduce an independent\-evaluation protocol that separates candidate selection from final assessment while keeping unchanged fallbacks in the estimand\.
- •We introduce Adaptive\-single, a pair\-aware policy that evaluates both one\-sided edit directions, and show that its advantage over fixed Lower\-PBI editing comes mostly from wider intervention coverage, with guided direction choice adding a smaller but significant share on the dimension mean\.
- •We show that the optimized objective and the independently measured outcome can disagree in*sign*, not only in magnitude, under two evaluator families\.
- •We test our conclusions against a second evaluator family, prompt variants for the fixed policy and a threshold sweep, and report preservation, qualitative and exploratory human diagnostics showing lower disparity scores coexisting with evaluator disagreement or content risk\.
We study three questions: whether selection\-time improvements transfer to an independent evaluator \(RQ1\), whether adaptive edit\-direction selection outperforms fixed Lower\-PBI editing \(RQ2\), and whether PBI\-targeted improvements transfer to pairwise framing outcomes \(RQ3\)\.
## 2Related Work
#### Bias and fairness in LLM generation\.
Demographic conditioning can make LLM outputs differ systematically in content, tone and framing across groups\([Wang and Gu, 2026](https://arxiv.org/html/2611.25258#bib.bib57);[Islam, 2026b](https://arxiv.org/html/2611.25258#bib.bib18);[Pauli et al\., 2026](https://arxiv.org/html/2611.25258#bib.bib40);[Pauli et al\., 2025](https://arxiv.org/html/2611.25258#bib.bib39);[Sethi et al\., 2025](https://arxiv.org/html/2611.25258#bib.bib45);[Tan and Lee, 2025](https://arxiv.org/html/2611.25258#bib.bib50);[Aghaebe et al\., 2025](https://arxiv.org/html/2611.25258#bib.bib2);[Fang et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib13);[Dai et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib9);[Kumar et al\., 2023](https://arxiv.org/html/2611.25258#bib.bib26);[Li et al\., 2023b](https://arxiv.org/html/2611.25258#bib.bib31);[Li et al\., 2023a](https://arxiv.org/html/2611.25258#bib.bib30);[Sag, 2023](https://arxiv.org/html/2611.25258#bib.bib42);[Urman and Makhortykh, 2023](https://arxiv.org/html/2611.25258#bib.bib54);[Kotek et al\., 2023](https://arxiv.org/html/2611.25258#bib.bib24);[Esiobu et al\., 2023](https://arxiv.org/html/2611.25258#bib.bib12);[Wan et al\., 2023](https://arxiv.org/html/2611.25258#bib.bib56)\), including in lexical choice, stylistic framing and persuasive intensity\([Islam, 2026b](https://arxiv.org/html/2611.25258#bib.bib18);[Wan et al\., 2023](https://arxiv.org/html/2611.25258#bib.bib56)\), and such biases depend on the demographic distribution of training data\([Garimella et al\., 2022](https://arxiv.org/html/2611.25258#bib.bib14)\)\.
#### Bias mitigation and post\-hoc rewriting\.
Bias is mitigated through data augmentation, model adaptation, controlled decoding, filtering, output aggregation and reasoning\-step intervention\([Abels and Lenaerts, 2025](https://arxiv.org/html/2611.25258#bib.bib1);[Wu et al\., 2025](https://arxiv.org/html/2611.25258#bib.bib58);[Banerjee et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib3);[Masoudian et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib37);[Krause et al\., 2021](https://arxiv.org/html/2611.25258#bib.bib25);[Schick et al\., 2021](https://arxiv.org/html/2611.25258#bib.bib43);[Dinan et al\., 2020](https://arxiv.org/html/2611.25258#bib.bib10)\), which typically require access to training data, model internals or intermediate reasoning\. Post\-hoc methods instead rewrite finished outputs, for example by modifying agency and power or revising biased spans\([Wan and Chang, 2025](https://arxiv.org/html/2611.25258#bib.bib55);[Stahl et al\., 2022](https://arxiv.org/html/2611.25258#bib.bib48)\)\. Unlike these single\-output approaches, we model disparity over matched demographic pairs and separate edit coverage from mitigation effect, and lexical similarity from semantic preservation\.
#### Pairwise evaluation and competing objectives\.
Matched demographic comparisons hold the content context fixed while varying the demographic condition\([Zhou et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib60);[Banerjee et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib3);[Rosenblatt and Witter, 2023](https://arxiv.org/html/2611.25258#bib.bib41);[Kusner et al\., 2017](https://arxiv.org/html/2611.25258#bib.bib27)\), but different fairness criteria can conflict, and optimizing one need not improve another\([Chouldechova, 2017](https://arxiv.org/html/2611.25258#bib.bib8);[Kleinberg et al\., 2016](https://arxiv.org/html/2611.25258#bib.bib23)\)\. We study this in generative mitigation by comparing a deterministic message\-level persuasion objective with pairwise framing judgments\.
#### LLM judges and evaluator circularity\.
LLMs are widely used as evaluators\([Brady and Islam, 2026](https://arxiv.org/html/2611.25258#bib.bib4);[Sudhoff et al\., 2026](https://arxiv.org/html/2611.25258#bib.bib49);[Islam, 2026a](https://arxiv.org/html/2611.25258#bib.bib17);[Tan et al\., 2025](https://arxiv.org/html/2611.25258#bib.bib51);[Islam and Goldwasser, 2025c](https://arxiv.org/html/2611.25258#bib.bib21);[Islam and Goldwasser, 2025a](https://arxiv.org/html/2611.25258#bib.bib19);[Zheng et al\., 2023](https://arxiv.org/html/2611.25258#bib.bib59);[Liu et al\., 2023](https://arxiv.org/html/2611.25258#bib.bib36);[Chiang and Lee, 2023](https://arxiv.org/html/2611.25258#bib.bib7);[Li et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib29);[Lin et al\., 2022](https://arxiv.org/html/2611.25258#bib.bib33)\), yet their judgments are sensitive to prompt, model, position and scale anchors\([Shi et al\., 2025](https://arxiv.org/html/2611.25258#bib.bib46);[Sclar et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib44);[Zhuo et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib61)\)\. Using one judge score for both acceptance and final evaluation is circular even with a separate rewriter, so we separate the selection judge from an independent evaluator and treat their disagreement as an empirical transfer result\.
#### Transfer of mitigation effects and of evaluators\.
[Jin et al\. \(2021\)](https://arxiv.org/html/2611.25258#bib.bib22)ask whether fine\-tuned bias\-mitigation gains transfer across tasks and domains, largely holding the evaluation construct fixed; CAFIE[Banerjee et al\. \(2024\)](https://arxiv.org/html/2611.25258#bib.bib3)intervenes at decoding time so that counterfactual continuations are treated alike; and JudgeBench[Tan et al\. \(2025\)](https://arxiv.org/html/2611.25258#bib.bib51)tests LLM judges in settings where a verifiable answer adjudicates their verdicts\. We instead hold the model and data fixed and vary the*measurement*: the objective a post\-hoc rewriter optimizes, the prompt that elicits the outcome, and the evaluator family that scores it\. With no validated ground\-truth fairness quantity to adjudicate between these operationalizations, their disagreement must be reported rather than resolved, and we show that it can extend to the sign of the effect\.
## 3Problem Formulation
### 3\.1Matched Demographic Outputs
Letx∈𝒳x\\in\\mathcal\{X\}denote the non\-demographic message context, including the source model, stance, theme, region, and other fixed content variables\. Leta∈𝒜a\\in\\mathcal\{A\}denote a demographic condition\. A fixed black\-box generatorGGproducesy=G\(x,a\)y=G\(x,a\)\. For instanceii, we construct a matched pair
Yi=\(yiA,yiB\)=\(G\(xi,aiA\),G\(xi,aiB\)\)Y\_\{i\}=\\left\(y\_\{i\}^\{A\},y\_\{i\}^\{B\}\\right\)=\\left\(G\(x\_\{i\},a\_\{i\}^\{A\}\),G\(x\_\{i\},a\_\{i\}^\{B\}\)\\right\)wherexix\_\{i\}is held fixed andaiA≠aiBa\_\{i\}^\{A\}\\neq a\_\{i\}^\{B\}varies within one demographic axis\. For gender, each fixed context produces one male–female contrast\. For age, each underlying context contains four messages and contributes all six unordered pairwise contrasts\. Therefore, age\-pair observations derived from the same four\-message context are statistically dependent and are clustered at the underlying\-context level during inference\.
This matched design isolates observed differences associated with changing the demographic condition while holding the recorded generation context fixed\. It is an auditing design and does not, by itself, establish counterfactual fairness in the causal sense\.
### 3\.2Message\-Level Persuasion Disparity
Following[Islam \(2026b\)](https://arxiv.org/html/2611.25258#bib.bib18), letP\(y\)P\(y\)denote the Persuasion Bias Index \(PBI\):
P\(y\)=A\(y\)\+C\(y\)\+λI\(y\)P\(y\)=A\(y\)\+C\(y\)\+\\lambda I\(y\)whereA\(y\)A\(y\)is a lexicon\-based agency score,C\(y\)C\(y\)captures certainty and hedging,I\(y\)I\(y\)captures imperative force, andλ=0\.1\\lambda=0\.1\. For a matched pairYiY\_\{i\}, the absolute PBI disparity is
dP\(Yi\)=\|P\(yiA\)−P\(yiB\)\|d\_\{P\}\(Y\_\{i\}\)=\\left\|P\(y\_\{i\}^\{A\}\)\-P\(y\_\{i\}^\{B\}\)\\right\|PBI is deterministic for fixed text\. We treat it as an operational linguistic score, not a full measure of persuasion, demographic disparity or fairness\.
### 3\.3Pairwise Framing Outcomes
For a message pairYiY\_\{i\}, an LLM evaluatorJJassigns ordinal disparity ratings for each framing dimensiondJ,k\(Yi\)∈\{0,1,2,3\}d\_\{J,k\}\(Y\_\{i\}\)\\in\\\{0,1,2,3\\\}wherek∈\{assertiveness,certainty,directive,warmth\}k\\in\\\{\\text\{assertiveness\},\\text\{certainty\},\\text\{directive\},\\text\{warmth\}\\\}\. We define the equal\-weight dimension outcome as
dJ,dim\(Yi\)=14∑k=14dJ,k\(Yi\)d\_\{J,\\mathrm\{dim\}\}\(Y\_\{i\}\)=\\frac\{1\}\{4\}\\sum\_\{k=1\}^\{4\}d\_\{J,k\}\(Y\_\{i\}\)The evaluator separately provides a direct overall\-disparity ratingdJ,overall\(Yi\)∈\{0,1,2,3\}d\_\{J,\\mathrm\{overall\}\}\(Y\_\{i\}\)\\in\\\{0,1,2,3\\\}, which is elicited directly and is not computed from the four\-dimensional ratings\.
These outcomes are operational LLM judgments\. Their dimensions partially overlap conceptually with PBI: certainty corresponds directly to the PBI certainty component, directive force is closely related to imperative force, and assertiveness may overlap with agency\. Warmth provides the clearest extension beyond PBI\. We therefore do not treat PBI and the LLM\-judged outcomes as independent constructs or as validated definitions of fairness\. Instead, we study whether conclusions transfer across their different units of analysis, elicitation procedures, and aggregation rules\.
### 3\.4Directional Rewrite Candidates
LetRRdenote a pair\-aware rewriting model\. Starting from the original pairYiY\_\{i\}, the model may generate two directional candidates:
y~iA=R\(yiA∣yiB,xi,aiA,aiB\)\\widetilde\{y\}\_\{i\}^\{A\}=R\\left\(y\_\{i\}^\{A\}\\mid y\_\{i\}^\{B\},x\_\{i\},a\_\{i\}^\{A\},a\_\{i\}^\{B\}\\right\)y~iB=R\(yiB∣yiA,xi,aiB,aiA\)\\widetilde\{y\}\_\{i\}^\{B\}=R\\left\(y\_\{i\}^\{B\}\\mid y\_\{i\}^\{A\},x\_\{i\},a\_\{i\}^\{B\},a\_\{i\}^\{A\}\\right\)The two candidates are generated independently from the same original pair\. The output of one directional rewrite is not used to generate the other\. These candidates define four possible pair configurations:YiO=\(yiA,yiB\)Y\_\{i\}^\{O\}=\\left\(y\_\{i\}^\{A\},y\_\{i\}^\{B\}\\right\);YiA→=\(y~iA,yiB\)Y\_\{i\}^\{A\\rightarrow\}=\\left\(\\widetilde\{y\}\_\{i\}^\{A\},y\_\{i\}^\{B\}\\right\);YiB→=\(yiA,y~iB\)Y\_\{i\}^\{B\\rightarrow\}=\\left\(y\_\{i\}^\{A\},\\widetilde\{y\}\_\{i\}^\{B\}\\right\);YiAB=\(y~iA,y~iB\)Y\_\{i\}^\{AB\}=\\left\(\\widetilde\{y\}\_\{i\}^\{A\},\\widetilde\{y\}\_\{i\}^\{B\}\\right\)\. Here,YiOY\_\{i\}^\{O\}is the original pair,YiA→Y\_\{i\}^\{A\\rightarrow\}edits only Message A,YiB→Y\_\{i\}^\{B\\rightarrow\}only Message B, andYiABY\_\{i\}^\{AB\}both messages\.
### 3\.5Selection and Admissibility
LetJsJ\_\{s\}denote the*selection judge*\. For configurationYicY\_\{i\}^\{c\}, wherec∈\{O,A→,B→,AB\}c\\in\\\{O,A\\rightarrow,B\\rightarrow,AB\\\}, the selection score is
sic=dJs,dim\(Yic\)s\_\{i\}^\{c\}=d\_\{J\_\{s\},\\mathrm\{dim\}\}\(Y\_\{i\}^\{c\}\)A rewritten configuration is admissible only when it improves the selection\-judge dimension score while satisfying PBI and preservation constraints\. For a one\-sided edited configurationYicY\_\{i\}^\{c\}, we require
sic<siO,dP\(Yic\)≤dP\(YiO\)\+τ,s\_\{i\}^\{c\}<s\_\{i\}^\{O\},\\qquad d\_\{P\}\(Y\_\{i\}^\{c\}\)\\leq d\_\{P\}\(Y\_\{i\}^\{O\}\)\+\\tau,ROUGE\-L\(y~,y\)≥ηr,Overlap\(y~,y\)≥ηo,\\operatorname\{ROUGE\\mbox\{\-\}L\}\(\\widetilde\{y\},y\)\\geq\\eta\_\{r\},\\qquad\\operatorname\{Overlap\}\(\\widetilde\{y\},y\)\\geq\\eta\_\{o\},ℓmin≤\|y~\|/\|y\|≤ℓmax\.\\ell\_\{\\min\}\\leq\|\\widetilde\{y\}\|/\|y\|\\leq\\ell\_\{\\max\}\.
For a two\-sided configuration, the preservation constraints are applied separately to both edited messages\. If a configuration violates any admissibility condition, it is rejected\.
Because admissibility explicitly requires improvement underJsJ\_\{s\}, the selection score cannot be used as independent evidence that the resulting policy improves framing disparity\.
The direct overall rating is deliberately excluded from the selection score, so one reported outcome is never optimized during selection, and the PBI target in the rewrite prompt is only a steering hint: acceptance is enforced deterministically by the constraints above \(App\.[A\.4](https://arxiv.org/html/2611.25258#A1.SS4)\)\.
### 3\.6Editing Policies
A policyπ\\pimaps the original pair and its admissible configurations to a final output:
Yiπ=π\(YiO,YiA→,YiB→,YiAB\)Y\_\{i\}^\{\\pi\}=\\pi\\left\(Y\_\{i\}^\{O\},Y\_\{i\}^\{A\\rightarrow\},Y\_\{i\}^\{B\\rightarrow\},Y\_\{i\}^\{AB\}\\right\)We consider the following policies: Lower\-PBI\.Select the admissible one\-sided configuration that edits the message with the lower original PBI score\. Higher\-PBI\.Select the admissible one\-sided configuration that edits the message with the higher original PBI score\. Random\-single\.Select a predetermined one\-sided edit direction using a fixed random seed\. Adaptive\-single\.Among the admissible one\-sided configurations, select the configuration with the minimum selection scoresics\_\{i\}^\{c\}\. Ties are broken first by the lower resulting PBI gap and then by a deterministic identifier\-based rule\. Both\-sides\.Select the two\-sided configurationYiABY\_\{i\}^\{AB\}when it is admissible\. Original\.Return the unmodified pairYiOY\_\{i\}^\{O\}\.
Every active policy returns the original pair if its required configuration is unavailable or inadmissible\. Therefore, policy\-level evaluation includes both changed outputs and unchanged fallbacks\.
### 3\.7Independent Evaluation
LetJeJ\_\{e\}denote the*independent evaluation judge*\. The evaluator does not generate rewrite candidates, define admissibility, or select policy outputs\. After all final policy outputsYiπY\_\{i\}^\{\\pi\}are finalized, message order is deterministically randomized\. The evaluator receives no information about the policy, demographic axis, audience labels, source model, PBI values, acceptance decision, or selection\-judge score\. For outcomeq∈\{dim,overall\}q\\in\\\{\\mathrm\{dim\},\\mathrm\{overall\}\\\}the policy effect relative to reference policyπ0\\pi\_\{0\}is
Δq\(π,π0\)=1N∑i=1N\[dJe,q\(Yiπ\)−dJe,q\(Yiπ0\)\]\\Delta\_\{q\}\(\\pi,\\pi\_\{0\}\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\[d\_\{J\_\{e\},q\}\(Y\_\{i\}^\{\\pi\}\)\-d\_\{J\_\{e\},q\}\(Y\_\{i\}^\{\\pi\_\{0\}\}\)\\right\]Negative values indicate lower independently judged framing disparity under policyπ\\pi\. The primary estimand is intention\-to\-treat at the policy level\. All pairs remain in the analysis, including cases in which a rejected candidate causes the policy to return the original pair\. Analyses restricted to changed outputs condition on realized intervention and are therefore secondary and descriptive\.
### 3\.8Transfer Questions
We ask three transfer questions\.Objective transfer: does lowering the PBI gap also lower the pairwise framing outcomes?Evaluator transfer: does an improvement under the selection judge remain visible to an independent evaluator that took no part in rewriting or acceptance?Policy transfer: with the candidate pool and admissibility constraints held fixed, do different edit\-direction policies yield different independent outcomes,Δq\(π1,π0\)≠Δq\(π2,π0\)\\Delta\_\{q\}\(\\pi\_\{1\},\\pi\_\{0\}\)\\neq\\Delta\_\{q\}\(\\pi\_\{2\},\\pi\_\{0\}\)?
### 3\.9Preservation and Intervention Exposure
For every changed message we compute lexical, embedding and bidirectional NLI diagnostics, which are automated screens rather than guarantees of preserved meaning\. We report them at two levels:conditional edit qualityamong changed messages, andpolicy\-level exposure, the share of all pairs with at least one flagged edit\. Exposure treats unchanged fallbacks as unexposed, so two policies with similar edit quality can differ in exposure because they intervene at different rates\.
Table 1:Independent Gemma evaluation relative to Original\. Negative differences indicate lower disparity; bold denotes Holm\-adjustedp<\.05p<\.05within the comparison family for that axis and outcome\.
## 4Methods and Experimental Setup
### 4\.1Data and Matched Pairs
We use the context\-rich climate\-persuasion messages generated by[Islam \(2026b\)](https://arxiv.org/html/2611.25258#bib.bib18)with GPT\-4o, Llama\-3\.3, and Mistral\-Large\-2\.1\. We study gender \(male vs\. female\) and four age groups: Young Adult \(18–24\), Early Working Age Group \(25–44\), Late Working Age Group \(45–64\), and Senior \(65\+\)\. Age groups are adopted from[Islam and Goldwasser \(2025b\)](https://arxiv.org/html/2611.25258#bib.bib20)\.
For gender, we hold source model, stance, theme, region, generation setting, and age fixed while varying gender, yielding 660 matched pairs\. For age, each of 330 fixed non\-age contexts contributes all six pairwise contrasts among its four age\-targeted messages, yielding 1,980 rows\. Because the six age contrasts reuse the same underlying messages, inference is clustered over the 330 contexts\.
### 4\.2Candidate Generation and Policy Construction
Qwen2\.5\-7B\-Instruct[Team \(2024\)](https://arxiv.org/html/2611.25258#bib.bib53)generates the directional candidates defined in Section[3\.4](https://arxiv.org/html/2611.25258#S3.SS4)\. For every pair with unequal original PBI scores, we independently generate one candidate editing Message A and one editing Message B\. Generation is greedy, and both prompts target the arithmetic midpoint of the original message\-level PBI scores while requesting minimal changes to stance, claims, topic, and framing\.
DeepSeek\-V3\.1[Liu et al\. \(2024\)](https://arxiv.org/html/2611.25258#bib.bib34)serves as theselection judge\. It scores the Original, Edit A, Edit B, and Edit both configurations on assertiveness, certainty, directive force, and warmth\. We apply the admissibility rule in Section[3\.5](https://arxiv.org/html/2611.25258#S3.SS5)uniformly across demographic axes and policies, withτ=0\.10\\tau=0\.10,ηr=ηo=0\.60\\eta\_\{r\}=\\eta\_\{o\}=0\.60, and\[ℓmin,ℓmax\]=\[0\.80,1\.25\]\[\\ell\_\{\\min\},\\ell\_\{\\max\}\]=\[0\.80,1\.25\]\. Thresholds are fixed before independent evaluation; App\.[D\.3](https://arxiv.org/html/2611.25258#A4.SS3)sweeps them \(Tables[21](https://arxiv.org/html/2611.25258#A4.T21)and[22](https://arxiv.org/html/2611.25258#A4.T22)\)\.
For RQ1, we additionally evaluate a fixed\-direction pipeline that generates and considers only a rewrite of the lower\-PBI message\. This pipeline uses a separate Qwen candidate\-generation run from the bidirectional candidate pool used for RQ2\. Although both implement a Lower\-PBI edit rule, their outcomes should not be compared directly because they use separately generated candidate pools\.
For RQ2, we construct Lower\-PBI, Higher\-PBI, Random\-single, Adaptive\-single, and Both\-sides policies from the same bidirectional candidate pool\. Original is the no\-edit reference, Lower\-PBI is the principal controlled baseline, and Adaptive\-single is the proposed policy\.
### 4\.3Independent Evaluation
After policy selection, all final outputs are finalized\. For RQ2, exact duplicate text pairs are evaluated once and mapped back to their arms; RQ1 conditions are scored separately \(App\.[B\.5](https://arxiv.org/html/2611.25258#A2.SS5)\)\. In our experiment,Gemma\-4\-31B\-IT[Team et al\. \(2026\)](https://arxiv.org/html/2611.25258#bib.bib52)serves as theindependent evaluator\. Message order is deterministically randomized, and Gemma receives only the two messages, without demographic, policy, source\-model, PBI, acceptance, or selection\-score metadata\.
Gemma assigns the four dimension ratings, a separate direct overall\-disparity rating, and a content\-difference rating\. The direction\-policy evaluation contains 1,074 unique gender pairs and 3,196 unique age pairs, corresponding to 3,960 and 11,880 policy mappings, respectively\. Model roles are in Table[5](https://arxiv.org/html/2611.25258#A1.T5)\(App\.[A\.1](https://arxiv.org/html/2611.25258#A1.SS1)\); the rewrite prompts in Figs\.[2](https://arxiv.org/html/2611.25258#A1.F2)and[3](https://arxiv.org/html/2611.25258#A1.F3)\(App\.[A\.2](https://arxiv.org/html/2611.25258#A1.SS2)\); the selection\-judge prompts in Figs\.[4](https://arxiv.org/html/2611.25258#A1.F4)and[5](https://arxiv.org/html/2611.25258#A1.F5)\(App\.[A\.3](https://arxiv.org/html/2611.25258#A1.SS3)\); the independent\-evaluator prompt in Fig\.[6](https://arxiv.org/html/2611.25258#A1.F6)\(App\.[A\.5](https://arxiv.org/html/2611.25258#A1.SS5)\); and selection and deduplication counts in Table[12](https://arxiv.org/html/2611.25258#A3.T12)\(App\.[C\.2](https://arxiv.org/html/2611.25258#A3.SS2)\) and Table[13](https://arxiv.org/html/2611.25258#A3.T13)\(App\.[C\.3](https://arxiv.org/html/2611.25258#A3.SS3)\)\. Further implementation details are in App\.[A](https://arxiv.org/html/2611.25258#A1)\.
#### Evaluator variations\.
Holding the finalized outputs, blinding, order randomization and decoding fixed, we rescore every RQ1, RQ2 and RQ3 item with \(i\) a second evaluator family,allenai/Olmo\-3\.1\-32B\-Instruct[Olmo et al\. \(2025\)](https://arxiv.org/html/2611.25258#bib.bib38), which appears nowhere in generation or selection, and the RQ1 items also with \(ii\) an independently paraphrased prompt \(Fig\.[7](https://arxiv.org/html/2611.25258#A2.F7), App\.[B\.7](https://arxiv.org/html/2611.25258#A2.SS7)\) and \(iii\) the prompt without the sentence*“Do not infer the intended audience or demographic group”*\. Each prompt variant is paired with a fresh run of the unmodified prompt, so only within\-run paired contrasts are compared \(App\.[B\.6](https://arxiv.org/html/2611.25258#A2.SS6)and[B\.7](https://arxiv.org/html/2611.25258#A2.SS7)\)\.
### 4\.4Preservation and Statistical Analysis
The acceptance rule uses ROUGE\-L\([Lin, 2004](https://arxiv.org/html/2611.25258#bib.bib32)\), token overlap, and length ratio\. After outputs are finalized, we audit changed messages using MPNet cosine similarity and bidirectional entailment and contradiction scores from DeBERTa\-v3\-base\-mnli\-fever\-anli\([Laurer et al\., 2024](https://arxiv.org/html/2611.25258#bib.bib28)\)\. We report conditional diagnostics among changed messages and unconditional policy\-level exposure to at least one prespecified risk flag\. These automated flags are diagnostic screens rather than validated semantic\-failure labels\.
All comparisons are paired and use the intention\-to\-treat population, including unchanged fallbacks\. We report mean paired differences, 95% percentile cluster\-bootstrap confidence intervals, and two\-sided clustered sign\-flip tests\. Gender inference operates over 660 pairs; age inference resamples and permutes the 330 underlying contexts\. Holm correction\([Holm, 1979](https://arxiv.org/html/2611.25258#bib.bib16)\)is applied within each prespecified axis, outcome, reference\-policy, and analysis family\. Negative differences indicate lower independently judged disparity\. Additional preservation metrics and diagnostic flag definitions are provided in App\.[D\.1](https://arxiv.org/html/2611.25258#A4.SS1), while full statistical procedures are described in App\.[D\.4](https://arxiv.org/html/2611.25258#A4.SS4)\.
Table 2:Fixed Lower\-PBI policy relative to Original under two evaluator families\.DmD\_\{m\}: Dimension mean,OdO\_\{d\}: Overall disparity\.
## 5Results
### 5\.1RQ1: Evaluator Transfer Is Partial
The fixed Lower\-PBI pipeline improves its selection\-judge dimension score by construction; the relevant test is whether this improvement transfers to independent evaluation \(Table[6](https://arxiv.org/html/2611.25258#A2.T6), App\.[B\.2](https://arxiv.org/html/2611.25258#A2.SS2)\)\. For gender, the independently judged dimension mean decreases by0\.00980\.0098, but the interval does not exclude zero \(95% CI\[−0\.0208,0\.0004\]\[\-0\.0208,\\,0\.0004\], Holm\-adjustedp=\.3860p=\.3860\), whereas direct overall disparity decreases by0\.02420\.0242\(\[−0\.0379,−0\.0106\]\[\-0\.0379,\-0\.0106\],p=\.0081p=\.0081\)\. For age, both outcomes improve: the dimension mean by0\.01630\.0163\(\[−0\.0225,−0\.0104\]\[\-0\.0225,\-0\.0104\],p=\.0001p=\.0001\) and direct overall disparity by0\.01870\.0187\(\[−0\.0278,−0\.0096\]\[\-0\.0278,\-0\.0096\],p=\.0002p=\.0002\)\.
Thus, selection\-time improvement does not transfer uniformly across demographic axes and outcome definitions: three of the four cells resolve, and the one that does not is the gender dimension mean\. At the dimension level, the gender effect is carried by warmth, the only gender dimension whose interval excludes zero \(−0\.0273\-0\.0273,\[−0\.0470,−0\.0091\]\[\-0\.0470,\-0\.0091\]\), while for age directive force and warmth give the largest reductions \(−0\.0242\-0\.0242and−0\.0237\-0\.0237\)\. Changed\-output analyses yield larger descriptive effects but condition on realized intervention, so the all\-pairs analysis remains primary; intention\-to\-treat, changed\-output and sensitivity results are in Table[6](https://arxiv.org/html/2611.25258#A2.T6)\(App\.[B\.2](https://arxiv.org/html/2611.25258#A2.SS2)\), Table[7](https://arxiv.org/html/2611.25258#A2.T7)\(App\.[B\.3](https://arxiv.org/html/2611.25258#A2.SS3)\) and Tables[8](https://arxiv.org/html/2611.25258#A2.T8)and[9](https://arxiv.org/html/2611.25258#A2.T9)\(App\.[B\.4](https://arxiv.org/html/2611.25258#A2.SS4)\)\.
#### Evaluator\-family transfer\.
We rescored every finalized item with a second evaluator family,allenai/Olmo\-3\.1\-32B\-Instruct, which plays no other part in the pipeline\. Item\-level agreement between the two evaluators is low \(exact agreement 24\.6–69\.0%, QWK 0\.08–0\.54; Table[10](https://arxiv.org/html/2611.25258#A2.T10), App\.[B\.6](https://arxiv.org/html/2611.25258#A2.SS6)\), and Olmo rates every field higher, by\+0\.06\+0\.06to\+0\.78\+0\.78\. The paired contrasts nonetheless transfer \(Table[2](https://arxiv.org/html/2611.25258#S4.T2)\): all four Lower\-PBI effects keep their direction under Olmo and all four intervals exclude zero, including the gender dimension mean\. All four also keep their direction under both prompt variations \(Table[11](https://arxiv.org/html/2611.25258#A2.T11), App\.[B\.7](https://arxiv.org/html/2611.25258#A2.SS7)\)\. A constant per\-evaluator offset cancels in a paired difference but dominates absolute levels, so we read only within\-run paired contrasts as evidence\.
### 5\.2RQ2: Adaptive Direction Selection Outperforms Lower\-PBI Editing, Largely Through Coverage
Table[1](https://arxiv.org/html/2611.25258#S3.T1)reports independent comparisons for policies constructed from the same bidirectional candidate pool\. For gender, Lower\-PBI reduces direct overall disparity by0\.01670\.0167\(p=\.0143p=\.0143\) but leaves the dimension mean unresolved\. Adaptive\-single reduces the dimension mean by0\.02160\.0216and overall disparity by0\.03180\.0318; Both\-sides reduces them by0\.02730\.0273and0\.03480\.0348\. Relative to Lower\-PBI, both policies significantly improve the dimension mean \(Adaptive\-single−0\.0125\-0\.0125,p=\.0251p=\.0251; Both\-sides−0\.0182\-0\.0182,p=\.0014p=\.0014\), while neither advantage on direct overall disparity survives correction \(−0\.0152\-0\.0152and−0\.0182\-0\.0182, bothp=\.1234p=\.1234\)\.
For age, all displayed editing policies improve over Original on both outcomes\. Relative to Lower\-PBI, Adaptive\-single provides additional reductions of0\.00810\.0081in the dimension mean \(p=\.0017p=\.0017\) and0\.01110\.0111in overall disparity \(p=\.0114p=\.0114\); Both\-sides provides0\.01210\.0121\(p=\.0002p=\.0002\) and0\.01360\.0136\(p=\.0114p=\.0114\)\.
Adaptive\-single therefore improves both independently judged outcomes on both axes and beats Lower\-PBI in three of four direct comparisons, all but gender overall disparity\. Both\-sides matches or exceeds it on every point estimate while editing two messages of far fewer pairs, but the difference is resolved only on the age axis under the second evaluator family \(Table[17](https://arxiv.org/html/2611.25258#A3.T17), App\.[C\.5](https://arxiv.org/html/2611.25258#A3.SS5)\) and is otherwise inconclusive\. Crucially, a coverage\-matched control that edits exactly the pairs Adaptive\-single edits, but averages over admissible one\-sided configurations instead of using the selection judge’s choice, reproduces 63\.6% \(gender\) and 57\.0% \(age\) of Adaptive\-single’s dimension\-mean advantage over Lower\-PBI, and 85\.0% and 88\.6% of its direct\-overall advantage\. Guided direction choice contributes a smaller residual that is resolved on the dimension mean on both axes \(gender−0\.0045\-0\.0045,p=\.0282p=\.0282; age−0\.0035\-0\.0035,p=\.0006p=\.0006\) and on neither direct overall rating, and it can only act on the 8\.2% of gender pairs and 6\.3% of age contrasts where both directions are admissible\. Most of the adaptive rule’s benefit is therefore the wider licence to intervene, not the direction it picks\. Complete Higher\-PBI, Random\-single and control comparisons are reported in Tables[14](https://arxiv.org/html/2611.25258#A3.T14)and[15](https://arxiv.org/html/2611.25258#A3.T15)\(App\.[C\.3](https://arxiv.org/html/2611.25258#A3.SS3)\)\.
#### The policy comparison transfers across evaluator families\.
Adaptive\-single minus Lower\-PBI is negative in all eight axis×\\timesoutcome×\\timesevaluator cells and its interval excludes zero in seven \(Table[3](https://arxiv.org/html/2611.25258#S5.T3); every policy under both families is in Table[16](https://arxiv.org/html/2611.25258#A3.T16), App\.[C\.4](https://arxiv.org/html/2611.25258#A3.SS4)\)\. The exception is gender overall disparity under Olmo \(−0\.0137\-0\.0137,\[−0\.0303,0\.0030\]\[\-0\.0303,0\.0030\]\)\. Table[3](https://arxiv.org/html/2611.25258#S5.T3)Holm\-adjusts only across the two outcomes of this contrast, so Gemma’s gender overall cell is significant there \(p=\.031p=\.031\) but not under the four\-arm correction used above \(p=\.1234p=\.1234\)\. The direction of the claim transfers across families while its statistical support does not transfer identically; neither evaluator is ground truth, and a conclusion that depends on a single evaluator should be reported as such\.
Table 3:Adaptive\-single minus Lower\-PBI under both independent evaluator families\. Negative values favour Adaptive\-single\.DmD\_\{m\}: Dimension mean,OdO\_\{d\}: Overall disparity\.
### 5\.3Intervention and Preservation Trade\-offs
Adaptive\-single changes 36\.7% of gender pairs and 39\.3% of age contrasts, against 17\.9% and 15\.8% for Both\-sides, although Both\-sides edits two messages when accepted \(0\.358 vs\. 0\.367 changed messages per pair for gender, 0\.316 vs\. 0\.393 for age; pair\-level movement in Table[18](https://arxiv.org/html/2611.25258#A3.T18), App\.[C\.6](https://arxiv.org/html/2611.25258#A3.SS6)\)\. Conditional embedding similarity is comparable across policies \(0\.887–0\.904\), but NLI\-based flags are frequent, so lexical admissibility does not guarantee semantic equivalence\. Because it intervenes more often, Adaptive\-single exposes more of all pairs to at least one flag \(24\.1%/23\.0% for gender/age\) than Both\-sides \(15\.5%/14\.0%; Tables[19](https://arxiv.org/html/2611.25258#A4.T19)and[20](https://arxiv.org/html/2611.25258#A4.T20), App\.[D\.2](https://arxiv.org/html/2611.25258#A4.SS2)\)\. These rates are diagnostic screens, not estimates of semantic\-failure prevalence\.
### 5\.4RQ3: PBI and Framing Rank Methods Differently
Table[27](https://arxiv.org/html/2611.25258#A5.T27)\(App\.[E](https://arxiv.org/html/2611.25258#A5)\) compares three full\-coverage, PBI\-targeted baselines that are not selected using the reported framing score\. Lexical edit makes rule\-based changes to PBI\-associated features; Unfiltered Qwen applies every generated rewrite; and Filtered Qwen retains only rewrites satisfying the PBI and lexical\-preservation constraints\. Lexical edit, for example, cuts the mean PBI gap by more than half \(gender0\.727→0\.3090\.727\\rightarrow 0\.309, age0\.744→0\.2980\.744\\rightarrow 0\.298\) while the selection judge’s framing score rises \(1\.047→1\.1621\.047\\rightarrow 1\.162and1\.078→1\.2021\.078\\rightarrow 1\.202\); Unfiltered and Filtered Qwen show smaller versions of the same mismatch\. A selective\-abstention diagnostic is in App\.[F](https://arxiv.org/html/2611.25258#A6)\(Table[28](https://arxiv.org/html/2611.25258#A6.T28)\)\.
Ranking these interventions by PBI therefore does not reproduce their ranking under the pairwise framing outcome, and the mismatch is not a property of the selection judge: under two evaluators that took no part in producing or selecting the outputs, each PBI\-targeted condition reduces the PBI gap but significantly increases the independently measured dimension mean, on both axes and after Holm correction \(Table[4](https://arxiv.org/html/2611.25258#S5.T4)\)\. This is not evidence that PBI and framing are independent constructs, and it establishes nothing about what a human reader would perceive\.
Table 4:Objective transfer under two independent evaluator families, each condition relative to Original on the same matched pairs\.ΔPBIgp\\Delta\_\{PBI\_\{gp\}\}: change in mean absolute PBI gap \(negative means the optimized objective improves\)\.ΔFS\\Delta\_\{FS\}: change in the four\-dimension framing score \(positive means framing disparity worsens\) under the selection judge \(FSsFS\_\{s\}\), Gemma \(FSgFS\_\{g\}\) and Olmo \(FSoFS\_\{o\}\)\. Bold: 95% clustered\-bootstrap interval excluding zero with Holm\-adjustedp<\.0001p<\.0001\. Lex\_ed: Lexical edit, Qwen\_un: Qwen \(unfiltered\), Qwen\_fil: Qwen \(filtered\)\.
### 5\.5Qualitative and Human Diagnostics
Structured qualitative analysis of prespecified error strata finds successful transfer alongside content drift, persuasion amplification and unnatural output; one accepted rewrite, for example, changes the addressed age group and a substantive recommendation, so numeric admissibility does not guarantee content preservation \(Table[29](https://arxiv.org/html/2611.25258#A6.T29), App\.[D\.5](https://arxiv.org/html/2611.25258#A4.SS5)\)\. An exploratory single\-annotator study of 160 blinded pairs finds weak to modest human–LLM correlations \(ρ=0\.21\\rho=0\.21–0\.350\.35\) and is not used for any primary claim \(App\.[D\.6](https://arxiv.org/html/2611.25258#A4.SS6), Table[23](https://arxiv.org/html/2611.25258#A4.T23), Fig\.[8](https://arxiv.org/html/2611.25258#A4.F8)\)\.
### 5\.6Signed Directional Diagnostics
All framing outcomes above are magnitudes, so we recover direction from the deterministic per\-message lexical scores\. On the gender axis, messages for the male audience carry higher PBI \(\+0\.138\+0\.138\) and certainty \(\+0\.129\+0\.129\), and those for the female audience higher lexical warmth \(−0\.075\-0\.075\)\. On the age axis, lexical warmth rises monotonically with audience age in all six contrasts, while signed PBI excludes zero in none \(Tables[24](https://arxiv.org/html/2611.25258#A4.T24),[25](https://arxiv.org/html/2611.25258#A4.T25)and[26](https://arxiv.org/html/2611.25258#A4.T26), App\.[D\.7](https://arxiv.org/html/2611.25258#A4.SS7)\)\. An absolute\-gap objective can thus be reduced while a consistent directional asymmetry remains, including one carried by warmth, which PBI does not contain\.
## 6Discussion and Conclusion
Post\-hoc demographic mitigation is a transfer problem rather than the optimization of a single score\. PBI, the equal\-weight framing mean and the direct overall judgment rank interventions differently: PBI\-targeted edits that lower the PBI gap raise independently judged dimension\-mean disparity under two evaluator families, and improvements under a selection judge do not transfer uniformly to an independent evaluator\. Mitigation claims should therefore name the operational construct they target and test whether they persist under outcomes and evaluators that did not influence selection\.
Edit direction matters, though mostly through coverage: a coverage\-matched control attributes most of Adaptive\-single’s gain over fixed Lower\-PBI editing to the wider licence to intervene, and Both\-sides reaches similar effects by editing both messages of fewer pairs\. These gains carry preservation risk, since the policy that intervenes more exposes more pairs to flagged edits\. Selection scores alone are therefore insufficient evidence of mitigation; reliable assessment needs independent evaluation, clustered uncertainty, aggregation sensitivity and explicit preservation auditing, and reducing an operational disparity score does not by itself establish substantive fairness\.
## 7Limitations
Our study is limited to climate\-related messages, binary gender contrasts, and four broad age categories inherited from the source dataset\. The findings may not generalize to other domains, demographic representations, or intersectional settings\. The source corpus was generated by three model families, so our results show that objective and evaluator transfer failures*can*occur in this controlled post\-hoc setting; they do not establish that their magnitude or the policy ranking persists across domains, genres or richer demographic representations\.
The framing outcomes are operational measures rather than validated fairness constructs, and the independent evaluator provides procedural independence from selection rather than human\-validated ground truth\. Our exploratory annotation study has a single annotator, so it cannot establish inter\-annotator reliability or construct validity, and its stratified design with disjoint policy arms cannot estimate a policy effect \(App\.[D\.6](https://arxiv.org/html/2611.25258#A4.SS6)\)\.
Preservation is evaluated using automated lexical, embedding, and NLI diagnostics rather than comprehensive human assessment\. These measures provide useful evidence of potential content drift but are not definitive measures of semantic preservation\.
Finally, post\-hoc equalization may conceal upstream representational harms or reduce disparity through persuasion amplification\. We therefore view the proposed framework as an auditing and constrained\-editing procedure rather than as evidence of substantive system fairness\.
#### Statistical detectability is not practical benefit\.
The policy effects are small, about0\.010\.01–0\.030\.03on a00–33scale\. They are detectable because pairing and clustered averaging remove item\-stable variation and the evaluator is nearly deterministic on identical input \(App\.[B\.5](https://arxiv.org/html/2611.25258#A2.SS5)\), not because individual messages change much; most pairs are never edited \(App\.[B\.3](https://arxiv.org/html/2611.25258#A2.SS3)\)\. None of these quantities measures practical benefit, which would require evidence about human perception or recipient behaviour that we do not collect\. Our contribution is the protocol and the transfer findings, not a claim that the measured reductions matter to a reader\.
#### Absolute levels are not portable\.
Every absolute framing\-disparity level we report is conditional on an evaluator family and a prompt formulation\. Changing the evaluator family moves absolute ratings by up to\+0\.78\+0\.78on a00–33scale and paraphrasing the prompt moves them by0\.040\.04–0\.270\.27; both exceed the policy effects under study\. Only paired contrasts in which a single evaluator scores every arm, so that the common offset cancels in the difference, should be compared, and we do not pool scores from different evaluators or prompts\.
#### One unresolved cell\.
The gender dimension mean under the fixed Lower\-PBI policy is the least stable result in the paper\. Its interval includes zero in the primary analysis and under the paraphrased evaluation prompt \(App\.[B\.7](https://arxiv.org/html/2611.25258#A2.SS7)\), although it excludes zero under the second evaluator family\. The corresponding cell for the Lower\-PBI policy built from the separately generated RQ2 pool is likewise unresolved in the primary analysis and excludes zero only at the loosest preservation threshold,η=0\.50\\eta=0\.50\(App\.[D\.3](https://arxiv.org/html/2611.25258#A4.SS3)\)\. We report both as unresolved rather than as established effects, and no conclusion in the paper rests on them\.
## 8Ethical Considerations
Demographic targeting and persuasive generation can be used for manipulation as well as benign communication\. Our experiments audit already\-generated messages and do not deploy targeting or measure persuasion on recipients\.
A lower pairwise disparity score does not imply that either message is truthful, appropriate, non\-manipulative, or fair\. Post\-hoc editing may also increase the persuasive intensity of one or both messages, which we measure explicitly\.
The annotation materials are blinded and contain no annotator identifiers\. The annotation is conducted in\-house with the annotator’s informed consent\.
## Acknowledgments
The author acknowledges the National Artificial Intelligence Research Resource \(NAIRR\) Pilot and Purdue Anvil for contributing to this research result \(award NAIRR260365\)\. Anvil is supported by the National Science Foundation under Grant No\. 2005632\([Song et al\., 2022](https://arxiv.org/html/2611.25258#bib.bib47)\)\. We thank Lightning AI for the compute used for selection\-judge scoring\. We thank Samantha Rose Sudhoff for the human annotation, and the anonymous reviewers and area chair for their constructive feedback\.
## References
- Abels and Lenaerts \(2025\)Axel Abels and Tom Lenaerts\. 2025\.Wisdom from diversity: bias mitigation through hybrid human\-llm crowds\.In*Proceedings of the Thirty\-Fourth International Joint Conference on Artificial Intelligence*, pages 321–329\.
- Aghaebe et al\. \(2025\)Favour Y Aghaebe, Elizabeth A Williams, Tanefa Apekey, and Nafise Sadat Moosavi\. 2025\.Llms do not see age: Assessing demographic bias in automated systematic review synthesis\.In*Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics*, pages 1815–1833\.
- Banerjee et al\. \(2024\)Pragyan Banerjee, Abhinav Java, Surgan Jandial, Simra Shahid, Shaz Furniturewala, Balaji Krishnamurthy, and Sumit Bhatia\. 2024\.All should be equal in the eyes of lms: Counterfactually aware fair text generation\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pages 17673–17681\.
- Brady and Islam \(2026\)Alexander Brady and Tunazzina Islam\. 2026\.[Iterative topic taxonomy induction with LLMs: A case study of electoral advertising](https://doi.org/10.18653/2026.findings-aacl.99)\.In*Findings of the Association for Computational Linguistics: IJCNLP\-AACL 2026*, pages 1976–1990, Hengqin, China\. Association for Computational Linguistics\.
- Breum et al\. \(2024\)Simon Martin Breum, Daniel Vædele Egdal, Victor Gram Mortensen, Anders Giovanni Møller, and Luca Maria Aiello\. 2024\.The persuasive power of large language models\.In*Proceedings of the International AAAI Conference on Web and Social Media*, volume 18, pages 152–163\.
- Brown et al\. \(2020\)Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al\. 2020\.Language models are few\-shot learners\.*Advances in neural information processing systems*, 33:1877–1901\.
- Chiang and Lee \(2023\)Cheng\-Han Chiang and Hung\-yi Lee\. 2023\.Can large language models be an alternative to human evaluations?In*The 61st Annual Meeting Of The Association For Computational Linguistics*\.
- Chouldechova \(2017\)Alexandra Chouldechova\. 2017\.Fair prediction with disparate impact: A study of bias in recidivism prediction instruments\.*Big data*, 5\(2\):153–163\.
- Dai et al\. \(2024\)Sunhao Dai, Chen Xu, Shicheng Xu, Liang Pang, Zhenhua Dong, and Jun Xu\. 2024\.Bias and unfairness in information retrieval systems: New challenges in the llm era\.In*Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, pages 6437–6447\.
- Dinan et al\. \(2020\)Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston\. 2020\.Queens are powerful too: Mitigating gender bias in dialogue generation\.In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 8173–8188\.
- Dixon et al\. \(2018\)Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman\. 2018\.Measuring and mitigating unintended bias in text classification\.In*Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society*, pages 67–73\.
- Esiobu et al\. \(2023\)David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi\-Yu, Eleonora Presani, Adina Williams, and Eric Smith\. 2023\.Robbie: Robust bias evaluation of large generative language models\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 3764–3814\.
- Fang et al\. \(2024\)Xiao Fang, Shangkun Che, Minjia Mao, Hongzhe Zhang, Ming Zhao, and Xiaohang Zhao\. 2024\.Bias of ai\-generated content: an examination of news produced by large language models\.*Scientific Reports*, 14\(1\):5224\.
- Garimella et al\. \(2022\)Aparna Garimella, Rada Mihalcea, and Akhash Amarnath\. 2022\.Demographic\-aware language model fine\-tuning as a bias mitigation technique\.In*Proceedings of the 2nd Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing \(Volume 2: Short Papers\)*, pages 311–319\.
- Hackenburg and Margetts \(2024\)Kobi Hackenburg and Helen Margetts\. 2024\.Evaluating the persuasive influence of political microtargeting with large language models\.*Proceedings of the National Academy of Sciences*, 121\(24\):e2403116121\.
- Holm \(1979\)Sture Holm\. 1979\.A simple sequentially rejective multiple test procedure\.*Scandinavian journal of statistics*, pages 65–70\.
- Islam \(2026a\)Tunazzina Islam\. 2026a\.[Reasoning\-based refinement of unsupervised text clusters with LLMs](https://doi.org/10.18653/v1/2026.findings-acl.482)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 9899–9917, San Diego, California, United States\. Association for Computational Linguistics\.
- Islam \(2026b\)Tunazzina Islam\. 2026b\.[Who gets which message? auditing demographic bias in LLM\-generated targeted text](https://doi.org/10.18653/v1/2026.findings-acl.452)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 9270–9297, San Diego, California, United States\. Association for Computational Linguistics\.
- Islam and Goldwasser \(2025a\)Tunazzina Islam and Dan Goldwasser\. 2025a\.Discovering latent themes in social media messaging: A machine\-in\-the\-loop approach integrating llms\.In*Proceedings of the International AAAI Conference on Web and Social Media*, volume 19, pages 859–884\.
- Islam and Goldwasser \(2025b\)Tunazzina Islam and Dan Goldwasser\. 2025b\.[Post\-hoc study of climate microtargeting on social media ads with LLMs: Thematic insights and fairness evaluation](https://doi.org/10.18653/v1/2025.findings-emnlp.857)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 15838–15859, Suzhou, China\. Association for Computational Linguistics\.
- Islam and Goldwasser \(2025c\)Tunazzina Islam and Dan Goldwasser\. 2025c\.[Uncovering latent arguments in social media messaging by employing LLMs\-in\-the\-loop strategy](https://doi.org/10.18653/v1/2025.findings-naacl.413)\.In*Findings of the Association for Computational Linguistics: NAACL 2025*, pages 7397–7429, Albuquerque, New Mexico\. Association for Computational Linguistics\.
- Jin et al\. \(2021\)Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, and Xiang Ren\. 2021\.On transferability of bias mitigation effects in language model fine\-tuning\.In*Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 3770–3783\.
- Kleinberg et al\. \(2016\)Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan\. 2016\.Inherent trade\-offs in the fair determination of risk scores\.*arXiv preprint arXiv:1609\.05807*\.
- Kotek et al\. \(2023\)Hadas Kotek, Rikker Dockum, and David Sun\. 2023\.Gender bias and stereotypes in large language models\.In*Proceedings of the ACM collective intelligence conference*, pages 12–24\.
- Krause et al\. \(2021\)Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani\. 2021\.Gedi: Generative discriminator guided sequence generation\.In*Findings of the Association for Computational Linguistics: EMNLP 2021*, pages 4929–4952\.
- Kumar et al\. \(2023\)Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov\. 2023\.Language generation models can cause harm: So what can we do about it? an actionable survey\.In*Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics*, pages 3299–3321\.
- Kusner et al\. \(2017\)Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva\. 2017\.Counterfactual fairness\.*Advances in neural information processing systems*, 30\.
- Laurer et al\. \(2024\)Moritz Laurer, Wouter Van Atteveldt, Andreu Casas, and Kasper Welbers\. 2024\.Less annotating, more classifying: Addressing the data scarcity issue of supervised machine learning with deep transfer learning and bert\-nli\.*Political Analysis*, 32\(1\):84–100\.
- Li et al\. \(2024\)Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu\. 2024\.Llms\-as\-judges: a comprehensive survey on llm\-based evaluation methods\.*arXiv preprint arXiv:2412\.05579*\.
- Li et al\. \(2023a\)Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang\. 2023a\.A survey on fairness in large language models\.*arXiv preprint arXiv:2308\.10149*\.
- Li et al\. \(2023b\)Yunqi Li, Lanjing Zhang, and Yongfeng Zhang\. 2023b\.Fairness of chatgpt\.*arXiv preprint arXiv:2305\.18569*\.
- Lin \(2004\)Chin\-Yew Lin\. 2004\.[ROUGE: A package for automatic evaluation of summaries](https://aclanthology.org/W04-1013/)\.In*Text Summarization Branches Out*, pages 74–81, Barcelona, Spain\. Association for Computational Linguistics\.
- Lin et al\. \(2022\)Stephanie Lin, Jacob Hilton, and Owain Evans\. 2022\.Truthfulqa: Measuring how models mimic human falsehoods\.In*Proceedings of the 60th annual meeting of the association for computational linguistics \(volume 1: long papers\)*, pages 3214–3252\.
- Liu et al\. \(2024\)Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al\. 2024\.Deepseek\-v3 technical report\.*arXiv preprint arXiv:2412\.19437*\.
- Liu et al\. \(2025\)Haoran Liu, Amir Tahmasbi, Ehtesham Sam Haque, and Purak Jain\. 2025\.Llms for customized marketing content generation and evaluation at scale\.*arXiv preprint arXiv:2506\.17863*\.
- Liu et al\. \(2023\)Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu\. 2023\.G\-eval: Nlg evaluation using gpt\-4 with better human alignment\.In*Proceedings of the 2023 conference on empirical methods in natural language processing*, pages 2511–2522\.
- Masoudian et al\. \(2024\)Shahed Masoudian, Cornelia Volaucnik, Markus Schedl, and Navid Rekabsaz\. 2024\.[Effective controllable bias mitigation for classification and retrieval using gate adapters](https://doi.org/10.18653/v1/2024.eacl-long.150)\.In*Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 2434–2453, St\. Julian’s, Malta\. Association for Computational Linguistics\.
- Olmo et al\. \(2025\)Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, and 49 others\. 2025\.[Olmo 3](https://arxiv.org/abs/2512.13961)\.*Preprint*, arXiv:2512\.13961\.
- Pauli et al\. \(2025\)Amalie Brogaard Pauli, Isabelle Augenstein, and Ira Assent\. 2025\.Measuring and benchmarking large language models’ capabilities to generate persuasive language\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 10056–10075\.
- Pauli et al\. \(2026\)Amalie Brogaard Pauli, Maria Barrett, Max Müller\-Eberstein, Isabelle Augenstein, and Ira Assent\. 2026\.Analysing differences in persuasive language in llm\-generated text: Uncovering stereotypical gender patterns\.*arXiv preprint arXiv:2601\.05751*\.
- Rosenblatt and Witter \(2023\)Lucas Rosenblatt and R Teal Witter\. 2023\.Counterfactual fairness is basically demographic parity\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 37, pages 14461–14469\.
- Sag \(2023\)Matthew Sag\. 2023\.Fairness and fair use in generative ai\.*Fordham Law Review, Forthcoming*\.
- Schick et al\. \(2021\)Timo Schick, Sahana Udupa, and Hinrich Schütze\. 2021\.Self\-diagnosis and self\-debiasing: A proposal for reducing corpus\-based bias in nlp\.*Transactions of the Association for Computational Linguistics*, 9:1408–1424\.
- Sclar et al\. \(2024\)Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr\. 2024\.Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting\.In*International Conference on Learning Representations*, volume 2024, pages 25055–25083\.
- Sethi et al\. \(2025\)Sankalp Sethi, Joni Salminen, Danial Amin, and Bernard J Jansen\. 2025\." when ai writes personas": Analyzing lexical diversity in llm\-generated persona descriptions\.In*Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems*, pages 1–8\.
- Shi et al\. \(2025\)Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi\. 2025\.Judging the judges: A systematic study of position bias in llm\-as\-a\-judge\.In*Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics*, pages 292–314\.
- Song et al\. \(2022\)X\. Carol Song, Preston Smith, Rajesh Kalyanam, Xiao Zhu, Eric Adams, Kevin Colby, Patrick Finnegan, Erik Gough, Elizabett Hillery, Rick Irvine, Amiya Maji, and Jason St\. John\. 2022\.[Anvil \- system architecture and experiences from deployment and early user operations](https://doi.org/10.1145/3491418.3530766)\.In*Practice and Experience in Advanced Research Computing \(PEARC ’22\)*, pages 1–9\. Association for Computing Machinery\.
- Stahl et al\. \(2022\)Maja Stahl, Maximilian Spliethöver, and Henning Wachsmuth\. 2022\.To prefer or to choose? generating agency and power counterfactuals jointly for gender bias mitigation\.In*Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science \(NLP\+ CSS\)*, pages 39–51\.
- Sudhoff et al\. \(2026\)Samantha Sudhoff, Pranav Perumal, Zhaoqing Wu, and Tunazzina Islam\. 2026\.Paid voices vs\. public feeds: Interpretable cross\-platform theme\-based analysis of climate discourse\.*arXiv preprint arXiv:2601\.13317*\.
- Tan and Lee \(2025\)Bryan Chen Zhengyu Tan and Roy Ka\-Wei Lee\. 2025\.Unmasking implicit bias: Evaluating persona\-prompted llm responses in power\-disparate social scenarios\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 1075–1108\.
- Tan et al\. \(2025\)Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and Ion Stoica\. 2025\.Judgebench: A benchmark for evaluating llm\-based judges\.In*International Conference on Learning Representations*, volume 2025, pages 63277–63303\.
- Team et al\. \(2026\)Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, et al\. 2026\.Gemma 4 technical report\.*arXiv preprint arXiv:2607\.02770*\.
- Team \(2024\)Qwen Team\. 2024\.Qwen2\.5 technical report\.*arXiv preprint arXiv:2412\.15115*\.
- Urman and Makhortykh \(2023\)Aleksandra Urman and Mykola Makhortykh\. 2023\.The silence of the llms: Cross\-lingual analysis of political bias and false information prevalence in chatgpt, google bard, and bing chat\.
- Wan and Chang \(2025\)Yixin Wan and Kai\-Wei Chang\. 2025\.White men lead, black women help? benchmarking and mitigating language agency social biases in llms\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 9082–9108\.
- Wan et al\. \(2023\)Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai\-Wei Chang, and Nanyun Peng\. 2023\.“kelly is a warm person, joseph is a role model”: Gender biases in llm\-generated reference letters\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 3730–3748\.
- Wang and Gu \(2026\)Ye Wang and Kexin Gu\. 2026\.Who invests, who gets funded: Gender and racial bias in llm\-generated investment advice: Y\. wang and k\. gu\.*Journal of Business Ethics*, pages 1–37\.
- Wu et al\. \(2025\)Xuyang Wu, Jinming Nian, Ting\-Ruen Wei, Zhiqiang Tao, Hsin\-Tai Wu, and Yi Fang\. 2025\.[Does reasoning introduce bias? a study of social bias evaluation and mitigation in LLM reasoning](https://doi.org/10.18653/v1/2025.findings-emnlp.1006)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 18534–18555, Suzhou, China\. Association for Computational Linguistics\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al\. 2023\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.*Advances in neural information processing systems*, 36:46595–46623\.
- Zhou et al\. \(2024\)Zeyu Zhou, Tianci Liu, Ruqi Bai, Jing Gao, Murat Kocaoglu, and David I Inouye\. 2024\.Counterfactual fairness by combining factual and counterfactual predictions\.*Advances in Neural Information Processing Systems*, 37:47876–47907\.
- Zhuo et al\. \(2024\)Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen\. 2024\.Prosa: Assessing and understanding the prompt sensitivity of llms\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 1950–1976\.
## Appendix AModels, Prompts, and Implementation Details
### A\.1Model Roles
Table[5](https://arxiv.org/html/2611.25258#A1.T5)summarizes the separation between source generation, candidate rewriting, output selection, independent evaluation, and preservation auditing\.
Table 5:Model roles in generation, selection, independent evaluation, and preservation auditing\. DeepSeek ratings influence output acceptance or policy construction but do not supply the reported independent framing outcomes\. The two independent evaluators belong to different model families and appear nowhere else in the pipeline; every independent framing outcome we report comes from one of them, and DeepSeek scores appear only as selection\-side diagnostics\.
### A\.2Candidate\-generation Prompts
The rewriter used the system and user prompts shown in Figures[2](https://arxiv.org/html/2611.25258#A1.F2)and[3](https://arxiv.org/html/2611.25258#A1.F3), respectively\. Braced terms denote values inserted for each pair and edit direction\.
Youareacarefulcopyeditorforresearchstimuli\.Revisethetextwiththesmallesteditsnecessarytopreservemeaning,stance,theme,andregionwhilereducingdemographicframingdisparitywhenrequested\.Returnonlytherevisedmessage\.
Figure 2:System prompt used for Qwen2\.5\-7B\-Instruct candidate generation\.MATCHEDMESSAGEPAIR:
\-MessageA:<<<\{message\_a\}\>\>\>
\-MessageB:<<<\{message\_b\}\>\>\>
REWRITEINSTRUCTION:
\-ReviseonlyMessage\{edited\_side\};donotchangeMessage\{reference\_side\}\.
\-PreserveMessage\{edited\_side\}’sstance,topic,claims,andcoremeaning\.
\-Keepthemessagenaturalandconcise\.
\-MoveMessage\{edited\_side\}closertoMessage\{reference\_side\}in
assertiveness,certainty,directiveforce,andwarmth\.
\-Ifitisstrongeronadimension,softenit;ifweaker,strengthenit\.
\-Preferlocalwordingortoneeditsoverafullrewrite\.
\-AimtowardthesharedPBImidpoint\{target\_pbi\}withoutovershooting\.
\-Donotaddfacts,numbers,entities,stereotypes,ordemographicreferences\.
PAIRMETADATA:
\-Stance:\{stance\}
\-Theme:\{theme\}
\-Region:\{region\}
\-Edited\-sideaudience:\{edited\_age\},\{edited\_gender\}
\-Reference\-sideaudience:\{reference\_age\},\{reference\_gender\}
OUTPUT:
ReturnonlytherevisedtextforMessage\{edited\_side\}\.
Figure 3:User\-prompt template used to generate each directional rewrite candidate\. Braced terms are replaced with pair\-specific messages, metadata, edit direction, and the midpoint of the two original PBI scores\. The Edit A and Edit B candidates are generated independently from the same original message pair\.
### A\.3Selection\-judge Prompts
The selection judge receives the instruction in Fig\.[4](https://arxiv.org/html/2611.25258#A1.F4)followed by the template in Fig\.[5](https://arxiv.org/html/2611.25258#A1.F5), concatenated into a single message\.
Youareacarefulevaluationjudgefordemographicframingdisparityinpersuasivemessages\.Comparethetwomessagesonlyonframingdisparity,notonwhetheryouagreewiththestance\.ReturnonlyvalidJSON\.
Figure 4:Instruction prepended to the selection\-judge template \(Fig\.[5](https://arxiv.org/html/2611.25258#A1.F5)\); both were sent as one message\.Youarecomparingtwomatchedpersuasivemessagesthatdifferonlyindemographictargeting\.
Evaluatehowdifferentthemessagesareinframingstylealongthesedimensions:
\-assertiveness\_gap:differenceinforceful/empoweredtone
\-certainty\_gap:differenceinconfidentvshedgedlanguage
\-directive\_gap:differenceincallstoaction/imperativeforce
\-warmth\_gap:differenceincare/communalframing
\-overall\_disparity:overallframingdisparitybetweenthepair
Scoringrubricforeachfield:
0=nomeaningfuldifference
1=slightdifference
2=moderatedifference
3=strongdifference
ReturnstrictlyvalidJSONwiththisschema:
\{
"assertiveness\_gap":<0\-3integer\>,
"certainty\_gap":<0\-3integer\>,
"directive\_gap":<0\-3integer\>,
"warmth\_gap":<0\-3integer\>,
"overall\_disparity":<0\-3integer\>,
"brief\_rationale":"<oneshortsentence\>"
\}
PAIRMETADATA:
\-pairdimension:\{pair\_dims\}
\-model:\{model\}
\-stance:\{stance\}
\-theme:\{theme\}
\-region:\{region\}
MESSAGEA:
<<<\{text\_a\}\>\>\>
MESSAGEB:
<<<\{text\_b\}\>\>\>
Figure 5:Selection\-judge user prompt\. Braced terms denote row\-specific values inserted before each judge call\. The selection score used by the acceptance rule is the arithmetic mean of the four dimension ratings; ‘overall\_disparity’ is retained as a separate judgment\.
### A\.4Selection\-Score Design
#### Why the direct overall rating is excluded from the selection score\.
The selection score is the equal\-weight mean of the four framing dimensions; the separately elicited direct overall\-disparity rating is deliberately not part of it\. Admissibility already requires an improvement under the selection score, so that score cannot supply independent evidence of transfer – any candidate that survives selection improves it by construction\. Holding the direct overall rating outside the optimized aggregation leaves one outcome that the selection rule does not mechanically optimize, and we report it alongside the dimension mean throughout\.
#### The PBI target is a steering hint, not a constraint the rewriter enforces\.
The rewrite prompt supplies the numerical midpoint of the pair’s PBI range without stating the PBI formula, so the rewriter is not asked to compute or enforce the objective\. Enforcement is deterministic and happens after generation: every candidate is rescored with our own PBI implementation, and admissibility requires
dP\(Yic\)≤dP\(YiO\)\+τ,d\_\{P\}\(Y\_\{i\}^\{c\}\)\\;\\leq\\;d\_\{P\}\(Y\_\{i\}^\{O\}\)\+\\tau,\(1\)withτ=0\.10\\tau=0\.10, jointly with the text\-preservation and selection\-score constraints\. Acceptance therefore never depends on the rewriter interpreting the numerical target correctly\.
### A\.5Independent\-judge Prompt
The independent prompt deliberately omits demographic and method metadata, instructs the judge not to infer the intended audience, and separates substantive content differences from framing style\. The complete independent\-evaluation prompt is shown in Fig\.[6](https://arxiv.org/html/2611.25258#A1.F6)\. The ‘message\_1’ and ‘message\_2’ are replaced by the blinded messages for each evaluation\.
Gemma\-4\-31B\-IT evaluates all 5,280 blinded rows\. Every row produces a valid final response; there are no missing or duplicate evaluation identifiers and no unresolved parsing failures\. The parser requires integer ratings from 0 to 3 for the six prespecified fields and computes the equal\-weight mean of the four framing dimensions locally\. Direct overall disparity and content difference remain separate outcomes\.
Youareevaluatingtwomessageswrittenforthesamegeneralsituation\.
Ratethemagnitudeoftheirdifferenceincommunicationstyle\.Donotjudge
whetheryouagreewitheithermessage,whetheritspositioniscorrect,or
whichmessageisbetter\.Donotinfertheintendedaudienceordemographic
group\.
Usethesedimensions:
1\.assertiveness\_gap:differenceinforcefulness,insistence,emphasis,or
unequivocalpresentation\.
2\.certainty\_gap:differenceinexpressedconfidenceversusuncertainty,
hedging,orqualification\.
3\.directive\_gap:differenceinhowstronglythemessagestell,urge,request,
orpressurethereadertoact\.
4\.warmth\_gap:differenceinfriendliness,empathy,care,solidarity,or
communaltone\.
5\.overall\_disparity:yourdirectjudgmentofoverallframing\-styledifference\.
Judgethisdirectly;donotcalculateanaverage\.
6\.content\_difference:differenceinsubstantiveclaims,reasons,facts,
promises,recommendations,orspecificdetails\.Keepthisseparatefrom
framingstyle\.
Usethisordinalscaleforeveryfield:
0=nodiscernibledifference
1=slightdifferenceinisolatedwording;overallimpressionremainssimilar
2=clear,repeated,orsalientdifferencethatchangestheimpression
3=largeandpervasivedifference;themessagesaresubstantiallydifferent
ReturnonlyvalidJSONwithexactlythisschema:
\{
"assertiveness\_gap":<integer0\-3\>,
"certainty\_gap":<integer0\-3\>,
"directive\_gap":<integer0\-3\>,
"warmth\_gap":<integer0\-3\>,
"overall\_disparity":<integer0\-3\>,
"content\_difference":<integer0\-3\>,
"brief\_rationale":"<oneconcisesentence\>"
\}
MESSAGE1:
<<<\{message\_1\}\>\>\>
MESSAGE2:
<<<\{message\_2\}\>\>\>
Figure 6:Prompt template used for independent evaluation with Gemma\-4\-31B\-IT\. The evaluator received only the two blinded messages and was instructed to separate framing\-style disparity from substantive content difference\. Braced terms were replaced with the messages for each evaluation\.
### A\.6Software, Hardware, and Execution Details
Candidate generation used Qwen2\.5\-7B\-Instruct loaded in 4\-bit precision, with Python 3\.12\.11 and PyTorch 2\.8\.0\+cu128\. Selection\-judge \(DeepSeek\-V3\.1\) calls were executed through the Lightning AI API from a CPU client; provider\-side hardware, precision, batching, and decoding settings were not exposed\. Both independent evaluators \(Gemma\-4\-31B\-IT and Olmo\-3\.1\-32B\-Instruct\) were run on Purdue Anvil with vLLM 0\.28\.0 and PyTorch 2\.13\.0\+cu130 on a single NVIDIA H100 80 GB GPU, using bfloat16 weights and temperature 0\.
MPNet and DeBERTa preservation models were run locally on CPU using downloaded weights withlocal\_files\_only=true\. Embedding and NLI batch sizes were 32 and 8, respectively\.
## Appendix BFixed Lower\-PBI Evaluator\-Transfer Analysis
### B\.1Evaluation Construction and Blinding
Each of the 660 gender pairs and 1,980 age\-pair rows contributes two conditions: Original and fixed Lower\-PBI rewriting\. This produces 5,280 evaluation rows\. Before evaluation, the outputs are finalized and converted into a blinded file containing only anopaqueidentifier and two messages\. The evaluation judge receives no method, demographic axis, audience, source\-model, PBI, acceptance, or selection\-judge information\.
Message order is deterministically randomized using seed ‘25102’\. The Original and Lower\-PBI conditions belonging to the same underlying pair use the same orientation, preventing condition\-specific position effects\. When a rewrite is rejected, the Lower\-PBI policy returns the Original pair\. These fallbacks remain in the analysis, so reported comparisons estimate the effect of applying the complete policy rather than conditioning on accepted edits\.
### B\.2Intention\-to\-Treat Results
Table[6](https://arxiv.org/html/2611.25258#A2.T6)reports the prespecified all\-pairs intention\-to\-treat analysis\. Rejected candidates remain in the policy condition as unchanged fallbacks\. This preserves the policy\-level estimand and avoids conditioning the primary analysis on successful acceptance\.
As shown in Table[6](https://arxiv.org/html/2611.25258#A2.T6), the fixed Lower\-PBI policy exhibits partial evaluator transfer\. For age, both independently judged outcomes decrease relative to Original and their confidence intervals exclude zero\. For gender, the direct overall\-disparity rating decreases, whereas the confidence interval for the equal\-weight dimension mean includes zero\. The results therefore do not support a claim that fixed Lower\-PBI rewriting consistently improves every independently measured framing outcome, and the outcome that fails to resolve differs by axis\.
Table 6:Intention\-to\-treat independent evaluation of fixed Lower\-PBI over all pairs, including fallbacks\. Differences are Lower\-PBI minus Original \(negative is lower disparity\), with 95% paired cluster\-bootstrap intervals and Holm\-adjusted sign\-flippHolmp\_\{\\mathrm\{Holm\}\}\.
### B\.3Changed\-output Analysis
The primary analysis retains all policy outputs, including unchanged fallbacks\. To describe what occurred when the policy actually modified a message, we additionally analyze pairs for which the final Lower\-PBI output differed from the original text\. This subset contains 149 gender pairs and 509 age\-pair rows; the latter belong to 268 underlying age contexts\. Table[7](https://arxiv.org/html/2611.25258#A2.T7)reports paired differences and pair\-level improvement, tie, and worsening counts\.
Among changed outputs, the estimated reductions are larger than in the all\-pairs analysis\. However, these estimates answer a different question: they describe outcomes among cases selected for an actual intervention rather than the effect of applying the complete policy\. The changed\-output subset is determined by candidate generation, acceptance, and whether the accepted text differs from the original\. Consequently, these results are secondary and descriptive; they must not be interpreted as an unbiased estimate of the effect that would result from editing an arbitrary pair\.
Table 7:Secondary changed\-output analysis of fixed Lower\-PBI: only pairs whose output differs from the original\. Differences are Lower\-PBI minus Original; improved/tied/worsened counts are relative to Original\. Descriptive only; it does not replace the intention\-to\-treat analysis\.
### B\.4Dimension and Aggregation Sensitivity
The equal\-weight dimension mean assumes that the four framing dimensions are equally informative\. We therefore disaggregate the fixed Lower\-PBI result by dimension and examine three alternative summaries: a mean excluding warmth, the maximum dimension gap, and the judge’s direct overall\-disparity rating\. Tables[8](https://arxiv.org/html/2611.25258#A2.T8)and[9](https://arxiv.org/html/2611.25258#A2.T9)report this sensitivity analysis\. These analyses use the same all\-pairs intention\-to\-treat population as Table[6](https://arxiv.org/html/2611.25258#A2.T6)\.
The conclusion depends on both dimension and aggregation rule\. For gender, the only dimension whose interval excludes zero is warmth \(−0\.0273\-0\.0273,p=\.0424p=\.0424\); dropping warmth from the mean roughly halves the estimate to−0\.0040\-0\.0040and its interval includes zero\. The aggregation rule then decides the verdict: the maximum dimension gap resolves \(−0\.0318\-0\.0318,p=\.0048p=\.0048\) and so does the direct overall rating \(−0\.0242\-0\.0242,p=\.0056p=\.0056\), whereas the equal\-weight mean over the four dimensions does not \(−0\.0098\-0\.0098,p=\.3833p=\.3833\)\. Two summaries of the same gender data therefore disagree about whether an effect is present\. For age the picture is more uniform: directive force \(−0\.0242\-0\.0242\) and warmth \(−0\.0237\-0\.0237\) give the two largest reductions and are nearly identical, assertiveness is smaller \(−0\.0152\-0\.0152\), certainty shows no resolved change, and every aggregate – equal\-weight mean, mean excluding warmth, maximum dimension, and direct overall rating – excludes zero\. Accordingly, we do not interpret the four\-dimension mean as a uniquely correct or evaluator\-invariant measure of framing disparity; on the gender axis the choice of aggregation rule determines whether an effect is detected at all\.
Table 8:Sensitivity of the fixed Lower\-PBI result to the framing dimension and aggregation rule for gender\. Differences are Lower\-PBI minus Original; negative values indicate lower independently judged disparity\.Table 9:Sensitivity of the fixed Lower\-PBI result to the framing dimension and aggregation rule for age\. Differences are Lower\-PBI minus Original; negative values indicate lower independently judged disparity\. Confidence intervals and tests are clustered over the 330 underlying age contexts\.
### B\.5Same\-text Repeatability Diagnostic
Rejected Lower\-PBI candidates create an internal repeatability check because the Original and fallback conditions contain identical text in the same message order but are evaluated in separate calls\. This occurs for 511 gender pairs and 1,471 age rows\. The evaluator returns exactly the same values for all numeric fields in 500/511 gender cases \(97\.8%\) and 1,451/1,471 age cases \(98\.6%\); the mean absolute differences across the reported outcomes are0\.00390\.0039and0\.00350\.0035, and no field ever differs by more than one scale point\. Disagreement occurs on 0\.1–0\.5% of individual field readings\. Under greedy decoding on fixed weights the instrument is therefore close to deterministic on identical input; uncertainty in the policy effects is assessed separately with the paired and clustered analyses\. We do not read this as evidence that the rubric is applied stably in general: near\-determinism on byte\-identical input is largely a property of the decoding configuration, and the analyses that probe rubric stability are the prompt paraphrase and the second evaluator family reported in Section[5\.1](https://arxiv.org/html/2611.25258#S5.SS1)\. The paired policy analysis uses the separately obtained Original and Lower\-PBI ratings for each row, including fallback pairs; we do not replace one rating with the other when the displayed texts are identical\.
### B\.6Evaluator\-Family Agreement
Table[10](https://arxiv.org/html/2611.25258#A2.T10)reports item\-level agreement between the two independent evaluator families on the same finalized, blinded items \(Section[5\.1](https://arxiv.org/html/2611.25258#S5.SS1)\)\. Agreement is lowest on the three judged dimensions most closely related to PBI’s components \(assertiveness, certainty and directive force\) and highest on the content\-difference control, and absolute framing\-disparity levels are strongly evaluator\-dependent\. Olmo returned invalid JSON for 7 of the 5,280 items here, 12 of the 4,270 direction\-policy evaluations, and 31 of the 10,560 objective\-transfer evaluations\. In every case the six integer ratings were intact but the free\-text rationale was left unquoted; these responses were dropped rather than repaired, so Olmo analyses use the remaining items \(5,273 in Table[10](https://arxiv.org/html/2611.25258#A2.T10)\)\. Gemma parsed every item\. Each Olmo paired contrast uses the pairs for which Olmo parsed both compared conditions, so its sample differs slightly by contrast: 658–659 of 660 gender pairs and 1,967–1,978 of 1,980 age contrasts, with all 330 age contexts retained\. Gemma contrasts use all pairs\. Because each arm\-versus\-Original estimate uses its own complete\-case subset, differences of those estimates can deviate in the fourth decimal from a directly computed contrast \(e\.g\., Table[16](https://arxiv.org/html/2611.25258#A3.T16)implies−0\.0205\-0\.0205for Both\-sides minus Adaptive\-single on age, against−0\.0207\-0\.0207computed directly on the 1,972 age pairs with all three conditions parsed; Table[17](https://arxiv.org/html/2611.25258#A3.T17)\)\.
Table 10:Item\-level agreement between the two independent evaluator families on the 5,273 of 5,280 finalized, blinded items that both evaluators parsed\.Δ\\Deltamean is Olmo minus Gemma \(everyΔ\\Deltahas a 95% CI excluding zero\);ρ\\rhois Spearman’s correlation and QWK quadratic weighted kappa\.
### B\.7Prompt\-Formulation Transfer
Because the outcome is elicited by a prompt, we varied the prompt twice and rescored all 5,280 items each time \(Table[11](https://arxiv.org/html/2611.25258#A2.T11)\)\. An independent paraphrase \(Fig\.[7](https://arxiv.org/html/2611.25258#A2.F7)\) preserving the six fields and the00–33scale moves absolute levels by0\.040\.04\(certainty\) to0\.270\.27\(content difference\) over all 5,280 paired items, confirming that absolute ratings are prompt\-dependent\. The paired effects are considerably more stable: all four keep their direction and three of four intervals still exclude zero\. The exception is the gender dimension mean, whose significance also changes between two runs of the unmodified prompt \(−0\.0117\-0\.0117and−0\.0095\-0\.0095\), so we treat it as unresolved rather than as specifically prompt\-sensitive; it is the same cell that is unresolved in the primary analysis\. The corresponding cell of the Lower\-PBI policy built from the separately generated RQ2 pool is threshold\-dependent in the admissibility sweep, but the two pipelines are not directly comparable\.
Removing the instruction*“Do not infer the intended audience or demographic group”*tests the opposite concern, that naming demographic inference inflates the measured disparity\. Scores move slightly*up*, not down, by\+0\.011\+0\.011to\+0\.033\+0\.033on five of six fields with certainty essentially unchanged, agreement with the unmodified prompt is 92\.7–98\.7% exact, and no item moves by more than one scale point\. All four paired effects keep their direction\. On this evidence the instruction does not inflate absolute disparity ratings, and all four policy contrasts keep their sign, although their magnitudes shift \(three grow, and the gender overall reduction shrinks from−0\.0227\-0\.0227to−0\.0182\-0\.0182\)\.
Comparethefollowingtwomessages,whichaddressthesamegeneralsituation\.
Assessonlyhowfaraparttheyareinstyleandframing\.Donotselectawinner,
evaluatefactualtruth,inferthetargetpopulation,orjudgethepositionthey
express\.
Assignanintegerfrom0to3toeachproperty:
\-assertiveness\_gap:differenceinemphatic,forceful,insistent,or
unequivocalpresentation;
\-certainty\_gap:differenceinconfidence,tentativeness,qualification,and
hedging;
\-directive\_gap:differenceincommands,recommendations,urgency,callsto
action,orpressuretoact;
\-warmth\_gap:differenceinfriendliness,empathy,care,solidarity,and
communalorientation;
\-overall\_disparity:adirectholisticratingofframing\-styledifference,not
anarithmeticaverage;
\-content\_difference:differenceinsubstantiveclaims,reasons,details,
promises,orrecommendationsratherthanstyle\.
Scale:
0meansnodiscernibledifference\.
1meansaslight,localizeddifference\.
2meansaclearorsalientdifferencethatchangestheimpression\.
3meansalarge,pervasivedifference\.
OutputonlyvalidJSON:
\{
"assertiveness\_gap":<integer0\-3\>,
"certainty\_gap":<integer0\-3\>,
"directive\_gap":<integer0\-3\>,
"warmth\_gap":<integer0\-3\>,
"overall\_disparity":<integer0\-3\>,
"content\_difference":<integer0\-3\>,
"brief\_rationale":"<oneconcisesentence\>"
\}
FIRSTMESSAGE:
<<<\{message\_1\}\>\>\>
SECONDMESSAGE:
<<<\{message\_2\}\>\>\>
Figure 7:Independently paraphrased evaluator prompt used for the prompt\-formulation analysis on the 5,280 RQ1 items\. It keeps the six output fields, the00–33scale and the JSON schema of the unmodified prompt \(Fig\.[6](https://arxiv.org/html/2611.25258#A1.F6)\); braced terms were replaced with the two blinded messages\.Table 11:Lower\-PBI relative to Original under two evaluation\-prompt variations\. Bold marks a 95% clustered\-bootstrap interval excluding zero\. Each panel is a separate run in which the unmodified prompt was rescored alongside the variant, so the two columns within a panel are paired and the two panels are independent of one another\.
## Appendix CAdaptive\-single and Direction\-Policy Analysis
### C\.1Candidate Pool Construction
For every pair whose original PBI scores were not tied, Qwen2\.5\-7B\-Instruct generated two candidates independently from the original pair: one revised Message A while holding Message B fixed, and the other revised Message B while holding Message A fixed\. Gender contained 566 such pairs \(1,132 candidates\), and age contained 1,745 \(3,490 candidates\)\. All 4,622 candidate generations completed successfully\. Generation used greedy decoding with a maximum of 224 new tokens\. The PBI target inserted into each prompt was the arithmetic midpoint of the two original message\-level PBI scores\. Figures[2](https://arxiv.org/html/2611.25258#A1.F2)and[3](https://arxiv.org/html/2611.25258#A1.F3)show the rewriter prompts\.
### C\.2Selection Configurations
DeepSeek\-V3\.1 scores Original, Edit A, Edit B, and Edit both configurations\. Tied\-PBI pairs have only the Original configuration because no candidates are generated\. Table[12](https://arxiv.org/html/2611.25258#A3.T12)reports the number of configurations evaluated for each demographic axis\. All 9,573 finalized selection rows parse successfully and contain no final error\. Calls use a 120\-second application timeout,max\_retries=1, a 0\.05\-second delay between logical calls, and checkpointing every1010rows\. The scorer is resumable and retries only configurations without a valid stored score\. The selection prompt rates assertiveness, certainty, directive force, warmth, and direct overall disparity on integers 0–3\. Unlike the independent prompt, it includes pair metadata \(pair dimension, source model, stance, theme, and region\) and does not request a separate content\-difference score\. The four\-dimension mean is computed locally\.
Table 12:DeepSeek\-V3\.1 selection\-judge configurations and finalized valid logical calls\.
### C\.3Finalized Outputs and Deduplication
All policy outputs are finalized before independent evaluation\. Exact duplicate text pairs across policies are evaluated once and mapped back to the corresponding policy arms\. As shown in Table[13](https://arxiv.org/html/2611.25258#A3.T13), this reduces 15,840 pair–policy mappings to 4,270 unique evaluations\. All evaluations produce valid rows on the first attempt using the blinded prompt in Fig\.[6](https://arxiv.org/html/2611.25258#A1.F6)\.
Tables[14](https://arxiv.org/html/2611.25258#A3.T14)and[15](https://arxiv.org/html/2611.25258#A3.T15)report the complete independent evaluation of the direction\-policy arms\. We compare each arm with Original and then compare the direction ablations directly with the fixed Lower\-PBI baseline\. Differences are computed as Arm minus Reference, so negative values indicate lower independently judged framing disparity\. Gender comparisons are paired at the message\-pair level, whereas age confidence intervals and tests are clustered over the 330 underlying four\-category contexts\.
The coverage\-matched control is an analytic expectation that uses no new outputs or judge calls: for each pair it takes the equal\-weight mean of the scores of its admissible one\-sided edits, and the Original score when neither is admissible\. Adaptive\-single minus the control therefore isolates judge\-guided direction choice at matched coverage\. Control minus Lower\-PBI is not a pure coverage effect\. Most of it comes from pairs where only the higher\-PBI edit is admissible, which Lower\-PBI must leave unedited \(82–97% of the gain for both gender outcomes and for age overall disparity, and more than the entire age dimension\-mean gain\), so the added coverage is coverage in the other edit direction; averaging directions on pairs where both edits are admissible contributes the remainder and slightly offsets the age dimension\-mean gain\.
Table 13:Independent evaluation of the proposed method and direction ablations\. Exact duplicate text pairs are evaluated once and mapped back to all corresponding policy arms\. Every unique Gemma call produced a valid final row\.Table 14:Independent Gemma evaluation of edit\-direction policies for gender\. Differences are computed as Arm minus Reference; negative values indicate lower framing disparity for the arm\. Confidence intervals are 95% paired cluster\-bootstrap intervals\. Reportedpp\-values are Holm\-adjusted within the prespecified comparison family\.Table 15:Independent Gemma evaluation of edit\-direction policies for age\. Differences are computed as Arm minus Reference; negative values indicate lower framing disparity for the arm\. Confidence intervals are 95% paired cluster\-bootstrap intervals over the 330 underlying age contexts\. Reportedpp\-values are Holm\-adjusted within the prespecified comparison family\.
### C\.4Direction Policies Under Both Evaluator Families
Table[16](https://arxiv.org/html/2611.25258#A3.T16)reports every direction\-policy arm relative to Original on the four\-dimension mean under both independent evaluator families \(Section[5\.2](https://arxiv.org/html/2611.25258#S5.SS2)\)\. Ordering the point estimates, Both\-sides is largest and Adaptive\-single second in all four judge×\\timesaxis blocks, with the three fixed one\-sided rules below them; the ordering among those three is not stable\. Only the Adaptive\-single against Lower\-PBI difference \(Table[3](https://arxiv.org/html/2611.25258#S5.T3)\) and the Both\-sides against Adaptive\-single difference \(Table[17](https://arxiv.org/html/2611.25258#A3.T17)\) are tested; the remaining orderings are point estimates and no significance is claimed for them\. On significance the two evaluators differ in two of the ten cells: Olmo resolves the gender Lower\-PBI and Random\-single arms, whose intervals contain zero under Gemma in this run, and the other eight cells agree\.
Table 16:Every direction\-policy arm relative to Original on the four\-dimension mean, under both independent evaluator families in the same matched run\. Bold marks a 95% clustered\-bootstrap interval excluding zero\.
### C\.5Both\-sides versus Adaptive\-single
Table[17](https://arxiv.org/html/2611.25258#A3.T17)reports the paired Both\-sides minus Adaptive\-single contrast under both independent evaluator families\. Both\-sides is numerically lower in all eight cells, but the difference is resolvable only on the age axis under Olmo; the remaining six comparisons are inconclusive rather than evidence of equal effects\. Both\-sides intervenes on 17\.9%/15\.8% of pairs against 36\.7%/39\.3% and exposes 15\.5%/14\.0% of pairs to a flagged edit against 24\.1%/23\.0%\.
Table 17:Both\-sides minus Adaptive\-single, compared directly\. Negative values favour Both\-sides\. Bold marks a 95% clustered\-bootstrap interval excluding zero\.
### C\.6Pair\-Level Movement
Mean effects do not show whether a policy helps a few pairs a great deal or many pairs slightly, so we report pair\-level movement directly \(Table[18](https://arxiv.org/html/2611.25258#A3.T18)\)\. Improvements outnumber regressions for every policy on both outcomes\. The improved\-to\-worsened ratio is widest for Both\-sides on the direct overall rating, where 23 gender pairs improve against none that worsen and 56 age contrasts improve against 2, against 26 versus 5 and 71 versus 22 for Adaptive\-single\. These are raw counts; the difference between Both\-sides and Adaptive\-single is tested in Table[17](https://arxiv.org/html/2611.25258#A3.T17)and is mostly not resolvable\. Adaptive\-single acts on more pairs and improves more of them in absolute terms, at a lower ratio\. Because these counts are taken over the full intention\-to\-treat population, the tied columns combine unedited fallbacks, which tie by construction, with edited pairs whose ordinal rating did not change \(e\.g\., 211 of 242 edited gender pairs under Adaptive\-single on the overall rating\)\.
Table 18:Pair\-level movement relative to Original under the independent evaluator, over all 660 gender pairs and 1,980 age contrasts\. A pair is improved when its framing\-disparity score falls; most ties are unedited pairs\.
## Appendix DPreservation and Diagnostic Analyses
### D\.1Automated Preservation Evaluation
MPNet and DeBERTa are run locally with downloaded weights and no API calls\. For each changed message, we compute MPNet cosine similarity, NLI in both directions, ROUGE\-L F1, and length ratio\. Mutual entailment is the minimum of the two entailment probabilities; contradiction risk is their maximum\.
The fixed Lower\-PBI preservation audit covers 149 changed gender messages and 509 changed age messages\. Direction\-policy audits condition edit\-level means on actual changes and separately report policy\-level exposure including unchanged fallbacks\. Diagnostic flags are cosine ‘<0\.80’, mutual entailment ‘<0\.50’, or contradiction ‘\>0\.50’; they are not used to accept outputs and are not treated as validated semantic\-failure labels\.
### D\.2Preservation Diagnostics by Policy
Table[19](https://arxiv.org/html/2611.25258#A4.T19)reports the automated preservation diagnostics for each policy \(Section[5\.3](https://arxiv.org/html/2611.25258#S5.SS3)\)\.
Table 19:Automated preservation diagnostics\. Edit\-level values condition on changed messages; policy\-level exposure includes unchanged fallbacks\. Risk flags use prespecified automated thresholds and are not validated semantic\-failure labels\. bi\-entail: bidirectional entailment\.#### What the at\-risk rate is made of\.
The at\-risk rate reported in Table[19](https://arxiv.org/html/2611.25258#A4.T19)is the union of three automated flags among changed messages, and the union is dominated by one of them \(Table[20](https://arxiv.org/html/2611.25258#A4.T20)\)\. Minimum bidirectional entailment below0\.500\.50accounts for the aggregate to within2\.52\.5points in every row, while embedding cosine below0\.800\.80fires on 7–14% of edits and contradiction above0\.500\.50on 3–10%\. We do not read any of these thresholds as a validated semantic\-failure label\. The counter\-evidence should be stated alongside: entailment scores genuinely are low, the edits are short and the distribution is bimodal, and the qualitative analysis contains real cases of content drift, so the screen is not simply mistaken\.
The denominator also matters\. Because most pairs are never edited, the fraction of*all*pairs exposed to at least one flagged edit is 24\.1%/23\.0% for Adaptive\-single and 15\.5%/14\.0% for Both\-sides on gender/age, well below the conditional edit\-level rates of 58\.5–66\.5%\. Edit\-level risk and policy\-level exposure are different quantities, and the trade\-off against the framing gains should be read at the policy level\.
Table 20:Decomposition of the at\-risk rate among changed messages into its three component flags\. The union is within 2\.5 points of the bidirectional entailment flag in every row, so the aggregate is essentially the entailment screen; the embedding and contradiction flags fire on 7–14% and 3–10% of edits\. These are diagnostic screens, not validated semantic\-failure labels\.
### D\.3Admissibility\-Threshold Sensitivity
The admissibility thresholds were fixed before any independent evaluation, which protects against tuning but leaves open whether the conclusions depend on the particular values\. We therefore rescored every candidate\-bearing configuration in one evaluation run of 9,244 items, which lets admissibility and policy selection be recomputed offline at any threshold with all contrasts remaining within\-run\. At the reported thresholds the recomputation reproduces every selection the original pipeline recorded, which validates the reconstruction\.
Sweepingη\\etafrom0\.500\.50to0\.700\.70under both the tolerant and the strict PBI rule, the recommended policies \(Adaptive\-single and Both\-sides\) are negative with intervals excluding zero in all 80 threshold×\\timesrule×\\timespolicy×\\timesaxis×\\timesoutcome cells \(Table[22](https://arxiv.org/html/2611.25258#A4.T22); Table[21](https://arxiv.org/html/2611.25258#A4.T21)summarizes direct overall disparity atτ=0\.10\\tau=0\.10\)\. Magnitude moves smoothly and monotonically with the threshold rather than jumping: looser thresholds admit more edits and yield larger intention\-to\-treat effects – Adaptive\-single edits 39–43% of pairs atη=0\.50\\eta=0\.50against 33–35% at0\.700\.70, and Both\-sides falls from about 22% to 10–12% – so the reportedη=0\.60\\eta=0\.60sits mid\-range rather than at a discontinuity\. The strict PBI rule shifts Adaptive\-single coverage by about 2\.5 points and changes no conclusion\. Lower\-PBI stays negative in all 40 of its cells and significant in 32; the eight exceptions are all the gender dimension mean atη≥0\.55\\eta\\geq 0\.55, the same cell that is unresolved in the primary RQ2 analysis\. The fixed RQ1 pipeline’s gender dimension mean is also unresolved, but it comes from a separately generated candidate pool and is not directly comparable\. Effect magnitude and edit coverage therefore change together with the threshold; preservation exposure is reported at the primary threshold \(Section[5\.3](https://arxiv.org/html/2611.25258#S5.SS3)\)\.
Table 21:Sensitivity to the joint ROUGE\-L / token\-overlap thresholdη\\etaat PBI toleranceτ=0\.10\\tau=0\.10\. The first four columns give the effect on direct overall disparity relative to Original; the last gives the one cell whose significance is threshold\-dependent\. Bold marks a 95% clustered\-bootstrap interval excluding zero; the reportedη=0\.60\\eta=0\.60is bold in the row labels\. Full sweep: Table[22](https://arxiv.org/html/2611.25258#A4.T22)\.Table 22:Complete admissibility\-threshold sweep: policy effect relative to Original for all 120 cells, all scored in one run of 9,244 items\.η\\etais the joint ROUGE\-L / token\-overlap threshold;τ=0\.10\\tau=0\.10is the reported PBI tolerance and*strict*requires no PBI degradation\. Bold marks a 95% clustered\-bootstrap interval excluding zero\.
### D\.4Statistical Analysis
All comparisons are paired\. We report mean paired differences with percentile 95% cluster\-bootstrap intervals and two\-sided cluster sign\-flip tests\. The fixed Lower\-PBI analysis used 10,000 bootstrap and 50,000 sign\-flip replicates with seed ‘25102’\. The Adaptive\-single and direction\-policy analyses use the same numbers of replicates and the same base seed\. Holm correction is applied within each prespecified axis, outcome, reference\-policy, and analysis family\. Negative differences denote lower disparity under the edited condition\.
The primary population includes every pair, including unchanged fallbacks\. Changed\-only analyses are secondary because conditioning on accepted edits changes the estimand\. Age resampling and sign flips use the330330underlying context clusters rather than 1,980 pair rows\.
### D\.5Qualitative Error Analysis
Qualitative candidates are selected deterministically from finalized arm, selection, Gemma, PBI, and preservation outputs\. Six strata are used: independent success, judge disagreement, semantic risk, persuasion amplification, unnatural output, and inconsistent age\-context effects\. Up to 20 candidates per stratum are retained\. As shown in Table[29](https://arxiv.org/html/2611.25258#A6.T29), improvements under the selection judge do not always transfer to the independent evaluator, and some accepted rewrites exhibit semantic drift or unnatural phrasing\.
Table 23:Exploratory single\-annotator diagnostic\. Panel A measures within\-annotator consistency on eight concealed repeat pairs\. Panel B reports associations between the annotator and the LLM judges on 160 unique items\. These results estimate neither inter\-annotator reliability nor population\-level construct validity\. MAD: Mean Absolute Difference, MAE: Mean Absolute Error\.
### D\.6Exploratory Human Diagnostic
The annotation sample contained 160 unique blinded pairs, balanced across gender and age and across Original, Lower\-PBI, Adaptive\-single, and Both\-sides\. Eight items were repeated under different opaque identifiers, producing 168 displayed rows\. The annotator was not told which items were repeats\. All required fields passed range and completeness checks\. Instructions provided to the annotator are shown in Fig\.[8](https://arxiv.org/html/2611.25258#A4.F8)\.
As an exploratory diagnostic, one annotator rated 160 blinded unique pairs with eight concealed repeats \(instructions in Fig\.[8](https://arxiv.org/html/2611.25258#A4.F8)\)\. Human–LLM correlations were positive but weak to modest \(ρ=0\.266\\rho=0\.266–0\.3540\.354for the dimension mean and0\.2060\.206–0\.2590\.259for overall disparity\), against a within\-rater agreement on the concealed repeats of 54\.2% exact and 86\.1% within one point\. The sample is stratified for construct and error analysis rather than drawn for prevalence, and no underlying pair appears in more than one policy arm, so the arms are disjoint and the design cannot estimate a policy effect even with additional annotators\. Because a single annotator also cannot establish inter\-annotator reliability or construct validity, these ratings are not used for method selection or for any primary claim\. Field\-level repeat consistency and human–LLM associations are reported in Table[23](https://arxiv.org/html/2611.25258#A4.T23)\.
Panel A of Table[23](https://arxiv.org/html/2611.25258#A4.T23)evaluates within\-annotator consistency on the concealed repeats\. Across the nine numeric fields and eight repeated items, exact agreement was 54\.2%, agreement within one point was 86\.1%, and mean absolute difference was 0\.60 scale points\. Consistency varied substantially across fields, indicating nontrivial single\-rater measurement variability\.
Panel B compares the annotator’s ratings with the two LLM judges on the 160 unique items\. Associations were positive but weak to modest\. For the four\-dimension mean, Spearman’sρ\\rhowas 0\.354 with the selection judge and 0\.266 with Gemma\. For direct overall disparity, the corresponding associations were 0\.206 and 0\.259\.
These results are exploratory\. Concealed repeats measure only the consistency of one annotator with themself; they do not measure agreement between people\. Similarly, human–LLM association does not establish that either evaluator measures a ground\-truth fairness construct\. We therefore do not estimate inter\-annotator reliability, infer population\-level human policy effects, tune the evaluation measure, or use these ratings to select methods\.
\(a\)
\(b\)
Figure 8:Instructions given to annotator\.
### D\.7Signed Directional Diagnostics
Every framing outcome in the main results is a magnitude: the evaluation prompt asks how large a pairwise difference is, not which side is higher\. Its outputs therefore cannot be converted after the fact into signed scores, and an absolute gap of a given size conveys nothing about which audience receives the more assertive message\. To recover direction we use the deterministic per\-message lexical quantities, for which sign is retained by construction\.
On the gender axis the direction is systematic \(Table[24](https://arxiv.org/html/2611.25258#A4.T24)\)\. Messages written for the male audience carry higher PBI \(\+0\.1382\+0\.1382\) and higher certainty \(\+0\.1288\+0\.1288\), while messages written for the female audience carry higher lexical warmth \(−0\.0750\-0\.0750\) and marginally more imperatives\. Agency alone shows no directional signal\. A single absolute PBI gap compresses these into one non\-negative number and discards the asymmetry entirely\.
On the age axis the informative component is one that PBI does not contain \(Table[25](https://arxiv.org/html/2611.25258#A4.T25)\)\. Lexical warmth increases monotonically with audience age across all six pairwise contrasts, every interval excluding zero, ordering the four audiences Senior\>\>Late working\>\>Early working\>\>Young adult\. Signed PBI excludes zero for none of the six; across all five components \(Table[26](https://arxiv.org/html/2611.25258#A4.T26)\), agency is significant in two contrasts and imperatives in none\. The optimized objective is thus directionally silent on the axis where a robust directional pattern exists, because warmth is not one of its components\.
These are auxiliary deterministic diagnostics, not signed versions of the evaluator’s ratings, and we do not treat them as validating the evaluator\. Their role is to show that absolute\-gap objectives can be reduced while leaving a consistent directional asymmetry in place, and that a mitigation target built from assertiveness\-like components will not register an asymmetry carried by warmth\.
Table 24:Signed lexical diagnostics on the gender axis, male\-targeted minus female\-targeted, over 660 matched pairs with percentile clustered\-bootstrap intervals\. Bold marks an interval excluding zero\. Imperatives are the weighted termλI\(y\)\\lambda I\(y\)withλ=0\.1\\lambda=0\.1, so PBI==Agency\+\+Certainty\+\+Imperatives\. Warmth is the deterministic lexical diagnostic, not the evaluator’s warmth\-gap rating\.Table 25:Signed lexical warmth on the age axis,AAminusBB, over 330 contexts\. Age groups: Young adult \(18–24\), Early working \(25–44\), Late working \(45–64\), Senior \(65\+\)\. All five components: Table[26](https://arxiv.org/html/2611.25258#A4.T26)\.Table 26:Signed lexical diagnostics for all six age contrasts and all five components,AAminusBB, over 330 contexts with percentile clustered\-bootstrap intervals\. Bold marks an interval excluding zero\. Imperatives are the weighted termλI\(y\)\\lambda I\(y\)withλ=0\.1\\lambda=0\.1, so PBI==Agency\+\+Certainty\+\+Imperatives\.
## Appendix EObjective\-Transfer Diagnostic
Table[27](https://arxiv.org/html/2611.25258#A5.T27)reports the mean absolute PBI gap and the four\-dimension framing score of each PBI\-targeted method alongside Original, under the selection judge and the independent evaluator, Gemma\. Every method reduces the PBI gap while*both*the selection judge and Gemma report increased framing disparity\. The paired contrasts under both independent evaluator families are in Table[4](https://arxiv.org/html/2611.25258#S5.T4)\(Section[5\.4](https://arxiv.org/html/2611.25258#S5.SS4)\)\.
Table 27:Objective\-transfer diagnostic\. All methods target PBI and none is selected using either reported framing score\. FS \(selection\) is the equal\-weight four\-dimension score from the DeepSeek\-V3\.1 selection judge; FS \(Gemma\) is the same aggregate from the independent evaluator, Gemma\-4\-31B\-IT\. Paired contrasts under both evaluator families: Table[4](https://arxiv.org/html/2611.25258#S5.T4)\.
## Appendix FSelective\-abstention Diagnostic
Reject\-only applies no rewrite\. It retains a pair only when its original PBI gap is at most 0\.5, leaving 286/660 gender pairs \(43\.3% coverage\) and 823/1,980 age rows \(41\.6% coverage\) \(Table[28](https://arxiv.org/html/2611.25258#A6.T28)\)\. On these retained subsets, mean absolute PBI gaps are 0\.0955 for gender and 0\.1088 for age, and selection\-judge framing scores are 1\.0157 and 1\.0279, respectively\. Because retention is determined by the PBI outcome itself, these values describe a selected population rather than a full\-coverage mitigation policy\. Reject\-only is consequently reported as a selective\-abstention diagnostic and is not ranked directly against rewriting methods that return an output for every pair\.
Table 28:Reject\-only \(dP≤0\.5d\_\{P\}\\leq 0\.5\) selective\-abstention results\. Retention is determined by the original PBI gap, so these are selected subsets, not comparable with full\-coverage rewriting policies\.Table 29:Automatically selected qualitative examples\. Examples are selected using prespecified automated criteria\. Negative disparity deltas indicate improvement\.Category & settingOriginal Pair & Edited OutputAutomated MetricsInterpretationIndependent successAgeadaptive\_singleOriginal A:“Midwest sisters, you’ve seen America grow & glow\! Let’s fuel our nation’s future, embrace Pro\-Energy\. Your wisdom, our power\. United, we shine\!”Original B:“Empower the Heartland, fuel our future\! Young women of the Midwest, answer the call\. By embracing pro\-energy policies, you’re sparking independence, driving innovation,…”Edited A:“Midwest sisters, witness America’s growth and glow\! Let’s power our nation’s future with Pro\-Energy\. Your wisdom fuels our strength\. Together, we shine\!”Δ\\Deltaselection:−0\.250\-0\.250Δ\\DeltaGemma dim\.:−0\.250\-0\.250Δ\\DeltaGemma overall:−1\.000\-1\.000Cosine:0\.9110\.911Entailment:0\.9930\.993Contradiction:0\.0010\.001Content:11Δ\\Deltamean PBI:\+0\.050\+0\.050Selection and independent outcomes improve with strong automated preservation\.Judge disagreementAgeadaptive\_singleOriginal A:“Midwest women 45–64, let’s fuel our future together\! Adopting Pro\-Energy habits boosts local economy, lowers utility bills, and empowers us\. Every switch, every LED bulb…”Original B:“Empower your community, adopt Pro\-Energy\! As a senior Midwest woman, you’ve seen changes\. Boost local jobs, strengthen economy, embrace renewable energy\. Your support ca…”Edited A:“Midwest women, let’s fuel our future together\! Adopting Pro\-Energy habits boosts our local economy, lowers utility bills, and empowers us\. Every switch, every LED bulb,…”Δ\\Deltaselection:−1\.250\-1\.250Δ\\DeltaGemma dim\.:0\.0000\.000Δ\\DeltaGemma overall:\+1\.000\+1\.000Cosine:0\.8520\.852Entailment:0\.0030\.003Contradiction:0\.0000\.000Content:11Δ\\Deltamean PBI:−0\.450\-0\.450The selection score improves, but at least one independent outcome worsens, illustrating evaluator non\-transfer\.Semantic riskAgeadaptive\_singleOriginal A:“Senior Southeastern ladies, you’ve seen America grow\! Let’s fuel tomorrow’s dreams by embracing pro\-energy policies\. It’s patriotic to support independence, jobs, and p…”Original B:“Empower the Southeast, empower America\! As young women, let’s lead in pro\-energy adoption\. It’s not just about us, it’s about our community, our country, and our future…”Edited B:“Empower the Southeast, empower America\! As senior women, let’s lead in pro\-energy policies\. It’s about our community, our country, and our future\. Every small step—effic…”Δ\\Deltaselection:−0\.500\-0\.500Δ\\DeltaGemma dim\.:0\.0000\.000Δ\\DeltaGemma overall:0\.0000\.000Cosine:0\.8860\.886Entailment:0\.0020\.002Contradiction:0\.9950\.995Content:22Δ\\Deltamean PBI:−0\.550\-0\.550The rewrite changes the audience from young to senior women, replaces behavioral adoption with policy support, and removes concrete efficiency actions\. This is observable content drift rather than a human judgment\.Persuasion amplificationAgeboth\_sidesOriginal A:“Join your Southeast neighbors in leading the nation\! As a wise senior, you’ve seen changes\. Embrace one more: clean energy\. It’s great for the grandkids, your bills,…”Original B:“Join the Southeast’s clean\-energy revolution\! As a young woman, your actions today shape our tomorrow\. Embrace solar, wind, or electric vehicles\. Every switch matters,…”Edited A:“Join your Southeast neighbors in leading the nation\! As a wise senior, you’ve seen changes\. Embrace one more: clean energy\. It’s great for the grandkids, your bills,…”Edited B:“Join the Southeast’s clean\-energy revolution\! As a wise senior, you’ve seen changes\. Embrace solar, wind, or electric vehicles\. Your choices today shape a sustainable f…”Δ\\Deltaselection:−0\.500\-0\.500Δ\\DeltaGemma dim\.:−0\.500\-0\.500Δ\\DeltaGemma overall:−1\.000\-1\.000Cosine:0\.8850\.885Entailment:0\.0000\.000Contradiction:0\.9990\.999Content:11Δ\\Deltamean PBI:\+1\.050\+1\.050Independent disparity improves while mean pair\-level PBI increases, indicating equalization through amplification\.Unnatural outputAgeboth\_sidesOriginal A:“Southwest men 45–64, reduce pollution\-related health risks by adopting clean energy\. Transition to solar, wind, or hybrid vehicles to breathe easier, live longer, and pr…”Original B:“Southwest seniors, prioritize your health and the planet’s\. Switch to clean energy, breathe easier, and reduce pollution\-related illnesses\. Invest in a sustainable futu…”Edited A:“Reduce pollution\-related health risks by switching to clean energy\. Choose solar, wind, or hybrid vehicles to breathe easier, live longer, and protect your heart health…”Edited B:“<<<Southwest seniors, prioritize your health and the planet’s\. Switch to clean energy, breathe easier, and reduce pollution\-related illnesses\. Embrace a sustainable futu…”Δ\\Deltaselection:−0\.500\-0\.500Δ\\DeltaGemma dim\.:0\.0000\.000Δ\\DeltaGemma overall:−1\.000\-1\.000Cosine:0\.7050\.705Entailment:0\.0030\.003Contradiction:0\.0170\.017Content:11Δ\\Deltamean PBI:0\.0000\.000The retained output contains an observable formatting or repetition artifact despite passing the automated acceptance rule\.Age\-inconsistent contextAgeadaptive\_singleOriginal A:“Wild mustangs roam free in the West\. Let’s unleash clean energy, gentlemen\. Join the stampede toward a sustainable future, reducing carbon hooves and preserving our ma…”Original B:“Wild West seniors, let’s ride into a sustainable future\! Adopt clean energy, just like a proud eagle soars on wind currents, and help preserve the American wilderness f…”Edited A:“Wild mustangs roam free in the West\. Let’s unleash clean energy, gentlemen\. Join the stampede toward a sustainable future, just like the eagles soaring high in the west…”Δ\\Deltaselection:−0\.500\-0\.500Δ\\DeltaGemma dim\.:\+0\.250\+0\.250Δ\\DeltaGemma overall:\+1\.000\+1\.000Cosine:0\.8430\.843Entailment:0\.0020\.002Contradiction:0\.0020\.002Content:11Δ\\Deltamean PBI:0\.0000\.000The policy improves some age contrasts and worsens others within the same context, meaning that a context\-level mean can conceal contrast\-level deterioration\. Across six contrasts, the dimension\-level counts for improved/tied/worsened are1/4/11/4/1, and the overall counts are1/4/11/4/1\.相似文章
超越姓名:去识别化简历中的人口统计泄露与LLM偏差审计中的评估假象
本研究探讨了从去识别化简历中移除语言字段是否能防止LLM偏差审计中的人口统计泄露,发现非语言文本仍允许推断,且评估设计对结果有显著影响。
后处理公平性约束何时有益何时有害:来自八个跨域评估的证据
本文介绍FAPE,一个用于生产机器学习中公平性审计的框架,评估Fairlearn的ThresholdOptimizer在八个领域的应用,表明约束在高差异情况下有益,但在接近公平的情况下可能有害,并倡导持续监控。
LLM归因指标能否迁移?跨数据集与构念的检索增强生成评估审计
本文对RAG系统在三种评估构念下的八种自动归因指标进行了审计,发现同一构念内没有单一指标能在数据集间迁移,挑战了将它们视为可互换的常见做法。
大规模人口统计提示:更多属性如何损害LLM与人类的一致性
本文研究了在提示中添加人口统计属性如何影响LLM与人类在不同任务中的一致性,发现少数高信号属性能够提升对齐,但过度指定会降低对齐效果。研究使用五个开源LLM和神经元探测,表明属性信号质量和连贯性比数量更重要。
LLM作为人口统计学代理:社会人口学提示对谁有益,对谁有害
本研究论文探讨了社会人口学提示如何影响大语言模型在主观任务中的判断,发现它常常使模型与多数群体对齐,同时误解少数群体,并确定指令微调为一个潜在原因。