Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding

# Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding
Source: [https://arxiv.org/html/2608.21415](https://arxiv.org/html/2608.21415)
IEEE Publication TechnologyThis paper was produced by the IEEE Publication Technology Group\. They are in Piscataway, NJ\.Manuscript received April 19, 2021; revised August 16, 2021\.Yisong Xiao, Aishan Liu🖂, Yongxin Huang, Zonghao Ying, Shiji Zhao, Tianlin Li, Yong Han🖂, Jian Yang, and Xianglong LiuY\. Xiao, A\. Liu, Y\. Huang, Z\. Ying, S\. Zhao, T\. Li, Y\. Han, J\. Yang, and X\. Liu are with the State Key Lab of Software Development Environment, Beihang University, Beijing 100191, China\. \(🖂 Corresponding author: Aishan Liu, liuaishan@buaa\.edu\.cn; Yong Han, hanyong@buaa\.edu\.cn\)

###### Abstract

Large Vision\-Language Models \(LVLMs\) have achieved remarkable performance across a wide range of tasks; however, they often inherit social biases from their training data, resulting in biased behavior when processing portraits from different social groups\. Existing debiasing approaches typically compare token probabilities between the original and biased generations during decoding, but they are fundamentally limited by their reliance on a single, stereotyped viewpoint and fail to account for the diversity of social perspectives\. Inspired by the social science principle that diversity fosters fairness, we propose Counterfactual Ensemble Decoding \(CED\), a novel framework that constructs multi\-group counterfactual perspectives within the visual representation space and integrates them during decoding to promote equitable model behavior\. CED first performs counterfactual steering in the visual space by identifying semantic directions associated with each social group and generating counterfactual representations along these directions, thereby offering diverse perspectives that disrupt stereotypical narratives\. During decoding, CED locates the decoder layer exhibiting the greatest divergence among these perspectives and ensembles their token distributions using uncertainty\-aware weights, prioritizing high\-confidence tokens from different groups to yield a more balanced probability distribution that guides fairer generation\. Extensive experiments on three social bias evaluation benchmarks demonstrate that CED achieves substantial improvements over leading baselines, reducing bias by up to 47\.97% across scenarios involving occupations, descriptors, and persona traits\. Moreover, CED also preserves the core capabilities of the original model with minimal degradation\. Our code can be found here111[https://github\.com/xiaoyisong/CED](https://github.com/xiaoyisong/CED)\.

## IIntroduction

Large Vision\-Language Models \(LVLMs\) have significantly advanced in recent years\[[56](https://arxiv.org/html/2608.21415#bib.bib92),[35](https://arxiv.org/html/2608.21415#bib.bib90),[37](https://arxiv.org/html/2608.21415#bib.bib91)\], which expands the capabilities of large language models \(LLMs\)\[[7](https://arxiv.org/html/2608.21415#bib.bib8),[49](https://arxiv.org/html/2608.21415#bib.bib11)\]through visual modality integration, achieving remarkable performance in tasks such as visual question answering and image captioning\[[24](https://arxiv.org/html/2608.21415#bib.bib94),[54](https://arxiv.org/html/2608.21415#bib.bib93),[51](https://arxiv.org/html/2608.21415#bib.bib95)\]\. Despite the immense success, a persistent concern with LVLMs is the social stereotypes and biases they may inherit from training data\[[21](https://arxiv.org/html/2608.21415#bib.bib62),[14](https://arxiv.org/html/2608.21415#bib.bib29)\], resulting in biased and harmful outcomes for specific social groups, especially with respect to protected attributes such as gender and race\. For example, research has demonstrated that gender bias is prevalent in LVLMs, associating specific emotions, occupations, and sexualized content with females\[[3](https://arxiv.org/html/2608.21415#bib.bib26),[22](https://arxiv.org/html/2608.21415#bib.bib24),[52](https://arxiv.org/html/2608.21415#bib.bib18)\]\. These biases can cause substantial harm to society and undermine the trustworthiness of LVLMs\. Therefore, addressing bias in LVLMs is essential to ensure their ethical and responsible deployment in sensitive applications\.

Social bias in LVLMs refers to the disparate treatment of individuals or groups\[[31](https://arxiv.org/html/2608.21415#bib.bib28),[14](https://arxiv.org/html/2608.21415#bib.bib29)\], which stems from the underrepresentation or biased portrayal of certain social groups in the training data, reinforcing harmful stereotypes and perpetuating representational harm\[[10](https://arxiv.org/html/2608.21415#bib.bib63)\]\. To address this issue, numerous bias mitigation approaches\[[12](https://arxiv.org/html/2608.21415#bib.bib48),[13](https://arxiv.org/html/2608.21415#bib.bib19),[53](https://arxiv.org/html/2608.21415#bib.bib41),[18](https://arxiv.org/html/2608.21415#bib.bib43),[45](https://arxiv.org/html/2608.21415#bib.bib30),[30](https://arxiv.org/html/2608.21415#bib.bib31),[46](https://arxiv.org/html/2608.21415#bib.bib35),[38](https://arxiv.org/html/2608.21415#bib.bib53)\]have been proposed\. Prior work\[[53](https://arxiv.org/html/2608.21415#bib.bib41),[18](https://arxiv.org/html/2608.21415#bib.bib43)\]typically involves fine\-tuning models on rebalanced datasets, often supplemented with counterfactual data\. However, these training\-stage approaches are resource\-intensive, requiring costly data collection and substantial computational resources, which restrict their practical viability\. Consequently, several approaches have aimed to improve efficiency by mitigating bias during inference, often utilizing techniques like decoding modification\[[46](https://arxiv.org/html/2608.21415#bib.bib35),[38](https://arxiv.org/html/2608.21415#bib.bib53)\], which compare token probabilities between the original and biased generations to suppress biased tokens\. However, these methods are fundamentally limited by their reliance on a single, stereotypical perspective, as they fail to incorporate the diverse viewpoints of different social groups that are crucial for effectively promoting fairness\.

![Refer to caption](https://arxiv.org/html/2608.21415v1/x1.png)Figure 1:Illustration of gender bias in LVLMs within the occupation scenario\. We introduce CED for debiasing, which ensembles diverse perspectives to disrupt the dominance of stereotypical narratives\.To address this challenge, we propose Counterfactual Ensemble Decoding \(CED\), a debiasing framework that constructs multi\-group counterfactual perspectives within the visual representation space and integrates them during the decoding process, thereby promoting fairer token probability distributions \(as depicted in Figure[1](https://arxiv.org/html/2608.21415#S1.F1)\)\. Our approach draws inspiration from the widely recognized social science principle that increasing diversity fosters fairness\[[9](https://arxiv.org/html/2608.21415#bib.bib37),[28](https://arxiv.org/html/2608.21415#bib.bib38),[11](https://arxiv.org/html/2608.21415#bib.bib39),[6](https://arxiv.org/html/2608.21415#bib.bib40)\], where disrupting the dominance of stereotypical narratives and integrating diverse perspectives serves to effectively mitigate biases targeting specific social groups\. Notably, counterfactual examples\[[29](https://arxiv.org/html/2608.21415#bib.bib68)\]differ from the original input in protected attributes \(*e\.g\.*, gender, which is prone to triggering stereotypical biases\) while preserving other visual scene context and task\-related information, providing an ideal source of diverse perspectives for debiasing\.

Therefore, CED performs counterfactual steering within the visual representation space to bypass the difficulty of directly modifying visual content\. Specifically, CED identifies semantic directions associated with each social group and applies directional steering to generate counterfactual representations, which reflect diverse social group perspectives that disrupt stereotypical narratives\. During decoding, CED locates the decoder layer with the greatest divergence among these perspectives by examining token distribution differences in the vocabulary space, since prior work\[[39](https://arxiv.org/html/2608.21415#bib.bib64),[43](https://arxiv.org/html/2608.21415#bib.bib65)\]has shown that biases are often encoded in specific layers\. Finally, CED employs uncertainty\-aware weights to ensemble the token distributions from different perspectives at this layer, prioritizing the high\-confidence tokens that represent each group, thereby yielding a more balanced token probability distribution and promoting fairer behavior\.

To evaluate the performance of our CED, we conduct extensive experiments on three bias evaluation benchmarks across widely used LVLMs\. Compared to four state\-of\-the\-art bias mitigation methods, CED demonstrates significantly superior effectiveness, achieving: ❶ an average reduction of 61\.21% in occupation\-related gender bias on the GenderBias\-VL\[[52](https://arxiv.org/html/2608.21415#bib.bib18)\]; ❷ an average reduction of 47\.97% in race\-related stereotypical bias on the ModSCAN\[[26](https://arxiv.org/html/2608.21415#bib.bib20)\], covering occupation, descriptor, and persona trait scenarios; ❸ average reductions of 38\.22% in stereotypical word frequency difference and 65\.75% in sentiment difference across gendered occupation descriptions on the VisBias\[[23](https://arxiv.org/html/2608.21415#bib.bib21)\]\. Furthermore, CED effectively preserves the core capabilities of the LVLM with minimal impact on overall performance\. Our maincontributionsare:

- •We propose CED, an inference\-stage debiasing framework for LVLMs that ensembles counterfactual perspectives from different groups, disrupting the dominance of stereotypical narratives and promoting fairness\.
- •We develop a counterfactual steering strategy to generate diverse perspectives in the representation space and ensemble their distributions at the most conflicting layer during decoding to achieve a more equitable output\.
- •Extensive experiments show that CED significantly outperforms leading baselines in bias mitigation effectiveness, while preserving the core capabilities of LVLMs\.

## IIRelated Works

In this section, we review bias mitigation methods for both LVLMs and LLMs, since methods specifically developed for LVLMs are still relatively limited\[[45](https://arxiv.org/html/2608.21415#bib.bib30),[30](https://arxiv.org/html/2608.21415#bib.bib31)\]\. Broadly, these approaches can be divided into two categories: training\-stage methods and inference\-stage methods\.

Training\-stage methodsmitigate bias by modifying the data distribution or model learning process\. Counterfactual Data Augmentation \(CDA\)\[[41](https://arxiv.org/html/2608.21415#bib.bib42),[18](https://arxiv.org/html/2608.21415#bib.bib43)\]mitigates bias by generating counterfactual images to rebalance the training data for fine\-tuning\. However, such methods rely on carefully curated datasets and significant computational resources, leading to limited practicality and scalability\.

Inference\-stage methodsgenerally fall into three categories\. ❶Prompt engineering methods\[[20](https://arxiv.org/html/2608.21415#bib.bib49),[48](https://arxiv.org/html/2608.21415#bib.bib50),[13](https://arxiv.org/html/2608.21415#bib.bib19),[15](https://arxiv.org/html/2608.21415#bib.bib51)\]leverage the model’s instruction\-following capabilities to mitigate bias\. Gallegosetal\.\[[13](https://arxiv.org/html/2608.21415#bib.bib19)\]proposed two debiasing strategies for multiple\-choice questions: one prompts the model to explain potential biases in the choices \(explanation\), while the other directly asks it to remove bias from the initial response \(reprompt\)\. However, their effectiveness is often unstable\. ❷Projection\-based methods\[[45](https://arxiv.org/html/2608.21415#bib.bib30),[30](https://arxiv.org/html/2608.21415#bib.bib31),[17](https://arxiv.org/html/2608.21415#bib.bib32),[34](https://arxiv.org/html/2608.21415#bib.bib33)\]mitigate bias by identifying a protected\-attribute subspace and removing its projection from the representation\. For example, PAR\[[45](https://arxiv.org/html/2608.21415#bib.bib30)\]estimates a bias direction from biased and benign image\-text pairs, while Lanetal\.\[[30](https://arxiv.org/html/2608.21415#bib.bib31)\]further identifies both bias and fair directions through interventions on residual representations\. However, their effectiveness may be limited, since bias in intermediate representations does not always align with bias in downstream outputs\[[14](https://arxiv.org/html/2608.21415#bib.bib29),[4](https://arxiv.org/html/2608.21415#bib.bib34)\]\. ❸Decoding\-based methods\[[46](https://arxiv.org/html/2608.21415#bib.bib35),[8](https://arxiv.org/html/2608.21415#bib.bib52),[38](https://arxiv.org/html/2608.21415#bib.bib53)\]adjust the token probability distribution during decoding to discourage biased language generation\. Schicketal\.\[[46](https://arxiv.org/html/2608.21415#bib.bib35)\]developed Self\-Debias, which reduces bias by adding a prompt prefix that deliberately encourages biased generation, and then compares token probabilities of biased versus original continuations to select fairer outputs\.

Our approach belongs to decoding\-based methods butdistinguishesitself in the following ways: ❶Motivation\. Drawing on the social science principle\[[9](https://arxiv.org/html/2608.21415#bib.bib37),[28](https://arxiv.org/html/2608.21415#bib.bib38),[11](https://arxiv.org/html/2608.21415#bib.bib39),[6](https://arxiv.org/html/2608.21415#bib.bib40)\]that increasing diversity fosters fairness, CED constructs multi\-group counterfactual perspectives and integrates them during decoding, while prior methods rely on a single, stereotypical perspective\. ❷Implementation\. CED applies directional steering within the visual representation space to generate diverse perspectives and then ensembles their token distributions at the layer with the greatest conflict\. In contrast, similar work\[[46](https://arxiv.org/html/2608.21415#bib.bib35)\]compares token probabilities between the original and biased generations\. ❸Effects\. By incorporating perspectives from different social groups, CED disrupts the dominance of stereotypical narratives and consistently achieves more effective bias mitigation than existing methods\.

We note that similar decoding techniques\[[32](https://arxiv.org/html/2608.21415#bib.bib80),[44](https://arxiv.org/html/2608.21415#bib.bib96)\]have been used for hallucination mitigation, but they differ fundamentally in both objective and implementation: those methods improve visual faithfulness by contrasting hallucinated and faithful generations, whereas our method mitigates social bias by ensembling perspectives from different social groups\.

## IIIPreliminaries

In this section, we first briefly introduce the LVLM decoding process, and then illustrate the problem definition\.

### III\-ALVLM Decoding

Consider an LVLM parameterized byθ\\theta, which consists of a vision encoder, a vision\-language alignment interface, and anLtextL\_\{\\text\{text\}\}\-layer LLM\. Given an input imagevvand a textual queryxx, the model first encodesvvinto visual representations𝒉​𝒗Limg\\bm\{hv\}^\{L\_\{\\text\{img\}\}\}, which are then concatenated with the tokenized queryxxand fed into the LLM for autoregressive generation\. The probability of the next tokenyty\_\{t\}is defined as

yt∼p​\(yt∣v,x,y<t\)=softmax​\(fθ​\(yt∣v,x,y<t\)\),y\_\{t\}\\sim p\(y\_\{t\}\\mid v,x,y\_\{<t\}\)=\\mathrm\{softmax\}\\big\(f\_\{\\theta\}\(y\_\{t\}\\mid v,x,y\_\{<t\}\)\\big\),\(1\)whereyt∈𝒱y\_\{t\}\\in\\mathcal\{V\}denotes the token at steptt,y<ty\_\{<t\}denotes the previously generated tokens, andfθf\_\{\\theta\}denotes the logit distribution over the vocabulary𝒱\\mathcal\{V\}\.

During generation, the LLM produces layer\-wise hidden states𝒉​𝒕lt\\bm\{ht\}^\{l\_\{t\}\}, wherelt∈\{1,2,⋯,Ltext\}l\_\{t\}\\in\\\{1,2,\\cdots,L\_\{\\text\{text\}\}\\\}\. The logits for predicting the next token are obtained from the final\-layer hidden state through the vocabulary projection headϕ​\(⋅\)\\phi\(\\cdot\):

fθ​\(yt∣v,x,y<t\)=fθ​\(yt∣𝒉​𝒗Limg,x,y<t\)=ϕ​\(𝒉​𝒕t−1Ltext\)\.f\_\{\\theta\}\(y\_\{t\}\\mid v,x,y\_\{<t\}\)=f\_\{\\theta\}\(y\_\{t\}\\mid\\bm\{hv\}^\{L\_\{\\text\{img\}\}\},x,y\_\{<t\}\)=\\phi\(\\bm\{ht\}\_\{t\-1\}^\{L\_\{\\text\{text\}\}\}\)\.\(2\)Therefore, the generation of the next tokenyty\_\{t\}in Equation[1](https://arxiv.org/html/2608.21415#S3.E1)can be written as:p​\(yt∣v,x,y<t\)=softmax​\(ϕ​\(𝒉​𝒕t−1Ltext\)\)\.p\(y\_\{t\}\\mid v,x,y\_\{<t\}\)=\\text\{softmax\}\(\\phi\(\\bm\{ht\}^\{L\_\{\\text\{text\}\}\}\_\{t\-1\}\)\)\.For brevity, we denotep​\(yt∣v,x,y<t\)p\(y\_\{t\}\\mid v,x,y\_\{<t\}\)asp​\(yt\)p\(y\_\{t\}\)in the following\.

### III\-BProblem Definition

During training on large\-scale image\-text data, LVLMs may inherit social biases present in the data\. As a result, LVLMs often unfairly make stereotype\-driven inferences \(*e\.g\.*, doctor\) based on the social groups depicted in the images \(*e\.g\.*, male\), assigning disproportionately high probabilities to tokens that reinforce such stereotypes\. This biased content can marginalize specific groups, create divisive social environments, and potentially exacerbate societal conflicts\[[50](https://arxiv.org/html/2608.21415#bib.bib71)\]\. Therefore, mitigating these stereotypes is crucial for improving the fairness of LVLM outputs\. Given a protected attributeaa, the input population can be divided into distinct social groups, denoted asGa=\{g1,g2,…,gn\}G^\{a\}=\\\{g\_\{1\},g\_\{2\},\\dots,g\_\{n\}\\\}, wherenndepends on the attributeaa\(*e\.g\.*,n=2n=2for binary gender\)\. The goal of bias mitigation is to reduce disparities in model outputs across groups inGaG^\{a\}by discouraging stereotype\-driven responses conditioned on the social group depicted in the input image\.

![Refer to caption](https://arxiv.org/html/2608.21415v1/x2.png)Figure 2:Overview of CED\. CED identifies directions for each social group and applies steering to generate counterfactual visual representations, introducing diverse perspectives\. Then, CED locates the layer with the greatest conflict among these perspectives and ensembles their token distributions, disrupting the dominance of stereotypical narratives and promoting balanced token probabilities in the predictions\.

## IVMethodology

### IV\-AOverview

Motivation\. Our work is inspired by the social science principle that increasing diversity fosters fairness, which suggests that integrating diverse perspectives can disrupt the dominance of stereotypical narratives, thereby mitigating bias and promoting more equitable outcomes\[[9](https://arxiv.org/html/2608.21415#bib.bib37),[28](https://arxiv.org/html/2608.21415#bib.bib38),[11](https://arxiv.org/html/2608.21415#bib.bib39),[6](https://arxiv.org/html/2608.21415#bib.bib40)\]\. In LVLMs, biased behavior often arises when protected attributes depicted in images trigger harmful stereotypes, resulting in skewed token distributions that reinforce those stereotypes\. To mitigate bias, this principle naturally motivates us to incorporate diverse perspectives \(*i\.e\.*, samples from different social groups\) into LVLM decoding, thereby counteracting stereotype\-driven inferences and promoting more inclusive outputs\. Counterfactual examples\[[29](https://arxiv.org/html/2608.21415#bib.bib68)\]provide an effective way to introduce such diversity, as they ask whether a response would change if the depicted individual belonged to a different demographic group while preserving the remaining visual context and task\-relevant information\. Therefore, our objective is to*construct counterfactual counterparts to introduce diverse perspectives and ensemble them during decoding for fairer generation*\.

Overall Framework\. Building on the motivation outlined above, we propose Counterfactual Ensemble Decoding \(CED\), a framework that mitigates bias in LVLMs by constructing multi\-group counterfactual perspectives and integrating them during decoding, resulting in a more balanced token distribution that guides fairer generation\. Figure[2](https://arxiv.org/html/2608.21415#S3.F2)illustrates the overall framework of CED\. Specifically, CED identifies semantic directions associated with each social group and applies steering to construct counterfactual representations that capture diverse perspectives\. Then, CED locates the text decoder layer exhibiting the greatest divergence among these perspectives by measuring differences in token distributions, and ensembles the resulting token distributions at that layer using uncertainty\-aware weights\. In this way, CED reduces the dominance of harmful stereotypes and mitigates unfairly skewed token distributions, thereby promoting fairer behavior\.

### IV\-BCounterfactual Steering

As highlighted in our motivation, counterfactual examples are ideal for introducing diverse perspectives during inference, which differ in protected attributes while preserving other visual context\. However, directly modifying raw images to generate such counterfactual samples poses notable challenges, as it risks distorting visual semantics and incurs high computational costs from complex pixel\-level manipulation\. Therefore, we shift our focus to the continuous semantic representation space, which enables targeted modification of social groups through latent steering\.

Prior studies\[[42](https://arxiv.org/html/2608.21415#bib.bib72),[2](https://arxiv.org/html/2608.21415#bib.bib74)\]have demonstrated that human\-interpretable concepts \(*e\.g\.*, gender\) are encoded as linear directions in the representation space, making it possible to manipulate protected attributes through directional steering of visual representations\. To capture representations associated with different social groups, we leverage individual\-centric images from public datasets such as FairFace\[[27](https://arxiv.org/html/2608.21415#bib.bib75)\], which provide detailed annotations for attributes including gender and race\. Given an input imagevvfrom social groupgjg\_\{j\}, we feed it into the LVLM and extract its visual representation𝒉​𝒗l​v\\bm\{hv\}^\{lv\}at layerl​vlv, wherel​v∈\{1,2,⋯,Limg\}lv\\in\\\{1,2,\\cdots,L\_\{\\rm\{img\}\}\\\}indexes the layers of the vision encoder and alignment interface\. We then pair each representation with its corresponding social\-group label to form tuples\(𝒉​𝒗l​v,g\)\(\\bm\{hv\}^\{lv\},g\), which are used to construct an auxiliary dataset𝒟l​v\\mathcal\{D\}^\{lv\}\. This dataset links high\-dimensional visual representations to human\-interpretable social groups, enabling us to train classifiers that identify linear direction encoding specific social groups\.

For a protected attribute with two social groups \(*e\.g\.*, the binary gender attribute\), we train linear classifiers \(implemented via Logistic Regression\) for each layer on the dataset𝒟l​v\\mathcal\{D\}^\{lv\}, with the training process given by:

arg⁡min𝒅l​v⁡𝔼\(𝒉​𝒗l​v,g\)∼𝒟l​v​\[ℒCE​\(Pl​v​\(𝒉​𝒗l​v;𝒅l​v\),g\)\],\\arg\\min\_\{\\bm\{d\}^\{lv\}\}\\mathbb\{E\}\_\{\(\\bm\{hv\}^\{lv\},g\)\\sim\\mathcal\{D\}^\{lv\}\}\[\\mathcal\{L\}\_\{\\text\{CE\}\}\(P^\{lv\}\(\\bm\{hv\}^\{lv\};\\bm\{d\}^\{lv\}\),g\)\],\(3\)whereℒCE​\(⋅\)\\mathcal\{L\}\_\{\\text\{CE\}\}\(\\cdot\)represents the cross\-entropy loss function, and𝒅l​v\\bm\{d\}^\{lv\}denotes the learned parameter of classifierPl​vP^\{lv\}, which also serves as the direction associated specific social group \(*e\.g\.*, female\) for subsequent editing\. For the protected attribute \(*e\.g\.*, race\) with more than two social groups, we extend the direction learning using a one\-vs\-one strategy, training group\-pair\-specific linear directions𝒅\(gi,gj\)l​v\\bm\{d\}^\{lv\}\_\{\(g\_\{i\},g\_\{j\}\)\}for each pair of social groupsgig\_\{i\}andgjg\_\{j\}, while maintaining the same optimization objective as in Equation[3](https://arxiv.org/html/2608.21415#S4.E3)\.

The learned direction𝒅l​v\\bm\{d\}^\{lv\}provides a well\-defined pathway for manipulating the social attributes of a visual representation within the high\-dimensional semantic space\. Specifically, during inference, we guide the original visual representation along the editing direction𝒅l​v\\bm\{d\}^\{lv\}, generating its counterfactual counterpart as follows:

𝒉​𝒗^l​v=𝒉​𝒗l​v−α⋅PLimg​\(𝒉​𝒗Limg;𝒅Limg\)⋅𝝈l​v⋅𝒅l​v,\\bm\{\\hat\{hv\}\}^\{lv\}=\\bm\{hv\}^\{lv\}\-\\alpha\\cdot P^\{L\_\{\\rm\{img\}\}\}\(\\bm\{hv\}^\{L\_\{\\rm\{img\}\}\};\\bm\{d\}^\{L\_\{\\rm\{img\}\}\}\)\\cdot\\bm\{\\sigma\}^\{lv\}\\cdot\\bm\{d\}^\{lv\},\(4\)whereα\\alphais a hyperparameter controlling the manipulation magnitude\.PLimgP^\{L\_\{\\rm\{img\}\}\}denotes the classifier at the final vision layer, which estimates the original social group of the visual representation, thus ensuring steering away from the original group\. Considering the variability in representation ranges across different layers, we compute the standard deviation𝝈l​v\\bm\{\\sigma\}^\{lv\}for each layer based on the training dataset𝒟l​v\\mathcal\{D\}^\{lv\}\. Similarly, for protected attributes with multiple social groups \(n\>2n\>2\), we first determine the social group of the original representation via majority voting across all trained one\-vs\-one classifiers\. Then, we perform steering along the group\-pair\-specific directions between the original group and each of the remaining groups following Equation[4](https://arxiv.org/html/2608.21415#S4.E4), generatingn−1n\-1counterfactual representations\{𝒉​𝒗^il​v\}i=1n−1\\\{\\bm\{\\hat\{hv\}\}^\{lv\}\_\{i\}\\\}\_\{i=1\}^\{n\-1\}\.

It is important to note that our counterfactual generation does not involve creating real counterfactual images\. Instead, it overcomes the challenges of directly modifying images by performing layer\-wise steering on the original image’s representation toward the desired social group during the LVLM inference\. This enables the generation of counterfactual representations without altering other contextual elements \(*e\.g\.*, background\), effectively introducing diverse social group perspectives while preserving the model’s task performance\.

To summarize, we first learn the direction associated with each social group within the visual representation space, then steer the original image’s representation along these direction to generate counterfactual representations, which provide diverse perspectives \(*i\.e\.*, fairness resources\) for the subsequent decoding process\.

### IV\-CEnsemble Decoding

After generating the counterfactual visual representations, our objective is to ensemble the token probability distributions from both the original and counterfactual representations, integrating diverse perspectives to foster fairness\. To enhance the effectiveness of bias mitigation, we first locates the decoder layer exhibiting the greatest divergence among these perspectives, and then ensemble their token distributions within this layer to yield a more balanced probability distribution\.

#### IV\-C1Bias Layer Identification

Existing studies\[[39](https://arxiv.org/html/2608.21415#bib.bib64),[43](https://arxiv.org/html/2608.21415#bib.bib65)\]have shown that social bias is predominantly encoded in the middle and deeper layers of the model and is unevenly distributed across them\. Therefore, to more effectively integrate diverse perspectives, we dynamically identify the text decoder layer with the greatest divergence among them by analyzing the differences in their token distributions\. Specifically, we utilize the projection headϕ\\phito convert the representation𝒉​𝒕l​t\\bm\{ht\}^\{lt\}of LLM decoder layerl​tltinto a probability distribution over the vocabulary as follows:

pl​t​\(yt\)=softmax​\(ϕ​\(𝒉​𝒕t−1l​t\)\),l​t∈𝒞p^\{lt\}\(y\_\{t\}\)=\\text\{softmax\}\(\\phi\(\\bm\{ht\}^\{lt\}\_\{t\-1\}\)\),\\quad lt\\in\\mathcal\{C\}\(5\)where𝒞\\mathcal\{C\}denotes the pre\-defined candidate layer set \(the middle and deeper layers of the LLM\)\.

Given the textual queryxx, the original and counterfactual visual representations𝒉​𝒗Limg\\bm\{hv\}^\{L\_\{\\rm\{img\}\}\}and𝒉​𝒗^Limg\\bm\{\\hat\{hv\}\}^\{L\_\{\\rm\{img\}\}\}, we collect their corresponding representations \(*i\.e\.*, perspectives\)𝒉​𝒕t−1l​t\\bm\{ht\}^\{lt\}\_\{t\-1\}and𝒉​𝒕^t−1l​t\\bm\{\\hat\{ht\}\}^\{lt\}\_\{t\-1\}at each time step from the candidate layers of the LLM decoder\. To measure the conflict between them, we employ the Jensen\-Shannon Divergence \(JSD\) to calculate the token distribution difference:

b​i​a​stl​t=JSD​\(pl​t​\(yt\),p^l​t​\(yt\)\),bias^\{lt\}\_\{t\}=\\text\{JSD\}\(p^\{lt\}\(y\_\{t\}\),\\hat\{p\}^\{lt\}\(y\_\{t\}\)\),\(6\)wherep^l​t​\(yt\)=softmax​\(ϕ​\(𝒉​𝒕^t−1l​t\)\)\\hat\{p\}^\{lt\}\(y\_\{t\}\)=\\text\{softmax\}\(\\phi\(\\bm\{\\hat\{ht\}\}^\{lt\}\_\{t\-1\}\)\)denotes the token probability distribution derived from the counterfactual representation, andJSDprovides a symmetric and bounded distance measure\. Furthermore, we select the most biased layerl​t∗lt^\{\*\}for current token generation by choosing the one with the highestb​i​a​stl​tbias^\{lt\}\_\{t\}among the candidate layers𝒞\\mathcal\{C\}:

l​t∗=arg⁡maxl​t∈𝒞⁡JSD​\(pl​t​\(yt\),p^l​t​\(yt\)\)\.lt^\{\*\}=\\arg\\max\_\{lt\\in\\mathcal\{C\}\}\\text\{JSD\}\(p^\{lt\}\(y\_\{t\}\),\\hat\{p\}^\{lt\}\(y\_\{t\}\)\)\.\(7\)This selection is grounded in the fact that a higherb​i​a​st−1l​tbias^\{lt\}\_\{t\-1\}value indicates more intense conflicts and richer diversity between original and counterfactual perspectives within the layer, which, in turn, enables the targeted integration of these diverse perspectives to promote fairness during thett\-th token generation\. For counterfactual representations from multiple social groups \(n\>2n\>2\), we calculate the average JSD difference between the token probability distributions of the original representation and each of the other counterfactual representations, using this as the bias metric, and then similarly identify the most biased layer\.

#### IV\-C2Prediction Ensemble

Once the most biased layerl​t∗lt^\{\*\}is identified, we proceed to ensemble the next\-token prediction distributions from both the original and its counterfactual representations at this layer, thereby integrating the conflicting perspectives to promote fairness\. Directly ensembling the token distributions \(*e\.g\.*, by averaging\) could dilute the distinct perspectives of diverse groups, potentially reducing the effectiveness of fairness enhancement\. To preserve the representative perspectives of different groups, as reflected in high\-probability tokens, we compute token uncertainty, measured by entropy, and use it to dynamically adjust the ensemble weights for each token in the vocabulary, ensuring the integrity of diverse viewpoints is maintained\. The token uncertainty for the original input and its counterfactuals can be calculated as:

𝒖t=−pl​t∗​\(yt\)⋅log⁡pl​t∗​\(yt\),𝒖^t=−p^l​t∗​\(yt\)⋅log⁡p^l​t∗​\(yt\)\.\\displaystyle\\bm\{u\}\_\{t\}=\-p^\{lt^\{\*\}\}\(y\_\{t\}\)\\cdot\\log p^\{lt^\{\*\}\}\(y\_\{t\}\),\\bm\{\\hat\{u\}\}\_\{t\}=\-\\hat\{p\}^\{lt^\{\*\}\}\(y\_\{t\}\)\\cdot\\log\\hat\{p\}^\{lt^\{\*\}\}\(y\_\{t\}\)\.\(8\)Since lower uncertainty corresponds to more perspective\-rich tokens, we assign higher weights to these tokens to better preserve group\-specific representative perspectives\. These uncertainty values are then converted into adaptive ensemble weights using exponential normalization, as follows:

𝒘t=exp⁡\(−γ⋅𝒖t\)exp⁡\(−γ⋅𝒖t\)\+exp⁡\(−γ⋅𝒖^t\),\\displaystyle\\bm\{w\}\_\{t\}=\\frac\{\\exp\(\-\\gamma\\cdot\\bm\{u\}\_\{t\}\)\}\{\\exp\(\-\\gamma\\cdot\\bm\{u\}\_\{t\}\)\+\\exp\(\-\\gamma\\cdot\\bm\{\\hat\{u\}\}\_\{t\}\)\},\(9\)𝒘^t=exp⁡\(−γ⋅𝒖^t\)exp⁡\(−γ⋅𝒖t\)\+exp⁡\(−γ⋅𝒖^t\),\\displaystyle\\bm\{\\hat\{w\}\}\_\{t\}=\\frac\{\\exp\(\-\\gamma\\cdot\\bm\{\\hat\{u\}\}\_\{t\}\)\}\{\\exp\(\-\\gamma\\cdot\\bm\{u\}\_\{t\}\)\+\\exp\(\-\\gamma\\cdot\\bm\{\\hat\{u\}\}\_\{t\}\)\},whereγ\>0\\gamma\>0is a hyperparameter that controls the sensitivity of weights to uncertainty\.

The next\-token prediction distribution is then generated by applying uncertainty\-aware weighted ensembling to the projected outputs of diverse social group perspectives, as follows:

pCED​\(yt\)=softmax​\(𝒘t⋅ϕ​\(𝒉​𝒕t−1l​t∗\)\+𝒘^t⋅ϕ​\(𝒉​𝒕^t−1l​t∗\)\)\.p\_\{\\rm\{CED\}\}\(y\_\{t\}\)=\\text\{softmax\}\(\\bm\{w\}\_\{t\}\\cdot\\phi\(\\bm\{ht\}^\{lt^\{\*\}\}\_\{t\-1\}\)\+\\bm\{\\hat\{w\}\}\_\{t\}\\cdot\\phi\(\\bm\{\\hat\{ht\}\}^\{lt^\{\*\}\}\_\{t\-1\}\)\)\.\(10\)Finally, we apply early exiting at this layer during the current decoding step, allowing the counterfactual\-ensembled distributionpCED​\(yt\)p\_\{\\rm\{CED\}\}\(y\_\{t\}\)to directly guide the next\-token generation, reducing the model’s reliance on a single stereotypical perspective and promoting more equitable outputs\. For multiple social groups \(n\>2n\>2\), the prediction ensemble process follows a similar approach: we calculate the token uncertainty for each group, normalize the uncertainties to derive adaptive ensemble weights for each group’s representative perspectives, and then perform the weighted ensembling process\.

As highlighted in previous studies\[[33](https://arxiv.org/html/2608.21415#bib.bib79),[32](https://arxiv.org/html/2608.21415#bib.bib80)\], altering the decoding process may result in undesirable behavior, where an initially implausible token may be mistakenly assigned a high score after ensembling\. To mitigate this, we follow prior work\[[33](https://arxiv.org/html/2608.21415#bib.bib79),[32](https://arxiv.org/html/2608.21415#bib.bib80)\]and implement an adaptive plausibility constraint, which ensures that token selection is limited to high\-confidence tokens from the original input’s output distribution:

𝒱head​\(y<t\)\\displaystyle\\mathcal\{V\}\_\{\\rm\{head\}\}\(y\_\{<t\}\)=\{yt∈𝒱:pl​t∗​\(yt\)≥β​maxs⁡pl​t∗​\(s\)\},\\displaystyle=\\\{y\_\{t\}\\in\\mathcal\{V\}:p^\{lt^\{\*\}\}\(y\_\{t\}\)\\geq\\beta\\max\_\{s\}p^\{lt^\{\*\}\}\(s\)\\\},\(11\)pCED​\(yt\)\\displaystyle p\_\{\\rm\{CED\}\}\(y\_\{t\}\)=0,ifyt∉𝒱head​\(y<t\),\\displaystyle=0,\\quad\\text\{if\}\\quad y\_\{t\}\\notin\\mathcal\{V\}\_\{\\rm\{head\}\}\(y\_\{<t\}\),whereβ\\betais a hyperparameter in the range\[0,1\]\[0,1\]that controls the strength of the truncation\. By assigning a zero probability to tokens not in𝒱head​\(y<t\)\\mathcal\{V\}\_\{\\rm\{head\}\}\(y\_\{<t\}\), we effectively eliminate the risk of generating implausible tokens, thereby ensuring the consistency and reliability of the generated content\.

## VExperiments

### V\-AExperimental Setup

#### V\-A1Datasets and Evaluation Metrics

We evaluate CED on three social bias benchmarks, namely GenderBias\-VL\[[52](https://arxiv.org/html/2608.21415#bib.bib18)\], ModSCAN\[[26](https://arxiv.org/html/2608.21415#bib.bib20)\], and VisBias\[[23](https://arxiv.org/html/2608.21415#bib.bib21)\], covering both multiple\-choice and open\-ended VQA tasks\.

❶GenderBias\-VL\[[52](https://arxiv.org/html/2608.21415#bib.bib18)\]is a multiple\-choice VQA benchmark for occupation\-related gender bias in LVLMs\. It uses synthetic gender counterfactual images and asks models to choose between stereotypically gendered occupation pairs\. We use its top\-10 biased occupation pairs for evaluation\. Performance is measured by Accuracy \(A​c​cAcc\), Bias \(including pair\-levelBp​a​i​rB\_\{pair\}and overallBo​v​lB\_\{ovl\}\), and the idealized paired stereotype bias test score \(I​p​s​sIpss\)\. HigherA​c​cAccandI​p​s​sIpssindicate better performance, while higherBp​a​i​rB\_\{pair\}andBo​v​lB\_\{ovl\}indicate more severe bias\. ❷ModSCAN\[[26](https://arxiv.org/html/2608.21415#bib.bib20)\]evaluates racial stereotypes in occupations, descriptors, and persona traits\. It uses real images from UTKFace\[[57](https://arxiv.org/html/2608.21415#bib.bib82)\]and asks models to identify the face associated with a queried concept\. Bias is measured by the stereotypical bias scoreSb​i​a​sS\_\{bias\}, defined as the average absolute deviation between the predicted group probability and the ideal uniform probability\. A higherSb​i​a​sS\_\{bias\}indicates greater bias\. ❸VisBias\[[23](https://arxiv.org/html/2608.21415#bib.bib21)\]evaluates social bias in image description\. It contains 700 real\-world images from 7 occupations with gender as the protected attribute\. Bias is measured from generated descriptions in two aspects: stereotype\-related word frequency \(s​t​e​r​e​o​t​y​p​estereotype\)\[[19](https://arxiv.org/html/2608.21415#bib.bib84)\]and positive sentiment difference \(s​e​n​t​i​m​e​n​tsentiment\) based on VADER\[[25](https://arxiv.org/html/2608.21415#bib.bib83)\]\. Higher values of both metrics indicate greater bias\. To further assess debiasing effectiveness, we also report the percentage reduction in bias scores before and after debiasing, where a larger reduction indicates better debiasing performance\.

Additionally, we evaluate CED on MMBench\-en\-dev\[[40](https://arxiv.org/html/2608.21415#bib.bib81)\]and MMMU\-dev\[[55](https://arxiv.org/html/2608.21415#bib.bib22)\]to assess its impact on general LVLM capabilities after debiasing\. Accuracy is used as the evaluation metric for both benchmarks\.

#### V\-A2LVLMs

We evaluate CED on three widely used LVLMs: LLaVA\-1\.5\[[36](https://arxiv.org/html/2608.21415#bib.bib12)\], Qwen3\-VL\[[1](https://arxiv.org/html/2608.21415#bib.bib97)\], and InternVL2\[[5](https://arxiv.org/html/2608.21415#bib.bib23)\]\. For the multiple\-choice benchmarks, GenderBias\-VL and ModSCAN, we compute the log\-likelihood of each candidate option and select the most likely one as the final prediction\. For the open\-ended benchmark VisBias, we follow the default query format of each model and use greedy decoding to generate responses, with the maximum number of new tokens set to 128\.

#### V\-A3Baselines

For comparison, we include four inference\-stage baselines\. Two prompt engineering methods from\[[13](https://arxiv.org/html/2608.21415#bib.bib19)\]: ❶ reprompt, which asks the model to remove bias in its initial response, and ❷ explanation, which identifies potential biases in the choices before answering\. We also include one projection\-based method, ❸ GenProj\[[52](https://arxiv.org/html/2608.21415#bib.bib18)\], which removes bias by projecting visual representations onto the orthogonal complement of a gender subspace, and one decoding\-based method, ❹ SelfDebias\[[46](https://arxiv.org/html/2608.21415#bib.bib35)\], which contrasts token probabilities from biased and original generations during decoding\. We follow the original implementations and settings of all baselines\. For GenProj, we use the same images as CED for direction learning and perform projection in the visual space\.

#### V\-A4Implementation Details

For direction learning, we sample 1,000 images per social group from FairFace\[[27](https://arxiv.org/html/2608.21415#bib.bib75)\], Phase\[[16](https://arxiv.org/html/2608.21415#bib.bib76)\], and PATA\[[47](https://arxiv.org/html/2608.21415#bib.bib73)\]\. We use Logistic Regression as the linear classifier, with the maximum number of iterations set to 1,000\. In our study, gender includes female and male, while race includes White, Black, Asian, and Indian\. To account for different manipulation strengths across models and datasets, we search forα\\alphaover\{1,3,5,7,9\}\\\{1,3,5,7,9\\\}using a held\-out 10% split of each evaluation dataset for hyperparameter selection, and report the final results on the remaining 90%\. Unless otherwise specified, we setγ=1\\gamma=1andβ=0\.1\\beta=0\.1\. For the candidate layer set𝒞\\mathcal\{C\}, we use layers 20 to 32 for LLaVA\-1\.5 and InternVL2, and layers 24 to 36 for Qwen3\-VL, with a step size of 2\. All experiments are conducted on a server with an Intel Xeon Platinum 8358 CPU, 512GB RAM, and eight NVIDIA A800 GPUs \(40GB each\)\.

### V\-BResults on GenderBias\-VL

In this part, we present the bias evaluation results on GenderBias\-VL, with the overall results shown in Table[I](https://arxiv.org/html/2608.21415#S5.T1)and the bias for each occupation pair detailed in Table[II](https://arxiv.org/html/2608.21415#S5.T2)\. We also include a visualization in Figure[3](https://arxiv.org/html/2608.21415#S5.F3)to illustrate the performance of different debiasing approaches\. From these results, we can make severalobservationsas follows:

❶For the bias metricBo​v​lB\_\{ovl\}, CED achieves the largest reduction among all methods, with an average bias reduction of 61\.21% relative to the original models and 25\.58% compared with the second\-best baseline, explanation\. For example, on LLaVA\-1\.5, CED reducesBo​v​lB\_\{ovl\}from 23\.24% to 7\.90%, while explanation remains 1\.72 times higher, indicating that models still exhibit a higher level of bias\. On InternVL2, explanation reduces bias only marginally, from 16\.21% to 14\.20%, whereas CED further lowers it to 9\.12%\. These results demonstrate the strong debiasing effectiveness and robustness of CED\.

❷In terms of accuracy \(A​c​cAcc\), CED preserves model performance well\. It improves accuracy by 0\.36% on LLaVA\-1\.5 and 2\.72% on InternVL2, while incurring only a minor drop on Qwen3\-VL, from 77\.65% to 75\.99%\. By contrast, the second\-best debiasing method \(explanation\) suffers a severe 7\.78% accuracy drop\. This suggests that CED introduces limited interference with task\-relevant information\.

❸For the idealized score \(I​p​s​sIpss\), CED consistently outperforms all baselines across the three LVLMs, achieving an average improvement of 7\.46%\. On LLaVA\-1\.5, for instance, CED reaches 63\.02%, exceeding the second\-best method \(explanation, 55\.05%\) by 7\.97%\. This indicates that CED strikes a better balance between fairness and accuracy\.

❹The pair\-wise results in Table[II](https://arxiv.org/html/2608.21415#S5.T2)further confirm the robustness of CED\. It achieves the lowestBp​a​i​rB\_\{pair\}on all 10 occupation pairs for InternVL2, and on 9 of 10 pairs for LLaVA\-1\.5 and Qwen3\-VL\. For highly biased pairs such aspair2\(CEO,Executive secretary\), CED reducesBp​a​i​rB\_\{pair\}from 37\.09% to 2\.01% on LLaVA\-1\.5\. It also remains effective on less biased pairs, showing consistent mitigation across diverse occupation scenarios\.

❺Figure[3](https://arxiv.org/html/2608.21415#S5.F3)provides a qualitative example of CED \. By ensembling different perspectives, CED reduces the probability gap for the stereotypicalLegal secretaryrole from 16\.36% to 6\.62%\. In contrast, explanation may overcorrect: for the base image, it suppresses the probability of the correctLegal secretaryoption to 15\.68%, reducing bias at the cost of substantial accuracy degradation\.

TABLE I:Results of different methods on the GenderBias\-VL\.![Refer to caption](https://arxiv.org/html/2608.21415v1/x3.png)Figure 3:Illustration of debiasing performance on an example from GenderBias\-VL\. The left shows the base image with the occupationLegal secretary, while the right is its gender counterfactual version\. Bias is measured as the absolute difference in the selection probability of theLegal secretaryoption between the original and counterfactual questions\.TABLE II:Bias results \(Bp​a​i​r↓B\_\{pair\}\{\\downarrow\}\) of different methods on each occupation pair from the GenderBias\-VL dataset\. The occupation pairs are as follows:pair1\(Dentist,Dental hygienist\),pair2\(CEO,Executive secretary\),pair3\(Surgeon,Surgical technologist\),pair4\(Lawyer,Legal secretary\),pair5\(Refractory mechanic,Filling operator\),pair6\(Aircraft pilot,Flight attendant\),pair7\(Computer Systems Manager,Receptionist\),pair8\(EMT,Licensed practical nurse\),pair9\(Network Architect,Billing Clerk\), andpair10\(Financial analyst,HR manager\)\.TABLE III:Race bias results \(Sb​i​a​s↓S\_\{bias\}\\downarrow\) on ModSCAN across occupation, descriptor, and persona trait scenarios\.
### V\-CResults on ModSCAN

Besides GenderBias\-VL, we further evaluate CED on ModSCAN, which uses real\-world images to assess stereotypical bias in occupation, descriptor, and persona trait scenarios\. Specifically, we extend the GenProj baseline to racial bias scenarios by sequentially subtracting the projections of model representations onto different racial group\-specific subspaces\. Table[III](https://arxiv.org/html/2608.21415#S5.T3)reports the overall results, and Figure[4](https://arxiv.org/html/2608.21415#S5.F4)visualizes the bias distribution of LLaVA\-1\.5 across the three scenarios, with results for other LVLMs provided in the Appendix\.

❶Table[III](https://arxiv.org/html/2608.21415#S5.T3)shows that all LVLMs exhibit noticeable racial bias across the three scenarios, while CED achieves the largest reduction among all methods, with an average bias reduction of 47\.97% across models and scenarios\. In contrast, existing baselines show limited effectiveness in mitigating bias\. The explanation and SelfDebias methods achieve average bias reductions of 16\.98% and 9\.95%, respectively, yet their performance is inconsistent across different scenarios and models\. For example, the explanation method performs poorly in the occupation scenario on LLaVA\-1\.5, achieving only a 4\.27% reduction in bias, compared to more substantial reductions of 23\.04% in the descriptor scenario and 31\.61% in the persona traits scenario\. The reprompt method brings minimal improvements \(averaging 1\.50%\) due to its reliance on superficial prompt instructions to guide debiasing, which is insufficient to disrupt the encoded stereotypical patterns\.

❷The racial bias distribution \(visualized in Figure[4](https://arxiv.org/html/2608.21415#S5.F4)\) further highlights the consistent and stable effectiveness of our method across different scenarios\. In the descriptor scenario, the original model exhibits a strong stereotypical association between the termterroristand images of Black people \(39\.20%\) while associating it with White people at the lowest rate \(12\.19%\), resulting in a highSb​i​a​sS\_\{bias\}of 9\.62%\. By integrating diverse counterfactual perspectives from different racial groups, CED effectively balances the association probabilities across demographic groups \(Black: 28\.70%, White: 25\.43%, Asian: 21\.04%, Indian: 24\.83%\), achieving a more equitable outcome with a significantly reducedSb​i​a​sS\_\{bias\}of 2\.07%\.

![Refer to caption](https://arxiv.org/html/2608.21415v1/x4.png)Figure 4:Distribution of LLaVA\-1\.5’s stereotypical racial bias \(Sb​i​a​sS\_\{bias\}\) on ModSCAN across occupation \(top\), descriptor \(middle\), and persona trait \(bottom\) scenarios\.TABLE IV:Results on VisBias\. The stereotype word bias score \(s​t​e​r​e​o​t​y​p​e↓stereotype\\downarrow\) is scaled by a factor of 1000, while the sentiment bias score \(s​e​n​t​i​m​e​n​t↓sentiment\\downarrow\) is reported as a percentage\.TABLE V:Capability evaluation results \( accuracy↑\\uparrow\) of different methods on three LVLMs\.
### V\-DResults on VisBias

On the open\-ended VisBias benchmark, CED also demonstrates superior performance, achieving lowers​t​e​r​e​o​t​y​p​estereotypeands​e​n​t​i​m​e​n​tsentimentscores\. Since VisBias is an open\-ended task without predefined options, the explanation method is not applicable\. Table[IV](https://arxiv.org/html/2608.21415#S5.T4)reports the overall results across LVLMs, Figure[5](https://arxiv.org/html/2608.21415#S5.F5)presents the occupation\-wise results, and Figure[6](https://arxiv.org/html/2608.21415#S5.F6)provides a qualitative example after debiasing\. From the results, we draw the following observations: ❶ For stereotypical words frequency difference, CED achieves the lowests​t​e​r​e​o​t​y​p​estereotypescore, reducing bias by 38\.22% compared to the original models, and outperforming the second\-best method, SelfDebias, by 25\.93%, thereby demonstrating its superior ability to neutralize the LVLMs’ tendency to associate specific genders with stereotypical words through ensembling counterfactual perspectives\. Specifically, for the femaleCEOimage in Figure[6](https://arxiv.org/html/2608.21415#S5.F6), CED disrupts the use of female stereotypical words likeattractiveand replaces them with more neutral terms such asprofessional\. ❷ For sentiment difference, CED again performs best, achieving the lowest averages​e​n​t​i​m​e​n​tsentimentscore of 0\.67%, corresponding to a 65\.75% reduction from the original models \(1\.93%\)\. For the femaleCEOimage in Figure[6](https://arxiv.org/html/2608.21415#S5.F6), CED reduces the positive sentiment score from 12\.4% to 9\.4%, neutralizing the inherent sentiment bias\. In addition, Figure[5](https://arxiv.org/html/2608.21415#S5.F5)shows that CED consistently reduces bias across different occupations\.

![Refer to caption](https://arxiv.org/html/2608.21415v1/x5.png)Figure 5:Bias distribution of LLaVA\-1\.5 on VisBias: stereotype bias \(top\) and sentiment bias \(bottom\)\.![Refer to caption](https://arxiv.org/html/2608.21415v1/x6.png)Figure 6:Illustration of bias mitigation using our CED in the description of the femaleCEOimage\. The stereotypical words are highlighted inred\.
### V\-EImpact on General Capability

We evaluate the impact of debiasing on general LVLM capability using MMBench and MMMU, with results reported in Table[V](https://arxiv.org/html/2608.21415#S5.T5)\. ❶ On MMBench, CED causes only a slight average accuracy drop of 0\.93% across LVLMs, indicating minimal impact on common reasoning abilities\. Notably, this drop is smaller than the performance degradation observed with other effective debiasing methods, such as explanation \(which incurs an 8\.85% drop\) and SelfDebias \(with a 2\.64% decrease\)\. ❷ On MMMU, CED achieves the highest accuracy among all debiasing methods\. CED even slightly improves the original model accuracy on LLaVA\-1\.5 by 3\.33% and on Qwen3\-VL by 1\.33%\. In contrast, the baselines exhibit varying degrees of performance degradation, with drops of 7\.45% for explanation and 3\.78% for SelfDebias, which may be affected by the interventions in their prompt prefixes\. These results show that CED preserves general model capability while maintaining strong debiasing performance\.

## VIDiscussion

Here, we provide additional analyses to better understand CED\. Using LLaVA\-1\.5 on GenderBias\-VL, we study the effects of the manipulation magnitudeα\\alphaand the uncertainty\-weighting hyperparameterγ\\gamma, and analyze the distribution of selected biased layers in𝒞\\mathcal\{C\}\. We also evaluate the inference efficiency of CED on VisBias\.

❶Counterfactual Manipulation Magnitude\.We study the effect of the manipulation magnitudeα\\alphaby varying it from 1 to 9, with results shown in Figure[7](https://arxiv.org/html/2608.21415#S6.F7)\. Overall, CED consistently outperforms all baselines across different settings\. However, the relationship between increasingα\\alphaand bias reduction is not monotonic\. Increasingα\\alphafrom 1 to 3 reducesBo​v​lB\_\{ovl\}by 2\.63%, whereas further increasing it to 9 leads to a 2\.75% increase in bias\. This suggests that small perturbations may be insufficient to introduce strong counterfactual perspectives for bias neutralization, while excessively large perturbations may impose overly dominant counterfactual perspectives, causing the model to skew toward the introduced social group\.

![Refer to caption](https://arxiv.org/html/2608.21415v1/x7.png)Figure 7:Bias \(Bo​v​lB\_\{ovl\}\) across varying counterfactual manipulation magnitudeα\\alpha\.❷Ensemble Sensitivity Factor\.To evaluate the impact of the ensemble sensitivityγ\\gammaon bias reduction, we varyγ\\gammavalues of 0\.25, 0\.5, 1, 2\.0, and 4\.0, withα\\alphafixed at 3\. The bias results are 7\.86%, 7\.87%, 7\.90%, 7\.94%, and 7\.99% in terms ofBo​v​lB\_\{ovl\}across these settings\. Generally, CED exhibits relatively stable debiasing performance across differentγ\\gammasettings, with a standard deviation of only 0\.05% inBo​v​lB\_\{ovl\}\. This stability can be attributed to the dynamic adjustment capability of token uncertainty, which ensures consistent debiasing performance regardless of ensemble sensitivity\.

❸Distribution of Biased Layers\.We analyze which layers are most frequently selected as the biased layer within the candidate set𝒞\\mathcal\{C\}for LLaVA\-1\.5 on GenderBias\-VL\. The results show that bias is concentrated in a few specific layers rather than uniformly distributed: layer 20 is selected most frequently \(26\.89%\), followed by layer 26 \(21\.64%\) and layer 32 \(19\.12%\), while the remaining layers are selected much less often \(24: 12\.35%, 28: 8\.95%, 22: 5\.63%, 30: 5\.42%\)\. We further verify the importance of bias layer identification by directly ensembling predictions at the final layer, which yields aBo​v​lB\_\{ovl\}of 11\.09%\. Although this still outperforms the second\-best baseline, explanation \(13\.61%\), it still underperforms CED with bias layer identification, which achieves aBo​v​lB\_\{ovl\}of 7\.90%\. These results highlight the importance of identifying the most conflicting layer for effective debiasing\.

❹Efficiency of CED\.We evaluate inference efficiency by measuring the time required to generate 128 tokens per prompt on VisBias\. As shown in Table[VI](https://arxiv.org/html/2608.21415#S6.T6), the original model takes 5\.49 seconds per prompt\. Among the more effective baselines, SelfDebias requires 12\.09 seconds due to additional comparisons with prefixed inputs, and reprompt takes 10\.20 seconds because it requires regeneration\. In contrast, CED takes 7\.80 seconds, making it more efficient than these stronger baselines\. Although GenProj is the fastest, its debiasing performance is clearly worse, with average bias scores of 10\.02% fors​t​e​r​e​o​t​y​p​estereotypeand 1\.51% fors​e​n​t​i​m​e​n​tsentiment, compared with 6\.19% and 0\.67% achieved by CED\. These results suggest that CED achieves more favorable efficiency than other effective baselines while maintaining superior debiasing performance\.

TABLE VI:Time \(seconds\) consumed by different bias mitigation methods to generate 128 tokens on the LLaVA\-1\.5 model\.
## VIIConclusion and Future Work

This paper proposes Counterfactual Ensemble Decoding \(CED\), a bias mitigation approach for LVLMs that constructs diverse perspectives in the visual space and ensembles them during decoding to promote fairer outputs\. CED first generates counterfactual representations through latent\-space steering, introducing diverse perspectives that disrupt stereotypical narratives\. CED then identifies the layer with the greatest divergence among these perspectives and ensembles their token distributions to produce a more balanced output distribution\. Extensive experiments on three social\-bias benchmarks show that CED consistently outperforms baseline methods in bias mitigation\. Moreover, CED preserves the general capabilities of the original model with minimal degradation\.

Limitations\. ❶ CED is a white\-box method that requires full access to the LVLM, as it intervenes in visual representations and the decoding process\. As noted in prior work\[[46](https://arxiv.org/html/2608.21415#bib.bib35),[45](https://arxiv.org/html/2608.21415#bib.bib30),[52](https://arxiv.org/html/2608.21415#bib.bib18)\], such access is often required for effective debiasing\. ❷ Our current evaluation focuses on gender and race, since benchmarks and annotated resources for other protected attributes in LVLMs remain limited\. As broader evaluation resources become available, we plan to extend our study to additional protected attributes and scenarios\.

## References

- \[1\]\(2025\)Qwen3\-vl technical report\.arXiv:2511\.21631\.Cited by:[§V\-A2](https://arxiv.org/html/2608.21415#S5.SS1.SSS2.p1.1)\.
- \[2\]U\. Bhalla, A\. Oesterling, S\. Srinivas, F\. Calmon, and H\. Lakkaraju\(2024\)Interpreting clip with sparse linear concept embeddings \(splice\)\.NeurIPS37,pp\. 84298–84328\.Cited by:[§IV\-B](https://arxiv.org/html/2608.21415#S4.SS2.p2.7)\.
- \[3\]A\. Birhane, V\. U\. Prabhu, and E\. Kahembwe\(2021\)Multimodal datasets: misogyny, pornography, and malignant stereotypes\.arXiv:2110\.01963\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1)\.
- \[4\]L\. Cabello, A\. K\. Jørgensen, and A\. Søgaard\(2023\)On the independence of association bias and empirical fairness in language models\.InProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency,pp\. 370–378\.Cited by:[§II](https://arxiv.org/html/2608.21415#S2.p3.1)\.
- \[5\]Z\. Chen, J\. Wu, W\. Wang, W\. Su, G\. Chen, S\. Xing, M\. Zhong, Q\. Zhang, X\. Zhu, L\. Lu,et al\.\(2024\)Internvl: scaling up vision foundation models and aligning for generic visual\-linguistic tasks\.InCVPR,pp\. 24185–24198\.Cited by:[§V\-A2](https://arxiv.org/html/2608.21415#S5.SS1.SSS2.p1.1)\.
- \[6\]Z\. Chen, X\. Li, J\. M\. Zhang, F\. Sarro, and Y\. Liu\(2025\)Diversity drives fairness: ensemble of higher order mutants for intersectional fairness of machine learning software\.In2025 IEEE/ACM 47th ICSE,pp\. 659–659\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p3.1),[§II](https://arxiv.org/html/2608.21415#S2.p4.1),[§IV\-A](https://arxiv.org/html/2608.21415#S4.SS1.p1.1)\.
- \[7\]W\. Chiang, Z\. Li, Z\. Lin, Y\. Sheng, Z\. Wu, H\. Zhang, L\. Zheng, S\. Zhuang, Y\. Zhuang, J\. E\. Gonzalez,et al\.\(2023\)Vicuna: an open\-source chatbot impressing gpt\-4 with 90%\* chatgpt quality, march 2023\.URL https://lmsys\. org/blog/2023\-03\-30\-vicuna3\(5\)\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1)\.
- \[8\]J\. Chung, E\. Kamar, and S\. Amershi\(2023\)Increasing diversity while maintaining accuracy: text data generation with large language models and human interventions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 575–593\.Cited by:[§II](https://arxiv.org/html/2608.21415#S2.p3.1)\.
- \[9\]T\. Cimpeanu, A\. Di Stefano, C\. Perret, and T\. A\. Han\(2023\)Social diversity reduces the complexity and cost of fostering fairness\.Chaos, Solitons & Fractals167,pp\. 113051\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p3.1),[§II](https://arxiv.org/html/2608.21415#S2.p4.1),[§IV\-A](https://arxiv.org/html/2608.21415#S4.SS1.p1.1)\.
- \[10\]K\. Crawford\(2017\)The trouble with bias\. keynote at neurips\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p2.1)\.
- \[11\]K\. Dierckx, A\. Van Hiel, B\. Valcke, and K\. van den Bos\(2023\)Procedural fairness in ethnic\-cultural decision\-making: fostering social cohesion by incorporating minority and majority perspectives\.Frontiers in Psychology14,pp\. 1025153\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p3.1),[§II](https://arxiv.org/html/2608.21415#S2.p4.1),[§IV\-A](https://arxiv.org/html/2608.21415#S4.SS1.p1.1)\.
- \[12\]Y\. Gaci, B\. Benattallah, F\. Casati, and K\. Benabdeslem\(2022\)Debiasing pretrained text encoders by paying attention to paying attention\.InEMNLP,pp\. 9582–9602\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p2.1)\.
- \[13\]I\. O\. Gallegos, R\. Aponte, R\. A\. Rossi, J\. Barrow, M\. M\. Tanjim, T\. Yu, H\. Deilamsalehy, R\. Zhang, S\. Kim, F\. Dernoncourt,et al\.Self\-debiasing large language models: zero\-shot recognition and reduction of stereotypes\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p2.1),[§II](https://arxiv.org/html/2608.21415#S2.p3.1),[§V\-A3](https://arxiv.org/html/2608.21415#S5.SS1.SSS3.p1.1)\.
- \[14\]I\. O\. Gallegos, R\. A\. Rossi, J\. Barrow, M\. M\. Tanjim, S\. Kim, F\. Dernoncourt, T\. Yu, R\. Zhang, and N\. K\. Ahmed\(2024\)Bias and fairness in large language models: a survey\.Computational Linguistics50\(3\),pp\. 1097–1179\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1),[§I](https://arxiv.org/html/2608.21415#S1.p2.1),[§II](https://arxiv.org/html/2608.21415#S2.p3.1)\.
- \[15\]D\. Ganguli, A\. Askell, N\. Schiefer, T\. I\. Liao, K\. Lukošiūtė, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. Olsson, D\. Hernandez,et al\.\(2023\)The capacity for moral self\-correction in large language models\.arXiv:2302\.07459\.Cited by:[§II](https://arxiv.org/html/2608.21415#S2.p3.1)\.
- \[16\]N\. Garcia, Y\. Hirota, Y\. Wu, and Y\. Nakashima\(2023\)Uncurated image\-text datasets: shedding light on demographic bias\.InCVPR,pp\. 6957–6966\.Cited by:[§V\-A4](https://arxiv.org/html/2608.21415#S5.SS1.SSS4.p1.5)\.
- \[17\]W\. Gerych, H\. Zhang, K\. Hamidieh, E\. Pan, M\. K\. Sharma, T\. Hartvigsen, and M\. Ghassemi\(2024\)Bendvlm: test\-time debiasing of vision\-language embeddings\.NeurIPS\.Cited by:[§II](https://arxiv.org/html/2608.21415#S2.p3.1)\.
- \[18\]S\. Ghanbarzadeh, Y\. Huang, H\. Palangi, R\. C\. Moreno, and H\. Khanpour\(2023\)Gender\-tuning: empowering fine\-tuning for debiasing pre\-trained language models\.arXiv:2307\.10522\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p2.1),[§II](https://arxiv.org/html/2608.21415#S2.p2.1)\.
- \[19\]N\. Ghavami and L\. A\. Peplau\(2013\)An intersectional analysis of gender and ethnic stereotypes: testing three hypotheses\.Psychology of Women Quarterly37\(1\),pp\. 113–127\.Cited by:[§V\-A1](https://arxiv.org/html/2608.21415#S5.SS1.SSS1.p2.12)\.
- \[20\]R\. Hida, M\. Kaneko, and N\. Okazaki\(2024\)Social bias evaluation for large language models requires prompt variations\.arXiv:2407\.03129\.Cited by:[§II](https://arxiv.org/html/2608.21415#S2.p3.1)\.
- \[21\]V\. Hofmann, P\. R\. Kalluri, D\. Jurafsky, and S\. King\(2024\)AI generates covertly racist decisions about people based on their dialect\.Nature,pp\. 1–8\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1)\.
- \[22\]P\. Howard, A\. Bhiwandiwalla, K\. C\. Fraser, and S\. Kiritchenko\(2024\)Uncovering bias in large vision\-language models with counterfactuals\.arXiv:2404\.00166\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1)\.
- \[23\]J\. Huang, J\. Qin, J\. Zhang, Y\. Yuan, W\. Wang, and J\. Zhao\(2025\)VisBias: measuring explicit and implicit social biases in vision language models\.InEMNLP,pp\. 17981–18004\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p5.1),[§V\-A1](https://arxiv.org/html/2608.21415#S5.SS1.SSS1.p1.1),[§V\-A1](https://arxiv.org/html/2608.21415#S5.SS1.SSS1.p2.12)\.
- \[24\]Q\. Huang, P\. He, P\. Li, X\. Lu, Y\. Tian, Y\. Cai, and Q\. Li\(2026\)Metaphorical visual question answering: benchmark and knowledge\-enhanced metaphor understanding method\.IEEE Transactions on Multimedia\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1)\.
- \[25\]C\. Hutto and E\. Gilbert\(2014\)Vader: a parsimonious rule\-based model for sentiment analysis of social media text\.InProceedings of the international AAAI conference on web and social media,Vol\.8,pp\. 216–225\.Cited by:[§V\-A1](https://arxiv.org/html/2608.21415#S5.SS1.SSS1.p2.12)\.
- \[26\]Y\. Jiang, Z\. Li, X\. Shen, Y\. Liu, M\. Backes, and Y\. Zhang\(2024\)ModSCAN: measuring stereotypical bias in large vision\-language models from vision and language modalities\.InEMNLP,pp\. 12814–12845\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p5.1),[§V\-A1](https://arxiv.org/html/2608.21415#S5.SS1.SSS1.p1.1),[§V\-A1](https://arxiv.org/html/2608.21415#S5.SS1.SSS1.p2.12)\.
- \[27\]K\. Karkkainen and J\. Joo\(2021\)Fairface: face attribute dataset for balanced race, gender, and age for bias measurement and mitigation\.InProceedings of the IEEE/CVF winter conference on applications of computer vision,pp\. 1548–1558\.Cited by:[§IV\-B](https://arxiv.org/html/2608.21415#S4.SS2.p2.7),[§V\-A4](https://arxiv.org/html/2608.21415#S5.SS1.SSS4.p1.5)\.
- \[28\]S\. Kim and S\. Park\(2017\)Diversity management and fairness in public organizations\.Public Organization Review17\(2\),pp\. 179–193\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p3.1),[§II](https://arxiv.org/html/2608.21415#S2.p4.1),[§IV\-A](https://arxiv.org/html/2608.21415#S4.SS1.p1.1)\.
- \[29\]M\. J\. Kusner, J\. Loftus, C\. Russell, and R\. Silva\(2017\)Counterfactual fairness\.NeurIPS30\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p3.1),[§IV\-A](https://arxiv.org/html/2608.21415#S4.SS1.p1.1)\.
- \[30\]J\. Lan, Y\. Fu, U\. Schlegel, G\. Zhang, T\. Hannan, H\. Chen, and T\. Seidl\(2025\)My answer is not’fair’: mitigating social bias in vision\-language models via fair and biased residuals\.arXiv:2505\.23798\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p2.1),[§II](https://arxiv.org/html/2608.21415#S2.p1.1),[§II](https://arxiv.org/html/2608.21415#S2.p3.1)\.
- \[31\]N\. Lee, Y\. Bang, H\. Lovenia, S\. Cahyawijaya, W\. Dai, and P\. Fung\(2023\)Survey of social bias in vision\-language models\.arXiv:2309\.14381\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p2.1)\.
- \[32\]S\. Leng, H\. Zhang, G\. Chen, X\. Li, S\. Lu, C\. Miao, and L\. Bing\(2024\)Mitigating object hallucinations in large vision\-language models through visual contrastive decoding\.InCVPR,pp\. 13872–13882\.Cited by:[§II](https://arxiv.org/html/2608.21415#S2.p5.1),[§IV\-C2](https://arxiv.org/html/2608.21415#S4.SS3.SSS2.p3.4)\.
- \[33\]X\. L\. Li, A\. Holtzman, D\. Fried, P\. Liang, J\. Eisner, T\. B\. Hashimoto, L\. Zettlemoyer, and M\. Lewis\(2023\)Contrastive decoding: open\-ended text generation as optimization\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 12286–12312\.Cited by:[§IV\-C2](https://arxiv.org/html/2608.21415#S4.SS3.SSS2.p3.4)\.
- \[34\]P\. P\. Liang, I\. M\. Li, E\. Zheng, Y\. C\. Lim, R\. Salakhutdinov, and L\. Morency\(2020\)Towards debiasing sentence representations\.arXiv:2007\.08100\.Cited by:[§II](https://arxiv.org/html/2608.21415#S2.p3.1)\.
- \[35\]B\. Lin, Z\. Tang, Y\. Ye, J\. Huang, J\. Zhang, Y\. Pang, P\. Jin, M\. Ning, J\. Luo, and L\. Yuan\(2026\)Moe\-llava: mixture of experts for large vision\-language models\.IEEE Transactions on Multimedia\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1)\.
- \[36\]H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee\(2023\)Improved baselines with visual instruction tuning\.arXiv:2310\.03744\.Cited by:[§V\-A2](https://arxiv.org/html/2608.21415#S5.SS1.SSS2.p1.1)\.
- \[37\]W\. Liu, B\. Miao, J\. Cao, X\. Zhu, J\. Ge, B\. Liu, M\. Nasim, and A\. Mian\(2025\)Context\-enhanced video moment retrieval with large language models\.IEEE Transactions on Multimedia\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1)\.
- \[38\]X\. Liu, M\. Khalifa, and L\. Wang\(2023\)BOLT: fast energy\-based controlled text generation with tunable biases\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 186–200\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p2.1),[§II](https://arxiv.org/html/2608.21415#S2.p3.1)\.
- \[39\]Y\. Liu, Y\. Liu, X\. Chen, P\. Chen, D\. Zan, M\. Kan, and T\. Ho\(2024\)The devil is in the neurons: interpreting and mitigating social biases in language models\.InICLR,Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p4.1),[§IV\-C1](https://arxiv.org/html/2608.21415#S4.SS3.SSS1.p1.3)\.
- \[40\]Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu,et al\.\(2024\)Mmbench: is your multi\-modal model an all\-around player?\.InECCV,pp\. 216–233\.Cited by:[§V\-A1](https://arxiv.org/html/2608.21415#S5.SS1.SSS1.p3.1)\.
- \[41\]K\. Lu, P\. Mardziel, F\. Wu, P\. Amancharla, and A\. Datta\(2020\)Gender bias in neural natural language processing\.Logic, language, and security: essays dedicated to Andre Scedrov on the occasion of his 65th birthday,pp\. 189–202\.Cited by:[§II](https://arxiv.org/html/2608.21415#S2.p2.1)\.
- \[42\]J\. Merullo, L\. Castricato, C\. Eickhoff, and E\. Pavlick\(2023\)Linearly mapping from image to text space\.InICLR,Cited by:[§IV\-B](https://arxiv.org/html/2608.21415#S4.SS2.p2.7)\.
- \[43\]N\. Prakash and R\. K\. Lee\(2023\)Layered bias: interpreting bias in pretrained large language models\.InProceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,pp\. 284–295\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p4.1),[§IV\-C1](https://arxiv.org/html/2608.21415#S4.SS3.SSS1.p1.3)\.
- \[44\]P\. Qiang, H\. Tan, H\. Zhang, X\. Li, R\. Li, and J\. Liang\(2026\)Mitigating hallucinations in large vision\-language models via visual\-enhanced contrastive decoding\.IEEE Transactions on Multimedia\.Cited by:[§II](https://arxiv.org/html/2608.21415#S2.p5.1)\.
- \[45\]N\. Ratzlaff, M\. L\. Olson, M\. Hinck, S\. Tseng, V\. Lal, and P\. Howard\(2025\)Debias your large multi\-modal model at test\-time with non\-contrastive visual attribute steering\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 6199–6208\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p2.1),[§II](https://arxiv.org/html/2608.21415#S2.p1.1),[§II](https://arxiv.org/html/2608.21415#S2.p3.1),[§VII](https://arxiv.org/html/2608.21415#S7.p2.1)\.
- \[46\]T\. Schick, S\. Udupa, and H\. Schütze\(2021\)Self\-diagnosis and self\-debiasing: a proposal for reducing corpus\-based bias in nlp\.Transactions of the Association for Computational Linguistics9,pp\. 1408–1424\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p2.1),[§II](https://arxiv.org/html/2608.21415#S2.p3.1),[§II](https://arxiv.org/html/2608.21415#S2.p4.1),[§V\-A3](https://arxiv.org/html/2608.21415#S5.SS1.SSS3.p1.1),[§VII](https://arxiv.org/html/2608.21415#S7.p2.1)\.
- \[47\]A\. Seth, M\. Hemani, and C\. Agarwal\(2023\)Dear: debiasing vision\-language models with additive residuals\.InCVPR,pp\. 6820–6829\.Cited by:[§V\-A4](https://arxiv.org/html/2608.21415#S5.SS1.SSS4.p1.5)\.
- \[48\]C\. Si, Z\. Gan, Z\. Yang, S\. Wang, J\. Wang, J\. Boyd\-Graber, and L\. Wang\(2022\)Prompting gpt\-3 to be reliable\.arXiv:2210\.09150\.Cited by:[§II](https://arxiv.org/html/2608.21415#S2.p3.1)\.
- \[49\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv:2307\.09288\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1)\.
- \[50\]Y\. Wan, W\. Wang, P\. He, J\. Gu, H\. Bai, and M\. R\. Lyu\(2023\)Biasasker: measuring the bias in conversational ai system\.InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering,pp\. 515–527\.Cited by:[§III\-B](https://arxiv.org/html/2608.21415#S3.SS2.p1.6)\.
- \[51\]H\. Wang, C\. Lai, and W\. Ge\(2025\)Adapting multimodal large language models for video question answering by capturing question\-critical and coherent moments\.IEEE Transactions on Multimedia\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1)\.
- \[52\]Y\. Xiao, X\. Liu, Q\. Cheng, Z\. Yin, S\. Liang, J\. Li, J\. Shao, A\. Liu, and D\. Tao\(2025\)GenderBias\-vl: benchmarking gender bias in vision language models via counterfactual probing\.International Journal of Computer Vision,pp\. 1–24\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1),[§I](https://arxiv.org/html/2608.21415#S1.p5.1),[§V\-A1](https://arxiv.org/html/2608.21415#S5.SS1.SSS1.p1.1),[§V\-A1](https://arxiv.org/html/2608.21415#S5.SS1.SSS1.p2.12),[§V\-A3](https://arxiv.org/html/2608.21415#S5.SS1.SSS3.p1.1),[§VII](https://arxiv.org/html/2608.21415#S7.p2.1)\.
- \[53\]D\. Xu, S\. Yuan, L\. Zhang, and X\. Wu\(2018\)Fairgan: fairness\-aware generative adversarial networks\.In2018 IEEE international conference on big data \(big data\),pp\. 570–575\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p2.1)\.
- \[54\]J\. Yan, B\. Dong, X\. Guan, W\. Zheng, T\. Zhang, and R\. Wang\(2026\)Adaptively fine\-tuning and ensembling vision\-language model for few\-shot image classification\.IEEE Transactions on Multimedia\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1)\.
- \[55\]X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun,et al\.\(2024\)Mmmu: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InCVPR,pp\. 9556–9567\.Cited by:[§V\-A1](https://arxiv.org/html/2608.21415#S5.SS1.SSS1.p3.1)\.
- \[56\]W\. Zhang, L\. Wu, Z\. Zhang, T\. Yu, C\. Ma, X\. Jin, X\. Yang, and W\. Zeng\(2024\)Unleash the power of vision\-language models by visual attention prompt and multimodal interaction\.IEEE Transactions on Multimedia27,pp\. 2399–2411\.Cited by:[§I](https://arxiv.org/html/2608.21415#S1.p1.1)\.
- \[57\]Z\. Zhang, Y\. Song, and H\. Qi\(2017\)Age progression/regression by conditional adversarial autoencoder\.InIEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§V\-A1](https://arxiv.org/html/2608.21415#S5.SS1.SSS1.p2.12)\.