Selective Test-Time Debiasing for CLIP via Reward Gating

# Selective Test-Time Debiasing for CLIP via Reward Gating
Source: [https://arxiv.org/html/2607.00423](https://arxiv.org/html/2607.00423)
Jaeho Han, Jisoo Yang, Hyeondong Woo, Mingyu Jeon, Sunjae Yoon, Junyeong Kim Department of Artificial Intelligence, Chung\-Ang University \{wogh50, yjs229, hyeondong, smart2557, sunjaeyoon, junyeongkim\}@cau\.ac\.kr

###### Abstract

Vision language models \(VLMs\) demonstrate strong zero\-shot performance, but often perpetuate social stereotypes in person\-centric queries, yielding skewed demographic distributions\. Current debiasing methods apply uniform bias corrections across all input queries regardless of their bias sensitivity, creating a fundamental fairness–utility trade\-off\. Strong debiasing distorts semantically meaningful information in bias\-insensitive queries, while weak debiasing fails to mitigate stereotypes in bias\-sensitive ones\. This one\-size\-fits\-all approach hampers simultaneously achieving high utility on bias\-insensitive queries and fairness on bias\-sensitive queries\. We introduceReward\-Gated Test\-Time Adaptation \(RG\-TTA\), a reinforcement learning\-based test\-time adaptation framework that selectively applies debiasing based on input sensitivity\. RG\-TTA adaptively triggers fairness regularization based on the bias sensitivity of each input during test\-time policy adaptation, while focusing exclusively on optimizing cross\-modal alignment for bias\-insensitive inputs\. Experiments on fairness benchmarks \(e\.g\., FairFace, UTKFace\) demonstrate substantial bias reduction while simultaneously improving zero\-shot utility, resolving the trade\-off of uniform debiasing\.

Selective Test\-Time Debiasing for CLIP via Reward Gating

Jaeho Han, Jisoo Yang, Hyeondong Woo, Mingyu Jeon, Sunjae Yoon, Junyeong KimDepartment of Artificial Intelligence, Chung\-Ang University\{wogh50, yjs229, hyeondong, smart2557, sunjaeyoon, junyeongkim\}@cau\.ac\.kr

![[Uncaptioned image]](https://arxiv.org/html/2607.00423v1/x1.png)

Figure 1:\(a\) We categorize inputs intobias\-sensitiveandbias\-insensitive, where only the former requires debiasing intervention\. \(b\) Existing methods applyuniform mitigation, creating a structural trade\-off: weak debiasing retains bias in sensitive queries \(left\), while strong debiasing distorts insensitive queries, degrading utility \(right\)\. \(c\) Our approach employsselective mitigation via reward\-gating, which applies strong debiasing only to bias\-sensitive inputs while preserving insensitive ones, ensuring both fairness and utility\.
## 1Introduction

Vision Language Models \(VLMs\) have demonstrated exceptional zero\-shot capabilities across a wide range of multimodal tasks\(Denget al\.,[2009](https://arxiv.org/html/2607.00423#bib.bib5); Plummeret al\.,[2015](https://arxiv.org/html/2607.00423#bib.bib6)\), reaching the stage of real\-world applications\. By learning joint representations from web\-scale image\-text pairs, these models achieve strong cross\-modal alignment without task\-specific fine\-tuning\. However, this same training paradigm causes VLMs to internalize social stereotypes\(Birhaneet al\.,[2021](https://arxiv.org/html/2607.00423#bib.bib8)\)present in their training data, leading to biased outputs that reflect and potentially amplify social prejudices\(Hallet al\.,[2023](https://arxiv.org/html/2607.00423#bib.bib1); Hamidiehet al\.,[2024](https://arxiv.org/html/2607.00423#bib.bib2); Janghorbani and De Melo,[2023](https://arxiv.org/html/2607.00423#bib.bib3); Zhaoet al\.,[2021](https://arxiv.org/html/2607.00423#bib.bib4); Wolfeet al\.,[2023](https://arxiv.org/html/2607.00423#bib.bib40); Hausladenet al\.,[2025](https://arxiv.org/html/2607.00423#bib.bib39)\)\. These biases manifest most critically in person\-centric queries, where models produce skewed demographic distributions\. For instance, querying “a photo of a doctor” yields disproportionately male images, or certain occupations become strongly associated with specific racial groups\. Such behavior poses serious risk of reinforcing discriminatory decision\-making\.

Existing debiasing approaches for VLMs\(Wanget al\.,[2021b](https://arxiv.org/html/2607.00423#bib.bib31); Chuanget al\.,[2023](https://arxiv.org/html/2607.00423#bib.bib32); Zhanget al\.,[2025](https://arxiv.org/html/2607.00423#bib.bib9)\)share common design philosophy: they apply fixed bias correction uniformly across all language queries, regardless of whether individual queries are sensitive to demographic biases\. Although conceptually simple, this uniform mitigation strategy causes a fundamental fairness–utility trade\-off that we illustrated in Figure[1](https://arxiv.org/html/2607.00423#S0.F1)\(b\)\. Here, we use utility to refer to general\-purpose zero\-shot performance on downstream tasks \(e\.g\., image classification and cross\-modal retrieval\)\.

For bias\-sensitive queries, the model’s predictions are often entangled with demographic attributes, and strong debiasing is necessary for fair outcomes\. In contrast for bias\-insensitive queries, ground\-truth semantics are largely orthogonal to demographic attributes, and the model’s original predictions already reflect accurate cross\-modal alignment\. When a uniform debiasing framework is applied to both types of queries, one of two failure modes inevitably occurs\. As we demonstrate in Figure[2](https://arxiv.org/html/2607.00423#footnote2), this inflexibility creates a trade\-off where existing methods must compromise between fairness in sensitive queries and utility in general tasks, unable to excel at both simultaneously\.

![[Uncaptioned image]](https://arxiv.org/html/2607.00423v1/x2.png)

Figure 2:Fairness versus utility for Race and Age\.Accuracyis measured as ImageNet zero\-shot top\-1 accuracy \(%\), andFairnessis measured as1−MaxSkew​@​10001\-\\mathrm\{MaxSkew@1000\}222MaxSkew@1000 is computed from the protected\-attribute distribution within the top\-1000 retrieved samples for neutral queries; see Sec\.[3](https://arxiv.org/html/2607.00423#S3)for details\.\(higher is better\)\. Existing query\-independent debiasing baselines\(Chuanget al\.,[2023](https://arxiv.org/html/2607.00423#bib.bib32); Wanget al\.,[2021b](https://arxiv.org/html/2607.00423#bib.bib31); Zhanget al\.,[2025](https://arxiv.org/html/2607.00423#bib.bib9)\)exhibit a fairness–utility trade\-off, whereas our method improves fairness while achieving higher accuracy\.
We believe that the aforementioned limitation is a consequence of the query\-independent design paradigm\. Since uniform debiasing methods cannot distinguish between inputs that require mitigation and those that do not, they are constrained to operate in a compromise regime\. This observation motivates a paradigm shift toward adaptive debiasing that selectively activates mitigation based on the bias sensitivity of each input\. To this end, we proposeReward\-Gated Test\-Time Adaptation for CLIP \(RG\-TTA\), a reinforcement learning \(RL\)\-based framework designed for selective debiasing\. Our key insight is that debiasing should be treated as a per\-query decision rather than global transformation, enabling the model to adapt its behavior dynamically based on input characteristics\. As illustrated in Figure[3](https://arxiv.org/html/2607.00423#S1.F3), RG\-TTA operates through an episodic test\-time adaptation protocol\. For each incoming query, we first assess its bias sensitivity by quantifying the alignment discrepancy between the query semantics and a set of demographic attributes\. Based on this assessment, an adaptive reward\-gating strategy dynamically triggers the fairness\-regularized objective only for bias\-sensitive queries, ensuring the preservation of the original cross\-modal alignment for neutral inputs\.

Empirical evaluations on multiple fairness benchmarks–including FairFace\(Kärkkäinen and Joo,[2021](https://arxiv.org/html/2607.00423#bib.bib33)\), UTKFace\(Zhanget al\.,[2017](https://arxiv.org/html/2607.00423#bib.bib34)\), and the challenging FACET\(Gustafsonet al\.,[2023](https://arxiv.org/html/2607.00423#bib.bib35)\)dataset–demonstrate that RG\-TTA significantly reduces social bias across various demographic attributes\. Notably, our framework effectively resolves the fairness–utility trade\-off by achieving substantial bias reduction alongside higher accuracy on tasks such as ImageNet\-1K\(Denget al\.,[2009](https://arxiv.org/html/2607.00423#bib.bib5)\)compared to existing query\-independent baselines\. By performing episodic optimization with this gated reward, RG\-TTA provides a practical design principle for mitigating bias without broadly disrupting the alignment of vision\-language models\.

![Refer to caption](https://arxiv.org/html/2607.00423v1/x3.png)Figure 3:Overview of RG\-TTA: RL\-based episodic test\-time adaptation withreward gatingvia the indicatorδ​\(q\)\\delta\(q\)\. For each query, we update only the query\-modality encoder \(text encoder for text queries; image encoder for image queries\) with a few policy\-gradient steps on a truncated top\-KKcandidate set\. The indicatorδ​\(q\)\\delta\(q\)controls the episode reward: whenδ​\(q\)=0\\delta\(q\)=0, we optimize an alignment\-only reward; whenδ​\(q\)=1\\delta\(q\)=1, we add an attribute\-balancing reward\. The gate activates when the top\-1 matchy∗y^\{\*\}is sufficiently close to the attribute\-alignment distribution, indicating elevated attribute entanglement\.
## 2Related Work

#### Social debiasing in vision\-language models\.

Social debiasing for CLIP\-style VLMs is broadly categorized into \(i\)*training\-based*approaches\(Alabdulmohsinet al\.,[2024](https://arxiv.org/html/2607.00423#bib.bib12); Hirotaet al\.,[2025b](https://arxiv.org/html/2607.00423#bib.bib10); Zhanget al\.,[2025](https://arxiv.org/html/2607.00423#bib.bib9)\), which suppress sensitive\-attribute signals by introducing additional objectives or modules during learning, and \(ii\)*training\-free*approaches\(Chuanget al\.,[2023](https://arxiv.org/html/2607.00423#bib.bib32); Gerychet al\.,[2024](https://arxiv.org/html/2607.00423#bib.bib11)\), which keep the foundation model fixed and apply post\-hoc adjustments to embeddings or outputs\. The former includes joint debiasing methods that align and remove biases across both modalities, while the latter estimates bias directions from attribute prompts and removes them via lightweight projections\. However, most existing methods remain*query\-independent*, applying a single globally\-defined transformation to all queries\. A recent attempt toward query\-adaptive debiasing is SANER\(Hirotaet al\.,[2025a](https://arxiv.org/html/2607.00423#bib.bib41)\), which performs selective debiasing by restricting intervention to attribute\-neutral text descriptions; yet its adjustment is confined to text features at training time, limiting adaptability to diverse input distributions encountered at deployment\. Consequently, such fixed or partially selective rules still struggle to optimize for both objectives simultaneously, forcing a compromise between effective bias mitigation and the preservation of general model utility\.

#### Test\-Time Adaptation of CLIP via Reinforcement Learning\.

Test\-time Adaptation \(TTA\) aims to improve model performance on unlabeled test inputs by performing parameter updates at inference time, enabling models to adapt to distribution shifts without retraining\(Sunet al\.,[2020](https://arxiv.org/html/2607.00423#bib.bib16); Wanget al\.,[2021a](https://arxiv.org/html/2607.00423#bib.bib17)\)\. Early TTA methods relied on auxiliary self\-supervised objectives such as entropy minimization\(Sunet al\.,[2020](https://arxiv.org/html/2607.00423#bib.bib16); Liuet al\.,[2021](https://arxiv.org/html/2607.00423#bib.bib18); Wanget al\.,[2021a](https://arxiv.org/html/2607.00423#bib.bib17)\)\. However, these approaches often suffer from instability from objective mismatch or prediction collapse\(Parket al\.,[2025](https://arxiv.org/html/2607.00423#bib.bib14)\), motivating recent interest in utilizing VLM’s internal alignment signal as direct feedback\. In particular, treating CLIP similarity as explicit reward and performing RL\-based optimization\(Zancatoet al\.,[2023](https://arxiv.org/html/2607.00423#bib.bib19)\)at inference time\(Zhaoet al\.,[2024](https://arxiv.org/html/2607.00423#bib.bib13)\)has been shown to reduce the collapse behavior observed in entropy\-based updates and to improve zero\-shot generalization\. We build upon the feedback\-driven TTA paradigm and extend it to address social bias in VLMs\. Our key contributions are \(1\) a bias\-sensitivity gate that activates debiasing per input, and \(2\) a reward that combines cross\-modal alignment with a bias\-subspace debiasing signal\. With episodic test\-time updates and selective reward gating, RG\-TTA mitigates the fairness–utility trade\-off, improving fairness and general\-purpose utility\.

## 3Method

We proposeReward\-Gated Test\-Time Adaptation \(RG\-TTA\), an RL\-based framework for selective debiasing that adaptively updates model parameters during inference\. RG\-TTA is built on two key components\. First, aselective gating strategyevaluates query sensitivity to triggerfairness regularizationonly when necessary\. Second, anadaptive reward functionbalances CLIP alignment with anattribute\-balancing reward, which encourages a uniform representation by favoring under\-represented attributes\. We update the model using a tractable approximation of the REINFORCE\(Williams,[1992](https://arxiv.org/html/2607.00423#bib.bib15)\)algorithm\. By focusing updates on the most relevant candidates through top\-KKtruncation, this approach ensures that the optimization stays centered on query\-specific semantics and limits drift via episodic resets\.

### 3\.1Preliminaries

#### CLIP\.

A pretrained vision\-language model \(VLM\) consists of an image encoderf​\(⋅\)f\(\\cdot\)and a text encoderg​\(⋅\)g\(\\cdot\), which project both modalities into a shared embedding space\(Radfordet al\.,[2021](https://arxiv.org/html/2607.00423#bib.bib7)\)\. Given a queryqqand a candidatey∈𝒴y\\in\\mathcal\{Y\}selected from the opposite modality, we define an alignment scoreS​\(q,y\)S\(q,y\)as the cosine similarity between their embeddings\. Specifically, in the text\-to\-image setting we useS​\(q,y\)=cos⁡\(g​\(q\),f​\(y\)\)S\(q,y\)=\\cos\\\!\\big\(g\(q\),f\(y\)\\big\), whereas in the image\-to\-text setting we useS​\(q,y\)=cos⁡\(f​\(q\),g​\(y\)\)S\(q,y\)=\\cos\\\!\\big\(f\(q\),g\(y\)\\big\)\. We use the policy scoreSθ​\(q,y\)S\_\{\\theta\}\(q,y\)for candidate ranking and parameterizing the policy\. We use the reference scoreSref​\(q,y\)S\_\{\\mathrm\{ref\}\}\(q,y\)only to compute the CLIP reward;SrefS\_\{\\mathrm\{ref\}\}is obtained from a fixed CLIP ViT\-L/14 model throughout all experiments\.

#### Test\-time adaptation in vision–language tasks\.

Test\-time adaptation \(TTA\(Sunet al\.,[2020](https://arxiv.org/html/2607.00423#bib.bib16); Wanget al\.,[2021a](https://arxiv.org/html/2607.00423#bib.bib17)\)\) updates a trained model at inference time using a few unlabeled steps\. We follow an*episodic*protocol: each queryqqis adapted independently and the parameters are reset before the next query, which mitigates negative transfer\. For vision–language retrieval, whereqqcan be text or an image, we adapt only the query\-modality encoder and keep the opposite\-modality encoder fixed\.

### 3\.2Selective Gating Strategy

Applying debiasing uniformly to all queries is unnecessary and can induce parameter drift on bias\-insensitive queries\. To mitigate the structural fairness–utility trade\-off, we introduce a gating mechanism that measures the alignment discrepancy between query semantics and demographic attributes\. For each queryqq, we select the top\-1 candidatey∗=arg⁡maxy⁡Sθ​\(q,y\)y^\{\*\}=\\arg\\max\_\{y\}S\_\{\\theta\}\(q,y\)as ananchorand assess whether the query–candidate alignment is disproportionately driven by demographic associations by comparingy∗y^\{\*\}to the mean attribute similarity over a predefined attribute set𝒜\\mathcal\{A\}of protected\-group exemplars\. We instantiate𝒜\\mathcal\{A\}in the same modality as the query \(e\.g\., attribute prompts for text and exemplar images for vision\), so thatSθ​\(a,y∗\)S\_\{\\theta\}\(a,y^\{\*\}\)uses the same scoring function; concrete instantiations are given in Sec\.[4\.4](https://arxiv.org/html/2607.00423#S4.SS4)\.

The gating logic is based on the following intuition: a large discrepancy suggests that the match is driven by general, attribute\-independent semantic alignment, in which case we deactivate fairness regularization \(δ​\(q\)=0\\delta\(q\)=0\) and focus on refining standard cross\-modal alignment to enhance utility\. Conversely, ify∗y^\{\*\}is close to the attribute alignment distribution \(within a thresholdϵ\\epsilon\), the signal is likely entangled with a specific attribute category, triggering an attribute\-balancing reward \(δ​\(q\)=1\\delta\(q\)=1\)\. Accordingly, the gate is defined as:

δ​\(q\)=𝕀​\[S​\(q,y∗\)−1\|A\|​∑a∈AS​\(a,y∗\)<ϵ\]\.\\delta\(q\)=\\mathbb\{I\}\\Big\[\\,S\(q,y^\{\*\}\)\-\\frac\{1\}\{\|A\|\}\\sum\_\{a\\in A\}S\(a,y^\{\*\}\)<\\epsilon\\Big\]\.\(1\)
Here,𝕀​\[⋅\]\\mathbb\{I\}\[\\cdot\]denotes the indicator function andϵ\\epsilonis a fixed threshold \(0\.020\.02\) kept constant throughout the episode to prevent parameter drift\.

### 3\.3Adaptive Reward Function

In this section, we define a candidate\-wise rewardr​\(q,yk\)r\(q,y\_\{k\}\)for each candidateyk∈𝒴K​\(q\)y\_\{k\}\\in\\mathcal\{Y\}\_\{K\}\(q\), where𝒴K​\(q\)\\mathcal\{Y\}\_\{K\}\(q\)denotes the candidate set used for adaptation\. We describe the construction of𝒴K​\(q\)\\mathcal\{Y\}\_\{K\}\(q\)in Sec\.[3\.4](https://arxiv.org/html/2607.00423#S3.SS4)\. Our reward always includes a CLIP\-based alignment signal to preserve general cross\-modal matching ability, and adds an attribute\-balancing reward only when the gate is activated\. This design maintains a balance between alignment and debiasing within each episode\.

#### CLIP alignment reward\.

We follow prior work in defining a CLIP\-based alignment reward\(Zhaoet al\.,[2024](https://arxiv.org/html/2607.00423#bib.bib13)\)\. Concretely, for each candidateyky\_\{k\}we compute a nonnegative alignment score assk=max⁡\(Sref​\(q,yk\),0\)s\_\{k\}=\\max\\\!\\big\(S\_\{\\mathrm\{ref\}\}\(q,y\_\{k\}\),0\\big\), and use the episode means¯=1K​∑j=1Ksj\\bar\{s\}=\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}s\_\{j\}as a baseline\. The resulting baseline\-normalized alignment reward is

rclip​\(q,yk\)=sk−s¯\.r\_\{\\text\{clip\}\}\(q,y\_\{k\}\)=s\_\{k\}\-\\bar\{s\}\.\(2\)

#### Bias subspace and attribute\-balancing reward\.

Whenδ​\(q\)=1\\delta\(q\)=1, we incorporate an attribute\-balancing reward computed in a predefined*bias subspace*into the reward\. Letμc∈ℝD\\mu\_\{c\}\\in\\mathbb\{R\}^\{D\}denote the class\-mean embedding for classccin the shared embedding space, precomputed from a labeled source dataset and kept fixed during test\-time adaptation\. We further define a reference vectorμ¯\\bar\{\\mu\}, which can be instantiated as the average of\{μc\}c=1C\\\{\\mu\_\{c\}\\\}\_\{c=1\}^\{C\}, and construct the set of difference vectors\{μc−μ¯\}c=1C\\\{\\mu\_\{c\}\-\\bar\{\\mu\}\\\}\_\{c=1\}^\{C\}\. We then perform PCA on these differences and extract all nonzero principal components, yieldingddcomponents in total\. The resulting orthonormal basisU∈ℝD×dU\\in\\mathbb\{R\}^\{D\\times d\}defines the bias subspace333Projection\-based techniques have also been used for group robustness on spurious correlations\(Zhuet al\.,[2025](https://arxiv.org/html/2607.00423#bib.bib42)\), in a different \(training\-time\) setting and formulation\.

For each selected candidateyky\_\{k\}, letzkz\_\{k\}denote the embedding produced by the frozen encoder of the opposite modality\. We then map both candidates and class prototypes into the bias subspace using the projection operatorP=U​U⊤P=UU^\{\\top\}, yieldingz~k=P​\(zk−μ¯\)\\tilde\{z\}\_\{k\}=P\(z\_\{k\}\-\\bar\{\\mu\}\)andμ~c=P​\(μc−μ¯\)\\tilde\{\\mu\}\_\{c\}=P\(\\mu\_\{c\}\-\\bar\{\\mu\}\)\. Next, we compute a soft assignment over classes by normalizing Gaussian\-kernel similarities with a temperature parameterγ\\gamma\. Defining the squared distance between the projected candidate and class prototype asdk​c=‖z~k−μ~c‖22d\_\{kc\}=\\\|\\tilde\{z\}\_\{k\}\-\\tilde\{\\mu\}\_\{c\}\\\|\_\{2\}^\{2\}, the soft assignment is given by:

αc\(k\)=exp⁡\(−dk​c/γ\)∑c′=1Cexp⁡\(−dk​c′/γ\)\.\\alpha^\{\(k\)\}\_\{c\}=\\frac\{\\exp\(\-d\_\{kc\}/\\gamma\)\}\{\\sum\_\{c^\{\\prime\}=1\}^\{C\}\\exp\(\-d\_\{kc^\{\\prime\}\}/\\gamma\)\}\.\(3\)
To estimate the attribute distribution at the episode level, we compute the popularity for each class aspc=1K​∑j=1Kαc\(j\)p\_\{c\}=\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}\\alpha\_\{c\}^\{\(j\)\}\. We then define the attribute\-balancing reward for each candidate as

d​\(q,yk\)=∑c=1Cαc\(k\)​\(pc−1C\)\.d\(q,y\_\{k\}\)=\\sum\_\{c=1\}^\{C\}\\alpha\_\{c\}^\{\(k\)\}\\\!\\left\(p\_\{c\}\-\\frac\{1\}\{C\}\\right\)\.\(4\)This score reflects how strongly candidateyky\_\{k\}is associated with attribute classes that are over\- or under\-represented in the current episode, encouraging a more balanced set of selected candidates\.

#### Final combined reward\.

Finally, we define the overall reward for each selected candidate by combining the CLIP alignment reward with the query\-conditioned attribute\-balancing reward:

r​\(q,yk\)=rclip​\(q,yk\)−δ​\(q\)​λ​d​\(q,yk\),r\(q,y\_\{k\}\)=r\_\{\\text\{clip\}\}\(q,y\_\{k\}\)\-\\delta\(q\)\\,\\lambda\\,d\(q,y\_\{k\}\),\(5\)whereλ\\lambdais a hyperparameter that controls the strength of the attribute\-balancing reward\. Whenδ​\(q\)=0\\delta\(q\)=0, the reward reduces to the CLIP alignment term alone\. Whenδ​\(q\)=1\\delta\(q\)=1, candidates associated with over\-represented classes \(i\.e\.,pc\>1/Cp\_\{c\}\>1/C\) tend to have largerd​\(q,yk\)d\(q,y\_\{k\}\)and thus incur a larger balancing penalty, while candidates linked to under\-represented classes \(i\.e\.,pc<1/Cp\_\{c\}<1/C\) tend to have smaller or negatived​\(q,yk\)d\(q,y\_\{k\}\)and are relatively favored\. As a result, the episode\-level attribute distribution is encouraged toward the uniform prior\.

### 3\.4Optimization and Episodic Update

We optimize the proposed objective at test time via episodic policy\-gradient updates\(Wanget al\.,[2021a](https://arxiv.org/html/2607.00423#bib.bib17); Shuet al\.,[2022](https://arxiv.org/html/2607.00423#bib.bib20)\), treating each input queryqqas an episode\. We first evaluate the gating indicatorδ​\(q\)\\delta\(q\)to determine whether to include the attribute\-balancing term in the episode reward\. Conditioned on this decision, we choose a candidate budgetKK\(e\.g\.,K=10K=10whenδ​\(q\)=0\\delta\(q\)=0andK=1024K=1024whenδ​\(q\)=1\\delta\(q\)=1\), and construct a truncated candidate set𝒴K​\(q\)\\mathcal\{Y\}\_\{K\}\(q\)by selecting the top\-KKcandidates from the fixed pool𝒴\\mathcal\{Y\}according to the alignment scoreSθ​\(q,y\)S\_\{\\theta\}\(q,y\)\. We then compute the candidate\-wise rewardr​\(q,yk\)r\(q,y\_\{k\}\)for eachyk∈𝒴K​\(q\)y\_\{k\}\\in\\mathcal\{Y\}\_\{K\}\(q\)and update the query\-modality encoder parametersθ\\thetawith a small number of gradient steps\. After the episode ends, we reset the parameters to their initial state before processing the next query, mitigating negative transfer across queries\. We define the policyπθ​\(y∣q\)\\pi\_\{\\theta\}\(y\\mid q\)over the full candidate pool𝒴\\mathcal\{Y\}using a softmax of the alignment score:

πθ​\(y∣q\)=exp⁡\(Sθ​\(q,y\)\)∑y′∈𝒴exp⁡\(Sθ​\(q,y′\)\)\.\\pi\_\{\\theta\}\(y\\mid q\)=\\frac\{\\exp\\\!\\big\(S\_\{\\theta\}\(q,y\)\\big\)\}\{\\sum\_\{y^\{\\prime\}\\in\\mathcal\{Y\}\}\\exp\\\!\\big\(S\_\{\\theta\}\(q,y^\{\\prime\}\)\\big\)\}\.\(6\)Althoughπθ\\pi\_\{\\theta\}is normalized over the full candidate pool𝒴\\mathcal\{Y\}\(i\.e\., the denominator is computed over𝒴\\mathcal\{Y\}\), we approximate the policy\-gradient objective by summing only over the truncated set𝒴K​\(q\)\\mathcal\{Y\}\_\{K\}\(q\)for the current query\. Concretely, for each episode we minimize the following REINFORCE\-style objective:

ℒ​\(q\)=−1K​∑yk∈𝒴K​\(q\)r​\(q,yk\)​log⁡πθ​\(yk∣q\)\.\\mathcal\{L\}\(q\)=\-\\frac\{1\}\{K\}\\sum\_\{y\_\{k\}\\in\\mathcal\{Y\}\_\{K\}\(q\)\}r\(q,y\_\{k\}\)\\,\\log\\pi\_\{\\theta\}\(y\_\{k\}\\mid q\)\.\(7\)This top\-KKtruncation based onS​\(q,y\)S\(q,y\)yields a tractable approximation to the full policy\-gradient objective while focusing updates on the most relevant candidates for the query\.

Table 1:Gender and age debiasing performance across different source datasets\. We report results on in/out\-of\-domain fairness benchmarks and zero\-shot utility tasks\. ABLE is calculated based on in\-domain metrics\. Best results are shown inbold, and second\-best results areunderlined\.BackboneBiasesMethodsIn\-DomainOut\-of\-DomainIN1KFlickrABLE \(%\)↑\\uparrowSource DatasetCross DatasetFACETAcc\. \(%\)↑\\uparrowR@5 \(%\)↑\\uparrowMS↓\\downarrowNDKL↓\\downarrowMS↓\\downarrowNDKL↓\\downarrowMS↓\\downarrowNDKL↓\\downarrowTop\-1Top\-5TRIR\\rowcolorgray\!20Source: UTKFace \| Cross: FairFaceViT\-B/16GenderOriginal CLIP0\.1140\.0800\.2180\.0880\.4780\.21568\.3191\.8396\.485\.577\.39CLIP\-clip0\.0700\.0550\.1330\.0380\.4590\.19067\.8191\.4295\.483\.078\.52Biased\-prompts0\.1790\.0620\.1610\.0480\.4600\.21565\.0789\.3894\.386\.173\.18Joint V\-L0\.0480\.0430\.1010\.0320\.4560\.18167\.9991\.6495\.884\.679\.36Ours0\.0510\.0290\.0800\.0350\.0530\.06470\.3893\.0097\.288\.580\.87AgeOriginal CLIP0\.4210\.2290\.6570\.4330\.7440\.36768\.3191\.8396\.485\.566\.96CLIP\-clip0\.3930\.2150\.6430\.4300\.7450\.36467\.9391\.5896\.184\.567\.70Biased\-prompts0\.5780\.4510\.7770\.5500\.6350\.35566\.4390\.2894\.185\.260\.83Joint V\-L0\.4140\.2310\.6060\.4100\.7460\.36567\.6391\.4695\.684\.666\.86Ours0\.1510\.2260\.6410\.4200\.7420\.33070\.3692\.9997\.288\.377\.39ViT\-B/32GenderOriginal CLIP0\.0660\.0320\.1380\.0540\.4850\.22563\.3988\.8394\.783\.575\.60CLIP\-clip0\.0980\.0450\.2530\.1050\.5000\.24062\.2188\.2393\.081\.073\.79Biased\-prompts0\.0890\.0360\.0940\.0270\.4170\.16460\.3786\.7593\.682\.472\.74Joint V\-L0\.0430\.0330\.1080\.0390\.4690\.21262\.4688\.2394\.782\.975\.60Ours0\.0540\.0380\.0880\.0350\.0500\.02169\.7692\.3197\.086\.679\.60AgeOriginal CLIP0\.4120\.2530\.6170\.4160\.7520\.38863\.3988\.8394\.783\.564\.77CLIP\-clip0\.4150\.2640\.6590\.4350\.7540\.39762\.7088\.3194\.283\.264\.34Biased\-prompts0\.5220\.4090\.7010\.4970\.6630\.36661\.0786\.9292\.082\.260\.19Joint V\-L0\.4070\.2520\.6270\.4160\.7510\.37062\.9388\.6694\.182\.564\.69Ours0\.1270\.2830\.3850\.3640\.7410\.36969\.7592\.3096\.986\.477\.85\\rowcolorgray\!20Source: FairFace \| Cross: UTKFaceViT\-B/16GenderOriginal CLIP0\.2180\.0880\.1140\.0800\.4780\.21568\.3191\.8396\.485\.573\.87CLIP\-clip0\.1030\.0260\.0830\.0620\.4780\.19968\.0091\.5095\.483\.077\.55Biased\-prompts0\.1610\.0480\.1790\.0620\.4600\.21565\.0789\.3894\.386\.173\.78Joint V\-L0\.0800\.0250\.0400\.0230\.4460\.17068\.0591\.6396\.684\.378\.35Ours0\.0820\.0310\.0300\.0220\.1140\.04070\.3292\.9897\.288\.579\.75AgeOriginal CLIP0\.6570\.4330\.4210\.2290\.7440\.36768\.3191\.8396\.485\.558\.94CLIP\-clip0\.6470\.4320\.4020\.2150\.7420\.37367\.9791\.6196\.384\.459\.16Biased\-prompts0\.7770\.5500\.5780\.4510\.6350\.35566\.4390\.2894\.185\.254\.33Joint V\-L0\.6080\.2940\.3770\.1150\.7380\.34168\.3491\.7496\.084\.060\.61Ours0\.5260\.3180\.2450\.2220\.7420\.26570\.3692\.9997\.288\.364\.24ViT\-B/32GenderOriginal CLIP0\.1380\.0540\.0660\.0320\.4850\.22563\.3988\.8394\.783\.573\.37CLIP\-clip0\.1070\.0300\.0610\.0230\.4920\.21559\.6286\.2990\.976\.271\.68Biased\-prompts0\.0940\.0270\.0890\.0360\.4170\.16460\.3786\.7593\.682\.472\.59Joint V\-L0\.0900\.0300\.0500\.0210\.4660\.20462\.5288\.5694\.982\.974\.24Ours0\.0780\.0320\.0500\.0290\.1370\.03669\.7692\.3097\.086\.679\.54AgeOriginal CLIP0\.6170\.4160\.4120\.2530\.7520\.38863\.3988\.8394\.783\.558\.29CLIP\-clip0\.6350\.4250\.4000\.2520\.7490\.38762\.4088\.3094\.582\.557\.32Biased\-prompts0\.7010\.4970\.5220\.4090\.6630\.36661\.0786\.9292\.082\.254\.76Joint V\-L0\.5720\.3640\.3850\.1950\.7500\.38163\.1388\.7194\.182\.859\.60Ours0\.5230\.2290\.2450\.2220\.7430\.26569\.7592\.3096\.986\.464\.09

## 4Experiments

### 4\.1Datasets

To comprehensively evaluate both debiasing performance and generalization, we used a diverse set of benchmarks\. For fairness evaluation, we consider in\-domain settings on FairFace \(val\)\(Kärkkäinen and Joo,[2021](https://arxiv.org/html/2607.00423#bib.bib33)\)and UTKFace\(Zhanget al\.,[2017](https://arxiv.org/html/2607.00423#bib.bib34)\), and an out\-of\-domain setting on FACET\(Gustafsonet al\.,[2023](https://arxiv.org/html/2607.00423#bib.bib35)\)\. To assess whether adaptation preserves general\-purpose zero\-shot utility, we additionally evaluate on ImageNet\-1K\(Denget al\.,[2009](https://arxiv.org/html/2607.00423#bib.bib5)\)for image classification and Flickr1k\(Plummeret al\.,[2015](https://arxiv.org/html/2607.00423#bib.bib6)\)for retrieval\.

### 4\.2Metrics

#### Fairness metrics\.

Following prior work\(Berget al\.,[2022](https://arxiv.org/html/2607.00423#bib.bib36); Sethet al\.,[2023](https://arxiv.org/html/2607.00423#bib.bib37)\), we use retrieval\-based metrics: MaxSkew@kkand NDKL@kk\. These metrics quantify the disparity in the distribution of protected attributes \(e\.g\., gender, age, and race\) within the top\-k retrieved images for neutral queries\. MaxSkew measures the maximum representation of a dominant group, while NDKL measures the divergence from uniform distribution\. For both metrics, lower values indicate a fairer model\.

#### Utility \(V–L alignment\) metrics\.

To ensure that our selective debiasing does not compromise the intrinsic V\-L alignment of the pre\-trained model, we evaluated zero\-shot performance on standard benchmarks\. We report Top\-1 and Top\-5 accuracy on ImageNet\-1K\(Denget al\.,[2009](https://arxiv.org/html/2607.00423#bib.bib5)\)for classification, and Recall@5 for both Image\-to\-Text \(TR\) and Text\-to\-Image \(IR\) retrieval on Flickr1k\(Plummeret al\.,[2015](https://arxiv.org/html/2607.00423#bib.bib6)\)\.

#### Alignment and Bias Level Evaluation \(ABLE\)\.

Single metrics capture either fairness or utility, but not their trade\-off\. We use ABLE proposed by Zhanget al\.\(Zhanget al\.,[2025](https://arxiv.org/html/2607.00423#bib.bib9)\)for a holistic assessment\. ABLE is defined as the harmonic mean of the zero\-shot accuracy and the fairness score:

ABLE=21a​c​c\+1exp⁡\(−MaxSkew​@​k\)\\mathrm\{ABLE\}=\\frac\{2\}\{\\frac\{1\}\{acc\}\+\\frac\{1\}\{\\exp\(\-\\mathrm\{MaxSkew\}@k\)\}\}\(8\)wherea​c​c\{acc\}denotes the ImageNet Top\-1 accuracy\. Higher ABLE indicates a better balance between mitigating social bias and retaining zero\-shot accuracy; we report ABLE×\\times100 \(%\) in tables\.

Table 2:Race debiasing performance using UTKFace as the source\.BackboneMethodsUTKFaceIN1KFlickrABLE \(%\)↑\\uparrowAcc\. \(%\)↑\\uparrowR@5 \(%\)↑\\uparrowMS↓\\downarrowNDKL↓\\downarrowTop\-1Top\-5TRIRViT\-B/16Original CLIP0\.5750\.13768\.3191\.8396\.485\.563\.31CLIP\-clip0\.6130\.15767\.7491\.4895\.885\.160\.20Biased\-prompts0\.6040\.20867\.0090\.7294\.185\.860\.21Joint V\-L0\.3780\.06968\.0791\.6496\.583\.868\.30Ours0\.1500\.02270\.3593\.0097\.288\.377\.42ViT\-B/32Original CLIP0\.6980\.21363\.3988\.8394\.783\.555\.75CLIP\-clip0\.8400\.42662\.9088\.3293\.681\.7451\.20Biased\-prompts0\.3170\.14061\.8087\.4691\.983\.3460\.21Joint V\-L0\.6380\.23063\.0188\.5694\.682\.457\.48Ours0\.3510\.28169\.7492\.3497\.086\.570\.07

### 4\.3Baselines

We compare RG\-TTA against theOriginal CLIPand three representative debiasing baselines using ViT\-B/16 and ViT\-B/32 backbones\.CLIP\-clip\(Wanget al\.,[2021b](https://arxiv.org/html/2607.00423#bib.bib31)\)removes bias\-correlated embedding dimensions identified via mutual information with attribute labels\.Biased\-prompts\(Chuanget al\.,[2023](https://arxiv.org/html/2607.00423#bib.bib32)\)neutralizes bias directions by projecting embeddings using prompt\-derived attribute subspaces without retraining\.Joint V\-L\(Zhanget al\.,[2025](https://arxiv.org/html/2607.00423#bib.bib9)\)jointly debiases image and text representations to mitigate over\-debiasing effects\. All methods are evaluated under identical settings for Gender, Age, and Race\.

### 4\.4Implementation Details

We use the official pretrained CLIP checkpoints\(Radfordet al\.,[2021](https://arxiv.org/html/2607.00423#bib.bib7)\)with ViT\-B/16, ViT\-B/32, and ViT\-L/14 backbones\. Consistent with the episodic TTA protocol, we update only the encoder corresponding to the input query modality \(e\.g\., the text encoder for text\-to\-image retrieval\) while keeping the target modality encoder frozen\. Optimization is performed using AdamW\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2607.00423#bib.bib38)\)\. Crucially, to balance efficiency and performance, we dynamically adjust the computational budget based on the gating decisionδ​\(q\)\\delta\(q\): we perform a lightweight update withT=3T\{=\}3steps andK=10K\{=\}10candidates for bias\-insensitive queries \(δ​\(q\)=0\\delta\(q\)\{=\}0\), while expanding toT=10T\{=\}10steps andK=1024K\{=\}1024candidates for bias\-sensitive ones \(δ​\(q\)=1\\delta\(q\)\{=\}1\)\. Detailed hyperparameters, including learning rates, gating thresholds, and prompt templates, are provided in Appendix[A](https://arxiv.org/html/2607.00423#A1)\.

![Refer to caption](https://arxiv.org/html/2607.00423v1/x4.png)Figure 4:Ablation ofKKandλ\\lambdashowing their effect on MaxSkew@kk\.
### 4\.5Results

Table[3\.4](https://arxiv.org/html/2607.00423#S3.SS4)summarizes bias\-mitigation results across different source datasets \(UTKFace and FairFace\) for Gender and Age\. Overall, our method improves fairness in both in\-domain and out\-of\-domain settings while enhancing ImageNet zero\-shot performance\. For Gender, we observe a clear in\-domain improvement on UTKFace \(MaxSkew@kk: 0\.114→0\.051\), with particularly large mitigation under distribution shift on FACET \(MaxSkew@kk: 0\.478→0\.053\)\. Meanwhile, ImageNet Top\-1 accuracy increases from 68\.31 to 70\.38, and ABLE also rises from 77\.39 to 80\.87, indicating that fairness gains do not come at the expense of utility \(see Table[3\.4](https://arxiv.org/html/2607.00423#S3.SS4)for full results\)\.

This trend is consistent across attributes and source configurations: for Age and Race, we observe substantial in\-domain fairness gains while maintaining \(or slightly improving\) zero\-shot utility \(Tables[3\.4](https://arxiv.org/html/2607.00423#S3.SS4),[2](https://arxiv.org/html/2607.00423#S4.T2)\)\. Switching the source dataset to FairFace shows the same behavior across in\-domain and out\-of\-domain settings, aligning with our design—gating with episodic resets—that focuses updates on bias\-sensitive queries while preventing drift on others\.

Table 3:Ablation of reward variants for selective debiasing, evaluated on bias\-sensitive \(MS↓\\downarrow\) and bias\-insensitive \(IN1K Top\-1↑\\uparrow\) benchmarks\.RewardMS↓\\downarrowIN1K Top\-1↑\\uparrowABLE \(%\)↑\\uparrowCLIP0\.60470\.4161\.54CLIP\+Debias0\.14953\.9858\.23RG\-TTA\(Ours\)0\.15170\.3677\.39## 5Discussion

### 5\.1Ablation Study

We ablate key design choices to examine how selective reward\-gating mitigates the fairness–utility trade\-off\.

#### Candidate budget and debiasing strength\.

Figure[4](https://arxiv.org/html/2607.00423#S4.F4)ablates the hyperparameters used when debiasing is active, varying the candidate budgetKKand the debiasing strengthλ\\lambdain theδ​\(q\)=1\\delta\(q\)=1regime\. We observe that increasingKKgenerally improves MaxSkew@1000 \(notably for Age\), suggesting that a larger truncated set yields a more stable estimate of the episode\-level attribute distribution and thus a more reliable balancing signal\. Varyingλ\\lambdashows a similar trend: weak values yield limited mitigation, while larger values deliver consistent gains with diminishing returns, suggesting performance is not brittle onceλ\\lambdapasses a minimal effective threshold\.

![Refer to caption](https://arxiv.org/html/2607.00423v1/x5.png)Figure 5:Sensitivity to the gating thresholdϵ\\epsilon\. Left: MaxSkew@kk\(gender\) on UTKFace/FACET\. Right: ImageNet\-1K Top\-1 accuracy\.
#### Selective vs\. uniform debiasing\.

We next examine when to apply the debiasing signal versus how strongly to apply it\. Table[3](https://arxiv.org/html/2607.00423#S4.T3)compares \(i\) an alignment\-only reward, \(ii\) adding the attribute\-balancing term uniformly to all queries, and \(iii\) activating the balancing term only when the gating indicatorδ​\(q\)\\delta\(q\)triggers\. While uniform debiasing can substantially reduce MaxSkew, it risks unnecessary adaptation and notably degrades zero\-shot utility; in contrast, selective activation preserves utility while retaining the fairness gains, yielding a markedly better overall balance\.

#### Gating threshold and activation behavior\.

We characterize the gate through threshold sensitivity and activation patterns under a dataset\-level proxy setting\. Figure[5](https://arxiv.org/html/2607.00423#S5.F5)shows that across a broad range of the thresholdϵ\\epsilon, fairness improves rapidly and remains largely stable, and utility is stable around the defaultϵ=0\.02\\epsilon\{=\}0\.02, with noticeable degradation only whenϵ\\epsilonis overly large and activates debiasing too aggressively\. We also evaluate the gate’s activation behavior \(Table[6](https://arxiv.org/html/2607.00423#S5.T6)\) under a proxy labeling scheme: treating UTKFace/FACET queries as debiasing\-needed and ImageNet\-1K queries as debiasing\-not\-needed, the gate activates for almost all UTKFace/FACET queries while remaining inactive for nearly all ImageNet\-1K queries, yielding low false negative/positive rates in this proxy setting\.

Table 4:Same\-size reward model ablation on UTKFace \(Gender\)\. “Reward” indicates the model used to compute the CLIP reward\. Fairness improvements \(MS\) are preserved regardless of reward model size, while the larger ViT\-L/14 reward primarily benefits utility \(IN1K accuracy\)\.BackboneRewardMS↓\\downarrowIN1K↑\\uparrowABLE↑\\uparrowViT\-B/16Original CLIP0\.11468\.3177\.38RG\-TTA \(L/14\)0\.05170\.3880\.87RG\-TTA \(B/16\)0\.05068\.7479\.81ViT\-B/32Original CLIP0\.06663\.3975\.60RG\-TTA \(L/14\)0\.05469\.7680\.36RG\-TTA \(B/32\)0\.05764\.1876\.43ViT\-L/14Original CLIP0\.18575\.5579\.14RG\-TTA \(L/14\)0\.05575\.6784\.10

#### Reward model scale\.

A natural concern is that RG\-TTA relies on a larger reference model \(ViT\-L/14\) to compute the CLIP reward, which may be undesirable when the backbone itself is smaller\. We test whether fairness gains depend on this scale mismatch by replacing the reward model with a*same\-size*variant, matching the backbone \(Table[4](https://arxiv.org/html/2607.00423#S5.T4)\)\. The results reveal a clear pattern:*fairness gains are largely independent of reward model scale*\. For ViT\-B/16, the same\-size reward achieves MaxSkew of 0\.050, essentially matching the 0\.051 obtained with the larger ViT\-L/14 reward, and a similar trend holds for ViT\-B/32 \(0\.057 vs\. 0\.054\)\. The benefit of the larger reward model instead manifests in*utility preservation*: ImageNet accuracy improves more substantially when using ViT\-L/14 as reward \(e\.g\., B/16: 68\.31→\\rightarrow70\.38 vs\. 68\.74; B/32: 63\.39→\\rightarrow69\.76 vs\. 64\.18\)\. Importantly, the same\-size configuration still improves ABLE over Original CLIP in all cases, confirming that RG\-TTA remains practically deployable when an external larger reference model is unavailable, while the larger reward model offers additional utility headroom when available\.

### 5\.2Runtime Analysis

While RG\-TTA introduces per\-query test\-time optimization, understanding its computational cost is essential for assessing practical deployment\. We measure per\-query wall\-clock time on a single NVIDIA A6000 GPU using ViT\-B/16, averaged over 1000 queries\.

Table 5:Per\-query runtime on A6000 GPU with ViT\-B/16\.MethodInference/queryOriginal CLIP4\.49 msRG\-TTA,δ​\(q\)=0\\delta\(q\)\{=\}0110\.96 msRG\-TTA,δ​\(q\)=1\\delta\(q\)\{=\}1467\.41 ms#### Overhead analysis\.

As shown in Table[5](https://arxiv.org/html/2607.00423#S5.T5), RG\-TTA incurs additional latency over Original CLIP due to alignment discrepancy computation and policy gradient updates\. The overhead ranges from∼\\sim25×\\timesfor bias\-insensitive queries \(δ​\(q\)=0\\delta\(q\)\{=\}0\) to∼\\sim104×\\timesfor bias\-sensitive queries \(δ​\(q\)=1\\delta\(q\)\{=\}1\), reflecting the difference in candidate budgetKKand update stepsTT\.

#### Practical implications\.

Despite this overhead, three factors support the practical viability of RG\-TTA\. First, both conditions remain*sub\-second*, which is acceptable for fairness\-critical batch\-processing scenarios such as automated hiring systems or medical image retrieval, where per\-query latency of hundreds of milliseconds is tolerable\. Second,*adaptive budgeting*effectively reduces average overhead: as reported in Table[6](https://arxiv.org/html/2607.00423#S5.T6), the majority of general queries are classified as bias\-insensitive, so the average latency in practice approaches∼\\sim111 ms rather than the worst\-case∼\\sim467 ms\. Third, RG\-TTA is entirely*training\-free*; unlike offline debiasing methods that require per\-domain fine\-tuning, RG\-TTA can be instantly applied to new domains or attributes by replacing the bias subspace, offering a favorable amortized cost in deployment scenarios with evolving fairness requirements\.

### 5\.3Out\-of\-Domain Analysis on FACET

On FACET \(out\-of\-domain\), our method transfers well for Gender, maintaining lower MaxSkew@kkthan prior methods\. We hypothesize that Gender is associated with multiple robust visual cues \(e\.g\., face, hairstyle, clothing\), which preserve group separation even after bias\-subspace projection and lead to more stable debiasing signals\. In contrast, Age relies on fine\-grained facial cues that are sensitive to distance, resolution, and occlusion; in FACET’s unconstrained images, these cues are weakened, reducing the reliability of the debiasing signal and ultimately limiting gains\.

Table 6:FP/FN are computed under a proxy labeling scheme where UTKFace and FACET are treated as positives \(debiasing\-needed\) and ImageNet\-1K as negatives \(debiasing\-not\-needed\)\.Datasetδ=1\\delta=1\(%\)δ=0\\delta=0\(%\)FP/FNUTKFace99\.90\.1FN = 0\.1FACET99\.80\.2FN = 0\.2ImageNet\-1K0\.199\.9FP = 0\.1![Refer to caption](https://arxiv.org/html/2607.00423v1/figures/failure1.jpeg)

![Refer to caption](https://arxiv.org/html/2607.00423v1/figures/failure2.jpeg)

Figure 6:Gating failure cases on FACET\.Bias\-sensitive queries are incorrectly gated off \(δ​\(q\)=0\\delta\(q\)=0\), preventing fairness regularization from being activated\.
### 5\.4Gating Failure Case Analysis

Figure[6](https://arxiv.org/html/2607.00423#S5.F6)shows*false\-negative*cases on FACET, where bias\-sensitive queries are gated off \(δ​\(q\)=0\\delta\(q\)=0\) and thus do not trigger fairness regularization\. These failures often arise when salient object and scene semantics in the top\-1 anchory∗y^\{\*\}overwhelm demographic\-correlated cues, causing the semantic–attribute discrepancy to exceed the threshold\.

## 6Conclusion

We introduced Reward\-Gated Test\-Time Adaptation \(RG\-TTA\), a selective test\-time debiasing framework for CLIP\-style vision–language models\. RG\-TTA uses an input\-dependent reward gate to activate an attribute\-balancing term only when necessary, together with episodic updates and parameter resets to limit unintended drift\. Experiments across in\-domain and out\-of\-domain fairness benchmarks as well as standard zero\-shot utility tasks show that RG\-TTA consistently reduces demographic skew while maintaining competitive utility\. Overall, our results highlight selective, input\-conditioned adaptation as a practical design principle for mitigating bias without broadly disrupting model behavior\.

## Limitations

Our approach relies on test\-time adaptation \(TTA\) with per\-query parameter updates, which can increase computation and latency, especially when using many update steps or large candidate sets\. Because offline\-trained debiasing methods amortize cost during training whereas TTA incurs cost online, direct runtime comparisons across these paradigms are not always apples\-to\-apples and can vary with the deployment scenario\. Our experiments focus on a single protected attribute; extending the framework to multiple attributes requires more complex reward/constraint design and may introduce conflicting objectives\. The method also assumes access to an external reward signal from a stronger model, which raises availability and cost considerations and may transfer the reward model’s own biases into the adaptation signal\. Moreover, our fairness objective implicitly targets proximity to a chosen reference distribution \(e\.g\., uniform\), and both evaluation and reward depend on the accuracy and domain robustness of attribute estimators; TTA behavior can further be sensitive to hyperparameters and may be unstable for some queries\. Future work will develop more efficient update schemes to reduce online overhead\. We will also explore scalable multi\-attribute objectives/constraints and robustness techniques to mitigate sensitivity to reward sources and attribute estimators\.

## Ethics Statement

This work proposes a selective test\-time adaptation \(TTA\) approach to mitigate distributional biases over protected attributes \(e\.g\., gender, age, and race\) that can arise for person\-centric queries\. Our experiments use publicly available fairness evaluation datasets \(e\.g\., FairFace, UTKFace, and FACET\) together with standard utility benchmarks, and quantify bias using distribution\-based fairness metrics\. Because face images and protected\-attribute annotations can be sensitive, our study does not aim to identify individuals and assumes use strictly in accordance with the datasets’ licenses and usage conditions\. Our method further relies on an external reward signal from a stronger model \(e\.g\., a fixed CLIP ViT\-L/14 reference\), which introduces practical considerations about the availability and cost of such signals and raises the possibility that biases present in the reward model could be propagated through the adaptation process\. In addition, our fairness objective implicitly assumes a chosen target distribution \(e\.g\., a uniform prior\), which may not be appropriate for all tasks or domains\. We therefore recommend that any real\-world deployment be accompanied by careful auditing of the reward source and attribute estimators for bias and error, and that use in high\-stakes decision\-making contexts be avoided or subjected to additional, domain\-specific validation and oversight\.

## Acknowledgments

This work was partly supported by Institute of Information and Communications Technology Planning and Evaluation \(IITP\) grant funded by the Korea Government \(MSIT\) \(No\. RS\-2022\-II220184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics\), partly supported by the Institute of Information and Communications Technology Planning and Evaluation \(IITP\) grant funded by the Korea Government \(MSIT\) \[RS\-2021\-II211341, Artificial Intelligence Graduate School Program \(Chung\-Ang University\)\], and partly supported by the National Research Foundation of Korea \(NRF\) grant funded by the Korea government \(MSIT\) \[RS\-2026\-25498346\]\.

## References

- CLIP the bias: how useful is balancing data in multimodal learning?\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.00423#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Berg, S\. Hall, Y\. Bhalgat, H\. Kirk, A\. Shtedritski, and M\. Bain \(2022\)A prompt array keeps the bias away: debiasing vision\-language models with adversarial learning\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 806–822\.Cited by:[§4\.2](https://arxiv.org/html/2607.00423#S4.SS2.SSS0.Px1.p1.2)\.
- A\. Birhane, V\. U\. Prabhu, and E\. Kahembwe \(2021\)Multimodal datasets: misogyny, pornography, and malignant stereotypes\.arXiv preprint arXiv:2110\.01963\.External Links:[Link](https://arxiv.org/abs/2110.01963),[Document](https://dx.doi.org/10.48550/arXiv.2110.01963),2110\.01963Cited by:[§1](https://arxiv.org/html/2607.00423#S1.p1.1)\.
- C\. Chuang, V\. Jampani, Y\. Li, A\. Torralba, and S\. Jegelka \(2023\)Debiasing vision\-language models via biased prompts\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 4007–4016\.Cited by:[Figure 2](https://arxiv.org/html/2607.00423#S1.F2),[§1](https://arxiv.org/html/2607.00423#S1.p2.1),[§2](https://arxiv.org/html/2607.00423#S2.SS0.SSS0.Px1.p1.1),[§4\.3](https://arxiv.org/html/2607.00423#S4.SS3.p1.1)\.
- J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and L\. Fei\-Fei \(2009\)ImageNet: a large\-scale hierarchical image database\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 248–255\.Cited by:[§1](https://arxiv.org/html/2607.00423#S1.p1.1),[§1](https://arxiv.org/html/2607.00423#S1.p5.1),[§4\.1](https://arxiv.org/html/2607.00423#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2607.00423#S4.SS2.SSS0.Px2.p1.1)\.
- W\. Gerych, H\. Zhang, K\. Hamidieh, E\. Pan, M\. Sharma, T\. Hartvigsen, and M\. Ghassemi \(2024\)BendVLM: test\-time debiasing of vision\-language embeddings\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.00423#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Gustafson, C\. Rolland, N\. Ravi, Q\. Duval, A\. Adcock, C\. Fu, M\. Hall, and C\. Ross \(2023\)FACET: fairness in computer vision evaluation benchmark\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 20313–20325\.Cited by:[§1](https://arxiv.org/html/2607.00423#S1.p5.1),[§4\.1](https://arxiv.org/html/2607.00423#S4.SS1.p1.1)\.
- M\. Hall, L\. Gustafson, A\. Adcock, I\. Misra, and C\. Ross \(2023\)Vision\-language models performing zero\-shot tasks exhibit gender\-based disparities\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 2778–2785\.Cited by:[§1](https://arxiv.org/html/2607.00423#S1.p1.1)\.
- K\. Hamidieh, H\. Zhang, W\. Gerych, T\. Hartvigsen, and M\. Ghassemi \(2024\)Identifying implicit social biases in vision\-language models\.InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society \(AIES\),pp\. 547–561\.Cited by:[§1](https://arxiv.org/html/2607.00423#S1.p1.1)\.
- C\. I\. Hausladen, M\. Knott, C\. F\. Camerer, and P\. Perona \(2025\)Social perception of faces in a vision\-language model\.InProceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency \(FAccT\),pp\. 639–659\.External Links:[Document](https://dx.doi.org/10.1145/3715275.3732041)Cited by:[§1](https://arxiv.org/html/2607.00423#S1.p1.1)\.
- Y\. Hirota, M\. Chen, C\. Wang, Y\. Nakashima, Y\. F\. Wang, and R\. Hachiuma \(2025a\)SANER: annotation\-free societal attribute neutralizer for debiasing CLIP\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=x5hXkSMOd1)Cited by:[§2](https://arxiv.org/html/2607.00423#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Hirota, R\. Hachiuma, B\. Li, X\. Lu, M\. R\. Boone, B\. Ivanovic, Y\. Choi, M\. Pavone, Y\. F\. Wang, N\. Garcia, Y\. Nakashima, and C\. H\. Yang \(2025b\)Bias in gender bias benchmarks: how spurious features distort evaluation\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),Cited by:[§2](https://arxiv.org/html/2607.00423#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Janghorbani and G\. De Melo \(2023\)Multi\-modal bias: introducing a framework for stereotypical bias assessment beyond gender and race in vision–language models\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics \(EACL\),pp\. 1725–1735\.Cited by:[§1](https://arxiv.org/html/2607.00423#S1.p1.1)\.
- K\. Kärkkäinen and J\. Joo \(2021\)FairFace: face attribute dataset for balanced race, gender, and age for bias measurement and mitigation\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\),pp\. 1547–1557\.Cited by:[§1](https://arxiv.org/html/2607.00423#S1.p5.1),[§4\.1](https://arxiv.org/html/2607.00423#S4.SS1.p1.1)\.
- Y\. Liu, P\. Kothari, B\. Van Delft, B\. Bellot\-Gurlet, T\. Mordan, and A\. Alahi \(2021\)TTT\+\+: when does self\-supervised test\-time training fail or thrive?\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.00423#S2.SS0.SSS0.Px2.p1.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.InProceedings of the International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§A\.3](https://arxiv.org/html/2607.00423#A1.SS3.p1.4),[§4\.4](https://arxiv.org/html/2607.00423#S4.SS4.p1.7)\.
- M\. Park, H\. Won, W\. W\. Ro, and S\. Kim \(2025\)Rethinking entropy in test\-time adaptation: the missing piece from energy duality\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.00423#S2.SS0.SSS0.Px2.p1.1)\.
- B\. A\. Plummer, L\. Wang, C\. M\. Cervantes, J\. C\. Caicedo, J\. Hockenmaier, and S\. Lazebnik \(2015\)Flickr30k entities: collecting region\-to\-phrase correspondences for richer image\-to\-sentence models\.InProceedings of the IEEE International Conference on Computer Vision \(ICCV\),pp\. 2641–2649\.Cited by:[§1](https://arxiv.org/html/2607.00423#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.00423#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2607.00423#S4.SS2.SSS0.Px2.p1.1)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever \(2021\)Learning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 8748–8763\.External Links:[Link](https://proceedings.mlr.press/v139/radford21a.html)Cited by:[§A\.1](https://arxiv.org/html/2607.00423#A1.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.00423#S3.SS1.SSS0.Px1.p1.10),[§4\.4](https://arxiv.org/html/2607.00423#S4.SS4.p1.7)\.
- A\. Seth, M\. Hemani, and C\. Agarwal \(2023\)DEAR: debiasing vision\-language models with additive residuals\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 6820–6829\.Cited by:[§4\.2](https://arxiv.org/html/2607.00423#S4.SS2.SSS0.Px1.p1.2)\.
- M\. Shu, W\. Nie, D\. Huang, Z\. Yu, T\. Goldstein, A\. Anandkumar, and C\. Xiao \(2022\)Test\-time prompt tuning for zero\-shot generalization in vision\-language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§3\.4](https://arxiv.org/html/2607.00423#S3.SS4.p1.16)\.
- Y\. Sun, X\. Wang, Z\. Liu, J\. Miller, A\. A\. Efros, and M\. Hardt \(2020\)Test\-time training with self\-supervision for generalization under distribution shifts\.InProceedings of the International Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2607.00423#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2607.00423#S3.SS1.SSS0.Px2.p1.2)\.
- D\. Wang, E\. Shelhamer, S\. Liu, B\. A\. Olshausen, and T\. Darrell \(2021a\)Tent: fully test\-time adaptation by entropy minimization\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.00423#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2607.00423#S3.SS1.SSS0.Px2.p1.2),[§3\.4](https://arxiv.org/html/2607.00423#S3.SS4.p1.16)\.
- J\. Wang, Y\. Liu, and X\. Wang \(2021b\)Are gender\-neutral queries really gender\-neutral? mitigating gender bias in image search\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 1995–2008\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.151/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.151)Cited by:[Figure 2](https://arxiv.org/html/2607.00423#S1.F2),[§1](https://arxiv.org/html/2607.00423#S1.p2.1),[§4\.3](https://arxiv.org/html/2607.00423#S4.SS3.p1.1)\.
- R\. J\. Williams \(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning8\(3–4\),pp\. 229–256\.Cited by:[§3](https://arxiv.org/html/2607.00423#S3.p1.1)\.
- R\. Wolfe, Y\. Yang, B\. Howe, and A\. Caliskan \(2023\)Contrastive language\-vision AI models pretrained on web\-scraped multimodal data exhibit sexual objectification bias\.InProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency \(FAccT\),pp\. 1174–1185\.External Links:[Document](https://dx.doi.org/10.1145/3593013.3594072)Cited by:[§1](https://arxiv.org/html/2607.00423#S1.p1.1)\.
- L\. Zancato, A\. Achille, T\. Y\. Liu, M\. Trager, P\. Perera, and S\. Soatto \(2023\)Train/test\-time adaptation with retrieval\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§2](https://arxiv.org/html/2607.00423#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Zhang, Y\. Guo, and M\. Kankanhalli \(2025\)Joint vision\-language social bias removal for clip\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 4246–4255\.Cited by:[Figure 2](https://arxiv.org/html/2607.00423#S1.F2),[§1](https://arxiv.org/html/2607.00423#S1.p2.1),[§2](https://arxiv.org/html/2607.00423#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2607.00423#S4.SS2.SSS0.Px3.p1.3.1),[§4\.3](https://arxiv.org/html/2607.00423#S4.SS3.p1.1)\.
- Z\. Zhang, Y\. Song, and H\. Qi \(2017\)Age progression/regression by conditional adversarial autoencoder\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\) Workshops,pp\. 4352–4360\.Cited by:[§1](https://arxiv.org/html/2607.00423#S1.p5.1),[§4\.1](https://arxiv.org/html/2607.00423#S4.SS1.p1.1)\.
- D\. Zhao, A\. Wang, and O\. Russakovsky \(2021\)Understanding and evaluating racial biases in image captioning\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 14830–14840\.Cited by:[§1](https://arxiv.org/html/2607.00423#S1.p1.1)\.
- S\. Zhao, X\. Wang, L\. Zhu, and Y\. Yang \(2024\)Test\-time adaptation with clip reward for zero\-shot generalization in vision\-language models\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.00423#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2607.00423#S3.SS3.SSS0.Px1.p1.3)\.
- B\. Zhu, J\. Cui, H\. Zhang, and C\. Zhang \(2025\)Project\-probe\-aggregate: efficient fine\-tuning for group robustness\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 25487–25496\.Cited by:[footnote 3](https://arxiv.org/html/2607.00423#footnote3)\.

## Appendix AImplementation Details

In this section, we provide comprehensive details regarding the experimental setup, optimization hyperparameters, and the construction of the bias subspace, complementing the summary provided in the main text\.

### A\.1Computational Environment and Models

All experiments were conducted on a singleNVIDIA A6000 GPUwith automatic mixed precision \(AMP\) enabled to enhance efficiency\. We utilized the official pre\-trained CLIP checkpoints\(Radfordet al\.,[2021](https://arxiv.org/html/2607.00423#bib.bib7)\)for ViT\-B/16, ViT\-B/32, and ViT\-L/14 backbones\. Consistent with the episodic TTA protocol, the CLIP reward signal was computed using a separate, fixedCLIP ViT\-L/14model to provide stable guidance\.

### A\.2Modality\-Specific Update Strategy

During the episodic adaptation, we update only the encoder corresponding to theinput query modalityto align it with the frozen target modality:

- •Text\-to\-Image Retrieval:We adapt thetext encoderwhile keeping the image encoder frozen\.
- •Image\-to\-Text Retrieval & Classification:We adapt theimage encoderwhile keeping the text encoder frozen\.

### A\.3Optimization Hyperparameters

We used theAdamWoptimizer\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2607.00423#bib.bib38)\)with a learning rate of1×10−41\\times 10^\{\-4\}and a weight decay of0\. The balancing coefficients for the reward function were set toλ=1000\\lambda=1000\(debiasing penalty weight\)\. Additionally, we set the Gaussian kernel temperature parameterγ=0\.25\\gamma=0\.25to control the sharpness of soft assignments in the bias subspace\.

### A\.4Adaptive Computational Budgeting

The Reward Gate \(RG\) determines the bias sensitivity using a fixed threshold ofϵ=0\.02\\epsilon=0\.02\. Based on the gate’s outputδ​\(q\)\\delta\(q\), we dynamically adjust the number of update steps \(TT\) and the candidate pool size \(KK\):

- •Bias\-Insensitive \(δ​\(q\)=0\\delta\(q\)=0\):We perform a lightweight update withT=3T=3steps andK=10K=10candidates\.
- •Bias\-Sensitive \(δ​\(q\)=1\\delta\(q\)=1\):We increase the budget toT=10T=10steps andK=1024K=1024candidates\. In this case, candidates are ranked by their CLIP alignment scores to filter the most relevant samples for the update\.

Table 7:Race debiasing performance using FairFace as the source\.BackboneMethodsFairfaceIN1KFlickrABLE \(%\)↑\\uparrowAcc\. \(%\)↑\\uparrowR@5 \(%\)↑\\uparrowMS↓\\downarrowNDKL↓\\downarrowTop\-1Top\-5TRIRViT\-B/16Original CLIP0\.5280\.18268\.3191\.8396\.485\.563\.31CLIP\-clip0\.5440\.16167\.9791\.6295\.485\.362\.62Biased\-prompts0\.5180\.21967\.0090\.7294\.185\.863\.07Joint V\-L0\.3530\.12568\.0791\.6496\.583\.869\.14Ours0\.3720\.16770\.3292\.9697\.288\.469\.62ViT\-B/32Original CLIP0\.5680\.16563\.3988\.8394\.783\.559\.84CLIP\-clip0\.7130\.22762\.5188\.3392\.681\.254\.95Biased\-prompts0\.5950\.28261\.8087\.4691\.983\.358\.29Joint V\-L0\.5030\.14963\.0788\.6194\.283\.061\.74Ours0\.3890\.14869\.7392\.2697\.086\.568\.74

### A\.5Bias Subspace Construction Details

The bias subspace is constructed offline using the training split of the dataset:

- •Text Queries:We use attribute\-specific prompts formatted as “a photo of a\{attribute class\}person”\. One prompt is generated per attribute class\.
- •Image Queries:We construct the reference setAAby samplingM=5M=5images per attribute class uniformly from theUTKFacedataset\. To ensure reproducibility and consistency, this reference set is fixed once and reused across all test episodes\. The total size of the reference set is\|A\|=C×5\|A\|=C\\times 5, whereCCis the number of attribute classes\.

## Appendix BAdditional Results

### B\.1Additional Race Debiasing Results using FairFace

In the main text \(Table[2](https://arxiv.org/html/2607.00423#S4.T2)\), we utilized UTKFace as the source dataset for constructing the bias subspace to mitigate Race bias\. To verify the robustness of our framework across different source domains, we conducted an additional experiment usingFairFaceas the source dataset\.

The results are presented in Table[7](https://arxiv.org/html/2607.00423#A1.T7)\. Consistent with the findings in the main text, our RG\-TTA framework demonstrates a superior capability to balance fairness and utility\. Regarding fairness, our method effectively mitigates racial bias compared to the Original CLIP; whileJoint V\-Lshows competitive scores on ViT\-B/16, our method achieves the best performance on ViT\-B/32 \(MaxSkew: 0\.389\)\. Crucially, in terms of utility preservation, our method consistently outperforms all baselines in zero\-shot tasks \(ImageNet and Flickr\) across both backbones, confirming that our selective routing mechanism successfully prevents the over\-debiasing observed in static approaches\. Consequently, our method achieves the highest ABLE scores for both backbones \(69\.62% and 68\.74%\), proving that it maintains the optimal trade\-off between fairness and utility regardless of the source dataset used\.

![Refer to caption](https://arxiv.org/html/2607.00423v1/x6.jpeg)

![Refer to caption](https://arxiv.org/html/2607.00423v1/x7.jpeg)

Figure 7:Gating failure cases on ImageNet\.In ImageNet, queries are expected to be bias\-insensitive and thus gated off \(δ​\(q\)=0\\delta\(q\)=0\)\. Shown are false\-positive cases where the gate is incorrectly activated \(δ​\(q\)=1\\delta\(q\)=1\), causing debiasing and alignment rewards to be jointly applied\.
### B\.2False\-positive gating failures on ImageNet

Figure[7](https://arxiv.org/html/2607.00423#A2.F7)illustratesfalse\-positivecases on ImageNet\-1K, where bias\-insensitive object queries are incorrectly gated on \(δ​\(q\)=1\\delta\(q\)=1\), triggering unnecessary fairness regularization\. These failures typically arise when the queried object strongly co\-occurs with humans or human\-like features in the top\-1 retrieved anchory∗y^\{\*\}\. For instance, as shown in Figure[7](https://arxiv.org/html/2607.00423#A2.F7), human figurines on a cake \(left\) or bystanders in the background \(right\) provide strong demographic signals that reduce the semantic\-attribute discrepancy below the thresholdϵ\\epsilon\. This misleads the gate into treating the object query as bias\-sensitive\. Although our hybrid reward design minimizes semantic drift even when the gate is mistakenly active, these cases represent a computational inefficiency\.

### B\.3Attribute\-Specified Queries

A natural question is whether the gating mechanism may overreact to queries that explicitly specify a demographic attribute \(e\.g\., “a photo of a male doctor”\), thereby distorting query intent\. Since such queries are likely to align with the corresponding attribute prompts in𝒜\\mathcal\{A\}, the gate may activateδ​\(q\)=1\\delta\(q\)\{=\}1and apply the attribute\-balancing reward, which could in principle override the user’s specified attribute\. To quantitatively assess this behavior, we evaluate on FACET \(ViT\-B/16\) using gender\-specified queries\. We introduce the*Attribute Consistency Rate*\(ACR@kk\): the proportion of top\-kkretrieved results whose gender matches the query\-specified gender\. Higher values indicate better preservation of query intent\.

As expected, gender\-specified queries triggerδ​\(q\)=1\\delta\(q\)\{=\}1, since they are close to the corresponding attribute prompts\. Nevertheless, RG\-TTA limits ACR degradation to only 1\.5–2\.6 percentage points compared to Original CLIP, because the CLIP alignment rewardrclipr\_\{\\text\{clip\}\}in Eq\. 5 assigns higher scores to gender\-matching candidates, effectively counterbalancing the attribute\-balancing penalty\. To verify this, we evaluate a*Debiasing Only*variant that removesrclipr\_\{\\text\{clip\}\}and applies the attribute\-balancing reward alone: ACR@10 drops sharply by 0\.16 \(0\.692→\\rightarrow0\.532\), confirming that the CLIP reward is the key component preventing this failure mode\.

### B\.4Intersectional Debiasing

Real\-world fairness concerns often involve multiple protected attributes simultaneously \(e\.g\., gender and race\)\. RG\-TTA accommodates intersectional settings by defining the class set𝒞\\mathcal\{C\}in Eq\. 3–4 as a Cartesian product of attributes \(e\.g\., gender×\\timesrace→\\rightarrow\{male\-White, male\-Black,…\\ldots\}\), without modifying the pipeline\. Rather than introducing separate reward terms per attribute \(which can cause multi\-objective conflicts\), this consolidates multi\-attribute balancing into a single reward over cross\-product classes\.

Table 8:Attribute Consistency Rate \(ACR@kk\) for gender\-specified queries on FACET \(ViT\-B/16\)\. “Debiasing Only” removes the CLIP alignment reward and forcesδ​\(q\)=1\\delta\(q\)\{=\}1\.MethodACR@10ACR@25ACR@50Original CLIP0\.6920\.6660\.649RG\-TTA \(Ours\)0\.6720\.6510\.623Debiasing Only0\.5320\.5240\.513Table 9:Intersectional debiasing on UTKFace \(gender×\\timesrace, ViT\-B/16\)\.AttributeMethodMaxSkew↓\\downarrowNDKL↓\\downarrowGenderOriginal CLIP0\.1140\.080RG\-TTA \(Ours\)0\.0720\.053RaceOriginal CLIP0\.5750\.137RG\-TTA \(Ours\)0\.3810\.089As shown in Table[9](https://arxiv.org/html/2607.00423#A2.T9), RG\-TTA reduces Gender MaxSkew by 36\.8% and Race MaxSkew by 33\.7%*simultaneously*, confirming effective multi\-attribute balancing\. As a test\-time method, only the class definition needs to change—no retraining is required\. We acknowledge that exponential growth of cross\-product classes \(e\.g\., 2×\\times4×\\times3=24 for three attributes\) may degrade the popularity estimatepcp\_\{c\}, and a more scalable formulation for high\-dimensional intersectional settings remains an open direction\.

### B\.5Generalization to BLIP

To verify that RG\-TTA generalizes beyond CLIP to other contrastive vision\-language models, we apply it to BLIP\-base \(Salesforce/blip\-itm\-base\-coco\) using its image\-text contrastive \(ITC\) branch\. We evaluate on UTKFace with gender as the protected attribute\.

Table 10:RG\-TTA applied to BLIP\-base on UTKFace \(Gender\)\. RG\-TTA generalizes to BLIP without retraining; only the gating thresholdϵ\\epsilonrequires recalibration\.MethodMaxSkew↓\\downarrowNDKL↓\\downarrowOriginal BLIP0\.2360\.115RG\-TTA \(Ours\)0\.1060\.044RG\-TTA achieves a 55\.1% reduction in MaxSkew and a 61\.7% reduction in NDKL on BLIP, confirming applicability to contrastive VLMs beyond CLIP\. Notably, no retraining is required; the entire framework operates at test time\.

#### Threshold recalibration\.

Different architectures produce different similarity score distributions, so the gating thresholdϵ\\epsilondoes not transfer directly across models\. Applying CLIP’s defaultϵ=0\.02\\epsilon\{=\}0\.02to BLIP yields 86\.2%δ​\(q\)=1\\delta\(q\)\{=\}1on UTKFace and 8\.1% on ImageNet \(compared to 99\.9% / 0\.1% on CLIP\)\. However, recalibration only involves adjusting a single scalar with a small validation set—no architectural changes or retraining are needed\.

### B\.6Generalization Beyond Demographic Bias

While RG\-TTA primarily targets social fairness with demographic attributes, the underlying mechanism—selective debiasing in a bias subspace—is in principle applicable to other types of distributional bias\. To briefly probe this generalization, we apply RG\-TTA to the Waterbirds benchmark, where bias arises from spurious correlations between bird species and background scenes \(water vs\. land\), rather than from protected demographic attributes\. We set the background as the bias subspace attribute and measure the distribution bias in top retrieval results for neutral queries \(e\.g\., “a photo of a bird”\)\.

Table 11:RG\-TTA on Waterbirds \(ViT\-B/16\)\. The bias subspace attribute is set to background \(water vs\. land\)\.MethodMaxSkew↓\\downarrowNDKL↓\\downarrowZero\-shot CLIP0\.2390\.134RG\-TTA \(Ours\)0\.1860\.070By simply redefining the bias subspace attribute \(demographic→\\rightarrowbackground\), RG\-TTA reduces MaxSkew by 22\.2% and NDKL by 47\.8% without retraining\. While a thorough investigation in spurious\-correlation settings is beyond the scope of this work, this result suggests broader applicability of the proposed selective adaptation principle\.