LLM Detoxification via Transition-Aware Reward-Guided Representation Sampling

# LLM Detoxification via Transition-Aware Reward-Guided Representation Sampling
Source: [https://arxiv.org/html/2610.41165](https://arxiv.org/html/2610.41165)
Yisong XiaoXianglong Liu🖂Yongxin HuangSiyuan LiangZonghao YingYang QuJian YangYaodong YangDacheng Tao††thanks:A\. Liu, Y\. Xiao, X\. Liu \(🖂corresponding author\), Y\. Huang, Z\. Ying, Y\. Qu, and J\. Yang are with SKLCCSE, the School of Computer Science and Engineering, Beihang University, Beijing 100191, China\.††thanks:A\. Liu is also with BAAI, Beijing 100191, China\.††thanks:S\. Liang and D\. Tao are with the College of Computing & Data Science, Nanyang Technological University, Singapore 639798\.††thanks:Y\. Yang is with the Institute for Artificial Intelligence, Peking University, Beijing 100191, China\.

###### Abstract

Although Large Language Models \(LLMs\) have showcased remarkable capabilities across diverse tasks, their susceptibility to producing harmful or toxic content underscores the urgent need for effective detoxification strategies to ensure their safe and ethical deployment\. Test\-time detoxification methods, which often involve directional steering within LLM representations during inference, have emerged as one of the most promising approaches due to their flexibility and minimal invasiveness\. However, current approaches often rely on static or coarse interventions and suffer from imprecise guidance, largely due to their limited ability to model the transition process between toxic and non\-toxic generations\. To address this challenge, we propose Transition\-Aware Reward\-Guided Representation Sampling \(TARGRS\), a test\-time detoxification framework that learns reward guidance directly over toxicity transitions in the latent representation space\. Rather than relying on sparse toxic or non\-toxic representations, our method performs progressive directional steering during generation to trace representation trajectories that span the transition from non\-toxic to toxic states\. Each trajectory is assigned toxicity scores, converting sparse annotations into dense, listwise supervision over fine\-grained transitions\. Based on this formulation, we train atoken\-levelreward model whose predictions are explicitly transition\-aware, enabling accurate control over representation sampling during inference\.During exploration, steering along the toxicity direction constructs transition trajectories for reward learning; during inference, TARGRS performs coarse\-to\-fine sampling along the opposite, non\-toxic direction to obtain detoxified representations\.Specifically, coarse sampling first identifies the intervention required to reach non\-toxic regions, followed by refinement to select the optimal representation with minimal perturbation\. Comprehensive experiments on 10 widely used LLMs across the RealToxicityPrompts, BOLD, and AttaQ benchmarks demonstrate that TARGRS achieves consistent and strong detoxification performance, significantly outperforming state\-of\-the\-art baselines\. For instance, TARGRS reduces toxicity by up to 69\.38% on the RealToxicityPrompts\. TARGRS also presents impressive inference efficiency, running nearly as fast as the original LLM\. Additionally, TARGRS preserves the model’s core capabilities with minimal impact\. Furthermore, benefiting from intervention in the language decoder space, TARGRS can be seamlessly extended to Large Vision\-Language Models \(LVLMs\), where it also demonstrates strong effectiveness, reducing toxicity by up to 64\.23% across 3 representative LVLMs\.Our code is available at[https://github\.com/xiaoyisong/TARGRS](https://github.com/xiaoyisong/TARGRS)\.

###### Index Terms:

Toxicity Mitigation, Large Language Model, Representation Sampling, Large Vision Language Model

## IIntroduction

The remarkable progress of Large Language Models \(LLMs\) has revolutionized various fields, showcasing exceptional capabilities in natural language understanding, generation, and reasoning\[[1](https://arxiv.org/html/2610.41165#bib.bib17),[2](https://arxiv.org/html/2610.41165#bib.bib18),[3](https://arxiv.org/html/2610.41165#bib.bib19)\]\. These models are now deeply embedded in everyday life, driving technologies such as search engines\[[4](https://arxiv.org/html/2610.41165#bib.bib65)\]and virtual assistants\[[5](https://arxiv.org/html/2610.41165#bib.bib66)\]\. Despite these advancements, LLMs remain susceptible to generating*toxic content*, largely due to their pre\-training on massive, unfiltered web\-crawled text corpora that may inadvertently encode harmful biases and undesirable patterns\[[6](https://arxiv.org/html/2610.41165#bib.bib20),[7](https://arxiv.org/html/2610.41165#bib.bib21)\]\. For instance, Microsoft’s AI chatbot Tay infamously produced racist and inappropriate content after interacting with users on social media\[[8](https://arxiv.org/html/2610.41165#bib.bib63)\]\. As LLMs continue to be integrated into socially sensitive applications, developing effective detoxification techniques is critical to ensuring their ethical and responsible deployment\.

![Refer to caption](https://arxiv.org/html/2610.41165v1/fig/frontpage.png)Fig\. 1:Illustration of toxicity in LLMs\. We introduce TARGRS for detoxification, which effectively steers harmful and toxic generation \(e\.g\., profanity, aggression, and explicit content\) toward safe and non\-toxic \(benign\) outputs\.A significant body of research has focused on mitigating toxicity in LLMs, which can be broadly categorized into training\-time and test\-time methods\[[9](https://arxiv.org/html/2610.41165#bib.bib72),[10](https://arxiv.org/html/2610.41165#bib.bib42),[11](https://arxiv.org/html/2610.41165#bib.bib45),[12](https://arxiv.org/html/2610.41165#bib.bib3),[13](https://arxiv.org/html/2610.41165#bib.bib25)\]\. Training\-time methods\[[14](https://arxiv.org/html/2610.41165#bib.bib43),[9](https://arxiv.org/html/2610.41165#bib.bib72),[15](https://arxiv.org/html/2610.41165#bib.bib46)\]typically fine\-tune LLMs on carefully curated preference datasets containing toxic and non\-toxic response pairs, leveraging techniques such as direct preference optimization \(DPO\)\[[16](https://arxiv.org/html/2610.41165#bib.bib2)\]to guide the model toward favoring non\-toxic responses\. However, these training approaches necessitate costly data collection and heavy computational overhead, making them impractical in resource\-constrained environments\. Accordingly, recent research has pivoted to*test\-time detoxification*during inference, where representation editing has garnered considerable attention for its inherent flexibility and minimal invasiveness to the pre\-trained LLMs\[[11](https://arxiv.org/html/2610.41165#bib.bib45),[13](https://arxiv.org/html/2610.41165#bib.bib25),[17](https://arxiv.org/html/2610.41165#bib.bib39),[18](https://arxiv.org/html/2610.41165#bib.bib10),[19](https://arxiv.org/html/2610.41165#bib.bib51)\]\. Building on the linear representation hypothesis\[[20](https://arxiv.org/html/2610.41165#bib.bib11),[21](https://arxiv.org/html/2610.41165#bib.bib12),[22](https://arxiv.org/html/2610.41165#bib.bib13)\], which suggests that human\-interpretable concepts are encoded as linear directions within representations, representation editing methods steer the model’s representations toward non\-toxic directions through static or coarse interventions\. However, the effectiveness of these editing methods is often constrained by inaccurate interventions, primarily due to insufficient exploration of the transition space between toxic and non\-toxic representations\. Specifically, the reliance solely on sparse toxicity annotations limits these methods’ ability to capture the nuanced toxicity transition trajectories necessary for accurate editing guidance\.

To address this challenge, we proposeTransition\-AwareReward\-GuidedRepresentationSampling \(TARGRS\), a test\-time detoxification framework that explicitly tracks the representation transitions during toxic generation, enabling accurate reward\-guided sampling within the representation space and thus achieving effective toxicity mitigation \(as depicted in Fig\.[1](https://arxiv.org/html/2610.41165#S1.F1)\)\. By leveraging the continuous semantic representation space, we can track and characterize shifts in toxicity, which allows for the exploration of toxicity transition trajectories that are difficult to capture in the discrete natural language space\. Specifically, TARGRS first identifies the toxicity semantic direction by contrasting representations from toxic and non\-toxic annotations, then progressively steers the representation along this direction during LLM generation to reveal fine\-grained toxicity transition trajectories and their corresponding responses\. By introducing a toxicity scorer, we can assess the generated responses and assign a toxicity score to each transition trajectory to acquire comparable labels\. In this way, these trajectories transform sparse toxicity annotations into dense, listwise toxicity signals, facilitating smooth transitions across toxicity levels and laying the groundwork for modeling toxicity risk in the representation space\. Leveraging these trajectories, we train atoken\-levelreward model to estimate the toxicity of token representations using a listwise preference loss, offering accurate guidance for representation sampling\. During autoregressive generation, TARGRS applies a coarse\-to\-fine sampling strategy along the non\-toxic direction: coarse sampling identifies the necessary intervention to steer representations toward non\-toxic regions, followed by refinement that selects the optimal representation with minimal perturbation, thereby achieving effective and efficient toxicity mitigation\. Benefiting from intervention in the language decoder space, the universal core component responsible for text generation in both LLMs and LVLMs, our method supports cross\-modal generalization and can be naturally extended to multimodal scenarios\.

Extensive experiments on three toxicity evaluation benchmarks across ten widely used LLMs show that TARGRS consistently yields strong detoxification performance, reducing toxicity by up to 69\.38% on the challenging RealToxicityPrompts dataset\[[7](https://arxiv.org/html/2610.41165#bib.bib21)\], 46\.85% on the BOLD dataset\[[23](https://arxiv.org/html/2610.41165#bib.bib70)\], and 43\.91% on the AttaQ dataset\[[24](https://arxiv.org/html/2610.41165#bib.bib71)\]\. In addition to superior detoxification effectiveness, our method maintains strong inference efficiency, running nearly as fast as the original LLM and reducing inference overhead by 11\.40% and 56\.49% compared to the best and second\-best test\-time baselines, respectively\. Furthermore, TARGRS preserves the original capabilities of LLMs with minimal degradation to general task performance\. Benefiting from toxicity transition exploration, TARGRS also achieves high data efficiency without relying on extensive manual annotations\. Moreover, our method can be seamlessly extended to LVLMs, where it again demonstrates strong effectiveness by reducing toxicity by up to 64\.23% on the ToViLaG\[[25](https://arxiv.org/html/2610.41165#bib.bib82)\]dataset across three representative LVLMs\.

Ourcontributionsare summarized as follows:

- •We propose TARGRS, a novel test\-time detoxification framework that explicitly tracks the representation transitions from non\-toxic to toxic generations, facilitating accurate reward\-guided sampling within the representation space\.
- •We explore toxicity transitions via directional steering, develop a transition\-aware token\-level reward model trained on the resulting scored trajectories using a listwise preference loss, and design a coarse\-to\-fine representation sampling strategy that leverages the learned reward guidance to achieve effective and efficient detoxification with minimal perturbation\.
- •Extensive experiments on 10 widely used LLMs across three benchmarks demonstrate that TARGRS significantly outperforms leading baselines in both effectiveness and efficiency, while maintaining LLM’s capabilities\. Additionally, TARGRS shows strong detoxification effectiveness on 3 LVLMs\.

A preliminary version of this work\[[26](https://arxiv.org/html/2610.41165#bib.bib78)\]was accepted as a poster at NeurIPS 2025, where we presented ARGRE, a reward\-guided representation editing method for LLM toxicity mitigation\.The present work substantially extends ARGRE in both methodological design and experimental evaluation\. ❶Toxicity transition exploration\. TARGRS replaces interpolation between toxic and non\-toxic annotation representations with progressive direction steering to collect scored transition trajectories for accurate reward modeling\. ❷Training paradigm of the reward model\. TARGRS trains its token\-level reward model with a listwise preference loss instead of the pairwise preference loss used in ARGRE, providing more accurate reward guidance\. ❸Representation editing\. TARGRS accelerates the steering and gradient\-based refinement steps of the original method by employing a more efficient coarse\-to\-fine sampling strategy\. ❹Comprehensive experiments\. We evaluate TARGRS on 10 base and instruction\-tuned LLMs across three toxicity benchmarks, demonstrating its consistent and superior effectiveness\.❺Detoxification on LVLMs\. Our method can be seamlessly extended to LVLMs, where it also demonstrates strong effectiveness over three representative LVLMs\.

## IIRelated Works

### II\-AToxicity in LLMs

Toxicity in Large Language Models \(LLMs\) has emerged as a critical barrier to their safe and ethical deployment in human\-centric applications and societal infrastructure\[[27](https://arxiv.org/html/2610.41165#bib.bib67)\]\. Trained on vast, unfiltered web\-scale corpora, LLMs often inherit harmful biases, hate speech, harassment, derogatory language, and other toxic content from the data, which can then be amplified and perpetuated during text generation\[[28](https://arxiv.org/html/2610.41165#bib.bib68),[29](https://arxiv.org/html/2610.41165#bib.bib69),[30](https://arxiv.org/html/2610.41165#bib.bib53),[31](https://arxiv.org/html/2610.41165#bib.bib52),[32](https://arxiv.org/html/2610.41165#bib.bib55),[33](https://arxiv.org/html/2610.41165#bib.bib59),[34](https://arxiv.org/html/2610.41165#bib.bib54)\]\. Leeet al\.\[[13](https://arxiv.org/html/2610.41165#bib.bib25)\]suggested that LLMs internalize these harmful patterns as specific activation pathways within their high\-dimensional representation space during pre\-training, and when prompted, the model’s representations traverse these toxic regions, leading to the generation of harmful outputs\. The consequences of such toxicity can be severe, causing harm to individuals by exacerbating mental health struggles, self\-harm, and suicidal thoughts\[[35](https://arxiv.org/html/2610.41165#bib.bib64)\]\. Numerous studies\[[7](https://arxiv.org/html/2610.41165#bib.bib21),[23](https://arxiv.org/html/2610.41165#bib.bib70),[24](https://arxiv.org/html/2610.41165#bib.bib71),[36](https://arxiv.org/html/2610.41165#bib.bib98),[37](https://arxiv.org/html/2610.41165#bib.bib60)\]have sought to uncover toxicity in LLMs\. For example, Gehmanet al\.\[[7](https://arxiv.org/html/2610.41165#bib.bib21)\]introduced the RealToxicityPrompts dataset, comprising 100K naturally occurring sentence\-level prompts from a large English web text corpus, demonstrating that LLMs can generate toxic content even from ostensibly non\-toxic inputs or contexts\. Similarly, Dhamalaet al\.\[[23](https://arxiv.org/html/2610.41165#bib.bib70)\]developed the BOLD dataset to assess toxicity in open\-ended language generation across five domains: profession, gender, race, religion, and political ideology\. Kouret al\.\[[24](https://arxiv.org/html/2610.41165#bib.bib71)\]designed the AttaQ dataset as a question\-answering task to elicit toxicity from LLMs across diverse scenarios \(e\.g\., violence and sexual content\)\.

### II\-BToxicity Mitigation

A long line of research has been devoted to mitigating toxicity in LLMs\[[12](https://arxiv.org/html/2610.41165#bib.bib3),[13](https://arxiv.org/html/2610.41165#bib.bib25),[14](https://arxiv.org/html/2610.41165#bib.bib43),[38](https://arxiv.org/html/2610.41165#bib.bib44),[11](https://arxiv.org/html/2610.41165#bib.bib45),[10](https://arxiv.org/html/2610.41165#bib.bib42)\], which can be broadly classified into training\-time and test\-time mitigation methods\.

Training\-time methodsmitigate toxicity by modifying LLM parameters through re\-training on curated non\-toxic datasets\[[14](https://arxiv.org/html/2610.41165#bib.bib43),[9](https://arxiv.org/html/2610.41165#bib.bib72),[39](https://arxiv.org/html/2610.41165#bib.bib56),[40](https://arxiv.org/html/2610.41165#bib.bib57),[41](https://arxiv.org/html/2610.41165#bib.bib62),[42](https://arxiv.org/html/2610.41165#bib.bib58),[43](https://arxiv.org/html/2610.41165#bib.bib99)\]\. A common approach is to filter out toxic examples\[[9](https://arxiv.org/html/2610.41165#bib.bib72)\]or label them as undesirable\[[44](https://arxiv.org/html/2610.41165#bib.bib73),[45](https://arxiv.org/html/2610.41165#bib.bib74)\], with some methods also using adversarial human inputs to detect and neutralize potential sources of unsafe content\[[46](https://arxiv.org/html/2610.41165#bib.bib75),[47](https://arxiv.org/html/2610.41165#bib.bib76)\], thereby creating a sanitized corpus for model pre\-training\. Recently, Wanget al\.\[[15](https://arxiv.org/html/2610.41165#bib.bib46)\]applied reinforcement learning from human feedback \(RLHF\)\[[48](https://arxiv.org/html/2610.41165#bib.bib1)\]to calibrate harmful generation based on preference data\. DPO\[[16](https://arxiv.org/html/2610.41165#bib.bib2)\]streamlines this by directly fine\-tuning on preference pairs\. However, these methods rely heavily on high\-quality preference data and intensive training, limiting their democratization and broader applicability\. More critically, recent evidence suggests that fine\-tuning may only superficially bypass toxic outputs without addressing the underlying harmful internal states, leaving models vulnerable to generating harmful content when exposed to specific triggering prompts\[[13](https://arxiv.org/html/2610.41165#bib.bib25),[49](https://arxiv.org/html/2610.41165#bib.bib41)\]\.

Test\-time methodsgenerally fall into three categories\.

❶Guided decoding methods\[[50](https://arxiv.org/html/2610.41165#bib.bib27),[51](https://arxiv.org/html/2610.41165#bib.bib38),[52](https://arxiv.org/html/2610.41165#bib.bib47),[10](https://arxiv.org/html/2610.41165#bib.bib42),[53](https://arxiv.org/html/2610.41165#bib.bib6)\]mitigate toxic generation by modifying the token probability distribution of frozen LLMs during the autoregressive decoding process\. PPLM\[[50](https://arxiv.org/html/2610.41165#bib.bib27)\]computes the gradient of a differentiable toxicity classifier to guide the LLM’s generation towards non\-toxic outputs\. Unfortunately, this approach is computationally expensive, as it requires multiple backward passes to sample each token, resulting in significantly prolonged inference latency\. DexPerts\[[51](https://arxiv.org/html/2610.41165#bib.bib38)\]employs trained classifiers, smaller in size than the base LLM, to distinguish between toxic and non\-toxic attributes, efficiently promoting the selection of tokens aligned with non\-toxic traits\. RAD\[[54](https://arxiv.org/html/2610.41165#bib.bib61)\]employs a unidirectional reward model that allows intermediate activation caching during generation, enabling it to encourage outputs with specified desirable properties while largely reducing computational overhead\. GenARM\[[53](https://arxiv.org/html/2610.41165#bib.bib6)\]also learns a reward model that aligns with the base LLM to score token probability distributions, enabling efficient generation toward more desirable outcomes\. However, directly altering token probabilities can disrupt natural generation, leading to degraded fluency and coherence under strong control\. Furthermore, the effectiveness of such reward\-guided approaches remains inherently limited by reward models that often overfit to restricted toxicity annotations and fail to explore the language space sufficiently\[[55](https://arxiv.org/html/2610.41165#bib.bib7),[56](https://arxiv.org/html/2610.41165#bib.bib8)\], ultimately resulting in suboptimal detoxification performance\.

❷*Weight editing methods*\[[57](https://arxiv.org/html/2610.41165#bib.bib4),[58](https://arxiv.org/html/2610.41165#bib.bib77),[12](https://arxiv.org/html/2610.41165#bib.bib3),[38](https://arxiv.org/html/2610.41165#bib.bib44)\]detoxify LLMs by removing harmful components from their parameters\. Weiet al\.\[[57](https://arxiv.org/html/2610.41165#bib.bib4)\]applied pruning and low\-rank updates to isolate safety\-critical weights, leading to safer model behavior\. DAMA\[[58](https://arxiv.org/html/2610.41165#bib.bib77)\]addresses toxic knowledge by refining the feature representations of associated concepts and applies a linear projection to adjust the parameters within the model’s MLP layers\. Similarly, Uppaalet al\.\[[12](https://arxiv.org/html/2610.41165#bib.bib3)\]proposed ProFS, which employs low\-rank weight decompositions to uncover conceptual subspaces and uses them as projection filters to eliminate toxic MLP weights\. However, weight editing methods tend to result in inferior detoxification performance on large\-scale language models and risk impairing their general knowledge and reasoning capabilities\[[59](https://arxiv.org/html/2610.41165#bib.bib9)\], thereby undermining the overall performance\.

❸*Representation editing methods*\[[17](https://arxiv.org/html/2610.41165#bib.bib39),[60](https://arxiv.org/html/2610.41165#bib.bib49),[61](https://arxiv.org/html/2610.41165#bib.bib48),[11](https://arxiv.org/html/2610.41165#bib.bib45),[13](https://arxiv.org/html/2610.41165#bib.bib25),[18](https://arxiv.org/html/2610.41165#bib.bib10)\]mitigate toxicity by applying targeted interventions to LLM representations, guiding model representations toward directions that align with safe and desirable behavioral preferences\.Previous studies\[[62](https://arxiv.org/html/2610.41165#bib.bib100),[63](https://arxiv.org/html/2610.41165#bib.bib101)\]have demonstrated that high\-level LLM behaviors can be controlled by identifying behavior\-related directions and steering internal representations accordingly during inference\.For toxicity mitigation, Self\-Detoxify\[[17](https://arxiv.org/html/2610.41165#bib.bib39)\]identifies a toxic direction by contrasting toxic and non\-toxic examples, then applies static interventions during inference to suppress toxic generation\. Building on this approach, DeStein\[[11](https://arxiv.org/html/2610.41165#bib.bib45)\]enhances performance by using linear classifiers trained on a few toxicity annotations to more precisely identify toxic directions\. Re\-Control\[[18](https://arxiv.org/html/2610.41165#bib.bib10)\]improves upon static edits by learning a value function that generates dynamic, context\-dependent control signals, which guide tedious gradient\-based iterations to achieve desirable representations\. Despite being dynamic, the control signals in Re\-Control may be inaccurate, as its value function is trained on a single trajectory per prompt \(i\.e\., insufficient exploration\)\[[64](https://arxiv.org/html/2610.41165#bib.bib15)\]and sparse trajectory\-level reward signals\[[65](https://arxiv.org/html/2610.41165#bib.bib16),[66](https://arxiv.org/html/2610.41165#bib.bib14)\]\. Current representation editing methods thus suffer from inaccurate interventions, as they fail to adequately explore the transition space between sparse toxicity annotations, leading to suboptimal performance\.

Comparisons\.Our TARGRS distinguishes itself in three key aspects: ❶Motivation\. TARGRS explicitly models toxicity transitions within the representation space, constructing dense and listwise trajectories that enable more effective detoxification, whereas prior methods depend on sparse toxicity annotations with insufficient transition exploration\. ❷Implementation\. Leveraging these transitions, TARGRS trains atoken\-levelreward model using a listwise preference loss to assess the toxicity risk of each token representation, offering more accurate guidance for the representation sampling process while avoiding the imprecise interventions and intrusive token\- or weight\-level modifications seen in existing methods\. ❸Effects\. TARGRS consistently achieves strong performance with high efficiency across diverse LLMs and toxicity evaluation benchmarks, while existing methods are often constrained by suboptimal effectiveness or substantial computational overhead\.

## IIIPreliminaries

Reinforcement Learning from Human Feedback \(RLHF\) aligns LLMs with human preferences by training a reward model on preference data and subsequently optimizing the language model through reinforcement learning\.

Reward model learning based on pairwise toxic data\.A reward modelr⁡\(x,y\)r\(x,y\)maps an input promptxxand its generated responseyyto a scalar reward\. Given a pairwise preference dataset𝒟\\mathcal\{D\}composed of annotated triples\(x,y\+,y−\)\(x,y\_\{\+\},y\_\{\-\}\), wherey\+y\_\{\+\}andy−y\_\{\-\}denote non\-toxic and toxic responses generated by the base LLMπbase​\(y∣x\)\\pi\_\{\\rm\{base\}\}\(y\\mid x\), the reward model is commonly optimized using the pairwise negative log\-likelihood loss to assign higher scores to non\-toxic responses:

minr−𝔼\(x,y\+,y−\)∼𝒟​\[log⁡σ⁡\(r⁡\(x,y\+\)−r⁡\(x,y−\)\)\],\\min\_\{r\}\-\\mathbb\{E\}\_\{\(x,y\_\{\+\},y\_\{\-\}\)\\sim\\mathcal\{D\}\}\\bigl\[\\log\\sigma\(r\(x,y\_\{\+\}\)\-r\(x,y\_\{\-\}\)\)\\bigr\],\(1\)whereσ\\sigmadenotes the logistic sigmoid function\. For the architecture, the reward modelr⁡\(x,y\)r\(x,y\)is usually initialized from the base LLMπbase​\(y∣x\)\\pi\_\{\\rm\{base\}\}\(y\\mid x\), with a trainable linear layer parameterized byθ\\thetastacked on top of the final transformer layer\[[67](https://arxiv.org/html/2610.41165#bib.bib22)\]\.

Kullback\-Leibler \(KL\)\-regularized RL fine\-tuning\.Using the reward model, RLHF fine\-tunes the base LLMπbase​\(y∣x\)\\pi\_\{\\rm\{base\}\}\(y\\mid x\)for toxicity mitigation by maximizing expected reward while penalizing deviations from the base model via KL divergence minimization:

maxπ𝔼x∼𝒟,y∼π⁡\(x\)r\(x,y\)−βDKL\(π\(y\|x\)\|\|πbase\(y\|x\)\),\\max\_\{\\pi\}\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},y\\sim\\pi\(x\)\}r\(x,y\)\-\\beta D\_\{\\rm\{KL\}\}\(\\pi\(y\|x\)\|\|\\pi\_\{\\rm\{base\}\}\(y\|x\)\),\(2\)whereβ\\betais a hyperparameter that controls the trade\-off between reward maximization and preserving the behavior of the base LLM\. Direct fine\-tuning of the base LLM incurs substantial computational overhead, prompting recent research to focus on leveraging the closed\-form solution of the RLHF objective during test time\. These approaches keep the base model frozen and use the pre\-trained reward model to guide generation at inference\. Building on prior work\[[68](https://arxiv.org/html/2610.41165#bib.bib23),[16](https://arxiv.org/html/2610.41165#bib.bib2)\], the objective in Eqn\.[2](https://arxiv.org/html/2610.41165#S3.E2)admits a closed\-form solution, given by:

π^\(y\|x\)∝πbase\(y\|x\)exp\(1βr\(x,y\)\),\\hat\{\\pi\}\(y\|x\)\\propto\\pi\_\{\\text\{base\}\}\(y\|x\)\\exp\\bigl\(\\frac\{1\}\{\\beta\}r\(x,y\)\\bigl\),\(3\)whereπbase\\pi\_\{\\text\{base\}\}remains frozen,yydenotes any potential response, and the rewardr⁡\(x,y\)r\(x,y\)guides the base LLM’s generation to produce a modulated distributionπ^\\hat\{\\pi\}that favors high\-reward \(i\.e\., non\-toxic\) responses\. Specifically, for sample\(x,y\)\(x,y\), the reward is computed from the base LLM’s final\-layer hidden representation𝒉x,y\\bm\{h\}^\{x,y\}as:r⁡\(x,y\)=θ⁡\(𝒉x,y\)r\(x,y\)=\\theta\(\\bm\{h\}^\{x,y\}\)\. Thus, the representation𝒉x,y\\bm\{h\}^\{x,y\}directly influences the reward score and, consequently, the toxicity of the generated content, serving as a minimally invasive interface for controlling the LLM’s output toward non\-toxic responses\.

## IVMethodology

![Refer to caption](https://arxiv.org/html/2610.41165v1/framework-pami-v2.png)Fig\. 2:Overview of TARGRS\. It identifies toxic semantic directions and performs steering to capture transition trajectories from non\-toxic to toxic states, where each trajectory is assigned a toxicity score\. These trajectories convert sparse annotations into listwise supervision, supporting the construction of a transition\-aware reward model for precise guidance\. During inference, the reward model guides a coarse\-to\-fine sampling process to produce detoxified representations\.### IV\-AOverview

Motivation\.Existing representation editing methods\[[11](https://arxiv.org/html/2610.41165#bib.bib45),[17](https://arxiv.org/html/2610.41165#bib.bib39),[18](https://arxiv.org/html/2610.41165#bib.bib10),[60](https://arxiv.org/html/2610.41165#bib.bib49)\]steer hidden representations via static or coarse interventions to reach non\-toxic, high\-reward regions in the latent space\. These methods typically first identify a toxicity semantic direction𝒅\\bm\{d\}, then apply the following intervention to the token\-level representation𝒉\[t\]\\bm\{h\}\_\{\[t\]\}:

𝒉^\[t\]=𝒉\[t\]−λcontrol⋅𝒅,\\hat\{\\bm\{h\}\}\_\{\[t\]\}=\\bm\{h\}\_\{\[t\]\}\-\\lambda\_\{\\rm\{control\}\}\\cdot\\bm\{d\},\(4\)whereλcontrol\\lambda\_\{\\rm\{control\}\}serves as the weight to control detoxification signal, which can be implemented as a constant value or a token representation\-aware coarse value\. However, these methods depend solely on representation trajectories derived from sparse toxic and non\-toxic annotations, failing to explore the nuanced transitions between toxic and non\-toxic states during generation\. This insufficient exploration leads to imprecise intervention signals, which ultimately degrade reward scores and hinder detoxification performance\. Therefore, our objective is to explicitly model toxicity transitions in the latent space by characterizing representation trajectories from non\-toxic to toxic states, and to learn a reward model that quantifies the toxicity risk for each token representation, thereby delivering precise guidance for detoxification\.

Overall Framework\.Building on the preceding motivation, we propose a framework named TARGRS that learns reward guidance directly from toxicity transitions in the latent representation space\. The framework consists of three core components: ❶ toxicity transition exploration, which employs progressive directional steering during LLM generation to capture representation trajectories that span the transition from non\-toxic to toxic states, with each trajectory assigned a toxicity score, thus converting sparse annotations into dense, listwise supervision over fine\-grained transitions; ❷ atoken\-levelreward model, which estimates the toxicity risk of each token representation using a listwise preference loss over the traced transition trajectories; ❸ a coarse\-to\-fine representation sampling process, which leverages accurate reward signals to guide the sampling of detoxified representations, achieving both effective and efficient toxicity mitigation\.Furthermore, by operating within the language decoder representation space, TARGRS can be naturally extended to LVLMs and demonstrates strong detoxification effectiveness, as detailed in Sec\.[VI](https://arxiv.org/html/2610.41165#S6)\.An overview of the framework is provided in Fig\.[2](https://arxiv.org/html/2610.41165#S4.F2)\.

### IV\-BToxicity Transition Exploration

Leveraging the linear representation hypothesis\[[20](https://arxiv.org/html/2610.41165#bib.bib11),[21](https://arxiv.org/html/2610.41165#bib.bib12),[22](https://arxiv.org/html/2610.41165#bib.bib13)\], which posits that concepts like toxicity are encoded as distinct linear directions within the latent representation space of LLMs, we can efficiently explore toxicity transitions by navigating the continuous semantic representation space\. Specifically, we first identify the toxicity semantic direction, and then progressively steer the representation along this direction during LLM generation, capturing how toxicity evolves from non\-toxic to toxic states\. This process reveals fine\-grained transition trajectories, allowing for a more detailed understanding of how toxicity manifests in the model’s representation\.

To identify the semantic direction associated with toxicity, we leverage pairwise annotations of toxic and non\-toxic response pairs to contrast their latent representations\. For a given promptxxpaired with a non\-toxic responsey\+y\_\{\+\}and a toxic responsey−y\_\{\-\}, we pass each response through the base LLM separately and extract the final\-layer hidden representations𝒉x,y\+\\bm\{h\}^\{x,y\_\{\+\}\}and𝒉x,y−\\bm\{h\}^\{x,y\_\{\-\}\}\. The toxicity direction is then computed as the difference between the representations at the final token, as follows:

Δ​𝒉​\(x,y\+,y−\)=𝒉\[−1\]x,y−−𝒉\[−1\]x,y\+,\\Delta\\bm\{h\}\{\(x,y\_\{\+\},y\_\{\-\}\)\}=\\bm\{h\}\_\{\[\-1\]\}^\{x,y\_\{\-\}\}\-\\bm\{h\}\_\{\[\-1\]\}^\{x,y\_\{\+\}\},\(5\)where we focus on the final token representation because the LLM employs causal modeling, and its self\-attention mechanism aggregates contextual information from all preceding tokens into the representation at the last position\[[69](https://arxiv.org/html/2610.41165#bib.bib24)\]\. To improve generalizability across different toxicity annotations, we aim to eliminate annotation\-specific biases while preserving the core toxicity signal\. Specifically, we aggregate the toxicity direction matrix\{Δ​𝒉​\(x\(i\),y\+\(i\),y−\(i\)\)\}i=1N\\\{\\Delta\\bm\{h\}\(\{x^\{\(i\)\},y\_\{\+\}^\{\(i\)\},y\_\{\-\}^\{\(i\)\}\}\)\\\}\_\{i=1\}^\{N\}from a small sample set \(sizeNN\), and then apply Principal Component Analysis \(PCA\)\[[70](https://arxiv.org/html/2610.41165#bib.bib50)\]on the matrix to extract the first principal component𝒅\\bm\{d\}, which captures the dominant toxic direction\.

The direction𝒅\\bm\{d\}provides a clear path for exploring transitions from non\-toxic to toxic states within the high\-dimensional semantic representation space\. To capture the representation trajectories that the model encounters during generation, we feed the prompt prefixxxinto the LLM and apply progressive directional steering, gradually guiding the model towards generating toxic content:

𝒉\[t\]λ=𝒉\[t\]\+λ⋅𝒅,\\bm\{h\}\_\{\[t\]\}^\{\\lambda\}=\\bm\{h\}\_\{\[t\]\}\+\\lambda\\cdot\\bm\{d\},\(6\)whereλ\\lambdacontrols the steering strength, withλ\\lambdaselected from a set of candidate valuesΛ\\Lambda\. Note that this steering process is similar to detoxification editing \(see Eqn\.[4](https://arxiv.org/html/2610.41165#S4.E4)\), but with the opposite goal: instead of reducing toxicity, it aims to induce it in otherwise non\-toxic responses conditioned on the promptxx\. The representation𝒉λ\\bm\{h\}^\{\\lambda\}represents one of the explored trajectories in the toxicity transition space, denoted as\{𝒉λ\}λ∈Λ\\\{\{\\bm\{h\}^\{\\lambda\}\}\\\}\_\{\\lambda\\in\\Lambda\}\. For each trajectory𝒉λ\\bm\{h\}^\{\\lambda\}, there is a corresponding textual responseyλy^\{\\lambda\}, which links the representation space to the language space, enabling the evaluation of the toxicity level for each trajectory\. We then use an off\-the\-shelf toxicity scorer to evaluate the toxicity of\(x,yλ\)\(x,y^\{\\lambda\}\), denoted asψλ\\psi^\{\\lambda\}\.

These trajectories provide dense, listwise supervision signals, converting sparse toxicity annotations into detailed transitions from non\-toxic to toxic representations\. Based on the trajectories and their corresponding toxicity scores, we construct a listwise representation\-level dataset𝒟h\\mathcal\{D\}\_\{h\}as follows:

𝒟h=⋃\(x,y\+,y−\)∈𝒟\(CLOSE\\displaystyle\\mathcal\{D\}\_\{h\}=\\bigcup\_\{\(x,y\_\{\+\},y\_\{\-\}\)\\in\\mathcal\{D\}\}\\Bigl\(\{\(𝒉λ,ψλ\)\}λ∈Λ\\displaystyle\\\{\(\\bm\{h\}^\{\\lambda\},\\psi^\{\\lambda\}\)\\\}\_\{\\lambda\\in\\Lambda\}\(7\)OPEN∪\{\(𝒉x,y\+,ψx,y\+\),\(𝒉x,y−,ψx,y−\)\}\)\.\\displaystyle\\cup\\\{\(\\bm\{h\}^\{x,y\_\{\+\}\},\\psi^\{x,y\_\{\+\}\}\),\(\\bm\{h\}^\{x,y\_\{\-\}\},\\psi^\{x,y\_\{\-\}\}\)\\\}\\Bigr\)\.Compared to the original dataset𝒟\\mathcal\{D\},𝒟h\\mathcal\{D\}\_\{h\}incorporates finer\-grained toxicity transitions and toxicity preference labels, enabling more accurate toxicity modeling within the representation space, which in turn improves the reward model’s guidance for representation sampling\.

### IV\-CToken\-levelReward Model Construction

Leveraging the transition dataset𝒟h\\mathcal\{D\}\_\{h\}, we aim to train a reward model to capture and quantify the toxicity risk within the representation space\. Trajectory\-level reward models are trained on complete trajectories and assign the final reward only at the last token, resulting in imprecise control signals during generation\[[65](https://arxiv.org/html/2610.41165#bib.bib16),[66](https://arxiv.org/html/2610.41165#bib.bib14)\]\.In contrast, we train a token\-level reward modelθr\\theta\_\{r\}that maps each contextualized token representation to a scalar toxicity risk score, providing finer\-grained and more accurate guidance throughout the generation process\.The overall scorer⁡\(x,y\)r\(x,y\)is decomposed into a sum over token\-wise representation scores:

r⁡\(x,y\)=∑t=1θr​\(𝒉\[t\]x,y\),r\(x,y\)=\\sum\_\{t=1\}\\theta\_\{r\}\(\\bm\{h\}^\{x,y\}\_\{\[t\]\}\),\(8\)where𝒉\[t\]x,y\\bm\{h\}^\{x,y\}\_\{\[t\]\}is the representation of thett\-th token, which implicitly depends on all preceding representations𝒉x,y\[<t\]\\bm\{h\}^\{x,y\}\_\{\[<t\]\}due to the autoregressive nature of the base LLM\. For implementation, thetoken\-levelreward modelθr\\theta\_\{r\}is instantiated as a learnable two\-layer MLP, applied on top of the final transformer layer, mapping token representations to their corresponding toxicity risk scores\.

Recalling the learning objective of the trajectory\-level reward model in Eqn\.[1](https://arxiv.org/html/2610.41165#S3.E1), which employs a pairwise preference loss to distinguish between binary toxic and non\-toxic states\. However, such a pairwise preference formulation is inherently insufficient to capture the fine\-grained dynamics of the toxicity transition manifold\. As we observe, the transition trajectories exhibit non\-uniform toxicity variation: while the steering strengthλ\\lambdaincreases linearly, the corresponding toxicity score evolves in a non\-linear manner\. To better model these complex variations, we employ a listwise preference ranking loss\[[71](https://arxiv.org/html/2610.41165#bib.bib80)\]as the learning objective for our reward model, which introduces an advanced Lambda weighting scheme that assigns listwise\-aware weights to preference pairs\. On the listwise toxicity transition dataset𝒟h\\mathcal\{D\}\_\{h\}, our training objective can be formalized as:

minθr⁡𝔼\(𝒉,ψ\)∼𝒟h​\[∑ψλi\>ψλjΔλi,λj​log⁡\(1\+eβr⋅θr​\(hλj\)eβr⋅θr​\(hλi\)\)\],\\min\_\{\\theta\_\{r\}\}\\mathbb\{E\}\_\{\(\\bm\{h\},\\psi\)\\sim\\mathcal\{D\}\_\{h\}\}\\Bigl\[\\sum\\nolimits\_\{\\psi^\{\\lambda\_\{i\}\}\>\\psi^\{\\lambda\_\{j\}\}\}\\Delta\_\{\\lambda\_\{i\},\\lambda\_\{j\}\}\\log\(1\+\\frac\{e^\{\\beta\_\{r\}\\cdot\\theta\_\{r\}\(h^\{\\lambda\_\{j\}\}\)\}\}\{e^\{\\beta\_\{r\}\\cdot\\theta\_\{r\}\(h^\{\\lambda\_\{i\}\}\)\}\}\)\\Bigr\],\(9\)whereβr\\beta\_\{r\}is a hyperparameter that scales the preference difference between toxicity transitions, andΔλi,λj=\|Gλi−Gλj\|⋅\|1D⁡\(τ⁡\(λi\)\)−1D⁡\(τ⁡\(λj\)\)\|\\Delta\_\{\\lambda\_\{i\},\\lambda\_\{j\}\}=\|G\_\{\\lambda\_\{i\}\}\-G\_\{\\lambda\_\{j\}\}\|\\cdot\|\\frac\{1\}\{D\(\\tau\(\\lambda\_\{i\}\)\)\}\-\\frac\{1\}\{D\(\\tau\(\\lambda\_\{j\}\)\)\}\|denotes the Lambda weight\[[71](https://arxiv.org/html/2610.41165#bib.bib80)\]\. Following the implementation in\[[71](https://arxiv.org/html/2610.41165#bib.bib80)\],Gλi=2ψλi−1G\_\{\\lambda\_\{i\}\}=2^\{\\psi^\{\\lambda\_\{i\}\}\}\-1serves as the gain function, andD⁡\(τ⁡\(λi\)\)=log⁡\(1\+τ⁡\(λi\)\)D\(\\tau\(\\lambda\_\{i\}\)\)=\\log\(1\+\\tau\(\\lambda\_\{i\}\)\)as the rank discount function, whereτ⁡\(λi\)\\tau\(\\lambda\_\{i\}\)denotes the rank ofhλih^\{\\lambda\_\{i\}\}in the preference ordering induced by the reward scores from modelθr\\theta\_\{r\}\. By accounting for both the toxicity score labels and the preference ordering derived from the model’s predictions, the listwise preference loss endows ourtoken\-levelreward model with intrinsic transition awareness, facilitating more precise characterization of the nuanced toxicity variation across the latent space\.

### IV\-DCoarse\-to\-Fine Representation Sampling Strategy

After developing the reward modelθr\\theta\_\{r\}, our goal is to navigate the latent space to identify a detoxified representation for each token during inference, thereby reducing the toxicity in generations from the base LLMπbase\\pi\_\{\\rm\{base\}\}\. By replacing the trajectory\-level reward modelθ\\thetawith ourtoken\-levelreward modelθr\\theta\_\{r\}, the generation process in Eqn\.[3](https://arxiv.org/html/2610.41165#S3.E3)can be written as:

π^\(y\|x\)∝πbase\(y\|x\)exp\(1β∑t=1θr\(𝒉\[t\]x,y\)\),\\hat\{\\pi\}\(y\|x\)\\propto\\pi\_\{\\text\{base\}\}\(y\|x\)\\exp\\bigl\(\\frac\{1\}\{\\beta\}\\sum\_\{t=1\}\\theta\_\{r\}\(\\bm\{h\}^\{x,y\}\_\{\[t\]\}\)\\bigl\),\(10\)where the response’s toxicity is governed by the cumulative toxicity score over its token\-level representations\. Therefore, effective detoxification requires guiding each token representation𝒉\[t\]\\bm\{h\}\_\{\[t\]\}toward regions in the latent space that yield lower toxicity risk \(i\.e\., non\-toxic regions\), thereby reducing the likelihood of toxic continuations\.

Previous representation editing methods rely on steering or gradient\-based optimization to obtain detoxified representations, which often results in suboptimal detoxification performance or prohibitive computational overhead\. To achieve effective and efficient detoxification, we leverage the reward modelθr\\theta\_\{r\}to guide a coarse\-to\-fine representation sampling strategy, capitalizing on its capacity for accurate toxicity modeling across the latent space\.

First, the coarse sampling phase aims to identify the non\-toxic region\. For each token representation𝒉\[t\]\\bm\{h\}\_\{\[t\]\}, we perform a broad steering sweep by sampling candidate token representationsℋc​o​a​r​s​e\\mathcal\{H\}\_\{coarse\}along the negative toxicity direction−𝒅\-\\bm\{d\}with a coarse step sizeλcoarse\\lambda\_\{\\rm\{coarse\}\}:

ℋcoarse=\{𝒉^\[t\]i∣𝒉\[t\]−i⋅λcoarse⋅𝒅,i∈\{0,1,…,Kcoarse\}\},\\mathcal\{H\}\_\{\\rm\{coarse\}\}=\\\{\\hat\{\\bm\{h\}\}^\{i\}\_\{\[t\]\}\\mid\\bm\{h\}\_\{\[t\]\}\-i\\cdot\\lambda\_\{\\rm\{coarse\}\}\\cdot\\bm\{d\},\\quad i\\in\\\{0,1,\\dots,K\_\{\\rm\{coarse\}\}\\\}\\\},\(11\)whereKcoarseK\_\{\\rm\{coarse\}\}denotes the maximum sampling depth\. After evaluating the toxicity scoresθr​\(𝒉^\[t\]i\)\\theta\_\{r\}\(\\hat\{\\bm\{h\}\}^\{i\}\_\{\[t\]\}\)for all candidates inℋcoarse\\mathcal\{H\}\_\{\\rm\{coarse\}\}, we identify the optimal coarse candidate𝒉^\[t\]i∗\\hat\{\\bm\{h\}\}^\{i^\{\*\}\}\_\{\[t\]\}by seeking the minimal intervention intensity that satisfies a predefined toxicity thresholdss, formulated as:

i∗=min\{i∣θr\(𝒉^\[t\]i\)≤s,i∈\{0,1,…,Kcoarse\}\}\.i^\{\*\}=\\min\\\{i\\mid\\theta\_\{r\}\(\\hat\{\\bm\{h\}\}^\{i\}\_\{\[t\]\}\)\\leq s,\\quad i\\in\\\{0,1,\\dots,K\_\{\\rm\{coarse\}\}\\\}\\\}\.\(12\)If no candidate inℋcoarse\\mathcal\{H\}\_\{\\rm\{coarse\}\}satisfies the threshold, we default to the candidate with the minimum toxicity score within the explored range\.

While the coarse phase effectively locates the non\-toxic region, the resulting representation may be subject to over\-steering, where an excessively aggressive intervention compromises the representation’s semantic integrity\. To mitigate this, we conduct a fine\-grained sampling refinement to calibrate the trade\-off between detoxification and utility\. By employing a smaller step sizeλfine<λcoarse\\lambda\_\{\\rm\{fine\}\}<\\lambda\_\{\\rm\{coarse\}\}, we explore a localized candidate setℋfine\\mathcal\{H\}\_\{\\rm\{fine\}\}designed to back off from the coarse anchor toward the original representation:

ℋfine=\{𝒉^\[t\]j∣𝒉^\[t\]i∗\+j⋅λfine⋅𝒅,j∈\{0,1,…,Kfine\}\},\\mathcal\{H\}\_\{\\rm\{fine\}\}=\\\{\\hat\{\\bm\{h\}\}^\{j\}\_\{\[t\]\}\\mid\\hat\{\\bm\{h\}\}^\{i^\{\*\}\}\_\{\[t\]\}\+j\\cdot\\lambda\_\{\\rm\{fine\}\}\\cdot\\bm\{d\},\\quad j\\in\\\{0,1,\\dots,K\_\{\\rm\{fine\}\}\\\}\\\},\(13\)whereKfineK\_\{\\rm\{fine\}\}is the maximum back\-off depth of the fine\-grained phase\. The final detoxified representation𝒉^\[t\]j∗\\hat\{\\bm\{h\}\}^\{j^\{\*\}\}\_\{\[t\]\}is subsequently determined by identifying the maximal back\-off stepj∗j^\{\*\}that preserves the required safety margin:

j∗=max\{j∣θr\(𝒉^\[t\]j\)≤s,j∈\{0,1,…,Kfine\}\}\.j^\{\*\}=\\max\\\{j\\mid\\theta\_\{r\}\(\\hat\{\\bm\{h\}\}^\{j\}\_\{\[t\]\}\)\\leq s,\\quad j\\in\\\{0,1,\\dots,K\_\{\\rm\{fine\}\}\\\}\\\}\.\(14\)If no such candidate exists, we default to the one with the minimum toxicity score\.Algorithm[1](https://arxiv.org/html/2610.41165#alg1)summarizes the coarse\-to\-fine representation sampling procedure\.The selection effectively pinpoints the optimal boundary representation, which lies as close as possible to the original representation while remaining within the safe region specified byss\. By identifying the minimally invasive intervention, our coarse\-to\-fine strategy guarantees effective detoxification while minimizing the shift that leads to linguistic degeneration\.

Compared to existing methods\[[11](https://arxiv.org/html/2610.41165#bib.bib45),[17](https://arxiv.org/html/2610.41165#bib.bib39),[18](https://arxiv.org/html/2610.41165#bib.bib10)\]that rely on heuristic static or gradient\-based dynamic interventions, our coarse\-to\-fine sampling strategy delivers improvedeffectiveness\(by mitigating the risk of becoming trapped in local optima\) andefficiency\(by eliminating the need for computationally expensive gradient computations\)\.

Algorithm 1Coarse\-to\-fine representation sampling at generation steptt\.0:Token representation𝒉\[t\]\\bm\{h\}\_\{\[t\]\}, toxicity direction𝒅\\bm\{d\}, reward modelθr\\theta\_\{r\}, thresholdss, step sizesλcoarse\\lambda\_\{\\rm coarse\}andλfine\\lambda\_\{\\rm fine\}, and sampling depthsKcoarseK\_\{\\rm coarse\}andKfineK\_\{\\rm fine\}\.

0:Detoxified token representation𝒉\[t\]∗\\bm\{h\}^\{\*\}\_\{\[t\]\}\.

1:Initialize candidate arraysℋcoarse\\mathcal\{H\}\_\{\\rm coarse\}andℋfine\\mathcal\{H\}\_\{\\rm fine\}, and score arraysScorescoarse\\mathrm\{Scores\}\_\{\\rm coarse\}andScoresfine\\mathrm\{Scores\}\_\{\\rm fine\}

2:for

i=0,…,Kcoarsei=0,\\ldots,K\_\{\\rm coarse\}do

3:

ℋcoarse​\[i\]←𝒉\[t\]−i⋅λcoarse⋅𝒅\\mathcal\{H\}\_\{\\rm coarse\}\[i\]\\leftarrow\\bm\{h\}\_\{\[t\]\}\-i\\cdot\\lambda\_\{\\rm coarse\}\\cdot\\bm\{d\}//coarse sampling in Eq\.[11](https://arxiv.org/html/2610.41165#S4.E11)

4:

Scorescoarse​\[i\]←θr​\(ℋcoarse​\[i\]\)\\mathrm\{Scores\}\_\{\\rm coarse\}\[i\]\\leftarrow\\theta\_\{r\}\(\\mathcal\{H\}\_\{\\rm coarse\}\[i\]\)
5:endfor

6:

SafeIdxcoarse←\{i∣Scorescoarse​\[i\]≤s\}\\mathrm\{SafeIdx\}\_\{\\rm coarse\}\\leftarrow\\\{i\\mid\\mathrm\{Scores\}\_\{\\rm coarse\}\[i\]\\leq s\\\}
7:if

SafeIdxcoarse≠∅\\mathrm\{SafeIdx\}\_\{\\rm coarse\}\\neq\\emptysetthen

8:

i∗←min⁡\(SafeIdxcoarse\)i^\{\*\}\\leftarrow\\min\\bigl\(\\mathrm\{SafeIdx\}\_\{\\rm coarse\}\\bigr\)//coarse selection in Eq\.[12](https://arxiv.org/html/2610.41165#S4.E12)

9:else

10:

i∗←arg​mini⁡Scorescoarse​\[i\]i^\{\*\}\\leftarrow\\operatorname\*\{arg\\,min\}\_\{i\}\\mathrm\{Scores\}\_\{\\rm coarse\}\[i\]
11:endif

12:for

j=0,…,Kfinej=0,\\ldots,K\_\{\\rm fine\}do

13:

ℋfine​\[j\]←ℋcoarse​\[i∗\]\+j⋅λfine⋅𝒅\\mathcal\{H\}\_\{\\rm fine\}\[j\]\\leftarrow\\mathcal\{H\}\_\{\\rm coarse\}\[i^\{\*\}\]\+j\\cdot\\lambda\_\{\\rm fine\}\\cdot\\bm\{d\}//fine sampling in Eq\.[13](https://arxiv.org/html/2610.41165#S4.E13)

14:

Scoresfine​\[j\]←θr​\(ℋfine​\[j\]\)\\mathrm\{Scores\}\_\{\\rm fine\}\[j\]\\leftarrow\\theta\_\{r\}\(\\mathcal\{H\}\_\{\\rm fine\}\[j\]\)
15:endfor

16:

SafeIdxfine←\{j∣Scoresfine​\[j\]≤s\}\\mathrm\{SafeIdx\}\_\{\\rm fine\}\\leftarrow\\\{j\\mid\\mathrm\{Scores\}\_\{\\rm fine\}\[j\]\\leq s\\\}
17:if

SafeIdxfine≠∅\\mathrm\{SafeIdx\}\_\{\\rm fine\}\\neq\\emptysetthen

18:

j∗←max⁡\(SafeIdxfine\)j^\{\*\}\\leftarrow\\max\\bigl\(\\mathrm\{SafeIdx\}\_\{\\rm fine\}\\bigr\)//fine selection in Eq\.[14](https://arxiv.org/html/2610.41165#S4.E14)

19:else

20:

j∗←arg​minj⁡Scoresfine​\[j\]j^\{\*\}\\leftarrow\\operatorname\*\{arg\\,min\}\_\{j\}\\mathrm\{Scores\}\_\{\\rm fine\}\[j\]
21:endif

22:return

𝒉\[t\]∗←ℋfine​\[j∗\]\\bm\{h\}^\{\*\}\_\{\[t\]\}\\leftarrow\\mathcal\{H\}\_\{\\rm fine\}\[j^\{\*\}\]

## VExperiments

In this section, we evaluate the performance of TARGRS by examining its effectiveness and efficiency in mitigating toxicity, as well as its impact on the LLM’s capabilities, including perplexity and zero\-shot prediction\. Moreover, we perform additional experiments to demonstrate the robustness and generalizability of our method\.

### V\-AExperimental Setup

Datasets and Metrics\.We follow the experimental settings of ProFS\[[12](https://arxiv.org/html/2610.41165#bib.bib3)\]\. For toxicity annotations, we adopt the pairwise toxic dataset from\[[13](https://arxiv.org/html/2610.41165#bib.bib25)\], where non\-toxic sequences are sampled from Wikitext\-2\[[72](https://arxiv.org/html/2610.41165#bib.bib26)\], and toxic counterparts are generated using PPLM\[[50](https://arxiv.org/html/2610.41165#bib.bib27)\]\.

❶*Toxicity Evaluation*\. We evaluate toxicity using the challenge subsets from RealToxicityPrompts\[[7](https://arxiv.org/html/2610.41165#bib.bib21)\], BOLD\[[23](https://arxiv.org/html/2610.41165#bib.bib70)\], and AttaQ\[[24](https://arxiv.org/html/2610.41165#bib.bib71)\]as our benchmark datasets\. For BOLD, we randomly sample 1680 prompts for evaluation due to limited computational resources\. For each benchmark, we prompt the LLMs to generate responses, which are then scored for toxicity using the Detoxify tool\[[73](https://arxiv.org/html/2610.41165#bib.bib30)\], where higher scores indicate more toxic content\. Additionally, we evaluate response fluency by calculating the perplexity using the original LLM\. We report average toxicity and perplexity of generated responses across the test set, denoted asToxicandPPLg\\rm\{PPL\}\_\{\\rm\{g\}\}, where lower values↓\{\\color\[rgb\]\{0,0,1\}\\downarrow\}for both metrics indicate better performance in terms of toxicity reduction and fluency\.

❷*Capability Evaluation\.*To evaluate the impact of detoxification on model capabilities, we first measure the model’s perplexity on the WikiText\-2\[[72](https://arxiv.org/html/2610.41165#bib.bib26)\]development split, denoted asPPLw\\rm\{PPL\}\_\{\\rm\{w\}\}\. For larger language models with zero\-shot capabilities, we further evaluate their performance on seven tasks from the EleutherAI LM Harness\[[74](https://arxiv.org/html/2610.41165#bib.bib31)\], including BoolQ\[[75](https://arxiv.org/html/2610.41165#bib.bib32)\], RTE\[[76](https://arxiv.org/html/2610.41165#bib.bib33)\], HellaSwag\[[77](https://arxiv.org/html/2610.41165#bib.bib34)\], WinoGrande\[[78](https://arxiv.org/html/2610.41165#bib.bib35)\], ARC Easy and Challenge\[[79](https://arxiv.org/html/2610.41165#bib.bib36)\], and OpenbookQA\[[80](https://arxiv.org/html/2610.41165#bib.bib37)\], by calculating the average zero\-shot accuracy, denoted asA​C​CACC\(the higher the better↑\{\\color\[rgb\]\{1,0,0\}\\uparrow\}\)\.

Models\.Our experiments span ten widely used LLMs, including both base and instruction\-tuned variants: Mistral 7B\[[81](https://arxiv.org/html/2610.41165#bib.bib28)\]and its SFT variant Mistral\-SFT 7B\[[82](https://arxiv.org/html/2610.41165#bib.bib29)\], Mistral\-7B\-Instruct, LLaMA\-7B\[[83](https://arxiv.org/html/2610.41165#bib.bib81)\]and its SFT variant LLaMA\-7B\-SFT\[[84](https://arxiv.org/html/2610.41165#bib.bib5)\], LLaMA\-13B\[[83](https://arxiv.org/html/2610.41165#bib.bib81)\], LLaMA\-30B\[[83](https://arxiv.org/html/2610.41165#bib.bib81)\], LLaMA\-2\-Chat 7B\[[83](https://arxiv.org/html/2610.41165#bib.bib81)\], Qwen2\-7B\[[85](https://arxiv.org/html/2610.41165#bib.bib79)\], and Qwen2\-7B\-Instruct\[[85](https://arxiv.org/html/2610.41165#bib.bib79)\], all evaluated with their default configurations \(e\.g\., temperature\)\. We also evaluate on LVLMs, which can be found in Sec\.[VI](https://arxiv.org/html/2610.41165#S6)\.

Baselines\.We compare our TARGRS with five state\-of\-the\-art test\-time methods: ProFS\[[12](https://arxiv.org/html/2610.41165#bib.bib3)\]\(weight editing\), Re\-Control\[[18](https://arxiv.org/html/2610.41165#bib.bib10)\]and ARGRE\[[26](https://arxiv.org/html/2610.41165#bib.bib78)\]\(representation editing\), RAD\[[54](https://arxiv.org/html/2610.41165#bib.bib61)\]and GenARM\[[53](https://arxiv.org/html/2610.41165#bib.bib6)\]\(guided decoding\)\. We also include the training\-time method DPO\[[16](https://arxiv.org/html/2610.41165#bib.bib2)\], evaluated specifically on LLaMA\-7B, as prior work \(i\.e\., ProFS\[[12](https://arxiv.org/html/2610.41165#bib.bib3)\]\) has reported its detoxification performance to be inferior to ProFS\. Additionally, we incorporate two extra representation\-editing methods, Self\-Detoxify\[[17](https://arxiv.org/html/2610.41165#bib.bib39)\]and DeStein\[[11](https://arxiv.org/html/2610.41165#bib.bib45)\], which rely on static, inference\-time interventions to mitigate toxicity\.The two static representation\-editing methods are also evaluated specifically on LLaMA\-7B, as they serve as supplementary baselines and are less competitive than the dynamic representation\-editing baseline Re\-Control\.We adopt the implementations of these methods directly from their respective GitHub repositories\.Specifically, for Re\-Control, RAD, and GenARM, we perform a hyperparameter search on a 10% validation subset of each task to select the optimal inference settings, while for ProFS and ARGRE we directly use the recommended settings reported in the original papers, with both procedures following their intended evaluation protocols\.For DPO, we adopt the implementations from\[[12](https://arxiv.org/html/2610.41165#bib.bib3)\]as in prior work\. More details of the baselines are provided in the Appendix\.

Implementation Details\.In the toxicity transition exploration, the steering strength candidate setΛ\\Lambdais set to\{0,2,4,6,8\}\\\{0,2,4,6,8\\\}\. Ourtoken\-levelreward model is implemented using a two\-layer MLP with a hidden size of 1024\. We train the model for three epochs with a learning rate of5×10−45\\times 10^\{\-4\}andβr=0\.05\\beta\_\{r\}=0\.05\. For the coarse\-to\-fine representation sampling process, both the coarse and fine sampling depthsKcoarseK\_\{\\rm\{coarse\}\}andKfineK\_\{\\rm\{fine\}\}are set to 10, with the coarse step sizeλcoarse\\lambda\_\{\\rm\{coarse\}\}set to 1\.0 and the fine\-grained step sizeλfine\\lambda\_\{\\rm\{fine\}\}set to 0\.1\. The toxicity thresholdssis set according to the percentile of toxicity scores \(predicted by our reward model\) across all tokens in the toxicity annotations\. We search over percentiles from\{5,10,20,30,40,50,60\}\\\{5,10,20,30,40,50,60\\\}using a 10% validation subset of each task to identify the optimal trade\-off between detoxification and fluency\. To ensure fair comparisons, we standardize the number of toxicity annotations at 2,000, consisting of matched toxic and non\-toxic pairs\. Experiments are conducted on a server with Intel\(R\) Xeon\(R\) Gold 6336Y CPU @ 2\.40GHz, 512GB system memory, and six NVIDIA A100 GPUs with 40GB memory\.

### V\-BEffectiveness of TARGRS

TABLE I:Toxicity evaluation results of different methods on the challenging RealToxicityPrompts dataset across 10 LLMs\. The best and second\-best results among the methods are shown inboldandunderlined, respectively\.ModelOrigProFSRe\-ControlRADGenARMARGRETARGRS\(Ours\)Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}LLaMA\-7B43\.276\.9728\.0712\.3832\.5216\.5831\.1215\.4323\.8614\.7618\.0612\.3613\.4312\.22LLaMA\-7B\-SFT46\.506\.4934\.529\.9934\.2314\.0432\.9512\.8928\.5712\.6319\.2112\.6116\.1112\.69LLaMA\-13B41\.576\.7530\.8810\.8431\.5414\.2129\.5514\.8522\.3413\.9117\.2911\.9713\.2411\.73LLaMA\-30B41\.726\.4031\.9412\.6931\.2814\.4928\.4813\.6823\.7915\.6017\.6811\.4113\.3611\.17Mistral 7B42\.797\.1430\.1418\.3433\.4417\.2227\.0715\.7421\.5216\.4218\.3013\.2412\.5113\.02Mistral\-SFT 7B34\.807\.4424\.8618\.6927\.1917\.5223\.3715\.3718\.8718\.0314\.4312\.6611\.9212\.43Qwen2\-7B42\.257\.3012\.3712\.9433\.0715\.0320\.7814\.4419\.8623\.5110\.3014\.278\.8011\.66LLaMA\-2\-Chat 7B37\.336\.3424\.8212\.1330\.6715\.1228\.9113\.2521\.5614\.1513\.6012\.2611\.0411\.22Mistral\-7B\-Instruct44\.486\.7331\.9313\.2735\.7713\.7227\.6414\.2126\.2314\.6520\.5913\.1614\.6211\.71Qwen2\-7B\-Instruct31\.439\.8713\.5115\.2326\.5616\.1917\.0817\.4915\.4322\.7211\.9714\.199\.4013\.84

![Refer to caption](https://arxiv.org/html/2610.41165v1/fig/toxicity-show-2.png)Fig\. 3:Detoxified continuations generated by LLaMA\-7B for the most toxic prompt from the challenging RealToxicityPrompts dataset\.To demonstrate the effectiveness of TARGRS, we conduct experiments on 10 LLMs across three benchmarks: RealToxicityPrompts, BOLD, and AttaQ\. The quantitative results are reported in Tab\.[I](https://arxiv.org/html/2610.41165#S5.T1), Tab\.[II](https://arxiv.org/html/2610.41165#S5.T2), and Tab\.[III](https://arxiv.org/html/2610.41165#S5.T3), respectively\. Fig\.[3](https://arxiv.org/html/2610.41165#S5.F3)provides a representative example of detoxified continuations\. To facilitate comparison, we calculate the percentage reduction in toxicity before and after detoxification, with a larger reduction indicating better performance\. From the results, we can identify that:

❶TARGRS achieves the strongest detoxification performance across all three benchmarks, reducing toxicity by 69\.38% on RealToxicityPrompts, 46\.85% on BOLD, and 43\.91% on AttaQ\. It consistently outperforms the recent state\-of\-the\-art baseline, ARGRE, which achieves reductions of 60\.36%, 35\.34%, and 34\.94%, respectively\.The superior detoxification performance underscores the efficacy of our design, which leverages listwise toxicity transitions to support transition\-aware reward model learning, thus enabling precise and effective representation sampling\.

❷Comparison with training\-time method and additional representation editing methods\.On LLaMA\-7B with RealToxicityPrompts, the training\-time method DPO reduces toxicity by 20\.73%, resulting in aToxicscore of 34\.30%\. In comparison, TARGRS achieves a substantially greater toxicity reduction of 68\.96%\. We further compare TARGRS with additional representation editing methods, Self\-Detoxify\[[17](https://arxiv.org/html/2610.41165#bib.bib39)\]and DeStein\[[11](https://arxiv.org/html/2610.41165#bib.bib45)\], which achieve toxicity rates \(Toxic\\rm\{Toxic\}\) of37\.31%37\.31\\%and36\.28%36\.28\\%, respectively\.As both approaches rely on static interventions, their effectiveness is limited, falling short of Re\-Control’s dynamic approach, and lagging even further behind our method, which benefits from more precise, transition\-aware sampling\.

❸TARGRS strikes a favorable balance between toxicity mitigation and fluency retention\. Reducing toxicity often leads to a decline in language fluency, an inherent trade\-off also observed in prior work\[[51](https://arxiv.org/html/2610.41165#bib.bib38),[86](https://arxiv.org/html/2610.41165#bib.bib40)\]\.However, despite being the most effective method, our TARGRS incurs the least fluency degradation among test\-time baselines across all three benchmarks\. Specifically, TARGRS yields average perplexity increases of only 5\.03, 0\.57, and 0\.57 on RealToxicityPrompts, BOLD, and AttaQ, respectively, compared with 5\.67, 0\.64, and 0\.92 for the second\-best method, ARGRE\.This favorable trade\-off stems from our precise representation sampling within the original LLM’s latent space, which guides model outputs toward non\-toxic regions while preserving semantic continuity and natural generation quality\.

❹TARGRS consistently demonstrates strong effectiveness across models of varying architectures, highlighting its robustness in mitigating toxicity across diverse LLMs\.On RealToxicityPrompts, TARGRS achieves a 74\.63% toxicity reduction on the Qwen series and maintains an average reduction of 68\.17% across the LLaMA series\. In contrast, ProFS achieves 63\.87% on the Qwen series but drops sharply to 28\.71% on the LLaMA models\. For instruction\-fine\-tuned LLMs \(Mistral\-7B\-Instruct, LLaMA\-2\-Chat 7B, and Qwen2\-7B\-Instruct\), TARGRS also maintains robust effectiveness, achieving 69\.22% toxicity reduction, which further demonstrates its broad applicability to both base LLMs and instruction\-tuned chat models\.We notice that Mistral\-SFT and Qwen2\-7B\-Instruct show larger gaps in relative toxicity reduction than Mistral\-7B and Qwen2\-7B, respectively\. This is mainly due to their lower initial toxicity, while their final toxicity after TARGRS remains comparable to that of the base models \(11\.92% vs\. 12\.51%, and 9\.40% vs\. 8\.80%\)\.

❺Across different LLaMA sizes, TARGRS consistently surpasses other baselines in toxicity mitigation, while its reduction gain does not vary strictly monotonically with model size\. On RealToxicityPrompts, which focuses on general toxicity in naturally occurring web\-text prompts, TARGRS maintains stable performance across LLaMA\-7B, LLaMA\-13B, and LLaMA\-30B, with toxicity reductions of 68\.96%, 68\.15%, and 67\.98%, respectively\. By comparison, ProFS suffers a performance decline, falling from 35\.13% on LLaMA\-7B to 23\.44% on LLaMA\-30B\. In contrast, performance fluctuations become more noticeable on BOLD, which evaluates toxicity in demographic\-conditioned prompts\. On BOLD, TARGRS achieves 54\.66%, 38\.92%, and 49\.41% reductions on LLaMA\-7B, LLaMA\-13B, and LLaMA\-30B, respectively\. A similar non\-monotonic trend is also observed for RAD and ARGRE, while ProFS even increases toxicity on all three LLaMA models\. These results indicate that the observed variation reflects an interplay between benchmark\-specific characteristics and model\-specific behavior, rather than a simple monotonic relationship with model size\.

TABLE II:Toxicity evaluation results of different methods on the BOLD dataset across 10 LLMs\. The best and second\-best results among the methods are shown inboldandunderlined, respectively\.ModelOrigProFSRe\-ControlRADGenARMARGRETARGRS\(Ours\)Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}LLaMA\-7B8\.588\.6811\.5416\.687\.9813\.865\.8611\.896\.9811\.414\.799\.263\.899\.21LLaMA\-7B\-SFT9\.125\.4810\.8410\.938\.7910\.778\.698\.407\.018\.255\.236\.384\.086\.26LLaMA\-13B8\.178\.138\.6510\.427\.5410\.286\.299\.426\.079\.845\.688\.504\.998\.47LLaMA\-30B8\.547\.7510\.3517\.267\.989\.956\.288\.986\.169\.315\.428\.734\.328\.51Mistral 7B7\.256\.678\.4510\.766\.199\.695\.8410\.596\.7111\.345\.658\.013\.598\.07Mistral\-SFT 7B6\.457\.039\.369\.975\.6310\.085\.119\.404\.959\.674\.407\.863\.787\.69Qwen2\-7B6\.046\.883\.568\.365\.718\.323\.968\.863\.9110\.713\.467\.302\.697\.14LLaMA\-2\-Chat 7B5\.974\.656\.236\.805\.638\.845\.036\.635\.066\.493\.725\.223\.315\.25Mistral\-7B\-Instruct6\.486\.155\.456\.955\.379\.285\.468\.515\.327\.265\.066\.384\.236\.31Qwen2\-7B\-Instruct5\.535\.923\.217\.595\.078\.973\.587\.914\.2110\.383\.136\.073\.126\.10

TABLE III:Toxicity evaluation results of different methods on the AttaQ dataset across three instruction\-fine\-tuned LLMs\. The best and second\-best results among the methods are shown inboldandunderlined, respectively\.MethodLLaMA\-2\-Chat 7BMistral\-7B\-InstructQwen2\-7B\-InstructToxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Orig18\.845\.8622\.187\.2414\.635\.44ProFS17\.918\.3416\.538\.899\.787\.68Re\-Control17\.759\.0619\.6410\.2011\.049\.49RAD16\.488\.3516\.959\.3910\.617\.43GenARM15\.677\.8415\.508\.3511\.677\.08ARGRE12\.446\.8814\.567\.579\.296\.84TARGRS\(Ours\)11\.536\.7413\.297\.486\.906\.02

### V\-CEfficiency of TARGRS

We evaluate the inference efficiency of detoxification methods by measuring the time required to generate 128 tokens per prompt\. To highlight the efficiency of our approach, we report results on LLaMA\-30B, which is the largest model in our experiments and serves as the primary bottleneck for inference speed\. As shown in Tab\.[IV](https://arxiv.org/html/2610.41165#S5.T4), TARGRS demonstrates strong inference efficiency, running nearly as fast as the original LLM \(8\.24s vs\. 8\.14s\), indicating minimal overhead from the coarse\-to\-fine representation sampling process\. In contrast, Re\-Control incurs considerable latency \(58\.69s\) due to the tedious gradient\-based updates \(e\.g\., 200\) during inference\. RAD requires an independent small reward model to guide decoding and thus takes 16\.63s\. GenARM is also notably slower \(18\.94s\), as its reward model introduces extra computation through LoRA modules added to each layer of the base LLM, whereas our reward model adopts a lightweight 2\-layer MLP, improving the efficiency of reward calculation\. Compared to ARGRE \(9\.30s\), which relies on a few gradient\-based refinement steps, TARGRS achieves faster inference by leveraging direct sampling\. Unlike gradient\-based methods that require multiple forward and backward passes to iteratively compute gradients and update representations, our TARGRS only involves representation sampling \(via vector calculation\) and single forward passes for toxicity scoring, thus avoiding heavy computational overhead during inference\. ProFS achieves the fastest inference by directly editing model weights, but its detoxification performance is limited, with a toxicity score of 31\.94% on the challenging RealToxicityPrompts, much higher than the 13\.36% of TARGRS\. These results demonstrate that TARGRS achieves superior inference efficiency than other effective test\-time methods, with a 11\.40% reduction in inference time compared to the best\-performing baseline ARGRE , and a 56\.49% reduction compared to the second\-best GenARM\.

TABLE IV:Efficiency of test\-time methods, measured by the time \(seconds\) to generate 128 tokens on LLaMA\-30B\.MethodOrigProFSRe\-ControlRADGenARMARGREOursTime \(s\)↓\\downarrow8\.148\.1858\.6916\.6318\.949\.308\.24

### V\-DCapability Impact of TARGRS

An ideal detoxification method should ensure that the LLM retains its state\-of\-the\-art capabilities without any degradation\. We therefore analyze changes in perplexity and zero\-shot performance before and after detoxification\. The evaluation results are presented in Tab\.[V](https://arxiv.org/html/2610.41165#S5.T5)\.

❶For WikiText\-2 perplexity, our TARGRS results in only a slight increase inPPLw\\rm\{PPL\}\_\{\\rm\{w\}\}, averaging 0\.07, which indicates minimal degradation in language performance\. This increase is the smallest among test\-time baselines, with 0\.85 for ARGRE, 1\.10 for Re\-Control, 1\.52 for GenARM, and 2\.78 for ProFS\. The minimal degradation can be attributed to the accurate toxicity risk modeling, which effectively guides the representation sampling process with only slight intervention, as the text in WikiText\-2 is predominantly non\-toxic and safe\. ❷ For zero\-shot capabilities, TARGRS preserves or even slightly improves the accuracy of the original LLM, with an average increase of 0\.06% inA​C​CACC\. This outcome arises from reward\-guided representation sampling, which primarily adjusts toxic representations while preserving the rest\. Similarly, ARGRE also achieves a small accuracy improvement of 0\.05%\. In contrast, other test\-time baselines show varying degrees of degradation, with accuracy drops of 0\.07% for RAD, 0\.09% for GenARM, 0\.12% for Re\-Control, and a larger drop of 2\.09% for ProFS, which may suffer from its aggressive weight editing strategy\. Overall, our TARGRS effectively retains the core capabilities of the original model with negligible impact\.

TABLE V:Capability evaluation results of different methods across 10 LLMs\. The best and second\-best results among the methods are shown inboldandunderlined, respectively\.ModelOrigProFSRe\-ControlRADGenARMARGRETARGRS\(Ours\)PPLw↓\\rm\{PPL\}\_\{\\rm\{w\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}A​C​C↑ACC\{\\color\[rgb\]\{1,0,0\}\\uparrow\}PPLw↓\\rm\{PPL\}\_\{\\rm\{w\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}A​C​C↑ACC\{\\color\[rgb\]\{1,0,0\}\\uparrow\}PPLw↓\\rm\{PPL\}\_\{\\rm\{w\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}A​C​C↑ACC\{\\color\[rgb\]\{1,0,0\}\\uparrow\}PPLw↓\\rm\{PPL\}\_\{\\rm\{w\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}A​C​C↑ACC\{\\color\[rgb\]\{1,0,0\}\\uparrow\}PPLw↓\\rm\{PPL\}\_\{\\rm\{w\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}A​C​C↑ACC\{\\color\[rgb\]\{1,0,0\}\\uparrow\}PPLw↓\\rm\{PPL\}\_\{\\rm\{w\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}A​C​C↑ACC\{\\color\[rgb\]\{1,0,0\}\\uparrow\}PPLw↓\\rm\{PPL\}\_\{\\rm\{w\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}A​C​C↑ACC\{\\color\[rgb\]\{1,0,0\}\\uparrow\}LLaMA\-7B7\.1460\.0211\.4556\.197\.6959\.98\-60\.018\.5659\.947\.5760\.017\.1860\.01LLaMA\-7B\-SFT8\.1858\.8112\.8255\.609\.0358\.78\-58\.759\.7858\.648\.9958\.938\.2058\.98LLaMA\-13B6\.4862\.637\.8057\.967\.1962\.33\-62\.557\.4562\.466\.8862\.676\.6562\.74LLaMA\-30B5\.3665\.456\.1458\.845\.8365\.24\-65\.215\.9665\.395\.7065\.435\.3665\.54Mistral 7B7\.2164\.358\.9763\.528\.4364\.38\-64\.358\.4063\.898\.2064\.417\.2664\.36Mistral\-SFT 7B7\.8663\.639\.8463\.358\.6663\.61\-63\.628\.8163\.868\.5563\.907\.9163\.76Qwen2\-7B10\.0364\.5815\.4664\.0612\.4164\.60\-64\.5312\.3563\.6411\.4664\.6210\.0564\.59LLaMA\-2\-Chat 7B9\.1960\.7210\.5759\.8110\.7260\.56\-60\.7212\.1960\.7910\.5560\.739\.1960\.74Mistral\-7B\-Instruct9\.4862\.4410\.4062\.3410\.2762\.15\-62\.4110\.7163\.549\.9962\.439\.6462\.47Qwen2\-7B\-Instruct10\.4265\.9715\.7166\.0312\.1665\.75\-65\.7512\.3865\.5211\.9965\.9510\.6165\.98

### V\-EAblation Studies

To better understand TARGRS, we conduct ablation studies on LLaMA\-7B on the challenging RealToxicityPrompts dataset, focusing on three key components: number of toxicity annotations, toxicity transition trajectories, and toxicity threshold\.

❶Number of Toxicity Annotations\.To assess TARGRS’s effectiveness in low\-data scenarios, we reduce the number of toxicity annotations from 2000 \(as used in the main experiment\) to 100\. As shown in Fig\.[4](https://arxiv.org/html/2610.41165#S5.F4), TARGRS consistently outperforms all baselines across annotation scales, achieving substantial toxicity reduction even with very limited data\. With only 100 annotations, TARGRS reduces toxicity from 43\.27% to 17\.13%, outperforming baselines trained with 2000 annotations \(e\.g\., TARGRS at 18\.06% and GenARM at 23\.86%\) and approaching our best result of 13\.43% with full annotations\. These results highlight the effectiveness of toxicity transition exploration and demonstrate the strong data efficiency and practical applicability of TARGRS\.

Fig\. 4:Toxicity scores across varying annotation sizes\. TARGRS presents strong data efficiency, consistently outperforming baselines even with as few as 100 annotations\.Fig\. 5:Impact of toxicity transition trajectory on TARGRS’s detoxification performance\. Performance improves with the number of transitions, surpassing the baselines at 3\.❷Toxicity Transition Trajectories\.To investigate the impact of toxicity transition trajectory, we vary the steering strength candidate setΛ\\Lambda, which is set as\{0,2,4,6,8\}\\\{0,2,4,6,8\\\}in the main experiment\. Specifically, we setΛ\\Lambdaas\{0\},\{0,2\},\{0,2,4\},\{0,2,4,6\},\{0,2,4,6,8\}\\\{0\\\},\\\{0,2\\\},\\\{0,2,4\\\},\\\{0,2,4,6\\\},\\\{0,2,4,6,8\\\}respectively, corresponding to a range of one to five transition trajectories\. We also include the case whereΛ=∅\\Lambda=\\emptyset, where no transitions are explored \(i\.e\., 0 transitions\) and the reward model is trained solely on the trajectories of raw annotations\. From the results shown in Fig\.[5](https://arxiv.org/html/2610.41165#S5.F5), we observe the following: \(1\) when no transition is explored, GenARM achieves a marginally better performance than TARGRS, with a 1\.9% lower toxicity score\. ARGRE reaches 26\.64%Toxic\\rm\{Toxic\}without toxicity transition exploration, while TARGRS achieves 25\.76%\. When setΛ\\Lambdaas\{0,2,4\}\\\{0,2,4\\\}, TARGRS surpass the leading methods including ARGRE and GenARM\. \(2\) Increasing the depth of the toxicity transition exploration improves detoxification performance, although the improvement gradually plateaus\. Specifically, raising transition exploration from 0 to 4 reducesToxicby 12\.12%, while further increasing it to 5 yields only a marginal gain of 0\.21%\. These results demonstrate the effectiveness of toxicity transition exploration, which provides denser and listwise supervision beyond sparse annotations and guides representation sampling toward non\-toxic regions\.

To better understand the transition trajectories from non\-toxic to toxic states, we further analyze their corresponding toxicity scores, with results presented in Tab\.[VI](https://arxiv.org/html/2610.41165#S5.T6)\. We also provide examples of textual responses generated during transition exploration in Tab\.[VII](https://arxiv.org/html/2610.41165#S5.T7)\. We observe that as the model transitions from the original non\-toxic state to the steered toxic state, the toxicity scores rise gradually at first and then surge sharply\. This pattern demonstrates that the representation shifts from a non\-toxic region to a toxic region along the interpolation trajectory in a non\-linear manner\. Specifically, the toxicity score increases only from 0\.55% to 10\.78% fromλ=0\\lambda=0toλ=4\\lambda=4, but jumps drastically from 10\.78% to 65\.09% betweenλ=4\\lambda=4andλ=8\\lambda=8\. The non\-linear toxicity variation also explains why TARGRS achieves substantial performance gains and outperforms competing methods whenΛ\\Lambdais set to\{0,2,4\}\\\{0,2,4\\\}, as exploration up toλ=4\\lambda=4can effectively elicit toxic behavior\.

TABLE VI:Average toxicity scores of the explored transition trajectories on LLaMA\-7B\.Strengthλ=0\\lambda=0λ=2\\lambda=2λ=4\\lambda=4λ=6\\lambda=6λ=8\\lambda=8Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}0\.551\.9510\.7833\.9465\.09

TABLE VII:Textual responses generated during transition exploration on LLaMA\-7B, with the prompt “Despite the ruthlessness and”\.StrengthTextual responseToxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}λ=0\\lambda=0brutality of the Nazi regime, the German people were not responsible for the crimes of the Third Reich\.0\.48λ=2\\lambda=2brutality of the Nazi regime, it was not without its humorous side\. Hitler’s sense of humour was notorious\. He was fond of0\.55λ=4\\lambda=4brutality of gangster life, gangster flicks have always been big box\-office fodder\. Gangster flicks have flickered9\.48λ=6\\lambda=6brutality of Hitler’s regime, Hitler’s murderous brutality was nothing like that of Stalin’s murderous brutality\. Hitler murdered51\.49λ=8\\lambda=8nastiness you guys f\*\*king hate me\! f\*\*k yeah\! f\*\*k yeah\! f\*\*k yeah\! f\*\*k yeah\! f\*\*k yeah\! f\*\*k99\.80

❸Toxicity Threshold\.We vary the percentile\-based toxicity thresholdssin\{5,10,20,30,40,50\}\\\{5,10,20,30,40,50\\\}, and the results are shown in Tab\.[VIII](https://arxiv.org/html/2610.41165#S5.T8)\.Across all three benchmarks, we observe a consistent trade\-off between toxicity mitigation and generation fluency:stricter thresholds yield greater toxicity reduction but impair generation quality, whereas more relaxed thresholds lead to slightly increased toxicity with better fluency\.On RealToxicityPrompts,as the toxicity threshold decreases from the 50th to 5th percentile, the toxicity score drops from 16\.09% to 6\.34%, indicating that a lower threshold imposes stricter constraints on toxic representations and effectively suppresses harmful content during generation\. However, lower thresholds also lead to higher generation perplexity, which rises from 11\.59 to 16\.87\.Similar trends are also observed on BOLD and AttaQ: on BOLD, the toxicity score decreases from 4\.73% to 3\.81% while generation perplexity increases from 9\.05 to 9\.42; on AttaQ, the toxicity score decreases from 12\.65% to 10\.84% while generation perplexity increases from 6\.43 to 7\.06\. These results suggest that the effect of the toxicity threshold is consistent across different prompt distributions, and that our approach achieves favorable results compared with baseline methods on most threshold settings\.From a deployment perspective, a small held\-out validation subset can be used to select the desired trade\-off between toxicity mitigation and generation fluency\. When validation data are unavailable, intermediate percentile thresholds, such as the 20th–30th percentiles, provide reasonable defaults with relatively balanced toxicity and fluency results across the three benchmarks\. The threshold can be further adjusted toward a stricter or more relaxed setting according to deployment\-specific safety and fluency requirements, without retraining the reward model\.

TABLE VIII:Toxicity results under different toxicity thresholds \(5th\-50th percentiles\) on RealToxicityPrompts and BOLD with LLaMA\-7B, and on AttaQ with LLaMA\-2\-Chat 7B\.ThresholdRTPBOLDAttaQToxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}5th6\.3416\.873\.819\.4210\.847\.0610th7\.2216\.163\.899\.2111\.106\.9220th8\.8314\.493\.959\.1911\.536\.7430th10\.0813\.844\.189\.1311\.946\.6840th13\.4312\.224\.469\.1012\.166\.5850th16\.0911\.594\.739\.0512\.656\.43

TABLE IX:Toxicity results under pairwise and listwise losses across different models and benchmarks\.DatasetLLaMA\-7BQwen2\-7BPairwiseListwisePairwiseListwiseToxic↓\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm g\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm g\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm g\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm g\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}RTP18\.6512\.3413\.4312\.2210\.7412\.758\.8011\.66BOLD4\.939\.193\.899\.213\.347\.382\.697\.14

❹Listwise Loss\.To assess the contribution of the listwise preference ranking loss in reward model learning, we replace it with a pairwise loss on the same transition data\. Specifically, we first sort the samples by toxicity score and then construct pairwise preference signals from adjacent samples in the sorted order\.As shown in Tab\.[IX](https://arxiv.org/html/2610.41165#S5.T9), the listwise loss yields lower toxicity scores than the pairwise loss for LLaMA\-7B and Qwen2\-7B on RealToxicityPrompts and BOLD\. Specifically, for LLaMA\-7B, the listwise loss reduces the toxicity score from 18\.65% to 13\.43% on RealToxicityPrompts and from 4\.93% to 3\.89% on BOLD\. For Qwen2\-7B, it reduces the toxicity score from 10\.74% to 8\.80% on RealToxicityPrompts and from 3\.34% to 2\.69% on BOLD\. Meanwhile, generation perplexity remains comparable or improves in these settings\.These results demonstrate the effectiveness of our listwise loss design, which better captures the fine\-grained preference structure in the toxicity trajectories and thereby provides more accurate guidance for effective detoxification\.

### V\-FInvestigations

❶SAE\-based Direction Extraction\. Sparse autoencoders \(SAEs\) are increasingly utilized for concept steering, as they factorize hidden representations into sparse and localized latent features\[[87](https://arxiv.org/html/2610.41165#bib.bib88),[88](https://arxiv.org/html/2610.41165#bib.bib89)\]\. Therefore, we conduct an additional experiment on LLaMA\-7B, replacing PCA with SAE for toxicity direction extraction while keeping the rest of the TARGRS pipeline unchanged\. Specifically, we leverage the SAELens framework\[[89](https://arxiv.org/html/2610.41165#bib.bib87)\]to train a standard SAE on the last\-layer representations of LLaMA\-7B, using the same toxicity annotations as in our main experiments and an expansion factor of 8\. To identify the toxicity direction, we rank SAE features according to the activation gap between toxic and non\-toxic samples, retain the top\-10 features with the largest gaps, and zero out the remaining activations\. We then decode the resulting sparse latent vector back into the original representation space to obtain the SAE\-derived toxicity direction\. We evaluate this direction on the challenging RealToxicityPrompts dataset\. SAE\-based direction extraction achieves a toxicity score of 13\.12% and aP​P​LwPPL\_\{\\rm\{w\}\}of 15\.61, compared to 13\.43% and 12\.22 under the original PCA\-based setup, respectively\. These results suggest that SAE yields a less favorable trade\-off between toxicity mitigation and fluency retention\. Although SAE slightly improves toxicity reduction, it substantially increasesP​P​LwPPL\_\{\\rm\{w\}\}\. One possible explanation is that the SAE\-derived direction, obtained through a sparser feature selection procedure, may induce a less smooth intervention, thereby causing a larger degradation in fluency\.

❷Evaluation with Perspective API\.In addition to employing Detoxify for toxicity evaluation in our main experiments, we conduct a cross\-metric validation via Perspective API\[[90](https://arxiv.org/html/2610.41165#bib.bib90)\]on the RealToxicityPrompts outputs of LLaMA\-7B to demonstrate the reliability of our evaluation\. The results are presented in Tab\.[X](https://arxiv.org/html/2610.41165#S5.T10)\. Across the seven methods, the average difference in toxicity scores between Detoxify and Perspective API is only 2\.32%\. Furthermore, we assess the correlation between toxicity score values and rankings across the two evaluation methods\. The Pearson correlation coefficient is 0\.9960 with a p\-value of1\.91×10−61\.91\\times 10^\{\-6\}, and Kendall’s tau is 1\.0000 with a p\-value of3\.97×10−43\.97\\times 10^\{\-4\}\. These results indicate a strong positive correlation between the toxicity results from both evaluation methods, demonstrating that our toxicity evaluation is reliable and convincing\.

TABLE X:Toxicity scores \(↓\{\\color\[rgb\]\{0,0,1\}\\downarrow\}\) on RealToxicityPrompts for LLaMA\-7B using Detoxify and Perspective API\.MetricOrigProFSRe\-ControlRADGenARMARGREOursDetoxify43\.2728\.0732\.5231\.1223\.8618\.0613\.43Perspective38\.0326\.1430\.1529\.4825\.1419\.3515\.90

❸Generalizability on Broader Safety Aspects\.To assess the generalizability of TARGRS, we extend our evaluation beyond detoxification to broader safety aspects\. We adopt the harmful queries from HarmBench\[[91](https://arxiv.org/html/2610.41165#bib.bib93)\]and Do\-Not\-Answer\[[92](https://arxiv.org/html/2610.41165#bib.bib94)\]to evaluate Mistral\-7B\-Instruct, and employ attack success rate \(ASR\) to measure safety \(lower is better\)\. For HarmBench, we utilize GPT\-4\[[93](https://arxiv.org/html/2610.41165#bib.bib95)\]to classify harmful and harmless responses; for Do\-Not\-Answer, we use its Longformer\-based refusal classifier\.We train a harm\-oriented reward model using the 128 pairwise benign\-harmful annotations from\[[94](https://arxiv.org/html/2610.41165#bib.bib97)\]and use the same trained model for both benchmarks\.We employ GPT\-4 with a five\-level harm scale\[[95](https://arxiv.org/html/2610.41165#bib.bib96)\]to score responses along transition trajectories, which are then mapped to harm scores from 0 to 1\. Details are provided in the Appendix\. The results are shown in Tab\.[XI](https://arxiv.org/html/2610.41165#S5.T11)and Tab\.[XII](https://arxiv.org/html/2610.41165#S5.T12)\. TARGRS achieves the lowest overall ASR on both benchmarks, reducing it from 47\.19% to 23\.75% on HarmBench and from 27\.05% to 16\.83% on Do\-Not\-Answer\.Despite these improvements, residual ASR remains high for General Harm \(50\.00%\) and Illegal Activities \(41\.51%\) on HarmBench, and Malicious Uses \(39\.09%\) on Do\-Not\-Answer\.

We further evaluate the original toxicity reward model, which reduces overall ASR to 27\.81% on HarmBench and 18\.00% on Do\-Not\-Answer\. Compared with the harm\-oriented reward model, it achieves lower ASR on harassment \(14\.29% vs\. 38\.10%\), discrimination\-related harms \(6\.82% vs\. 11\.36%\), and human–chatbot interaction harms \(7\.69% vs\. 12\.82%\), but higher ASR on malicious uses and misinformation\-related categories\. This variation may reflect differences between toxicity and other harms, since harmful instructions or misleading claims need not contain toxic language\. These results support applying TARGRS to the evaluated safety tasks, while the lower overall ASR of the harm\-oriented reward model suggests that target\-relevant training data can further improve reward guidance\.

TABLE XI:HarmBench ASR \(%,↓\{\\color\[rgb\]\{0,0,1\}\\downarrow\}\) on Mistral\-7B\-Instruct\.Ours uses the harm\-oriented reward model; Ours \(toxicity RM\) directly reuses the original toxicity reward model\.MethodOverallChem\./Bio\.CopyrightCybercrimeHarass\.HarmfulIllegalMisinfo\.Orig47\.1957\.141\.2534\.6280\.9577\.7883\.0261\.11ProFS45\.3154\.762\.5030\.7785\.7177\.7877\.3657\.41GenARM35\.6342\.861\.2528\.8561\.9055\.5662\.2644\.44ARGRE30\.6335\.710\.0025\.0042\.8661\.1152\.8340\.74Ours23\.7521\.430\.0017\.3138\.1050\.0041\.5135\.19Ours \(toxicity RM\)27\.8130\.950\.0025\.0014\.2955\.5649\.0644\.44

TABLE XII:Do\-Not\-Answer ASR \(%,↓\{\\color\[rgb\]\{0,0,1\}\\downarrow\}\) on Mistral\-7B\-Instruct\.Ours uses the harm\-oriented reward model; Ours \(toxicity RM\) reuses the original toxicity reward model\.MethodOverallDiscrim\.Human–ChatbotInfo\. HazardsMaliciousMisinfo\.Orig27\.0519\.3224\.7910\.0858\.0216\.13ProFS24\.2817\.0522\.228\.8752\.2614\.84GenARM21\.6215\.9118\.808\.8745\.6812\.90ARGRE19\.4914\.2017\.097\.6642\.3910\.32Ours16\.8311\.3612\.826\.0539\.098\.39Ours \(toxicity RM\)18\.006\.827\.698\.0645\.2711\.61

❹Evaluation on AlpacaEval under Clean and Toxic\-Wrapped Prompts\.Beyond perplexity and zero\-shot accuracy, we further assess whether TARGRS preserves instruction\-following capability using AlpacaEval\[[96](https://arxiv.org/html/2610.41165#bib.bib92)\]on Mistral\-7B\-Instruct\. We consider two prompt settings:*clean instructions*and*toxic\-wrapped instructions*\. For the toxic\-wrapped setting, we wrap the core task content with abusive text using one of five randomly assigned templates \(prefix, suffix, or sandwich\), to test whether the model can still follow the core instruction under toxic prompts\. Template details are provided in the Appendix\. We employ GPT\-4 to judge the Win Rate against the reference outputs of Alpaca\-7B, and report generation toxicity score \(via Detoxify\)\.As shown in Tab\.[XIII](https://arxiv.org/html/2610.41165#S5.T13), TARGRS achieves win rates of 65\.84% and 57\.89% under clean and toxic\-wrapped prompts, respectively, compared with 66\.09% and 56\.52% for the original model, while reducing toxicity in both settings\. Head\-to\-head win rates against the original model are 52\.05% and 51\.93%, respectively, indicating comparable instruction\-following performance\.

We further analyze 46 creative\-writing instructions from the same evaluation, covering stories, poetry and lyrics, scripts and role\-based dialogue, and humor\. As shown in Tab\.[XIII](https://arxiv.org/html/2610.41165#S5.T13), TARGRS achieves win rates of 60\.87% and 50\.00% under clean and toxic\-wrapped prompts, respectively, compared with 54\.35% and 47\.83% for the original model, while reducing toxicity in both settings\.

We complement the automatic evaluation with a human assessment of the same 46 creative\-writing instructions and 46 randomly sampled general instructions under both prompt settings\. Three human evaluators independently select the higher\-quality response from anonymized pairs of TARGRS and original\-model outputs, with final preferences determined by majority vote\. Human win rates are 47\.83% and 56\.52% for instruction following, and 52\.17% and 54\.35% for creative writing, under clean and toxic\-wrapped prompts, respectively, suggesting comparable performance to the original model on both tasks\.

TABLE XIII:AlpacaEval results \(%\) on Mistral\-7B\-Instruct\. Win rate is judged by GPT\-4 against the AlpacaEval reference outputs\. Toxic\-wrapped prompts preserve the core instruction while adding abusive language around it\.SettingOrigOursWin Rate↑\{\\color\[rgb\]\{1,0,0\}\\uparrow\}Toxic↓\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Win Rate↑\{\\color\[rgb\]\{1,0,0\}\\uparrow\}Toxic↓\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Full setClean66\.090\.5065\.840\.39Toxic\-wrapped56\.520\.5957\.890\.40CreativewritingClean54\.352\.8260\.871\.83Toxic\-wrapped47\.833\.4150\.001\.79

❺Distribution Gap across Toxicity Types\.To examine the distribution gap between the synthetic training annotations and the evaluation benchmarks, we use DeepSeek\-V4\-Flash to identify toxicity patterns in individual samples, assign them descriptive labels, and consolidate semantically related labels into broader types\. This analysis identifies five broad types in the training annotations: profanity and obscenity, insult and harassment, identity\-based hate and discrimination, threat and violence, and sexually explicit content\. RealToxicityPrompts spans these types in naturally occurring web\-text contexts and additionally includes patterns such as body shaming and self\-harm\. BOLD comprises prompts concerning gender, political ideology, profession, race, and religion, whereas AttaQ comprises harmful questions involving deception, discrimination, harmful information, substance abuse, explicit content, personally identifiable information, and violence\. Compared with the training annotations, these benchmarks differ in their data sources, toxicity types, and prompt contexts\. Using a fixed reward model for each LLM, TARGRS consistently reduces toxicity across these benchmarks, supporting its generalizability\.

## VIExtension to LVLMs

LVLMs are increasingly deployed in real\-world applications such as visual question answering and multimodal dialogue, making their detoxification crucial to prevent the propagation of harmful information\. Therefore, we further investigate the generalizability and effectiveness of our method in mitigating toxic content generation in LVLMs\.

### VI\-AMethodology

Beyond text\-only LLMs, TARGRS can be seamlessly extended to Large Vision\-Language Models \(LVLMs\) by intervening within their language decoder representation space\. We first explain how the shared reliance of LLMs and LVLMs on language decoders enables this extension and then introduce the transition\-aware reward\-guided representation sampling process for LVLMs\.

#### VI\-A1Shared Language Decoder Architecture

LVLMs\[[97](https://arxiv.org/html/2610.41165#bib.bib83),[98](https://arxiv.org/html/2610.41165#bib.bib84),[99](https://arxiv.org/html/2610.41165#bib.bib85)\]typically consist of two core components: a vision encoder\[[100](https://arxiv.org/html/2610.41165#bib.bib86)\]that extracts visual features from input images, and a language decoder \(i\.e\., an LLM backbone\)\[[1](https://arxiv.org/html/2610.41165#bib.bib17)\]that serves as the core engine for generating text responses based on the fused vision\-text features\. The shared reliance of LLMs and LVLMs on language decoders for text generation provides the architectural foundation for extending TARGRS to LVLMs\.

For an image\-grounded text generation task, given an input imagevvand a textual queryxx, the LVLMπLVLM​\(y\|v,x\)\\pi\_\{\\rm\{LVLM\}\}\(y\|v,x\)autoregressively generates a contextually coherent textual response sequenceyy\. Specifically, each tokenyty\_\{t\}is sampled from the probability distribution as follows:

yt∼πLVLM\(⋅∣v,x,y<t\)=softmax\(ϕ\(𝒉\[t\]\)\),y\_\{t\}\\sim\\pi\_\{\\rm LVLM\}\(\\cdot\\mid v,x,y\_\{<t\}\)=\\operatorname\{softmax\}\\\!\\left\(\\phi\(\\bm\{h\}\_\{\[t\]\}\)\\right\),\(15\)whereϕ\\phirepresents the language head, which maps the token representation𝒉\[t\]\\bm\{h\}\_\{\[t\]\}at timestepttto a probability distribution over the vocabulary\. The token representation𝒉\[t\]\\bm\{h\}\_\{\[t\]\}is computed by the language decoder based on both the encoded visual features and the previous tokens generated in the autoregressive process\. Critically, the toxicity of the generated text is determined by the properties of these token representations in the language decoder space\.

Therefore, for LVLM detoxification, our objective is to steer the decoder’s token representation𝒉\[t\]\\bm\{h\}\_\{\[t\]\}towards a non\-toxic, safe region of the latent space, similar to how we approach detoxification for LLMs\. Our detoxification operations are confined entirely to the language decoder, which is the universal core component responsible for text generation in both LLMs and LVLMs\. This decoder\-centric design allows our method to operate independently of the visual encoder and the specifics of the visual modality, enabling natural and unmodified adaptation to multimodal scenarios\.

#### VI\-A2Transition\-Aware Detoxification

To explore toxicity transition trajectories, we employ pairwise annotations of toxic and non\-toxic textual responses, paired with a blank image as visual input to the LVLM\. The blank image ensures that no meaningful visual context interferes with the analysis\. We then compute the toxicity direction𝒅\\bm\{d\}by contrasting the corresponding toxic and non\-toxic representations using PCA\. By steering representations along this direction, we trace trajectories from non\-toxic to toxic states, thereby constructing the representation\-level listwise dataset𝒟h\\mathcal\{D\}\_\{h\}used to train the reward model\. Under the training objective in Eqn\.[9](https://arxiv.org/html/2610.41165#S4.E9), we learn a transition‑aware reward model that accurately captures and quantifies toxicity risk in the representation space, and this reward model operates within the language decoder space that is unified across different modalities\. During inference of the LVLM, we deploy the learned reward model to guide the coarse\-to\-fine representation sampling procedure detailed in Sec\.[IV\-D](https://arxiv.org/html/2610.41165#S4.SS4), yielding detoxified representations and thus enabling effective and efficient toxicity mitigation for LVLMs\. Notably, this entire process requires no adaptation to the visual encoder or multimodal fusion mechanism, as our method’s focus on the language decoder space inherently aligns with the text generation pipeline of LVLMs regardless of the input image modality\.

### VI\-BExperiments

TABLE XIV:Toxicity evaluation results on the ToViLaG dataset across three LVLMs\. The best and second\-best results are shown inboldandunderlined, respectively\.MethodQwen\-VL\-ChatLLaVA\-1\.5Llava\-next\-8BToxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Toxic↓\\text\{Toxic\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}PPLg↓\\rm\{PPL\}\_\{\\rm\{g\}\}\{\\color\[rgb\]\{0,0,1\}\\downarrow\}Orig44\.8819\.1340\.139\.1318\.2513\.67ProFS31\.5125\.1827\.4613\.8214\.6316\.41Re\-Control35\.6230\.2030\.9717\.7415\.9418\.29GenARM24\.3627\.6621\.0915\.6813\.5918\.36ARGRE18\.0124\.9515\.1113\.049\.3315\.35Ours15\.4523\.2513\.2212\.317\.2915\.03

#### VI\-B1Evaluation Setup

To demonstrate the generalizability of our method to LVLMs, we evaluate toxicity using 1,000 toxic images \(e\.g\., pornography, violence, bloody scenes\) from the ToViLaG dataset\[[25](https://arxiv.org/html/2610.41165#bib.bib82)\], and prompt LVLMs with “Please describe the image content\.” to generate responses\. Following the same evaluation protocol as for LLMs, we assess the toxicity \(Toxic\\rm\{Toxic\}\) and fluency \(PPLg\\rm\{PPL\}\_\{\\rm\{g\}\}\) of the generated responses\. We conduct experiments on three representative LVLMs: Qwen\-VL\-chat\[[97](https://arxiv.org/html/2610.41165#bib.bib83)\], LLaVA\-1\.5\[[98](https://arxiv.org/html/2610.41165#bib.bib84)\], and Llava\-next\-8B\[[99](https://arxiv.org/html/2610.41165#bib.bib85)\]\.We compare TARGRS with ProFS, Re\-Control, GenARM, and ARGRE, all adapted to the LVLM scenario\.For implementation, we pair each toxicity annotation with a blank image as visual input to the LVLM and apply each detoxification method to the LLM component within each LVLM, following the same experimental settings used for LLM detoxification\.

#### VI\-B2Results and Analysis

Tab\.[XIV](https://arxiv.org/html/2610.41165#S6.T14)presents the toxicity evaluation results for three representative LVLMs tasked with generating image descriptions from toxic images, while Fig\.[6](https://arxiv.org/html/2610.41165#S6.F6)provides examples of the detoxified descriptions generated by our method\. From the results, we can identify:

❶Regarding detoxification performance, TARGRS achieves a substantial average toxicity reduction across all three LVLMs, outperforming all baseline methods by a significant margin\. Specifically, the original toxicity rates for Qwen\-VL\-Chat, LLaVA\-1\.5, and Llava\-next\-8B on the ToViLaG dataset are 44\.88%, 40\.13%, and 18\.25%, respectively\. After applying TARGRS , these toxicity rates decrease to 15\.45%, 13\.22%, and 7\.29%, resulting in an average toxicity reduction of approximately 64\.23% across the three models\.In comparison, ProFS, Re\-Control, GenARM, and ARGRE achieve average toxicity reductions of 27\.07%, 18\.71%, 39\.57%, and 57\.03%, respectively\.This substantial difference highlights that our method is more effective in mitigating toxic content generation in LVLMs, regardless of the underlying model architecture or initial toxicity levels\. As shown in Fig\.[6](https://arxiv.org/html/2610.41165#S6.F6), TARGRS substantially reduces explicit wording and toxicity scores in the generated descriptions\. For example, it transforms the text “A woman in fishnets is getting her ass fingered” \(97\.80%\) into “Two women in lingerie are on a couch with a large member” \(15\.61%\)\.Although both the explicitness and Detoxify score are substantially reduced, the resulting description still contains potentially inappropriate implications, showing that the lower automated score does not fully reflect the remaining safety concern in this example\.Clearer improvements can be observed in other examples, where explicit phrases such as “licking each other” \(87\.20%\) are rewritten into milder expressions like “in a passionate embrace” \(0\.60%\)\.

![Refer to caption](https://arxiv.org/html/2610.41165v1/LVLMs.png)Fig\. 6:Examples of descriptions generated by Qwen\-VL\-Chat for toxic images from the ToViLaG dataset\. Theredtexts are responses generated by the original LVLM, while thegreentexts are responses generated by TARGRS with reduced toxicity,which substantially softens explicit wording\.❷More importantly, TARGRS maintains an excellent balance between toxicity mitigation and the fluency and naturalness of the generated text, which is a critical requirement for practical deployment of LVLMs\. While all detoxification methods lead to a slight increase in perplexity compared to the original model outputs, our method exhibits the smallest average increase inPPLg\\rm\{PPL\}\_\{\\rm\{g\}\}\. Specifically, the average perplexity of the original models across the three LVLMs is 13\.98, and our method increases this average to 16\.86, representing a slight average increase of 2\.88\.In comparison, ProFS, Re\-Control, GenARM, and ARGRE incur averagePPLg\\rm\{PPL\}\_\{\\rm\{g\}\}increases of 4\.49, 8\.10, 6\.59, and 3\.80, respectively\.This indicates that our method can effectively mitigate toxic content without significantly compromising the fluency and readability of the generated text\. The excellent balance is further confirmed in Fig\.[6](https://arxiv.org/html/2610.41165#S6.F6), where TARGRS transforms toxic content into contextually accurate descriptions without sacrificing relevance to the visual content\. For example, it changes “A woman in a white leotard and pink knee socks” into “A beautiful young woman in a white top and pink and white striped socks”, even offering a more detailed and accurate description\.

These results demonstrate that our proposed method not only outperforms existing baseline methods in detoxification but also preserves the fluency of the generated text\. Additionally, it exhibits strong generalizability across different LVLM architectures, offering a practical and effective solution for ensuring the safe deployment of LVLMs\.

Moreover, we conduct an additional validation on Qwen\-VL\-Chat to examine the potential modality gap introduced by the blank\-image setup and whether this setup is reasonable in our LVLM extension\. Specifically, we sample 100 real benign images from the neutral class of the NSFW classification dataset\[[101](https://arxiv.org/html/2610.41165#bib.bib91)\]\. For each pair of toxic and non\-toxic textual responses, we randomly sample one image and pair both responses with the same image as visual input to the LVLM\. The real\-image setting yields a toxicity score of 16\.32% and aPPLg\\rm\{PPL\}\_\{\\rm\{g\}\}of 23\.17, which are comparable to the results under the blank\-image setup \(15\.45% toxicity and 23\.25PPLg\\rm\{PPL\}\_\{\\rm\{g\}\}\)\. This may be attributed to the design of our method, which operates in a language\-decoder representation space unified across modalities\. Therefore, the blank\-image setup is reasonable and does not undermine the effectiveness of our method\.

## VIIConclusions and Future Work

This paper presents TARGRS, a test\-time detoxification method that learns reward guidance directly from toxicity transitions within the latent representation space\. By performing progressive directional steering during generation to trace representation trajectories spanning from non\-toxic to toxic states, TARGRS transforms sparse annotations into dense, listwise supervision\. This enables the learning of a transition\-aware reward model that provides precise and reliable control over representation sampling during inference\. Extensive experiments on 10 LLMs and 3 LVLMs show that TARGRS consistently outperforms competitive baselines in terms of detoxification performance, achieves superior inference efficiency compared to state\-of\-the\-art methods, and maintains the original model’s capabilities with minimal degradation\.

Limitations\.Despite the promising results, TARGRS is a white\-box method that requires access to the internal representations of LLMs\. This is a common assumption in prior work\[[11](https://arxiv.org/html/2610.41165#bib.bib45),[12](https://arxiv.org/html/2610.41165#bib.bib3),[17](https://arxiv.org/html/2610.41165#bib.bib39),[18](https://arxiv.org/html/2610.41165#bib.bib10),[53](https://arxiv.org/html/2610.41165#bib.bib6),[60](https://arxiv.org/html/2610.41165#bib.bib49)\]where model transparency is essential for effective controllable generation\.This access requirement limits the applicability of TARGRS in closed\-source or API\-restricted deployment scenarios\. Although such systems may provide grey\-box access through token\-level log\-probabilities, this interface exposes only vocabulary\-space outputs and does not grant access to the internal hidden states needed for our representation\-level intervention\. Consequently, TARGRS is primarily intended for open\-weight models with full internal access, and extending it to closed\-source systems remains an open problem\.In addition, some outputs with substantially reduced toxicity scores may still retain potentially inappropriate implications, suggesting that improvements in automated metrics may not always fully reflect actual output safety\.Despite the improvements on broader safety tasks, residual attack success rates indicate room for further improvement in some evaluated risk categories, including general harm, illegal activities, and malicious uses\.

Ethical Statement and Broader Impact\.This work aims to enhance the safety of LLMs by mitigating toxic outputs through representation editing\. However, the same techniques could be misused to intentionally steer models toward harmful or toxic behavior, underscoring the importance of cautious and responsible deployment\.

## VIIIAcknowledgments

This work was supported by the New Generation Artificial Intelligence National Science and Technology Major Project \(Grant No\. 2025ZD0123201\), the National Natural Science Foundation of China \(Grant No\. 62576019, 62525601, 625B2012, T2588101\)\.

## References

- \[1\]W\. Chiang, Z\. Li, Z\. Lin, Y\. Sheng, Z\. Wu, H\. Zhang, L\. Zheng, S\. Zhuang, Y\. Zhuang, J\. E\. Gonzalez,et al\.\(2023\)Vicuna: an open\-source chatbot impressing gpt\-4 with 90%\* chatgpt quality, march 2023\.URL https://lmsys\. org/blog/2023\-03\-30\-vicuna3\(5\)\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p1.1),[§VI\-A1](https://arxiv.org/html/2610.41165#S6.SS1.SSS1.p1.1)\.
- \[2\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv:2307\.09288\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p1.1)\.
- \[3\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p1.1)\.
- \[4\]H\. Xiong, J\. Bian, Y\. Li, X\. Li, M\. Du, S\. Wang, D\. Yin, and S\. Helal\(2024\)When search engine services meet large language models: visions and challenges\.IEEE Transactions on Services Computing\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p1.1)\.
- \[5\]X\. L\. Dong, S\. Moon, Y\. E\. Xu, K\. Malik, and Z\. Yu\(2023\)Towards next\-generation intelligent assistants leveraging llm techniques\.InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 5792–5793\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p1.1)\.
- \[6\]A\. Deshpande, V\. Murahari, T\. Rajpurohit, A\. Kalyan, and K\. R\. NarasimhanToxicity in chatgpt: analyzing persona\-assigned language models\.InThe 2023 Conference on Empirical Methods in Natural Language Processing,Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p1.1)\.
- \[7\]S\. Gehman, S\. Gururangan, M\. Sap, Y\. Choi, and N\. A\. Smith\(2020\)RealToxicityPrompts: evaluating neural toxic degeneration in language models\.InEMNLP,pp\. 3356–3369\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p1.1),[§I](https://arxiv.org/html/2610.41165#S1.p4.1),[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p2.1)\.
- \[8\]N\. BBC\(2019\)Taylor swift ’tried to sue’ microsoft over racist chatbot tay\.\.Note:[https://www\.bbc\.com/news/newsbeat\-49645508](https://www.bbc.com/news/newsbeat-49645508)Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p1.1)\.
- \[9\]S\. Gururangan, A\. Marasović, S\. Swayamdipta, K\. Lo, I\. Beltagy, D\. Downey, and N\. A\. Smith\(2020\)Don’t stop pretraining: adapt language models to domains and tasks\.InACL,pp\. 8342–8360\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[10\]X\. Zhang and X\. Wan\(2023\)Mil\-decoding: detoxifying language models at token\-level via multiple instance learning\.InACL,pp\. 190–202\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p1.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p4.1)\.
- \[11\]Y\. Li, H\. Jiang, C\. Gong, and Z\. WeiDeStein: navigating detoxification of language models via universal steering pairs and head\-wise activation fusion\.InFirst Conference on Language Modeling,Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p1.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p6.1),[§IV\-A](https://arxiv.org/html/2610.41165#S4.SS1.p1.1),[§IV\-D](https://arxiv.org/html/2610.41165#S4.SS4.p5.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p5.1),[§V\-B](https://arxiv.org/html/2610.41165#S5.SS2.p3.1.1),[§VII](https://arxiv.org/html/2610.41165#S7.p2.1)\.
- \[12\]R\. Uppaal, A\. Dey, Y\. He, Y\. Zhong, and J\. HuModel editing as a robust and denoised variant of dpo: a case study on toxicity\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p1.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p5.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p1.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p5.1),[§VII](https://arxiv.org/html/2610.41165#S7.p2.1)\.
- \[13\]A\. Lee, X\. Bai, I\. Pres, M\. Wattenberg, J\. K\. Kummerfeld, and R\. Mihalcea\(2024\)A mechanistic understanding of alignment algorithms: a case study on dpo and toxicity\.InICML,Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p1.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p6.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p1.1)\.
- \[14\]B\. Wang, W\. Ping, C\. Xiao, P\. Xu, M\. Patwary, M\. Shoeybi, B\. Li, A\. Anandkumar, and B\. Catanzaro\(2022\)Exploring the limits of domain\-adaptive training for detoxifying large\-scale language models\.NeurIPS35,pp\. 35811–35824\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p1.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[15\]B\. Wang, R\. Zheng, L\. Chen, Y\. Liu, S\. Dou, C\. Huang, W\. Shen, S\. Jin, E\. Zhou, C\. Shi,et al\.\(2024\)Secrets of rlhf in large language models part ii: reward modeling\.arXiv preprint arXiv:2401\.06080\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[16\]R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn\(2023\)Direct preference optimization: your language model is secretly a reward model\.NeurIPS36,pp\. 53728–53741\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1),[§III](https://arxiv.org/html/2610.41165#S3.p3.2.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p5.1)\.
- \[17\]C\. T\. Leong, Y\. Cheng, J\. Wang, J\. Wang, and W\. Li\(2023\)Self\-detoxifying language models via toxification reversal\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 4433–4449\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p6.1),[§IV\-A](https://arxiv.org/html/2610.41165#S4.SS1.p1.1),[§IV\-D](https://arxiv.org/html/2610.41165#S4.SS4.p5.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p5.1),[§V\-B](https://arxiv.org/html/2610.41165#S5.SS2.p3.1.1),[§VII](https://arxiv.org/html/2610.41165#S7.p2.1)\.
- \[18\]L\. Kong, H\. Wang, W\. Mu, Y\. Du, Y\. Zhuang, Y\. Zhou, Y\. Song, R\. Zhang, K\. Wang, and C\. Zhang\(2024\)Aligning large language models with representation editing: a control perspective\.NeurIPS37,pp\. 37356–37384\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p6.1),[§IV\-A](https://arxiv.org/html/2610.41165#S4.SS1.p1.1),[§IV\-D](https://arxiv.org/html/2610.41165#S4.SS4.p5.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p5.1),[§VII](https://arxiv.org/html/2610.41165#S7.p2.1)\.
- \[19\]Y\. Xiao, A\. Liu, S\. Liang, X\. Liu, and D\. Tao\(2025\)Fairness mediator: neutralize stereotype associations to mitigate bias in large language models\.InISSTA,Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1)\.
- \[20\]K\. Park, Y\. J\. Choe, Y\. Jiang, and V\. Veitch\(2025\)The geometry of categorical and hierarchical concepts in large language models\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§IV\-B](https://arxiv.org/html/2610.41165#S4.SS2.p1.1)\.
- \[21\]K\. Park, Y\. J\. Choe, and V\. Veitch\(2024\)The linear representation hypothesis and the geometry of large language models\.InICML,pp\. 39643–39666\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§IV\-B](https://arxiv.org/html/2610.41165#S4.SS2.p1.1)\.
- \[22\]N\. Nanda, A\. Lee, and M\. Wattenberg\(2023\)Emergent linear representations in world models of self\-supervised sequence models\.InProceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,pp\. 16–30\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p2.1),[§IV\-B](https://arxiv.org/html/2610.41165#S4.SS2.p1.1)\.
- \[23\]J\. Dhamala, T\. Sun, V\. Kumar, S\. Krishna, Y\. Pruksachatkun, K\. Chang, and R\. Gupta\(2021\)Bold: dataset and metrics for measuring biases in open\-ended language generation\.InProceedings of the 2021 ACM conference on fairness, accountability, and transparency,pp\. 862–872\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p4.1),[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p2.1)\.
- \[24\]G\. Kour, M\. Zalmanovici, N\. Zwerdling, E\. Goldbraich, O\. Fandina, A\. A\. Tavor, O\. Raz, and E\. Farchi\(2023\)Unveiling safety vulnerabilities of large language models\.InProceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics \(GEM\),pp\. 111–127\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p4.1),[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p2.1)\.
- \[25\]X\. Wang, X\. Yi, H\. Jiang, S\. Zhou, Z\. Wei, and X\. Xie\(2023\)Tovilag: your visual\-language generative model is also an evildoer\.arXiv preprint arXiv:2312\.11523\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p4.1),[§VI\-B1](https://arxiv.org/html/2610.41165#S6.SS2.SSS1.p1.1)\.
- \[26\]Y\. Xiao, A\. Liu, S\. Liang, Z\. Ying, X\. Liu, and D\. Tao\(2025\)Detoxifying large language models via autoregressive reward guided representation editing\.NeurIPS\.Cited by:[§I](https://arxiv.org/html/2610.41165#S1.p6.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p5.1)\.
- \[27\]M\. M\. Ferdaus, M\. Abdelguerfi, E\. Loup, K\. N\. Niles, K\. Pathak, and S\. Sloan\(2026\)Towards trustworthy ai: a review of ethical and robust large language models\.ACM Computing Surveys58\(7\),pp\. 1–43\.Cited by:[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1)\.
- \[28\]A\. Xu, E\. Pathak, E\. Wallace, S\. Gururangan, M\. Sap, and D\. Klein\(2021\)Detoxifying language models risks marginalizing minority voices\.InNAACL,pp\. 2390–2397\.Cited by:[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1)\.
- \[29\]D\. Dale, A\. Voronov, D\. Dementieva, V\. Logacheva, O\. Kozlova, N\. Semenov, and A\. Panchenko\(2021\)Text detoxification using large pre\-trained neural models\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 7979–7996\.Cited by:[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1)\.
- \[30\]Y\. Xiao, X\. Liu, Q\. Cheng, Z\. Yin, S\. Liang, J\. Li, J\. Shao, A\. Liu, and D\. Tao\(2025\)GenderBias\-vl: benchmarking gender bias in vision language models via counterfactual probing: y\. xiao et al\.\.International Journal of Computer Vision\.Cited by:[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1)\.
- \[31\]Y\. Xiao, A\. Liu, X\. Zhang, T\. Zhang, T\. Li, S\. Liang, X\. Liu, Y\. Liu, and D\. Tao\(2025\)BDefects4NN: a backdoor defect database for controlled localization studies in neural networks\.InICSE,Cited by:[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1)\.
- \[32\]Y\. Xiao, A\. Liu, T\. Li, and X\. Liu\(2023\)Latent imitator: generating natural individual discriminatory instances for black\-box fairness testing\.InISSTA,Cited by:[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1)\.
- \[33\]Z\. Ying, D\. Zhang, Z\. Jing, Y\. Xiao, Q\. Zou, A\. Liu, S\. Liang, X\. Zhang, X\. Liu, and D\. Tao\(2025\)Reasoning\-augmented conversation for multi\-turn jailbreak attacks on large language models\.arXiv preprint arXiv:2502\.11054\.Cited by:[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1)\.
- \[34\]Y\. Xiao, A\. Liu, T\. Zhang, H\. Qin, J\. Guo, and X\. Liu\(2023\)Robustmq: benchmarking robustness of quantized models\.Visual Intelligence\.Cited by:[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1)\.
- \[35\]H\. Wang, W\. Fu, Y\. Tang, Z\. Chen, Y\. Huang, J\. Piao, C\. Gao, F\. Xu, T\. Jiang, and Y\. Li\(2025\)A survey on responsible llms: inherent risk, malicious use, and mitigation strategy\.arXiv preprint arXiv:2501\.09431\.Cited by:[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1)\.
- \[36\]A\. Liu, X\. Liu, X\. Zhang, Y\. Xiao, Y\. Zhou, S\. Liang, J\. Wang, X\. Cao, and D\. Tao\(2025\)Pre\-trained trojan attacks for visual recognition\.International Journal of Computer Vision133\(6\),pp\. 3568–3585\.Cited by:[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1)\.
- \[37\]Q\. Zou, Z\. Ying, M\. Chen, W\. Xu, Y\. Xiao, Y\. Li, D\. Zhang, D\. Yang, Z\. Liu, and X\. Zhang\(2025\)PRISM: programmatic reasoning with image sequence manipulation for lvlm jailbreaking\.arXiv preprint arXiv:2507\.21540\.Cited by:[§II\-A](https://arxiv.org/html/2610.41165#S2.SS1.p1.1)\.
- \[38\]M\. Wang, N\. Zhang, Z\. Xu, Z\. Xi, S\. Deng, Y\. Yao, Q\. Zhang, L\. Yang, J\. Wang, and H\. Chen\(2024\)Detoxifying large language models via knowledge editing\.arXiv preprint arXiv:2403\.14472\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p1.1),[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p5.1)\.
- \[39\]A\. Liu, S\. Tang, X\. Chen, L\. Huang, H\. Qin, X\. Liu, and D\. Tao\(2023\)Towards defending multiple lp\-norm bounded adversarial perturbations via gated batch normalization\.International Journal of Computer Vision\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[40\]A\. Liu, S\. Tang, S\. Liang, R\. Gong, B\. Wu, X\. Liu, and D\. Tao\(2023\)Exploring the relationship between architecture and adversarially robust generalization\.InCVPR,Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[41\]S\. Zhao, M\. Wang, S\. Xiong, F\. Chen, Q\. Zhu, S\. Ruan, Y\. Xiao, R\. Duan, X\. Chen, and X\. Wei\(2026\)Improving safety alignment via balanced direct preference optimization\.arXiv preprint arXiv:2603\.22829\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[42\]A\. Liu, X\. Liu, H\. Yu, C\. Zhang, Q\. Liu, and D\. Tao\(2021\)Training robust deep neural networks via adversarial noise propagation\.TIP\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[43\]Z\. Yin, J\. Wang, Y\. Xiao, H\. Zhao, T\. Li, W\. Zhou, A\. Liu, and X\. Liu\(2024\)Improving deepfake detection generalization by invariant risk minimization\.IEEE transactions on multimedia26,pp\. 6785–6798\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[44\]T\. Korbak, K\. Shi, A\. Chen, R\. V\. Bhalerao, C\. Buckley, J\. Phang, S\. R\. Bowman, and E\. Perez\(2023\)Pretraining language models with human preferences\.InICML,pp\. 17506–17533\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[45\]S\. Prabhumoye, M\. Patwary, M\. Shoeybi, and B\. Catanzaro\(2023\)Adding instructions during pretraining: effective way of controlling toxicity in language models\.InEACL,pp\. 2636–2651\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[46\]E\. Dinan, S\. Humeau, B\. Chintagunta, and J\. Weston\(2019\)Build it break it fix it for dialogue safety: robustness from adversarial human attack\.InEMNLP\-IJCNLP,pp\. 4537–4546\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[47\]J\. Xu, D\. Ju, M\. Li, Y\. Boureau, J\. Weston, and E\. Dinan\(2020\)Recipes for safety in open\-domain chatbots\.arXiv preprint arXiv:2010\.07079\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[48\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.NeurIPS35,pp\. 27730–27744\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[49\]X\. Yang, X\. Wang, Q\. Zhang, L\. Petzold, W\. Y\. Wang, X\. Zhao, and D\. Lin\(2023\)Shadow alignment: the ease of subverting safely\-aligned language models\.arXiv preprint arXiv:2310\.02949\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p2.1)\.
- \[50\]S\. Dathathri, A\. Madotto, J\. Lan, J\. Hung, E\. Frank, P\. Molino, J\. Yosinski, and R\. Liu\(2019\)Plug and play language models: a simple approach to controlled text generation\.InInternational Conference on Learning Representations,Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p4.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p1.1)\.
- \[51\]A\. Liu, M\. Sap, X\. Lu, S\. Swayamdipta, C\. Bhagavatula, N\. A\. Smith, and Y\. Choi\(2021\)DExperts: decoding\-time controlled text generation with experts and anti\-experts\.InACL,pp\. 6691–6706\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p4.1),[§V\-B](https://arxiv.org/html/2610.41165#S5.SS2.p4.1)\.
- \[52\]B\. Krause, A\. D\. Gotmare, B\. McCann, N\. S\. Keskar, S\. Joty, R\. Socher, and N\. F\. Rajani\(2020\)Gedi: generative discriminator guided sequence generation\.arXiv preprint arXiv:2009\.06367\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p4.1)\.
- \[53\]Y\. Xu, U\. M\. Sehwag, A\. Koppel, S\. Zhu, B\. An, F\. Huang, and S\. Ganesh\(2025\)GenARM: reward guided generation with autoregressive reward model for test\-time alignment\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p4.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p5.1),[§VII](https://arxiv.org/html/2610.41165#S7.p2.1)\.
- \[54\]H\. Deng and C\. Raffel\(2023\)Reward\-augmented decoding: efficient controlled text generation with a unidirectional reward model\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 11781–11791\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p4.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p5.1)\.
- \[55\]S\. Zhang, D\. Yu, H\. Sharma, Z\. Yang, S\. Wang, H\. H\. Awadalla, and Z\. WangSelf\-exploring language models: active preference elicitation for online alignment\.InAutomated Reinforcement Learning: Exploring Meta\-Learning, AutoML, and LLMs,Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p4.1)\.
- \[56\]R\. Rafailov, Y\. Chittepu, R\. Park, H\. S\. Sikchi, J\. Hejna, B\. Knox, C\. Finn, and S\. Niekum\(2024\)Scaling laws for reward model overoptimization in direct alignment algorithms\.NeurIPS37,pp\. 126207–126242\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p4.1)\.
- \[57\]B\. Wei, K\. Huang, Y\. Huang, T\. Xie, X\. Qi, M\. Xia, P\. Mittal, M\. Wang, and P\. Henderson\(2024\)Assessing the brittleness of safety alignment via pruning and low\-rank modifications\.InICML,pp\. 52588–52610\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p5.1)\.
- \[58\]T\. Limisiewicz, D\. Mareček, and T\. Musil\(2023\)Debiasing algorithm through model adaptation\.arXiv preprint arXiv:2310\.18913\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p5.1)\.
- \[59\]J\. Gu, H\. Xu, J\. Ma, P\. Lu, Z\. Ling, K\. Chang, and N\. Peng\(2024\)Model editing harms general abilities of large language models: regularization to the rescue\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 16801–16819\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p5.1)\.
- \[60\]N\. Panickssery, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. M\. Turner\(2023\)Steering llama 2 via contrastive activation addition\.arXiv preprint arXiv:2312\.06681\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p6.1),[§IV\-A](https://arxiv.org/html/2610.41165#S4.SS1.p1.1),[§VII](https://arxiv.org/html/2610.41165#S7.p2.1)\.
- \[61\]S\. Liu, H\. Ye, L\. Xing, and J\. Zou\(2023\)In\-context vectors: making in context learning more effective and controllable through latent space steering\.arXiv preprint arXiv:2311\.06668\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p6.1)\.
- \[62\]A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski,et al\.\(2023\)Representation engineering: a top\-down approach to ai transparency\.arXiv preprint arXiv:2310\.01405\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.01405)Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p6.1.2)\.
- \[63\]A\. M\. Turner, L\. Thiergart, D\. Udell, G\. Leech, U\. Mini, and M\. MacDiarmid\(2023\)Activation addition: steering language models without optimization\.arXiv preprint arXiv:2308\.10248\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2308.10248)Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p6.1.2)\.
- \[64\]Z\. Liu, L\. Li, R\. Chen, Y\. Jiang, T\. Zhu, Z\. Su, W\. Chen, and J\. Shao\(2025\)Iterative value function optimization for guided decoding\.arXiv preprint arXiv:2503\.02368\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p6.1)\.
- \[65\]R\. S\. Sutton A\. G\. Bartoet al\.\(1998\)Reinforcement learning: an introduction\.Vol\.1,MIT press Cambridge\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p6.1),[§IV\-C](https://arxiv.org/html/2610.41165#S4.SS3.p1.1)\.
- \[66\]E\. Pignatelli, J\. Ferret, M\. Geist, T\. Mesnard, H\. van Hasselt, and L\. ToniA survey of temporal credit assignment in deep reinforcement learning\.Transactions on Machine Learning Research\.Cited by:[§II\-B](https://arxiv.org/html/2610.41165#S2.SS2.p6.1),[§IV\-C](https://arxiv.org/html/2610.41165#S4.SS3.p1.1)\.
- \[67\]D\. M\. Ziegler, N\. Stiennon, J\. Wu, T\. B\. Brown, A\. Radford, D\. Amodei, P\. Christiano, and G\. Irving\(2019\)Fine\-tuning language models from human preferences\.arXiv preprint arXiv:1909\.08593\.Cited by:[§III](https://arxiv.org/html/2610.41165#S3.p2.2.1)\.
- \[68\]D\. Go, T\. Korbak, G\. Kruszewski, J\. Rozen, N\. Ryu, and M\. Dymetman\(2023\)Aligning language models with preferences through f\-divergence minimization\.InICML,pp\. 11546–11583\.Cited by:[§III](https://arxiv.org/html/2610.41165#S3.p3.2.1)\.
- \[69\]K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov\(2022\)Locating and editing factual associations in gpt\.NeurIPS35,pp\. 17359–17372\.Cited by:[§IV\-B](https://arxiv.org/html/2610.41165#S4.SS2.p2.2)\.
- \[70\]J\. Shlens\(2014\)A tutorial on principal component analysis\.arXiv preprint arXiv:1404\.1100\.Cited by:[§IV\-B](https://arxiv.org/html/2610.41165#S4.SS2.p2.2)\.
- \[71\]T\. Liu, Z\. Qin, J\. Wu, J\. Shen, M\. Khalman, R\. Joshi, Y\. Zhao, M\. Saleh, S\. Baumgartner, J\. Liu,et al\.\(2025\)Lipo: listwise preference optimization through learning\-to\-rank\.InNAACL,pp\. 2404–2420\.Cited by:[§IV\-C](https://arxiv.org/html/2610.41165#S4.SS3.p2.1),[§IV\-C](https://arxiv.org/html/2610.41165#S4.SS3.p2.2)\.
- \[72\]S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\(2017\)Pointer sentinel mixture models\.InInternational Conference on Learning Representations,Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p1.1),[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p3.1)\.
- \[73\]L\. Hanu and Unitary team\(2020\)Detoxify\.Note:Github\. https://github\.com/unitaryai/detoxifyCited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p2.1)\.
- \[74\]L\. Gao, J\. Tow, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, K\. McDonell, N\. Muennighoff,et al\.\(2021\)A framework for few\-shot language model evaluation\.Version v0\. 0\.1\. Sept,pp\. 8\.Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p3.1)\.
- \[75\]C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova\(2019\)BoolQ: exploring the surprising difficulty of natural yes/no questions\.InNAACL,pp\. 2924–2936\.Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p3.1)\.
- \[76\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. Bowman\(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,pp\. 353–355\.Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p3.1)\.
- \[77\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\(2019\)HellaSwag: can a machine really finish your sentence?\.InACL,pp\. 4791–4800\.Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p3.1)\.
- \[78\]K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi\(2021\)Winogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p3.1)\.
- \[79\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p3.1)\.
- \[80\]T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal\(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 2381–2391\.Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p3.1)\.
- \[81\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed\(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p4.1)\.
- \[82\]L\. Tunstall, E\. Beeching, N\. Lambert, N\. Rajani, K\. Rasul, Y\. Belkada, S\. Huang, L\. von Werra, C\. Fourrier, N\. Habib,et al\.\(2023\)Zephyr: direct distillation of lm alignment\.arXiv preprint arXiv:2310\.16944\.Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p4.1)\.
- \[83\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p4.1)\.
- \[84\]M\. Khanov, J\. Burapacheep, and Y\. LiARGS: alignment as reward\-guided search\.InThe Twelfth International Conference on Learning Representations,Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p4.1)\.
- \[85\]Q\. Teamet al\.\(2024\)Qwen2 technical report\.arXiv preprint arXiv:2407\.106712\(3\)\.Cited by:[§V\-A](https://arxiv.org/html/2610.41165#S5.SS1.p4.1)\.
- \[86\]D\. Wingate, M\. Shoeybi, and T\. Sorensen\(2022\)Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models\.InEMNLP,pp\. 5621–5634\.Cited by:[§V\-B](https://arxiv.org/html/2610.41165#S5.SS2.p4.1)\.
- \[87\]A\. Templeton, T\. Conerly, J\. Marcus, J\. Lindsey, T\. Bricken, B\. Chen, A\. Pearce, C\. Citro, E\. Ameisen, A\. Jones,et al\.\(2026\)Scaling monosemanticity: extracting interpretable features from claude 3 sonnet\.arXiv preprint arXiv:2605\.29358\.Cited by:[§V\-F](https://arxiv.org/html/2610.41165#S5.SS6.p1.1.1)\.
- \[88\]K\. O’Brien, D\. Majercak, X\. Fernandes, R\. G\. Edgar, B\. Bullwinkel, J\. Chen, H\. Nori, D\. Carignan, E\. Horvitz, and F\. Poursabzi\-SangdehSteering language model refusal with sparse autoencoders\.InICML 2025 Workshop on Reliable and Responsible Foundation Models,Cited by:[§V\-F](https://arxiv.org/html/2610.41165#S5.SS6.p1.1.1)\.
- \[89\]J\. Bloom, C\. Tigges, A\. Duong, and D\. Chanin\(2024\)SAELens\.Note:[https://github\.com/decoderesearch/SAELens](https://github.com/decoderesearch/SAELens)Cited by:[§V\-F](https://arxiv.org/html/2610.41165#S5.SS6.p1.1.1)\.
- \[90\]Perspective api\.Note:https://perspectiveapi\.com/Cited by:[§V\-F](https://arxiv.org/html/2610.41165#S5.SS6.p2.1.1)\.
- \[91\]M\. Mazeika, L\. Phan, X\. Yin, A\. Zou, Z\. Wang, N\. Mu, E\. Sakhaee, N\. Li, S\. Basart, B\. Li,et al\.\(2024\)Harmbench: a standardized evaluation framework for automated red teaming and robust refusal\.arXiv preprint arXiv:2402\.04249\.Cited by:[§V\-F](https://arxiv.org/html/2610.41165#S5.SS6.p3.1)\.
- \[92\]Y\. Wang, H\. Li, X\. Han, P\. Nakov, and T\. Baldwin\(2023\)Do\-not\-answer: a dataset for evaluating safeguards in llms\.arXiv preprint arXiv:2308\.13387\.Cited by:[§V\-F](https://arxiv.org/html/2610.41165#S5.SS6.p3.1)\.
- \[93\]B\. Wang, W\. Chen, H\. Pei, C\. Xie, M\. Kang, C\. Zhang, C\. Xu, Z\. Xiong, R\. Dutta, R\. Schaeffer,et al\.\(2023\)DecodingTrust: a comprehensive assessment of trustworthiness in\{\\\{gpt\}\\\}models\.Cited by:[§V\-F](https://arxiv.org/html/2610.41165#S5.SS6.p3.1)\.
- \[94\]L\. Phan\(2023\)Harmful harmless instructions\.\.Note:[\{HuggingFace\.https://huggingface\.co/datasets/justinphan3110/harmful\_harmless\_instructions\}](https://{huggingface.https//huggingface.co/datasets/justinphan3110/harmful_harmless_instructions%7D)Cited by:[§V\-F](https://arxiv.org/html/2610.41165#S5.SS6.p3.1.2)\.
- \[95\]L\. Li, B\. Dong, R\. Wang, X\. Hu, W\. Zuo, D\. Lin, Y\. Qiao, and J\. Shao\(2024\)SALAD\-bench: a hierarchical and comprehensive safety benchmark for large language models\.arXiv preprint arXiv:2402\.05044\.Cited by:[§V\-F](https://arxiv.org/html/2610.41165#S5.SS6.p3.1)\.
- \[96\]X\. Li, T\. Zhang, Y\. Dubois, R\. Taori, I\. Gulrajani, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto\(2023\)AlpacaEval: an automatic evaluator of instruction\-following models\.GitHub\.Note:[https://github\.com/tatsu\-lab/alpaca\_eval](https://github.com/tatsu-lab/alpaca_eval)Cited by:[§V\-F](https://arxiv.org/html/2610.41165#S5.SS6.p5.1)\.
- \[97\]J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.\(2023\)Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§VI\-A1](https://arxiv.org/html/2610.41165#S6.SS1.SSS1.p1.1),[§VI\-B1](https://arxiv.org/html/2610.41165#S6.SS2.SSS1.p1.1)\.
- \[98\]H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee\(2024\)Improved baselines with visual instruction tuning\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 26296–26306\.Cited by:[§VI\-A1](https://arxiv.org/html/2610.41165#S6.SS1.SSS1.p1.1),[§VI\-B1](https://arxiv.org/html/2610.41165#S6.SS2.SSS1.p1.1)\.
- \[99\]H\. Liu, C\. Li, Y\. Li, B\. Li, Y\. Zhang, S\. Shen, and Y\. J\. Lee\(2024\)Llavanext: improved reasoning, ocr, and world knowledge\.Cited by:[§VI\-A1](https://arxiv.org/html/2610.41165#S6.SS1.SSS1.p1.1),[§VI\-B1](https://arxiv.org/html/2610.41165#S6.SS2.SSS1.p1.1)\.
- \[100\]M\. Cherti, R\. Beaumont, R\. Wightman, M\. Wortsman, G\. Ilharco, C\. Gordon, C\. Schuhmann, L\. Schmidt, and J\. Jitsev\(2023\)Reproducible scaling laws for contrastive language\-image learning\.InCVPR,pp\. 2818–2829\.Cited by:[§VI\-A1](https://arxiv.org/html/2610.41165#S6.SS1.SSS1.p1.1)\.
- \[101\]NSFW image classification\.Note:[https://www\.kaggle\.com/datasets/360fbfce26b59056e60d5e9cd1cfa884c2d66c5b6f3b350254651cd136a41322](https://www.kaggle.com/datasets/360fbfce26b59056e60d5e9cd1cfa884c2d66c5b6f3b350254651cd136a41322)Cited by:[§VI\-B2](https://arxiv.org/html/2610.41165#S6.SS2.SSS2.p5.1.1)\.

## IXBiography

![[Uncaptioned image]](https://arxiv.org/html/2610.41165v1/fig/photo/Aishan_Liu.jpg)Aishan Liuis an Associate Professor in School of Computer Science and Engineering, Beihang University\. His current research interests include adversarial examples and robust deep learning models\. He has authored 100\+ papers in top\-tier conferences and journals\. He has been awarded 2025 ACM MM Most Popular Demo Award, 2024 ACM CCS Distinguished Artifact Award, 2025 ICML MAS Best Paper Award, etc\.![[Uncaptioned image]](https://arxiv.org/html/2610.41165v1/fig/photo/yisong.jpg)Yisong Xiaois a Ph\.D student at the School of Computer Science and Engineering, Beihang University\. His research interests lie in Trustworthy Artificial Intelligence, with a particular focus on the fairness and toxicity of large language models \(LLMs\)\. He has published 6 first\-author papers in top venues including IJCV, NeurIPS, ISSTA, and ICSE\. He was awarded the Outstanding Graduate of Beihang University in 2022 and received the National Scholarship in 2025\.![[Uncaptioned image]](https://arxiv.org/html/2610.41165v1/fig/photo/Xianglong_liu.jpg)Xianglong Liuis a Full Professor in School of Computer Science and Engineering at Beihang University\. He has published over 100 research papers at top venues like the IEEE Transactions on Image Processing, the IEEE Transactions on Cybernatics, the Conference on Computer Vision and Pattern Recognition, the International Conference on Computer Vision, and the Association for the Advancement of Artificial Intelligence\. His research interests include machine learning, computer vision and multimedia information retrieval\.![[Uncaptioned image]](https://arxiv.org/html/2610.41165v1/fig/photo/zonghao_ying.jpg)Zonghao Yingis currently pursuing the Ph\.D\. degree with the School of Artificial Intelligence, Beihang University, Beijing, China\. His research interests focus on Trustworthy AI\. He has published several peer\-reviewed articles in highly regarded journals and conferences\.![[Uncaptioned image]](https://arxiv.org/html/2610.41165v1/fig/photo/jianyang.jpg)Jian Yangis an Associate Professor at Beihang University, and a jointly\-appointed Associate Researcher at Shanghai Artificial Intelligence Laboratory\. His main research areas include multimodal and multilingual natural language understanding and generation, pre\-training and fine\-tuning of large language models, and code understanding and generation\. He has published over one hundred papers in prestigious international journals and conferences and over 16,000 citations on Google Scholar\.![[Uncaptioned image]](https://arxiv.org/html/2610.41165v1/fig/photo/yaodongyang.png)Yaodong Yangis an Assistant Professor \(Boya Young Scholar\) at the Institute for Artificial Intelligence, Peking University and Chief Scientist of the PKU\-PsiBot Joint Laboratory\. His research focuses on experience learning and alignment of AI/Embodied agents\. He has published over 200 papers in leading journals and conferences, with more than 13,000 Google Scholar citations\.![[Uncaptioned image]](https://arxiv.org/html/2610.41165v1/fig/photo/Dacheng_Tao.jpeg)Dacheng Tao\(Fellow, IEEE\) is currently a Distinguished University Professor in the College of Computing & Data Science at Nanyang Technological University\. He mainly applies statistics and mathematics to artificial intelligence and data science, and his research is detailed in one monograph and over 200 publications in prestigious journals and proceedings at leading conferences, with best paper awards, best student paper awards, and test\-of\-time awards\. He is a Fellow of the Australian Academy of Science, AAAS, ACM and IEEE\.