Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback

# Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback
Source: [https://arxiv.org/html/2608.15530](https://arxiv.org/html/2608.15530)
\[2pt\] Instituto Politécnico Nacional \(IPN\)Centro de Investigación en Computación \(CIC\)\[2pt\] mkrasitskii2023, gelbukh, kolesnikova, sidorov@cic\.ipn\.mx

## 1Introduction

The advent of Reinforcement Learning from Human Feedback \(RLHF\) has fundamentally reshaped natural language generation, enabling LLMs to outperform supervised baselines in helpfulness, safety, and fluency\[[12](https://arxiv.org/html/2608.15530#bib.bib11)\]\. However, this alignment introduces a hidden cost:The Neutrality Trap\. As models optimize to avoid controversial outputs, they inadvertently suppress affective content that is essential for faithfully conveying sentiment, a critical issue for downstream tasks such as sentiment analysis\.

### 1\.1Sentiment Drift in RLHF

As RL training progresses, summaries become increasingly neutral and affectively flattened\. While factual correctness improves, emotional markers \(intensifiers, evaluative adjectives\) are progressively attenuated\. For instance, where an SFT summary might describe a situation as "deeply concerning," an RLHF\-trained model may replace this with a neutral factual statement\. This systematic divergence between original and summarized affect is what we termsentiment drift\. By sentiment drift, we specifically refer to the reduction in both thevarianceof sentiment scores \(indicating a pull towards neutrality\) and theintensityof individual affective markers\.

### 1\.2Diagnostic Challenge

Why does the policy choose neutrality? Prior work attributes this to conservative reward modeling or overregularization\[[4](https://arxiv.org/html/2608.15530#bib.bib4)\], but explanations remain qualitative\. We hypothesize that affective tokens introduce higher reward variance, making them less attractive under optimization regimes prioritizing stability\. Despite anecdotal evidence, systematic methodologies to measure and attribute this effect, particularly in multilingual settings, are lacking\.

### 1\.3Our Contribution: Policy Attribution

To move beyond anecdotal observations, we proposePolicy Attribution, a framework that decomposes RL objective components to identify drivers of affective suppression\. Our contributions are:

- •We formally define and quantifySentiment Driftacross major benchmarks, with multilingual validation across eight typologically diverse languages\.
- •We introducePolicy Attribution, decomposing policy gradients by RL component \(reward, KL, clipping\) to identify causal drivers\. Our analysis reveals KL regularization accounts for≈\\approx82% of negative attribution on sentiment\-bearing tokens\.
- •We demonstrate sentiment loss is a strategic "low\-risk" outcome consistent with RLHF dynamics\[[4](https://arxiv.org/html/2608.15530#bib.bib4)\], extending analysis toDirect Preference Optimization \(DPO\)to show suppression persists across alignment methods\.
- •We propose and validateSentiment\-Aware KL Regularization, reducing drift by 18\-22% while maintaining ROUGE\-L within 0\.5 points of standard RLHF\.
- •We release ourPolicy Attribution toolkitto support reproducibility and effect\-aware alignment research\.

## 2Background and Related Work

### 2\.1RLHF in Text Summarization

The standard pipeline for aligning Large Language Models \(LLMs\) with human preferences consists of three stages: Supervised Fine\-Tuning \(SFT\), Reward Modeling \(RM\), and Reinforcement Learning \(RL\)\[[12](https://arxiv.org/html/2608.15530#bib.bib11)\]\. In the context of summarization, the RL phase typically employs Proximal Policy Optimization \(PPO\)\[[10](https://arxiv.org/html/2608.15530#bib.bib9)\]to maximize the expected reward while maintaining a constraint on the Kullback\-Leibler \(KL\) divergence from the SFT model\. The objective function is defined as:

J⁡\(θ\)=Ex∼𝒟,y∼πθ⁡\[Rϕ​\(x,y\)−β​log⁡πθ​\(y\|x\)πref​\(y\|x\)\]J\(\\theta\)=\\operatorname\{E\}\_\{x\\sim\\mathcal\{D\},y\\sim\\pi\_\{\\theta\}\}\\big\[R\_\{\\phi\}\(x,y\)\-\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}\\big\]\(1\)
whereRϕ​\(x,y\)R\_\{\\phi\}\(x,y\)is the scalar reward, andβ\\betacontrols the strength of the KL\-divergence penalty\. While this formulation stabilizes training, it has been shown to induce degenerate behaviors when the model over\-optimizes for summary length to satisfy the RM\[[4](https://arxiv.org/html/2608.15530#bib.bib4)\]\.

Recent work has extended beyond PPO\-based RLHF to direct alignment methods such as Direct Preference Optimization \(DPO\)\[[9](https://arxiv.org/html/2608.15530#bib.bib8)\], and Kahneman\-Tversky Optimization \(KTO\)\[[3](https://arxiv.org/html/2608.15530#bib.bib3)\]\. These approaches eliminate the explicit reward model and KL penalty by directly optimizing policy parameters against preference data\. However, whether these methods exhibit similar sentiment suppression remains an open question\. Our work provides the first systematic comparison of sentiment drift across both PPO and DPO frameworks\.

### 2\.2The Phenomenon of Reward Over\-optimization

A significant body of research has documented "Reward Hacking", where models exploit weaknesses in the reward function to maximize scores without improving true task performance\. In the context of summarization, this manifests as verbosity bias, excessive hedging, or stylistic flattening\. While prior work has examined factual degradation and hallucination under reward over\-optimization\[[4](https://arxiv.org/html/2608.15530#bib.bib4)\], the systematic suppression of affective intensity in favor of neutral alternatives remains under\-explored\.

\[[15](https://arxiv.org/html/2608.15530#bib.bib14)\]demonstrates that KL\-regularized RLHF systematically biases policies toward conservative action distributions under preference uncertainty\. Our work extends this theoretical finding by providing empirical evidence that sentiment\-bearing tokens are disproportionately affected by this conservatism, particularly in multilingual settings where affective encoding varies typologically\.

### 2\.3Interpretability and Attribution in LLMs

Understanding the decision\-making processes of neural models has long been a central concern in NLP research\. Feature attribution methods such as Integrated Gradients\[[13](https://arxiv.org/html/2608.15530#bib.bib12)\], SHAP\[[8](https://arxiv.org/html/2608.15530#bib.bib7)\], and gradient\-based analyses have been widely applied to interpret predictions in deep neural networks\. More recently, interpretability research has shifted toward mechanistic, theory\-grounded explanations of large\-scale language models, particularly in generative settings, where understanding stylistic and affective behavior is critical\[[2](https://arxiv.org/html/2608.15530#bib.bib2)\]\.

Recent theoretical work has provided formal perspectives on attribution methods, clarifying the assumptions under which gradient\-based explanations reliably capture model behavior\[[14](https://arxiv.org/html/2608.15530#bib.bib13)\]\. These approaches enable token\-level and component\-level analysis of generation dynamics, making them suitable for diagnosing subtle phenomena such as affective suppression\.

### 2\.4Sentiment and Style Preservation

Maintaining the stylistic and emotional integrity of the source text has long been recognized as a challenge in text generation\. Prior research in style transfer\[[11](https://arxiv.org/html/2608.15530#bib.bib10)\], sentiment\-controlled generation\[[7](https://arxiv.org/html/2608.15530#bib.bib6)\], and affect\-aware summarization\[[5](https://arxiv.org/html/2608.15530#bib.bib5)\]demonstrates that preserving emotional tone requires explicit modeling constraints\. In multilingual settings, this challenge is further compounded by typological variation and culturally grounded sentiment expression\[[1](https://arxiv.org/html/2608.15530#bib.bib1)\]\.

Our work bridges these lines of research by framing sentiment preservation as a policy\-level optimization problem and proposing an attribution\-based approach to preserving intent fidelity in RL\-tuned summarizers\. Unlike prior work that modifies the reward function heuristically, we provide a mechanistic diagnosis of which objective components drive sentiment loss, enabling targeted intervention\.

## 3Methodology

Our methodology evaluates the impact of RLHF optimization on sentiment preservation and attributes observed drift to specific training objective components\. The framework consists of three modules: \(1\) RLHF training pipeline with variable regularization, \(2\) multilingual sentiment evaluation suite, and \(3\)Policy Attributionmechanism for gradient decomposition\.

### 3\.1RLHF Training Pipeline

We follow the standard PPO pipeline\[[10](https://arxiv.org/html/2608.15530#bib.bib9)\]\. Given promptxx, policyπθ\\pi\_\{\\theta\}generates summaryyy\. The objective functionJ⁡\(θ\)J\(\\theta\)maximizes expected reward while penalizing deviation from reference modelπref\\pi\_\{\\text\{ref\}\}\(SFT checkpoint\):

J\(θ\)=Ex∼𝒟,y∼πθ\[Rϕ​\(x,y\)−−β⋅KL\(πθ\(⋅\|x\)∥πref\(⋅\|x\)\)\]\\begin\{split\}J\(\\theta\)=\\operatorname\{E\}\_\{x\\sim\\mathcal\{D\},y\\sim\\pi\_\{\\theta\}\}\\big\[&R\_\{\\phi\}\(x,y\)\-\\\\ \-\\beta\\cdot\\text\{KL\}\(\\pi\_\{\\theta\}\(\\cdot\|x\)\\\|\\pi\_\{\\text\{ref\}\}\(\\cdot\|x\)\)\\big\]\\end\{split\}\(2\)
whereRϕR\_\{\\phi\}is scalar reward andβ\\betacontrols KL\-divergence penalty strength\. We train models acrossβ∈\{0\.05,0\.1,0\.2\}\\beta\\in\\\{0\.05,0\.1,0\.2\\\}\(Table[1](https://arxiv.org/html/2608.15530#S4.T1)\)\. Unlike prior work, we explicitly track gradient contributions of each term during backpropagation\.

### 3\.2Multilingual Sentiment Evaluation

We employ pretrained transformer\-based sentiment classifiers for eight typologically diverse languages\. For source textSSand summaryS^\\hat\{S\}, we compute sentiment distributionsP⁡\(y\|S\)P\(y\|S\)andP⁡\(y\|S^\)P\(y\|\\hat\{S\}\)\.

Sentiment Variance \(SV\)measures the variance of sentiment scores across the dataset; reduction indicates convergence toward neutral outputs\. We also measure distributional shift via Jensen\-Shannon Divergence:

JSD\(P∥Q\)=12DKL\(P∥M\)\+12DKL\(Q∥M\)\\begin\{split\}\\text\{JSD\}\(P\\\|Q\)=\\frac\{1\}\{2\}D\_\{\\text\{KL\}\}\(P\\\|M\)\+\\frac\{1\}\{2\}D\_\{\\text\{KL\}\}\(Q\\\|M\)\\end\{split\}\(3\)
whereM=12​\(P\+Q\)M=\\frac\{1\}\{2\}\(P\+Q\)\. Lower JSD indicates better preservation of the original sentiment distribution\. Classifiers remain frozen to ensure shifts are attributable solely to policy updates\.

Building on recent advances in multilingual sentiment analysis, our evaluation framework follows the methodology established by\[[6](https://arxiv.org/html/2608.15530#bib.bib15)\], who demonstrated that sentiment preservation in summarization varies systematically across languages and is particularly challenging for morphologically rich languages\. Their cross\-lingual analysis provides the empirical foundation for our language selection and evaluation protocol\.

### 3\.3Policy Attribution Framework

Policy Attributiondecomposes the total policy gradient into contributions from distinct objective components\. Unlike standard attribution methods that attribute predictions to input features, we attributepolicy updatestoloss components\.

Let∇θℒt​o​t​a​l\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{total\}be the total gradient\. Based on Equation[2](https://arxiv.org/html/2608.15530#S3.E2), this decomposes as:

∇θℒt​o​t​a​l=∇θℒr​e​w​a​r​d\+β⋅∇θℒK​L\+∇θℒc​l​i​p\\begin\{split\}\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{total\}=\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{reward\}\+\\beta\\cdot\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{KL\}\+\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{clip\}\\end\{split\}\(4\)
whereℒr​e​w​a​r​d=−Rϕ​\(x,y\)\\mathcal\{L\}\_\{reward\}=\-R\_\{\\phi\}\(x,y\),ℒK​L=log⁡πθ​\(y\|x\)πref​\(y\|x\)\\mathcal\{L\}\_\{KL\}=\\log\\frac\{\\pi\_\{\\theta\}\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}, andℒc​l​i​p\\mathcal\{L\}\_\{clip\}is the PPO surrogate loss clipping term\.

Token\-Level Attribution Score\.For tokentit\_\{i\}in sequenceyy, we compute attribution scoreAc​\(ti\)A\_\{c\}\(t\_\{i\}\)for componentc∈\{reward,KL,clip\}c\\in\\\{\\text\{reward\},\\text\{KL\},\\text\{clip\}\\\}\. We adapt Integrated Gradients\[[13](https://arxiv.org/html/2608.15530#bib.bib12)\]:

Ac​\(ti\)=\(θ−θ′\)⋅∫α=01∂ℒc​\(θ′\+α⁡\(θ−θ′\)\)∂log⁡πθ​\(ti\|x,y<i\)​𝑑α\\begin\{split\}A\_\{c\}\(t\_\{i\}\)=\(\\theta\-\\theta^\{\\prime\}\)\\cdot\\int\_\{\\alpha=0\}^\{1\}\\frac\{\\partial\\mathcal\{L\}\_\{c\}\(\\theta^\{\\prime\}\+\\alpha\(\\theta\-\\theta^\{\\prime\}\)\)\}\{\\partial\\log\\pi\_\{\\theta\}\(t\_\{i\}\|x,y\_\{<i\}\)\}d\\alpha\\end\{split\}\(5\)
We approximate this integral using 50 Riemann steps\. Negative scoreAK​L​\(ti\)<0A\_\{KL\}\(t\_\{i\}\)<0indicates KL penalty suppressed tokentit\_\{i\}\. Aggregating over sentiment\-bearing tokens quantifies suppression proportion per component \(Table[5](https://arxiv.org/html/2608.15530#S5.T5)\)\.

### 3\.4Experimental Scope

Our attribution analysis focuses on abstractive generation where policy updates reshape output distribution\. Extractive methods preserve source tokens and sentiment by design, serving as upper bounds rather than optimization targets\. Comparative results appear in Table[2](https://arxiv.org/html/2608.15530#S5.T2)\.

## 4Experimental Setup

To empirically investigate sentiment drift, we conduct experiments using state\-of\-the\-art LLMs and established summarization benchmarks\. Our setup is designed to track the model’s evolution from a supervised baseline to an RL\-optimized policy, and to validate proposed mitigation strategies\.

### 4\.1Datasets

We evaluate our framework on two datasets with high emotional variance:

- •Reddit TL;DR: A dataset of informal posts as used in\[[12](https://arxiv.org/html/2608.15530#bib.bib11)\]
- •CNN/DailyMail: A news summarization benchmark used to observe drift in a formal context

We filter for examples where the source text has a clear sentiment polarity using an external classifier to ensure the drift is measurable\. For multilingual analysis, we use parallel subsets translated and validated for eight typologically diverse languages \(English, Arabic, Finnish, French, German, Hungarian, Italian, and Spanish\)\.

### 4\.2Model Configuration

Our base models include Llama\-3\-8B and Mistral\-7B\-v0\.1\. We follow the standard RLHF pipeline using the TRL library\. The KL\-penalty coefficientβ\\betais varied \(β∈\{0\.05,0\.1,0\.2\}\\beta\\in\\\{0\.05,0\.1,0\.2\\\}\) to study its impact on neutrality\. Hyperparameters are summarized in Table[1](https://arxiv.org/html/2608.15530#S4.T1)\.

Table 1:Hyperparameters used for RLHF training\.
### 4\.3Evaluation Metrics

To quantify both summarization quality and sentiment preservation, we employ a combination of discrete and distributional metrics:

- •Sentiment Variance \(SV\): Captures changes in affective intensity across the dataset\.
- •Jensen\-Shannon Divergence \(JSD\): Measures distributional shifts between sentiment predictions of the source text and its summary \(Equation[3](https://arxiv.org/html/2608.15530#S3.E3)\)\.
- •ROUGE\-L: Assesses content overlap and overall summarization quality\.

Metrics are reported at both the aggregate level and per language\. Figure[1](https://arxiv.org/html/2608.15530#S5.F1)illustrates how increasing the KL coefficientβ\\betasystematically reduces sentiment variance\.

### 4\.4Policy Attribution Implementation

For gradient\-based attribution, we use the Captum library\[[13](https://arxiv.org/html/2608.15530#bib.bib12)\]\. We compute Integrated Gradients for the top\-10% of tokens with the highest sentiment mass\. The integration is performed over 50 steps to ensure convergence\. Attribution scores are aggregated over sentiment\-bearing tokens identified via a multilingual sentiment lexicon and classifier confidence threshold \(p\>0\.8p\>0\.8\)\.

### 4\.5DPO Experimental Setup

To test whether sentiment drift is specific to PPO\-based RLHF or generalizes to direct alignment methods, we train additional models using Direct Preference Optimization \(DPO\)\[[9](https://arxiv.org/html/2608.15530#bib.bib8)\]\. We use the same base models \(Llama\-3\-8B, Mistral\-7B\) and datasets, with preference pairs constructed from the Reddit TL;DR and CNN/DailyMail validation sets following the protocol of\[[9](https://arxiv.org/html/2608.15530#bib.bib8)\]\. The DPO objective is:

ℒDPO\(θ\)=−E\(x,yw,yl\)\[logσ\(β​log⁡πθ​\(yw\|x\)πref​\(yw\|x\)−βlogπθ​\(yl\|x\)πref​\(yl\|x\)\)\]\\begin\{split\}\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\theta\)=\-\\operatorname\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\}\\big\[\\log\\sigma\\big\(&\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\\\\ &\-\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\\big\)\\big\]\\end\{split\}\(6\)
whereywy\_\{w\}andyly\_\{l\}are preferred and dispreferred summaries, respectively\. We varyβ∈\{0\.1,0\.2,0\.5\}\\beta\\in\\\{0\.1,0\.2,0\.5\\\}and evaluate sentiment drift using the same metrics\. Results are reported in Table[6](https://arxiv.org/html/2608.15530#S5.T6)\.

### 4\.6Mitigation Experiment: Sentiment\-Aware Regularization

Building on our attribution analysis, we propose and validate aSentiment\-Aware KL Regularizationtechnique\. The key insight is that KL penalty should be relaxed for tokens identified as sentiment\-bearing\. We modify Equation[2](https://arxiv.org/html/2608.15530#S3.E2)as follows:

JSA\(θ\)=Ex,y\[Rϕ\(x,y\)−β⋅\(1−γ⋅Isent\(ti\)\)⋅KL\(πθ∥πref\)\]\\begin\{split\}J\_\{\\text\{SA\}\}\(\\theta\)=\\operatorname\{E\}\_\{x,y\}\\big\[R\_\{\\phi\}\(x,y\)\\\\ \-\\beta\\cdot\(1\-\\gamma\\cdot\\operatorname\{I\}\_\{\\text\{sent\}\}\(t\_\{i\}\)\)\\cdot\\text\{KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)\\big\]\\end\{split\}\(7\)
whereIsent⁡\(ti\)=1\\operatorname\{I\}\_\{\\text\{sent\}\}\(t\_\{i\}\)=1if tokentit\_\{i\}is identified as sentiment\-bearing \(via lexicon or classifier confidence\>0\.8\>0\.8\), andγ∈\[0,1\]\\gamma\\in\[0,1\]controls the strength of relaxation\. We setγ=0\.3\\gamma=0\.3based on a small validation sweep\. This formulation reduces the KL penalty for affective tokens, allowing the policy to retain emotional expressivity when it aligns with the reward signal\.

We train models with this modified objective on Reddit TL;DR and CNN/DailyMail, keeping all other hyperparameters identical to the baseline RLHF setup\. Results demonstrating reduced sentiment drift \(18\-22% improvement\) without ROUGE\-L degradation are reported in Table[7](https://arxiv.org/html/2608.15530#S5.T7)\.

## 5Results and Analysis

This section analyzes sentiment drift under different training regimes, focusing on systematic patterns induced by RLHF optimization\.

### 5\.1Quantitative Sentiment Drift

Figure[1](https://arxiv.org/html/2608.15530#S5.F1)shows that higher KL coefficientβ\\betavalues induce stronger neutrality bias across both datasets\. Table[2](https://arxiv.org/html/2608.15530#S5.T2)confirms that abstractive summaries exhibit the largest sentiment variance reduction, while extractive approaches better preserve affective cues\. Cross\-lingual results \(Table[3](https://arxiv.org/html/2608.15530#S5.T3)\) reveal that morphologically rich languages \(Finnish, Hungarian\) show more pronounced suppression, suggesting inflectional sentiment encoding is particularly vulnerable\. These findings align with the observations of\[[6](https://arxiv.org/html/2608.15530#bib.bib15)\], who reported that sentiment attenuation in summarization is most severe for languages with complex morphological systems that encode affective meaning through inflectional patterns\. Overall, RLHF reduces sentiment variance by 30\-40% versus SFT baseline, with the magnitude of reduction correlating with morphological complexity as identified in prior cross\-lingual work\.

Table 2:Effect of summarization strategy on sentiment \(SV\)\.Table 3:Sentiment drift across languages after RLHF optimization\.![Refer to caption](https://arxiv.org/html/2608.15530v1/sv_vs_beta.png)Figure 1:Sentiment variance as a function of the KL coefficientβ\\beta\. Higherβ\\betavalues induce stronger neutrality bias\.![Refer to caption](https://arxiv.org/html/2608.15530v1/cross_language_drift.png)Figure 2:Distribution of sentiment drift across eight languages\. Morphologically rich languages \(Finnish, Hungarian\) exhibit more pronounced suppression\.
### 5\.2Impact of KL Regularization

Table[4](https://arxiv.org/html/2608.15530#S5.T4)isolates the KL effect: increasingβ\\betaconsistently reduces sentiment variance while marginally improving ROUGE\-L, confirming neutrality as a low\-risk optimization strategy\.

Table 4:Effect of KL regularization on sentiment drift and summarization quality\.
### 5\.3Policy Attribution Results

Table[5](https://arxiv.org/html/2608.15530#S5.T5)shows that KL\-divergence accounts for≈\\approx82% of negative attribution on sentiment\-bearing tokens\. Figure[3](https://arxiv.org/html/2608.15530#S5.F3)visualizes this: tokens with strong emotional polarity consistently receive negative gradients, indicating active suppression\. These findings align with theoretical analyses of KL\-regularized RL\[[15](https://arxiv.org/html/2608.15530#bib.bib14)\]\.

Table 5:Attribution mass assigned to components of the RL objective\.![Refer to caption](https://arxiv.org/html/2608.15530v1/attribution_heatmap.png)Figure 3:Token\-level attribution scores aggregated over sentiment\-bearing tokens\. Negative values indicate suppression by the KL penalty\.
### 5\.4DPO and Mitigation Experiments

Table[6](https://arxiv.org/html/2608.15530#S5.T6)shows sentiment suppression persists under DPO \(25\-30% variance reduction vs\. 30\-40% for PPO\), indicating neutrality bias is a broader consequence of preference optimization\. Our Sentiment\-Aware KL Regularization \(Table[7](https://arxiv.org/html/2608.15530#S5.T7)\) reduces drift by 18\-22% while maintaining ROUGE\-L within 0\.5 points of baseline\. Mitigation is most effective for morphologically rich languages\.

Table 6:Comparison of sentiment drift between PPO and DPO alignment methods\.Table 7:Compact version of mitigation results\.
### 5\.5Human and Qualitative Evaluation

Human evaluation \(50 pairs, 3 annotators\) confirms RLHF summaries are perceived as more neutral \(mean=2\.1\) vs\. SFT \(mean=3\.4\), while SA\-KL partially restores expressivity \(mean=2\.9; Fleiss’κ\\kappa=0\.62\)\. Figure[4](https://arxiv.org/html/2608.15530#S5.F4)shows representative examples: RLHF preserves factual content while replacing emotionally charged expressions with neutral paraphrases\.

![Refer to caption](https://arxiv.org/html/2608.15530v1/qualitative_comparison.png)Figure 4:Qualitative examples of sentiment attenuation\. Emotionally charged expressions \(highlighted\) are replaced with neutral paraphrases after RLHF\. \(a\) Reddit TL;DR example 1; \(b\) Reddit TL;DR example 2; \(c\) CNN/DailyMail example 1; \(d\) CNN/DailyMail example 2\.

## 6Discussion

### 6\.1The Neutrality Trap as a Statistical Equilibrium

The observed sentiment drift can be interpreted as a statistical equilibrium induced by RLHF optimization\. When reward models aggregate heterogeneous human preferences, emotionally neutral expressions become invariant features that minimize expected penalty across annotators\. From this perspective, neutrality functions as a stable attractor in policy space, favoring reward predictability over expressive fidelity\. Our attribution analysis confirms that the KL\-divergence penalty is the primary driver of this equilibrium, accounting for≈\\approx82% of negative attribution on sentiment\-bearing tokens\.

### 6\.2The Role of Regularization and KL\-Divergence

KL\-divergence plays a central role in shaping policy behavior during RLHF training\. By penalizing deviations from the reference model, the KL term suppresses low\-frequency and high\-entropy tokens\. Recent analyses demonstrate that this mechanism systematically biases policies toward conservative action distributions\[[15](https://arxiv.org/html/2608.15530#bib.bib14)\]\. Our results indicate that sentiment\-bearing tokens disproportionately fall into this suppressed category, explaining why affective intensity is reduced even when overall summarization quality improves\. This suggests that future work should explore token\-level adaptive regularization rather than uniform constraints\.

### 6\.3Generalization Across Alignment Methods

Our DPO analysis reveals that sentiment suppression persists across alignment methods, though the magnitude varies \(25\-30% for DPO vs\. 30\-40% for PPO\)\. This suggests that the neutrality bias is not solely an artifact of the PPO\-KL penalty structure, but a broader consequence of preference optimization favoring conservative outputs\. The implicit regularization in DPO induces similar pressures, indicating that affective flattening is a fundamental challenge in aligning LLMs, not just a quirk of PPO\.

The cross\-lingual consistency of this phenomenon further reinforces the findings of\[[6](https://arxiv.org/html/2608.15530#bib.bib15)\], who demonstrated that sentiment attenuation during summarization is a robust cross\-linguistic effect, independent of the specific summarization architecture\. Their work established that the degree of sentiment preservation correlates with morphological complexity, a pattern we observe consistently across our RLHF\-optimized models\. This suggests that the fundamental challenge of preserving affective content during text compression is amplified by preference optimization, regardless of the specific alignment algorithm employed\.

### 6\.4Effectiveness of Mitigation Strategies

Our proposed Sentiment\-Aware KL Regularization demonstrates that targeted intervention can partially reverse sentiment drift \(18\-22% reduction\) without compromising summarization quality\. However, mitigation is incomplete: SA\-KL restores only part of the lost variance\. This implies that the reward model itself may contribute to neutrality bias, either through training data favoring neutral responses or architectural limitations\. Future work should explore joint optimization of reward models and policy regularization for affect\-aware alignment\.

### 6\.5Implications for Downstream Applications

The systematic suppression of affective content poses challenges for applications relying on emotional fidelity: \(1\)Sentiment Analysis: Biased inputs for classifiers; \(2\)News/Media: Failure to convey urgency; \(3\)Customer Feedback: Obscured critical signals; \(4\)Mental Health: Harmful affective flattening\. These implications underscore the need for affect\-sensitive objectives in RL\-based pipelines\.

### 6\.6Adherence to Reporting Standards

All experiments follow standards for transparent reporting\. We report aggregate and per\-language metrics and visualize key trends\. Our Policy Attribution toolkit will be made publicly available to support reproducibility\.

## 7Conclusion

In this work, we investigated the phenomenon of sentiment drift in summarization systems optimized with Reinforcement Learning from Human Feedback \(RLHF\)\. We demonstrated that, while RLHF improves fluency, safety, and overall perceived quality, it systematically suppresses affective content, leading to increasingly neutral summaries\. This effect poses a significant challenge for downstream tasks such as sentiment analysis, where emotional fidelity is critical\.

Through extensive experiments across multiple datasets and training configurations, we showed that sentiment drift is not incidental but emerges as a direct consequence of the optimization objective\. In particular, KL regularization plays a central role in discouraging sentiment\-bearing tokens, effectively steering the policy toward low\-variance, neutral outputs\. Our quantitative and qualitative analyses confirm that this behavior is consistent across models, domains, and eight typologically diverse languages, corroborating and extending the cross\-lingual analysis of\[[6](https://arxiv.org/html/2608.15530#bib.bib15)\], who first identified systematic patterns of sentiment attenuation across languages in standard summarization systems\.

To better understand this phenomenon, we introducedPolicy Attribution, a framework that enables fine\-grained analysis of how different components of the RL objective influence affective expression\. Our attribution analysis reveals that the KL\-divergence penalty accounts for≈\\approx82% of negative attribution on sentiment\-bearing tokens, establishing it as the primary driver of neutrality bias\.

We extended our analysis to Direct Preference Optimization \(DPO\), showing that sentiment suppression persists across alignment methods, though with slightly reduced magnitude \(25\-30% for DPO vs\. 30\-40% for PPO\)\. This suggests that the neutrality bias is a broader consequence of preference optimization rather than an artifact specific to PPO\.

Finally, we proposed and experimentally validated aSentiment\-Aware KL Regularizationstrategy that reduces sentiment drift by 18\-22% without degrading summary quality \(ROUGE\-L within 0\.5 points of baseline\)\. Our findings underscore the importance of incorporating affect\-sensitive objectives into RL\-based text generation pipelines, particularly for applications in which sentiment preservation is essential\.

We release our Policy Attribution toolkit and evaluation scripts to support reproducibility and future research on affect\-aware alignment\. We believe that this work lays the foundation for developing alignment methods that balance safety, fluency, and emotional expressivity in large language models\.

## Acknowledgments

The work was done with partial support from grants 20260626, 20260643, 20260367, and 20260496 by the Secretary of Research and Postgraduate Studies \(SIP\) of Instituto Politécnico Nacional, Mexico\.

## Limitations

While our study provides a systematic diagnosis of sentiment drift in RLHF\-based summarization, several limitations merit discussion\.

1. 1\.Scope of summarization tasks\.Our experiments focus exclusively on single\-document summarization over Reddit TL;DR and CNN/DailyMail\. We do not examine multi\-document, query\-focused, or dialogue summarization, where affective dynamics may differ, for instance, due to inter\-speaker sentiment interaction or topical framing effects\.
2. 2\.Sentiment modeling assumptions\.We rely on pretrained, static sentiment classifiers for eight languages\. Although we fixed the models to isolate summarizer\-induced drift, their architectures, training data, and label schemata \(e\.g\., 3\-class vs\. fine\-grained intensity scales\) inevitably introduce variance\. Moreover, these classifiers capture explicit sentiment but may overlook implicit, context\-dependent, or culturally nuanced affect \(e\.g\., irony, understatement\), especially in morphologically rich languages like Finnish or Arabic\. A limitation, noted by reviewers, is the lack of explicit correlation analysis between JSD and human judgments of sentiment preservation, which we leave for future work\.
3. 3\.Human evaluation scale\.While we conducted a small\-scale human evaluation \(50 summary pairs, 3 annotators\), larger\-scale studies with diverse annotator pools would provide more robust validation of perceived neutrality and emotional fidelity\.
4. 4\.Generalization beyond summarization\.Though neutrality bias is likely pervasive in RLHF\-tuned generators \(e\.g\., chatbots, translators\), our conclusions are grounded in summarization\. Whether the same "low\-risk token" explanation holds for open\-ended generation, where length, coherence, and safety interact differently, remains an open question\.
5. 5\.Computational costs\.Policy Attribution requires gradient decomposition over 50 integration steps per token, which adds computational overhead\. Future work should explore efficient approximation methods for real\-time diagnostics\.

Addressing these limitations presents promising avenues for future research, particularly in developing human\-in\-the\-loop diagnostics and cross\-task generalizations of affect\-aware alignment\.

## Referencias

- \[1\]C\. Braud, G\. Carenini, and M\. Amblard\(2024\)Semi\-supervised Discourse Parsing with Self\-Training Strategies\.InProceedings of the 2024 Conference on Computational Natural Language Learning / COLING,Stroudsburg, PA\.Cited by:[§2\.4](https://arxiv.org/html/2608.15530#S2.SS4.p1.1)\.
- \[2\]N\. Calderon and R\. Reichart\(2025\)Trends in NLP Model Interpretability in the Era of LLMs\.InProceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Stroudsburg, PA,pp\. 656–693\.Cited by:[§2\.3](https://arxiv.org/html/2608.15530#S2.SS3.p1.1)\.
- \[3\]K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. Kiela\(2024\)KTO: Model Alignment as Prospect Theoretic Optimization\.InProceedings of the 41st International Conference on Machine Learning,ICML ’24,Cambridge, MA\.Cited by:[§2\.1](https://arxiv.org/html/2608.15530#S2.SS1.p4.1)\.
- \[4\]L\. Gao, J\. Schulman, and J\. Hilton\(2023\)Scaling Laws for Reward Model Overoptimization\.InProceedings of the 40th International Conference on Machine Learning,ICML ’23,Cambridge, MA,pp\. 10835–10866\.Cited by:[3rd item](https://arxiv.org/html/2608.15530#S1.I1.i3.p1.1),[§1\.2](https://arxiv.org/html/2608.15530#S1.SS2.p1.1),[§2\.1](https://arxiv.org/html/2608.15530#S2.SS1.p3.1),[§2\.2](https://arxiv.org/html/2608.15530#S2.SS2.p1.1)\.
- \[5\]S\. Gehrmann, Y\. Deng, and A\. Rush\(2019\)Summarization with Facet\-Aware Attention\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Stroudsburg, PA,pp\. 1–11\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1001)Cited by:[§2\.4](https://arxiv.org/html/2608.15530#S2.SS4.p1.1)\.
- \[6\]M\. Krasitskii, G\. Sidorov, O\. Kolesnikova, L\. Chanona\-Hernández, and A\. F\. Gelbukh\(2026\)Multilingual sentiment analysis of summarized texts: a cross\-language study of text shortening effects\.PeerJ Computer Science12,pp\. e3406\.External Links:[Document](https://dx.doi.org/10.7717/peerj-cs.3406),[Link](https://peerj.com/articles/cs-3406/),ISSN 2376\-5992Cited by:[§3\.2](https://arxiv.org/html/2608.15530#S3.SS2.p5.1),[§5\.1](https://arxiv.org/html/2608.15530#S5.SS1.p1.1),[§6\.3](https://arxiv.org/html/2608.15530#S6.SS3.p2.1),[§7](https://arxiv.org/html/2608.15530#S7.p2.1)\.
- \[7\]X\. Liu, Y\. Liu, and Y\. Chen\(2021\)Sentiment\-Controllable Text Generation\.InProceedings of the 35th AAAI Conference on Artificial Intelligence,Vol\.35,Palo Alto, CA,pp\. 13267–13275\.Cited by:[§2\.4](https://arxiv.org/html/2608.15530#S2.SS4.p1.1)\.
- \[8\]S\. M\. Lundberg and S\. Lee\(2017\)A Unified Approach to Interpreting Model Predictions\.InAdvances in Neural Information Processing Systems,Vol\.30,Red Hook, NY,pp\. 4765–4774\.Cited by:[§2\.3](https://arxiv.org/html/2608.15530#S2.SS3.p1.1)\.
- \[9\]R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn\(2023\)Direct Preference Optimization: Your Language Model is Secretly a Reward Model\.InAdvances in Neural Information Processing Systems,Vol\.36,Red Hook, NY\.Cited by:[§2\.1](https://arxiv.org/html/2608.15530#S2.SS1.p4.1),[§4\.5](https://arxiv.org/html/2608.15530#S4.SS5.p1.1)\.
- \[10\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal Policy Optimization Algorithms\.arXivabs/1707\.06347\.External Links:1707\.06347Cited by:[§2\.1](https://arxiv.org/html/2608.15530#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.15530#S3.SS1.p1.1)\.
- \[11\]T\. Shen, T\. Lei, R\. Barzilay, and T\. Jaakkola\(2017\)Style Transfer from Non\-Parallel Text by Cross\-Alignment\.InAdvances in Neural Information Processing Systems,Vol\.30,Red Hook, NY,pp\. 6830–6841\.Cited by:[§2\.4](https://arxiv.org/html/2608.15530#S2.SS4.p1.1)\.
- \[12\]N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. Christiano\(2020\)Learning to Summarize from Human Feedback\.InAdvances in Neural Information Processing Systems,Vol\.33,Red Hook, NY,pp\. 3008–3021\.Cited by:[§1](https://arxiv.org/html/2608.15530#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.15530#S2.SS1.p1.1),[1st item](https://arxiv.org/html/2608.15530#S4.I1.i1.p1.1)\.
- \[13\]M\. Sundararajan, A\. Taly, and Q\. Yan\(2017\)Axiomatic Attribution for Deep Networks\.InProceedings of the 34th International Conference on Machine Learning,ICML ’17,Cambridge, MA,pp\. 3319–3328\.Cited by:[§2\.3](https://arxiv.org/html/2608.15530#S2.SS3.p1.1),[§3\.3](https://arxiv.org/html/2608.15530#S3.SS3.p5.1),[§4\.4](https://arxiv.org/html/2608.15530#S4.SS4.p1.1)\.
- \[14\]Q\. Zhang and M\. Du\(2025\)Attribution Explanations for Deep Neural Networks: A Theoretical Perspective\.arXivabs/2508\.07636\.External Links:2508\.07636,[Link](https://arxiv.org/html/2508.07636v1)Cited by:[§2\.3](https://arxiv.org/html/2608.15530#S2.SS3.p2.1)\.
- \[15\]H\. Zhao, C\. Ye, Q\. Gu, and T\. Zhang\(2024\)Sharp Analysis for KL\-Regularized Contextual Bandits and RLHF\.arXivabs/2411\.04625\.External Links:2411\.04625,[Link](https://arxiv.org/abs/2411.04625)Cited by:[§2\.2](https://arxiv.org/html/2608.15530#S2.SS2.p2.1),[§5\.3](https://arxiv.org/html/2608.15530#S5.SS3.p1.1),[§6\.2](https://arxiv.org/html/2608.15530#S6.SS2.p1.1)\.