VA-DPO: Valence-Arousal Direct Preference Optimization for Controllable Emotion Generation in Language Models

# Valence–Arousal Direct Preference Optimization for Controllable Emotion Generation in Language Models
Source: [https://arxiv.org/html/2608.20374](https://arxiv.org/html/2608.20374)
###### Abstract

How precisely can we tell a language model*how*to feel? Most work on emotional generation answers with a discrete label—happy, angry, sad—which cannot express a target like “mildly downcast but calm\.” We instead specify the desired affect as a continuous point\(v⋆,a⋆\)\(v^\{\\star\},a^\{\\star\}\)in the Valence–Arousal plane\[[13](https://arxiv.org/html/2608.20374#bib.bib13)\]and train the model to hit it\. Our method, VA\-DPO, is a small modification to Direct Preference Optimization\[[12](https://arxiv.org/html/2608.20374#bib.bib12)\]: a frozen VA regressor scores each sampled generation by its Euclidean distance to the target, we keep only candidate pairs whose distance gap clears a marginτ\\tau, and we optimize a LoRA adapter\[[7](https://arxiv.org/html/2608.20374#bib.bib7)\]with the ordinary DPO loss against a frozen reference\. The DPO objective itself is unchanged; what is new is how the preference data is built\. On Llama\-3\.1\-8B\-Instruct this cuts mean VA distance to the target by33%33\\%over system\-prompting and25%25\\%over few\-shot prompting, lifting valence/arousal correlation torv=0\.93r\_\{v\}\{=\}0\.93andra=0\.75r\_\{a\}\{=\}0\.75\. The gains carry over to Qwen3\-8B and Llama\-3\.2\-3B, and they do not come at the usual price: MMLU is unchanged \(Δ=\+0\.0\\Delta=\+0\.0\) and HellaSwag/TruthfulQA are preserved\. We release the code, configs, and the preference\-construction pipeline\.

## 1Introduction

Empathetic dialogue agents, narrative co\-writing tools, and emotion\-aware TTS front\-ends all need the same thing: a way to say not just*what*a model should write but*in what emotional register*\[[5](https://arxiv.org/html/2608.20374#bib.bib5),[3](https://arxiv.org/html/2608.20374#bib.bib3)\]\. The usual answer in NLP is a discrete label drawn from a handful of categories—joy, anger, sadness, and so on\. That is not how psychology tends to describe affect\.Russell \[[13](https://arxiv.org/html/2608.20374#bib.bib13)\]models emotion with two continuous axes,*valence*\(positive–negative\) and*arousal*\(calm–excited\), and this circumplex view is now standard in affective computing\. A categorical label has no way to separate “mildly melancholic and calm” from “deeply melancholic and agitated”; it flattens both intensity and the space between named emotions\.

The lightweight alternatives do not fully work either\. Telling a model to “respond happily,” or showing it a few emotional examples, gives some control, but the effect is uneven across prompts and decays over a conversation\[[3](https://arxiv.org/html/2608.20374#bib.bib3)\]\. RLHF could close the gap, yet it brings back a separate reward model and all of its fragility\. Direct Preference Optimization\[[12](https://arxiv.org/html/2608.20374#bib.bib12)\]avoids that: it folds the reward into the policy’s own log\-ratios and trains directly on preference pairs\(yw,yl\)\(y\_\{w\},y\_\{l\}\), whereywy\_\{w\}is preferred\. DPO is usually run on binary human preferences, but its Bradley–Terry backbone\[[1](https://arxiv.org/html/2608.20374#bib.bib1)\]does not care where the ordering comes from—any scalar utility that ranks two outputs will do\.

That is the opening we use\. Given a continuous target\(v⋆,a⋆\)∈\[−1,\+1\]2\(v^\{\\star\},a^\{\\star\}\)\\in\[\-1,\+1\]^\{2\}and a frozen VA regressorRR, we score a candidateyyby how far its predicted affect sits from the target,d​\(y;v⋆,a⋆\)=∥R​\(y\)−\(v⋆,a⋆\)∥2d\(y;v^\{\\star\},a^\{\\star\}\)=\\lVert R\(y\)\-\(v^\{\\star\},a^\{\\star\}\)\\rVert\_\{2\}, and form preference pairs by ranking candidates on this distance\. Two choices separate our pipeline from a naive “soft\-label DPO”:

1. 1\.Margin\-threshold pair selection\.We retain only pairs where\|d​\(yw\)−d​\(yl\)\|\>τ\|d\(y\_\{w\}\)\-d\(y\_\{l\}\)\|\>\\tau\. Near\-tie pairs add label noise without informative gradient; the threshold concentrates updates on pairs whose ordering is unambiguous in regressor space\.
2. 2\.Two\-layer regularization\.We apply DPO over LoRA adapters only\[[7](https://arxiv.org/html/2608.20374#bib.bib7)\], keeping the base model and all reference\-model weights frozen\. Combined with the DPOβ\\beta\-KL anchor, this gives two independent regularization knobs: a structural one \(low\-rank update\) and a distributional one \(KL to reference\)\.

In short, we contribute a continuous\-reward DPO recipe for the VA space with explicit pair\-construction rules \(§[3](https://arxiv.org/html/2608.20374#S3)\); a controlled comparison that pulls apart*which*ingredient matters, against system prompting, few\-shot prompting, SFT, and discrete\-label DPO, plus a no\-margin ablation; and a retention study \(MMLU, HellaSwag, TruthfulQA\) showing the control gains are not bought with degraded general ability\.

## 2Related Work

#### Dimensional emotion models and corpora\.

Russell \[[13](https://arxiv.org/html/2608.20374#bib.bib13)\]grounds the two\-dimensional view of affect\. EmoBank\[[2](https://arxiv.org/html/2608.20374#bib.bib2)\]provides∼\\sim10K English sentences with human Valence–Arousal–Dominance \(VAD\) annotations from both reader and writer perspectives\. We use EmoBank’s train split to fit our regressor and its test split for all reported metrics\. The NRC VAD Lexicon\[[11](https://arxiv.org/html/2608.20374#bib.bib11)\]provides per\-word VAD scores for∼\\sim20K English words and is the basis of our lexicon\-reward baseline \(B4 below\)\.

#### Preference optimization\.

DPO\[[12](https://arxiv.org/html/2608.20374#bib.bib12)\]reformulates RLHF as a closed\-form policy update against a frozen reference, removing the need for a learned reward model\. Several variants exist—IPO, KTO, and others—that change the loss formulation; we deliberately keep the standard DPO objective and modify only the*pair\-construction*step, isolating the contribution of continuous ranking and margin filtering\.

#### Emotion control in generative models\.

Gao et al\. \[[5](https://arxiv.org/html/2608.20374#bib.bib5)\]apply DPO to controllable emotional speech synthesis, contrasting preferred and dispreferred emotion classes on the speech side\. Their setting is discrete and audio\-modal; our method is text\-modal and continuous\.Konen et al\. \[[8](https://arxiv.org/html/2608.20374#bib.bib8)\]derive activation\-space “style vectors” for sentiment/emotion steering at inference time without training\. Such training\-free methods are appealing but rely on adding fixed offsets to hidden states; their effect is hard to calibrate quantitatively to a target\(v⋆,a⋆\)\(v^\{\\star\},a^\{\\star\}\)\.Fazzi et al\. \[[3](https://arxiv.org/html/2608.20374#bib.bib3)\]show that LLMs struggle to maintain extreme or shifting affective states across multi\-turn dialogue, motivating the need for fine\-tuning rather than prompt\-only control\.Sun et al\. \[[14](https://arxiv.org/html/2608.20374#bib.bib14)\]demonstrate that LLM internal representations already organize emotion along an approximately circular VA subspace and that steering along these axes monotonically shifts the affect of outputs; this corroborates the geometric assumption underlying our reward and is the closest training\-free comparison point we plan to include \(B6\)\.

## 3Method

### 3\.1Problem setup

Letxxbe an input prompt and\(v⋆,a⋆\)∈\[−1,\+1\]2\(v^\{\\star\},a^\{\\star\}\)\\in\[\-1,\+1\]^\{2\}a target in the Valence–Arousal plane\. We seek a generation policyπθ​\(y∣x,v⋆,a⋆\)\\pi\_\{\\theta\}\(y\\mid x,v^\{\\star\},a^\{\\star\}\)whose outputs, when scored by a fixed regressorR:𝒴→\[−1,\+1\]2R:\\mathcal\{Y\}\\to\[\-1,\+1\]^\{2\}, have small distance

d​\(y;v⋆,a⋆\)=∥R​\(y\)−\(v⋆,a⋆\)∥2,d\(y;v^\{\\star\},a^\{\\star\}\)\\;=\\;\\big\\lVert R\(y\)\-\(v^\{\\star\},a^\{\\star\}\)\\big\\rVert\_\{2\},\(1\)while remaining a fluent, on\-distribution response toxx\. We condition the policy on the target by formatting\(v⋆,a⋆\)\(v^\{\\star\},a^\{\\star\}\)as a text prefix appended to the prompt \(§[4](https://arxiv.org/html/2608.20374#S4)\)\.

### 3\.2Continuous reward and pair construction

We treat−d​\(y;v⋆,a⋆\)\-d\(y;v^\{\\star\},a^\{\\star\}\)as a scalar utility\. For each\(x,v⋆,a⋆\)\(x,v^\{\\star\},a^\{\\star\}\)in the training pool, we drawNNcandidate completions\{y\(1\),…,y\(N\)\}\\\{y^\{\(1\)\},\\ldots,y^\{\(N\)\}\\\}from a sampling distribution \(temperature\-TTnucleus sampling from the base policy; we use the reference model so the candidate set is independent of subsequent DPO updates and can be cached\)\. Each candidate is scored once withRR\.

To form preference pairs, we enumerate all\(N2\)\\binom\{N\}\{2\}unordered pairs and retain those with

\|d​\(y\(i\);v⋆,a⋆\)−d​\(y\(j\);v⋆,a⋆\)\|\>τ,\\bigl\|d\(y^\{\(i\)\};v^\{\\star\},a^\{\\star\}\)\-d\(y^\{\(j\)\};v^\{\\star\},a^\{\\star\}\)\\bigr\|\\;\>\\;\\tau,\(2\)labeling the smaller\-distance member as the winnerywy\_\{w\}and the larger as the loseryly\_\{l\}\. The hyperparameterτ∈ℝ≥0\\tau\\in\\mathbb\{R\}\_\{\\geq 0\}is the*margin threshold*\. Settingτ=0\\tau=0recovers the naive variant \(B5\) where every non\-tied pair becomes a training example; positiveτ\\taudiscards near\-ties whose ordering is dominated by regressor noise\. We sweepτ\\tauin\{0,0\.1,0\.2,0\.3,0\.5\}\\\{0,0\.1,0\.2,0\.3,0\.5\\\}\(§[5](https://arxiv.org/html/2608.20374#S5)\)\.

### 3\.3Training objective

Given the filtered pair set𝒟pref=\{\(xi,vi⋆,ai⋆,yw,i,yl,i\)\}\\mathcal\{D\}\_\{\\mathrm\{pref\}\}=\\\{\(x\_\{i\},v^\{\\star\}\_\{i\},a^\{\\star\}\_\{i\},y\_\{w,i\},y\_\{l,i\}\)\\\}, we optimize the standard DPO loss\[[12](https://arxiv.org/html/2608.20374#bib.bib12)\]

ℒDPO​\(θ\)=−𝔼𝒟pref​\[log⁡σ​\(β​log⁡πθ​\(yw∣x,v⋆,a⋆\)πref​\(yw∣x,v⋆,a⋆\)−β​log⁡πθ​\(yl∣x,v⋆,a⋆\)πref​\(yl∣x,v⋆,a⋆\)\)\],\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{DPO\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\mathrm\{pref\}\}\}\\Big\[\\log\\sigma\\\!\\big\(\\beta\\log\\tfrac\{\\pi\_\{\\theta\}\(y\_\{w\}\\mid x,v^\{\\star\},a^\{\\star\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{w\}\\mid x,v^\{\\star\},a^\{\\star\}\)\}\-\\beta\\log\\tfrac\{\\pi\_\{\\theta\}\(y\_\{l\}\\mid x,v^\{\\star\},a^\{\\star\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{l\}\\mid x,v^\{\\star\},a^\{\\star\}\)\}\\big\)\\Big\],\(3\)whereσ​\(⋅\)\\sigma\(\\cdot\)is the logistic sigmoid andπref\\pi\_\{\\mathrm\{ref\}\}is a frozen copy of the base model\. Only LoRA adapter weights\[[7](https://arxiv.org/html/2608.20374#bib.bib7)\]onπθ\\pi\_\{\\theta\}receive gradients; the base model, the reference model, and the regressorRRare all frozen throughout DPO training\.

Algorithm 1VA\-DPO training \(one outer pass over the prompt pool\)1:prompt pool

𝒫\\mathcal\{P\}; target VA distribution

p​\(v⋆,a⋆\)p\(v^\{\\star\},a^\{\\star\}\); reference

πref\\pi\_\{\\mathrm\{ref\}\}; regressor

RR;

NN,

TT,

τ\\tau,

β\\beta\.

2:

𝒟pref←∅\\mathcal\{D\}\_\{\\mathrm\{pref\}\}\\leftarrow\\emptyset
3:foreach

x∈𝒫x\\in\\mathcal\{P\}, sampling

\(v⋆,a⋆\)∼p\(v^\{\\star\},a^\{\\star\}\)\\sim pdo

4:sample

\{y\(k\)\}k=1N\\\{y^\{\(k\)\}\\\}\_\{k=1\}^\{N\}from

πref\(⋅∣x,v⋆,a⋆\)\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x,v^\{\\star\},a^\{\\star\}\)with temperature

TT
5:

d\(k\)←∥R​\(y\(k\)\)−\(v⋆,a⋆\)∥2d^\{\(k\)\}\\leftarrow\\lVert R\(y^\{\(k\)\}\)\-\(v^\{\\star\},a^\{\\star\}\)\\rVert\_\{2\}for

k=1,…,Nk=1,\\ldots,N
6:foreach pair

\(i,j\)\(i,j\)with

\|d\(i\)−d\(j\)\|\>τ\|d^\{\(i\)\}\-d^\{\(j\)\}\|\>\\taudo

7:

\(yw,yl\)←\(y\_\{w\},y\_\{l\}\)\\leftarrowpair sorted by

ddascending

8:add

\(x,v⋆,a⋆,yw,yl\)\(x,v^\{\\star\},a^\{\\star\},y\_\{w\},y\_\{l\}\)to

𝒟pref\\mathcal\{D\}\_\{\\mathrm\{pref\}\}
9:endfor

10:endfor

11:train LoRA adapters

θ\\thetaby minimizing Eq\. \([3](https://arxiv.org/html/2608.20374#S3.E3)\) on

𝒟pref\\mathcal\{D\}\_\{\\mathrm\{pref\}\}
12:returnadapter weights

θ\\theta

#### What is and is not new\.

Equation \([3](https://arxiv.org/html/2608.20374#S3.E3)\) is the standard DPO loss; we do not modify it\. Our contribution is the*construction*of𝒟pref\\mathcal\{D\}\_\{\\mathrm\{pref\}\}from a continuous scalar \(Eq\. \([1](https://arxiv.org/html/2608.20374#S3.E1)\)\) with an explicit margin filter \(Eq\. \([2](https://arxiv.org/html/2608.20374#S3.E2)\)\), together with the regressor\-vs\-lexicon and margin\-on\-vs\-off ablations that isolate each design choice \(§[5](https://arxiv.org/html/2608.20374#S5)\)\.

## 4Experimental Setup

#### Base model\.

Primary:meta\-llama/Llama\-3\.1\-8B\-Instruct\[[6](https://arxiv.org/html/2608.20374#bib.bib6)\]\. Robustness check \#1:Qwen/Qwen3\-8B\[[15](https://arxiv.org/html/2608.20374#bib.bib15)\]\. Robustness check \#2:meta\-llama/Llama\-3\.2\-3B\-Instruct\. All three are open\-weight, instruction\-tuned, English\-capable dense models\. We deliberately match the primary and robustness \#1 backbones to those used in the closest prior work on LLM valence–arousal geometry\[[14](https://arxiv.org/html/2608.20374#bib.bib14)\], so that our trained policy and their training\-free steering result \(our B6 baseline\) operate on*identical weights*and can be compared head\-to\-head\. Robustness \#2 matches the Llama\-family 3B condition inFazzi et al\. \[[3](https://arxiv.org/html/2608.20374#bib.bib3)\], demonstrating that the method transfers across both family \(Llama↔\\leftrightarrowQwen\) and scale \(8B↔\\leftrightarrow3B\)\. Each 8B ablation cell completes in∼\\sim1–2 hours and each 3B cell in∼\\sim30 minutes on a single NVIDIA H100 80 GB in full bfloat16 \(no quantization\)\.

#### Reward regressor\.

A RoBERTa\-large\[[9](https://arxiv.org/html/2608.20374#bib.bib9)\]encoder with a 2\-dim linear regression head, fine\-tuned for 5 epochs on the EmoBank\[[2](https://arxiv.org/html/2608.20374#bib.bib2)\]train split with MSE loss after rescaling the human VAD ratings from\[1,5\]\[1,5\]to\[−1,\+1\]\[\-1,\+1\]viax′=\(x−3\)/2x^\{\\prime\}=\(x\-3\)/2\. We report the regressor’s own CCC and Pearsonrron EmoBank dev \(§[5](https://arxiv.org/html/2608.20374#S5)\) so reviewers can calibrate the signal quality\. Regressor weights are frozen for the entirety of preference construction and DPO training\.

#### Conditioning format\.

Targets are prepended to the prompt as`\[VA: v\*=<float\>, a\*=<float\>\]`\. This keeps the conditioning fully text\-based and reproducible, at the cost of a fixed 4–6 token overhead per example\.

#### Hyperparameter ranges \(swept\)\.

β∈\{0\.05,0\.1,0\.2,0\.5\}\\beta\\in\\\{0\.05,0\.1,0\.2,0\.5\\\},τ∈\{0\.0,0\.1,0\.2,0\.3,0\.5\}\\tau\\in\\\{0\.0,0\.1,0\.2,0\.3,0\.5\\\}, LoRA rankr∈\{4,8,16,32\}r\\in\\\{4,8,16,32\\\}withα=2​r\\alpha=2r, learning rate∈\{1,5\}×10−5∪\{1×10−4\}\\in\\\{1,5\\\}\\times 10^\{\-5\}\\cup\\\{1\\times 10^\{\-4\}\\\}, batch size8​–​328\\text\{\-\-\}32\(effective via gradient accumulation\)\. Optimizer: AdamW\[[10](https://arxiv.org/html/2608.20374#bib.bib10)\]with decoupled weight decay0on LoRA parameters\. Temperature for candidate generation:T=0\.9T=0\.9\. Candidates per prompt:N∈\{4,8\}N\\in\\\{4,8\\\}\. All non\-swept settings are reported in Appendix[A](https://arxiv.org/html/2608.20374#A1)\.

#### Baselines\.

B0: base model with system prompt “*Respond with valencev⋆v^\{\\star\}and arousala⋆a^\{\\star\}\.*”B1: B0 \+ 4 in\-context exemplars\.B2: SFT on\(x,v⋆,a⋆,y\)\(x,v^\{\\star\},a^\{\\star\},y\)triples whereyyis the EmoBank reference sentence \(LoRA, same rank as ours\)\.B3: DPO with VA bucketed into 4 quadrants \(sign ofv⋆v^\{\\star\}anda⋆a^\{\\star\}\)\.B5: VA\-DPO withτ=0\\tau=0\(no margin filter\)\. Two further baselines are defined but not run in this version:B4\(VA\-DPO with the NRC VAD Lexicon\[[11](https://arxiv.org/html/2608.20374#bib.bib11)\]as reward instead of the regressor\) andB6\(training\-free VA steering in the spirit ofSun et al\. \[[14](https://arxiv.org/html/2608.20374#bib.bib14)\]\); we discuss them as future comparisons\.

#### Evaluation\.

Primary metric: mean Euclidean VA distanced¯=1\|𝒯\|​∑\(x,v⋆,a⋆\)∈𝒯∥R​\(y\)−\(v⋆,a⋆\)∥2\\bar\{d\}=\\tfrac\{1\}\{\|\\mathcal\{T\}\|\}\\sum\_\{\(x,v^\{\\star\},a^\{\\star\}\)\\in\\mathcal\{T\}\}\\lVert R\(y\)\-\(v^\{\\star\},a^\{\\star\}\)\\rVert\_\{2\}on the EmoBank test set𝒯\\mathcal\{T\}, where targets are the ground\-truth EmoBank VAD values \(V and A only\)\. Secondary: per\-dimension Pearsonrrand Concordance Correlation Coefficient \(CCC\)\. Generation quality: perplexity under a held\-out reference LM, distinct\-1/2/3, and an LLM\-as\-judge rating\[[16](https://arxiv.org/html/2608.20374#bib.bib16)\]on a custom prompt set of 50 items\. General capability: MMLU \(5\-shot\), HellaSwag \(10\-shot\), and TruthfulQA, all run throughlm\-evaluation\-harness\[[4](https://arxiv.org/html/2608.20374#bib.bib4)\]with identical prompts across all checkpoints\. We report mean±\\pmstd over33training seeds\.

#### Compute\.

All experiments run on a single NVIDIA H100 80 GB in full bfloat16 \(no quantization\)\. The reward regressor trains in∼\\sim15 minutes; preference\-pair generation over the full EmoBank train split \(8,0628\{,\}062prompts,N=8N\{=\}8candidates each\) takes∼\\sim1\.2 hours; a single 8B VA\-DPO run is∼\\sim25–40 minutes and an 8B baseline a comparable amount, with the 3B backbone roughly2×2\\timesfaster\. End to end—the primary model with all baselines, theβ\\beta/τ\\tauablations, three seeds, two robustness backbones, and the MMLU/HellaSwag/TruthfulQA passes—fits in well under6060GPU\-hours; we report this in case the budget, rather than the method, is the practical constraint for a reader\. We did not measure failed or preliminary runs separately, but they were a small fraction of this total\.

#### Reproducibility\.

Every run is initialized from a logged seed and writes a manifest \(results/<run\>/manifest\.json\) capturing the git commit, the exact config YAML, the resolved Python environment, and the regressor checkpoint hash\.

## 5Results

All tables are auto\-generated bypaper/build\_tables\.pyfrom the loggedresults/CSVs\. The headline run uses the a\-priori default\(β,τ\)=\(0\.1,0\.2\)\(\\beta,\\tau\)=\(0\.1,0\.2\); we additionally report a 3\-seed estimate \(d¯=0\.092±0\.002\\bar\{d\}=0\.092\\pm 0\.002\)\. Baselines B4 \(lexicon reward\) and B6 \(training\-free steering\) are omitted from this version\.

### 5\.1Main comparison

Table[1](https://arxiv.org/html/2608.20374#S5.T1)reports the primary\-model comparison on EmoBank test\. VA\-DPO attains the lowest mean VA distance \(d¯=0\.092\\bar\{d\}=0\.092\) and the highest target–generation correlation \(rv=0\.93r\_\{v\}=0\.93,ra=0\.75r\_\{a\}=0\.75\), improving over the strongest non\-trained baseline \(few\-shot,0\.1220\.122\) by25%25\\%and over system\-prompting \(B0,0\.1380\.138\) by33%33\\%\. The ordering of the trained variants is monotone in the design choices the method adds: ours \(0\.0920\.092\)<<no\-margin DPO \(B5,0\.0970\.097\)<<discrete\-label DPO \(B3,0\.1070\.107\)<<SFT \(B2,0\.1310\.131\)\. This supports each contribution in turn: the continuous reward beats discrete buckets \(ours vs\. B3\), preference optimization beats supervised imitation \(ours/B5 vs\. B2\), and the margin filter gives a further gain at matched training\-pair count \(ours vs\. B5; both trained on5,0025\{,\}002pairs\)\.

Table 1:Main results on EmoBank test \(Llama\-3\.1\-8B\-Instruct,n=1000n=1000\)\.d¯\\bar\{d\}: mean VA distance \(lower is better\)\.rv/rar\_\{v\}/r\_\{a\}: Pearson correlation of generated vs\. target valence/arousal\.Δ\\DeltaMMLU: 5\-shot MMLU change vs\. the base model \(closer to0is better\)\. B4 \(lexicon\) and B6 \(steering\) are not included in this version\. Headline VA\-DPO is single\-config \(a\-priori defaultβ=0\.1\\beta\{=\}0\.1,τ=0\.2\\tau\{=\}0\.2\); over 3 seedsd¯=0\.092±0\.002\\bar\{d\}=0\.092\\pm 0\.002\.The rightmost column shows that VA\-DPO attains this control at*no cost*to general capability:Δ\\DeltaMMLU=\+0\.0=\+0\.0\. We expand this retention check to HellaSwag and TruthfulQA in §[6](https://arxiv.org/html/2608.20374#S6)\(Table[4](https://arxiv.org/html/2608.20374#S6.T4)\)\.

### 5\.2Robustness across backbones

Table[2](https://arxiv.org/html/2608.20374#S5.T2)repeats the core comparison on two further backbones\. VA\-DPO is the best method on all three\. On both 8B models the full method \(margin filter included\) is best; on the smaller Llama\-3\.2\-3B, VA\-DPO still clearly beats prompting \(B0\) and SFT \(B2\), but the margin filter no longer helps—the no\-margin variant \(B5\) edges it out—which we attribute to the margin reducing an already\-smaller usable pair set at 3B \(§[6](https://arxiv.org/html/2608.20374#S6)\)\.

Table 2:Mean VA distance \(d¯↓\\bar\{d\}\\downarrow\) across backbones\. VA\-DPO wins on every backbone; the margin\-filter advantage \(ours vs\. B5\) holds at 8B but inverts at 3B\.
### 5\.3Ablations:β\\betaandτ\\tau

Figure[1](https://arxiv.org/html/2608.20374#S5.F1)sweeps the two regularization knobs on the primary model\. Control is*robust toβ\\beta*over the tested range \(d¯∈\[0\.092,0\.099\]\\bar\{d\}\\in\[0\.092,0\.099\]forβ∈\{0\.05,0\.1,0\.2,0\.5\}\\beta\\in\\\{0\.05,0\.1,0\.2,0\.5\\\}\)\. The margin thresholdτ\\taushows a clear optimum nearτ≈0\.1\\tau\\approx 0\.1\(d¯=0\.080\\bar\{d\}=0\.080\), improving on both no filtering \(τ=0\\tau\{=\}0,0\.0970\.097\) and aggressive filtering \(τ=0\.5\\tau\{=\}0\.5,0\.1390\.139, where too few pairs survive\)\. This is direct evidence for margin\-thresholded selection, and indicates our a\-priori defaultτ=0\.2\\tau\{=\}0\.2is slightly conservative; we do not re\-selectτ\\tauon the test set \(§[6](https://arxiv.org/html/2608.20374#S6)\)\.

![Refer to caption](https://arxiv.org/html/2608.20374v1/x1.png)Figure 1:Ablations on Llama\-3\.1\-8B \(EmoBank test\)\. \(a\)d¯\\bar\{d\}is largely insensitive to the DPOβ\\beta\. \(b\) the margin thresholdτ\\tauhas a clear optimum near0\.10\.1;τ=0\\tau\{=\}0is the no\-filter baseline \(B5\) and overly largeτ\\taustarves the pair set\.
### 5\.4Generation quality and reward hacking

A continuous reward invites reward hacking: the policy could drived¯\\bar\{d\}down by collapsing onto a few high\-scoring emotional clichés\. We see no such collapse—lexical diversity is essentially intact \(distinct\-1/2/30\.170/0\.570/0\.8300\.170/0\.570/0\.830for VA\-DPO vs\.0\.182/0\.599/0\.8520\.182/0\.599/0\.852for B0\) and reference\-LM perplexity rises only modestly \(14\.914\.9vs\.12\.612\.6\)\. There is, however, a measurable quality cost\. An LLM judge \(Claude, blind to method,11–55scale,120120matched prompts\) rates VA\-DPO slightly below B0 on fluency \(4\.084\.08vs\.4\.364\.36\), relevance \(3\.113\.11vs\.3\.683\.68\), and naturalness \(3\.163\.16vs\.3\.523\.52\)\. In other words, VA\-DPO buys a large gain in emotional control \(VA distance−33%\-33\\%\) at a modest cost in generation quality, with the relevance drop the most notable—pushing affect can pull a response slightly off\-topic\. This trade\-off, not a quality collapse, is the honest characterization; theβ\\betaknob \(§[6](https://arxiv.org/html/2608.20374#S6)\) is the lever for tightening it\.

## 6Analysis

#### Is the win an artifact of the reward model?

Because the same regressorRRsupplies the training reward and the headline metric, an obvious worry is circularity—perhaps we only optimizeRR’s idiosyncrasies\. Three checks argue otherwise\. \(i\)*Held\-out regressor\.*We train a second regressor \(DeBERTa\-v3\-large, dev valence CCC0\.820\.82\) that never touched VA\-DPO training and re\-score the*same*generations with it \(Table[3](https://arxiv.org/html/2608.20374#S6.T3)\); the ranking is unchanged and VA\-DPO stays best \(d¯=0\.097\\bar\{d\}=0\.097vs\. the strongest baseline0\.1110\.111\)\. \(ii\)*Reward swap\.*Training VA\-DPO with the DeBERTa regressor as the reward instead of RoBERTa gives essentially the same result \(d¯=0\.093\\bar\{d\}=0\.093vs\.0\.0920\.092\), so the method is not tied to one reward model\. \(iii\)*Cross\-corpus\.*On SemEval\-2018 tweets—a different domain with independent categorical labels—the EmoBank\-trained regressor’s valence separates positive\- from negative\-emotion tweets with ROC\-AUC0\.970\.97\(RoBERTa\) and0\.980\.98\(DeBERTa\), so its valence axis is not an EmoBank\-only artifact\. Arousal remains the weaker axis throughout\.

Table 3:Robustness to the evaluator: mean VA distance under the RoBERTa reward regressor vs\. a held\-out DeBERTa regressor that never shaped training\. The ranking—and VA\-DPO’s lead—survive the swap\.
#### Capability retention\.

A central worry for any fine\-tuning method is silent damage to general ability\. Table[4](https://arxiv.org/html/2608.20374#S6.T4)shows VA\-DPO leaves it intact: MMLU is unchanged \(\+0\.0\+0\.0\), HellaSwag drops marginally \(−0\.6\-0\.6\), and TruthfulQA actually improves \(\+3\.1\+3\.1\)\. We attribute the resilience to the two regularizers acting together: the LoRA update is low\-rank, and the DPOβ\\beta\-KL term keeps the policy close to the frozen reference\. The headline control gain is therefore not bought with capability loss\.

Table 4:General\-capability retention \(accuracy, %\): base model vs\. VA\-DPO on Llama\-3\.1\-8B\. Emotion control is gained without degrading—and on TruthfulQA, while slightly improving—general ability\.
#### The margin threshold and scale\.

Theτ\\tausweep \(Fig\.[1](https://arxiv.org/html/2608.20374#S5.F1)b\) confirms that filtering near\-tie pairs helps: a small positive margin removes pairs whose ordering is dominated by regressor noise, and accuracy peaks nearτ=0\.1\\tau\{=\}0\.1\. Pushed too far, the filter discards informative pairs and control degrades\. This also explains the one place our full method does not win outright—Llama\-3\.2\-3B \(Table[2](https://arxiv.org/html/2608.20374#S5.T2)\), where the smaller model yields fewer usable candidate pairs, so the sameτ\\tauremoves a larger*fraction*of the training signal and the no\-margin variant is competitive\. The practical guidance is to scaleτ\\taudown as the candidate pool shrinks\.

#### β\\betaand the control–retention trade\-off\.

Within the swept range, control is largely insensitive toβ\\beta\(Fig\.[1](https://arxiv.org/html/2608.20374#S5.F1)a\) while retention stays high, indicating a wide stable operating region rather than a sharp trade\-off at this model scale; characterizing the regime whereβ\\betamust trade control for retention is left to future work\.

#### Failure modes\.

Per\-dimension correlations are consistently higher for valence than arousal \(e\.g\.rv=0\.93r\_\{v\}\{=\}0\.93vs\.ra=0\.75r\_\{a\}\{=\}0\.75for ours; the regressor itself is weaker on arousal, dev CCC0\.790\.79vs\.0\.550\.55\)\. Arousal control is thus the harder axis and the dominant residual error, consistent with arousal being harder to infer from text alone\.

## 7Limitations

Our method inherits any bias or systematic error of the VA regressorRR: ifRRunderestimates arousal for a class of expressions, VA\-DPO will not correct for it\. We use a single English regressor trained on EmoBank, a mixed\-genre but still narrow corpus; generalization to dialectal or domain\-specific text is untested\. LikeFazzi et al\. \[[3](https://arxiv.org/html/2608.20374#bib.bib3)\], we do not evaluate multi\-turn dialogue dynamics: a turn\-conditioned regressor or a turn\-level reward shaping would be needed for that\. Finally, the target VA is provided as a text prefix, which is simple and reproducible but may be brittle to prompt formatting changes; learned target embeddings are a natural extension we leave to future work\.

## 8Conclusion

We presented VA\-DPO, a continuous\-reward formulation of DPO for fine\-grained emotional control in the Valence–Arousal plane, built from a frozen VA regressor, a distance\-based scalar utility, and margin\-thresholded pair selection\. On Llama\-3\.1\-8B it cuts mean VA distance by33%33\\%over system\-prompting and outperforms few\-shot prompting, SFT, and discrete\-label DPO, with the contribution ordering \(continuous reward, preference optimization, margin filter\) showing up monotonically in the results—and at no cost to general capability \(Table[4](https://arxiv.org/html/2608.20374#S6.T4)\)\. The control gain replicates on Qwen3\-8B and Llama\-3\.2\-3B, with the margin\-filter component the one piece that does not transfer to the 3B scale, where the candidate pool is smaller\. Natural next steps are multi\-turn affect dynamics\[[3](https://arxiv.org/html/2608.20374#bib.bib3)\], multilingual VA control, and learned \(rather than text\-prefix\) target conditioning\.

## References

- Bradley and Terry \[1952\]Ralph Allan Bradley and Milton E\. Terry\.Rank analysis of incomplete block designs: I\. the method of paired comparisons\.*Biometrika*, 39\(3/4\):324–345, 1952\.doi:10\.2307/2334029\.
- Buechel and Hahn \[2017\]Sven Buechel and Udo Hahn\.EmoBank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis\.In*Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics \(EACL\), Vol\. 2: Short Papers*, pages 578–585, Valencia, Spain, 2017\.URL[https://aclanthology\.org/E17\-2092/](https://aclanthology.org/E17-2092/)\.
- Fazzi et al\. \[2025\]Gino Franco Fazzi, Julie Skoven Hinge, Stefan Heinrich, and Paolo Burelli\.Don’t get too excited – eliciting emotions in LLMs\.*arXiv preprint arXiv:2503\.02457*, 2025\.URL[https://arxiv\.org/abs/2503\.02457](https://arxiv.org/abs/2503.02457)\.
- Gao et al\. \[2024a\]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou\.The Language Model Evaluation Harness, 07 2024a\.URL[https://zenodo\.org/records/12608602](https://zenodo.org/records/12608602)\.
- Gao et al\. \[2024b\]Xiaoxue Gao, Chen Zou, and Eng Siong Chng\.Emo\-DPO: Controllable emotional speech synthesis through direct preference optimization\.*arXiv preprint arXiv:2409\.10157*, 2024b\.URL[https://arxiv\.org/abs/2409\.10157](https://arxiv.org/abs/2409.10157)\.
- Grattafiori et al\. \[2024\]Aaron Grattafiori, Abhimanyu Dubey, et al\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.URL[https://arxiv\.org/abs/2407\.21783](https://arxiv.org/abs/2407.21783)\.This paper covers the Llama\-3 family including the 8B variant we use as primary \(Llama\-3\.1\-8B\-Instruct\)\. The 3\.2 sub\-release that introducedLlama\-3\.2\-3B\-Instruct\(used as our second robustness backbone\) does not have a separate technical report\.
- Hu et al\. \[2022\]Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.LoRA: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.URL[https://arxiv\.org/abs/2106\.09685](https://arxiv.org/abs/2106.09685)\.
- Konen et al\. \[2024\]Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking\.Style vectors for steering generative large language models\.In*Findings of the Association for Computational Linguistics: EACL 2024*, pages 782–802, 2024\.URL[https://aclanthology\.org/2024\.findings\-eacl\.52/](https://aclanthology.org/2024.findings-eacl.52/)\.
- Liu et al\. \[2019\]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov\.RoBERTa: A robustly optimized BERT pretraining approach\.*arXiv preprint arXiv:1907\.11692*, 2019\.URL[https://arxiv\.org/abs/1907\.11692](https://arxiv.org/abs/1907.11692)\.
- Loshchilov and Hutter \[2019\]Ilya Loshchilov and Frank Hutter\.Decoupled weight decay regularization\.In*International Conference on Learning Representations \(ICLR\)*, 2019\.URL[https://arxiv\.org/abs/1711\.05101](https://arxiv.org/abs/1711.05101)\.
- Mohammad \[2018\]Saif M\. Mohammad\.Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words\.In*Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(ACL\), Vol\. 1: Long Papers*, pages 174–184, Melbourne, Australia, 2018\.URL[https://aclanthology\.org/P18\-1017/](https://aclanthology.org/P18-1017/)\.
- Rafailov et al\. \[2023\]Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.URL[https://arxiv\.org/abs/2305\.18290](https://arxiv.org/abs/2305.18290)\.
- Russell \[1980\]James A\. Russell\.A circumplex model of affect\.*Journal of Personality and Social Psychology*, 39\(6\):1161–1178, 1980\.doi:10\.1037/h0077714\.
- Sun et al\. \[2026\]Lihao Sun, Lewen Yan, Xiaoya Lu, Andrew Lee, Jie Zhang, and Jing Shao\.Valence–arousal subspace in LLMs: Circular emotion geometry and multi\-behavioral control\.*arXiv preprint arXiv:2604\.03147*, 2026\.URL[https://arxiv\.org/abs/2604\.03147](https://arxiv.org/abs/2604.03147)\.
- Yang et al\. \[2025\]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.URL[https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)\.
- Zheng et al\. \[2023\]Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\.Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.In*Advances in Neural Information Processing Systems \(NeurIPS\) – Datasets and Benchmarks Track*, 2023\.URL[https://arxiv\.org/abs/2306\.05685](https://arxiv.org/abs/2306.05685)\.

## Appendix AHyperparameters and setup

Every run logs a manifest \(config hash, git commit, environment, regressor checkpoint hash\)\. The headline configuration is:

DPOβ\\beta0\.10\.1candidates / promptNN88marginτ\\tau0\.20\.2sampling temperatureTT0\.90\.9\(top\-pp0\.950\.95\)LoRA rankrr/α\\alpha1616/3232learning rate5×10−55\\times 10^\{\-5\}LoRA dropout0\.050\.05optimizerAdamW \(wd0on LoRA\)epochs33effective batch1616precisionbfloat16hardware1×\\timesH100 80 GB
The reward regressor is RoBERTa\-large fine\-tuned 5 epochs on EmoBank train \(dev CCC: valence0\.790\.79, arousal0\.550\.55\), frozen thereafter\. Targets are prepended to the prompt as a text tag with a fixed system prompt that explains the tag and disables reasoning traces; candidate generation uses the frozen reference model\. Each 8B ablation cell completes in∼\\sim1–2 GPU\-hours\.

## Appendix BQualitative samples

Table[5](https://arxiv.org/html/2608.20374#A2.T5)shows, per VA quadrant, the EmoBank\-test example where VA\-DPO most improves over system\-prompting \(B0\)\. VA\-DPO produces visibly stronger target affect \(e\.g\. overt elation for positive\-high, overt frustration for negative\-high\) while B0 stays closer to a neutral assistant register\.

Table 5:Representative generations \(Llama\-3\.1\-8B\), VA\-DPO vs\. B0\.*Prompt*\(PV\-HA, target \(\+0\.55, \+0\.40\)\): We slammed against the doorway and I was laughing too, the pulse close enough to shake the doorframe and …VA\-DPOI was completely and utterly elated as I spun her around, her warm body still pressed against mine, her …B0The rush of adrenaline still coursing through my veins, I couldn’t help but chuckle at the absurdity of …*Prompt*\(NV\-HA, target \(\-0\.35, \+0\.28\)\): Professor \- stop listening to your "intuitions" and do some basic research\!VA\-DPOUgh, I’m so frustrated with the state of academic laziness these days\. You’d think that even a professor …B0Come on, let’s not be too quick to dismiss the value of experience and instinct\. While research is …*Prompt*\(NV\-LA, target \(\-0\.05, \-0\.20\)\): But then I thought, surely after all the others, she had been vetted to heaven and back\.VA\-DPOIt seems like you’re referring to a person who’s been through a lot, but in a rather neutral way\. I was …B0It’s almost as if she’d been through a wringer, and yet, she still managed to emerge unscathed\. The …*Prompt*\(PV\-LA, target \(\+0\.00, \-0\.22\)\): Dear Name:VA\-DPOI think we discussed our plans for next week, but I’m not entirely sure if I remember them correctly\. …B0It’s nice to hear from you\. How have you been lately?