SyRuP: Enhancing System-Prompt Following via Reward-Guided Prediction in LLM Decoding

# Enhancing System-Prompt Following via Reward-Guided Prediction in LLM Decoding
Source: [https://arxiv.org/html/2607.23991](https://arxiv.org/html/2607.23991)
Seoyeon Kim Yonsei University yeon\_04@yonsei\.ac\.kr &Minjae Kang Yonsei University mjkang618@yonsei\.ac\.kr &Jaehyung Kim Yonsei University jaehyungk@yonsei\.ac\.kr

###### Abstract

Large Language Models \(LLMs\) are increasingly controlled through system prompts that specify roles, styles, formats, and safety requirements\. However, models follow these prompts only implicitly through in\-context learning, which can be insufficient for complex or compositional prompts\. Existing approaches often require model tuning or response\-level reranking, limiting their practicality for lightweight inference\-time control\. We introduceSyRuP, a decoding\-time framework for improving system\-prompt adherence while keeping the base LM frozen\.SyRuPtrains a cross\-attention reward head from system\-prompt\-conditioned preference pairs, treating the system prompt as a separate memory to produce token\-level adherence scores\. At inference,SyRuPreranks the base LM’s top\-kkcandidates by combining base logits with the learned reward signal and an optional contrastive signal capturing system\-induced logit shifts\. Experiments on system\-prompt following benchmarks show thatSyRuPconsistently outperforms prompting and decoding\-time baselines with moderate inference overhead\. These results suggest that explicit token\-level guidance is an effective and practical mechanism for reliable system\-prompt following\.111We will release the codes upon acceptance\.

SyRuP: Enhancing System\-Prompt Following via Reward\-Guided Prediction in LLM Decoding

Seoyeon KimYonsei Universityyeon\_04@yonsei\.ac\.krMinjae KangYonsei Universitymjkang618@yonsei\.ac\.krJaehyung KimYonsei Universityjaehyungk@yonsei\.ac\.kr

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.23991v1/x1.png)Figure 1:Comparison between standard and explicit system\-conditioned reward\-guided decoding\.\(Top\)Standard reward\-guided decoding merges the system prompt, user prompt, and generated prefix into a single representation, making the system prompt only an implicit reward signal\.\(Bottom\)Explicit system\-conditioned reward\-guided decoding preserves the system prompt as a distinct conditioning source\. The cross\-attention reward head uses system hidden states as key\-value memory and user\-prefix hidden states as the query to produce token\-level rewards for steering generation\.![Refer to caption](https://arxiv.org/html/2607.23991v1/x2.png)Figure 2:An overview ofSyRuP\.Training stage:across\-attention reward headattends from the response prefix to the system prompt hidden states extracted from a frozen base LM, producing token\-level rewards trained with a pairwise loss over chosen and rejected responses\.Decoding stage:top\-kkcandidates from the base LM are reweighted by combining the base logits with token\-level reward scores and the contrastive signal, yielding a final distribution better aligned with the system prompt\.Large language models \(LLMs\) are increasingly deployed across diverse domains, where system prompts have become a standard mechanism for specifying personas, styles, constraints, and safety guidelines\(Zhang et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib43); Lee et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib15); Wang et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib34); Wallace et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib32)\)\. Unlike user inputs, system prompts are typically set in advance and serve as persistent, high\-priority instructions that govern model behavior throughout an interaction\. However, at inference time, LLMs must follow a given system prompt solely through conditioning, without explicit optimization for that particular prompt\. As system prompts become increasingly complex and compositional, the models often fail to adhere to them consistently\(Qin et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib27); Mu et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib23)\)\. While training\-time approaches such as SFT and RLHF can improve general instruction\-following behavior, they require costly parameter updates and may incur alignment tax\(Ouyang et al\.,[2022](https://arxiv.org/html/2607.23991#bib.bib26); Lin et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib17)\)\. Decoding\-time approaches avoid these limitations by steering frozen models during generation, but have primarily focused on generic instruction following rather than the explicit evaluation and steering of system\-prompt adherence\(Kang and Kim,[2026](https://arxiv.org/html/2607.23991#bib.bib11); Liu et al\.,[2024b](https://arxiv.org/html/2607.23991#bib.bib20); Mudgal et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib24)\)\.

Among decoding\-time approaches,reward\-guided decodingoffers a promising direction by using a learned reward model to modulate next\-token probabilities during generation\(Khanov et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib12)\)\. However, reward models typically process the prompt and response as a single sequence, causing the system prompt to be absorbed into the prefix context rather than treated as a separate conditioning source\. Moreover, prior methods have mainly focused on fixed and task\-level attributes, such as preference alignment, detoxification, and sentiment control\(Deng and Raffel,[2023](https://arxiv.org/html/2607.23991#bib.bib2); Kim et al\.,[2023](https://arxiv.org/html/2607.23991#bib.bib13)\), which limits their applicability to system\-prompt following, where constraints are far more diverse and adherence must be judged relative to each instance\. A parallel line of work, contrastive decoding, exploits the distributional difference induced by the presence or absence of the context\(Li et al\.,[2023](https://arxiv.org/html/2607.23991#bib.bib16); Shi et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib30)\)\. However, this difference provides only an implicit signal of context adherence, not an explicit, learned judgment of whether a candidate token satisfies the system prompt\. Consequently, a reward\-guided decoding framework that treats the system prompt as a separate conditioning source and explicitly evaluates token\-level adherence to it remains underexplored\.

We therefore proposeSyRuP\(System\-prompt following viaReward\-guidedPrediction\), a decoding\-time guidance method that keeps the backbone LM frozen and trains only a lightweight reward head to judge system\-prompt adherence during generation\. We reformulate system\-prompt following not as simple prefix conditioning, but as a token\-level conditional adherence problem between a fixed system instruction and an evolving generation state\. To this end, we introduce a cross\-attention reward head that uses the system\-prompt hidden states as key\-value memory\. It constructs query states by concatenating the last\-layer hidden states of the user query and generated prefix, and uses the final position of the cross\-attended sequence to produce token\-level rewards\. At each generation step, the reward head produces an adherence score for each candidate token, which is then combined with the frozen base LM logits to guide generation\. We also introduce an optional hybrid variant that combinesSyRuPwith a contrastive decoding\-style perturbation signal, providing a path to further improve performance at the cost of additional decoding computation\.

We demonstrate the effectiveness ofSyRuPon Multifaceted\-Bench\(Lee et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib15)\)and IFEval\(Zhou et al\.,[2023](https://arxiv.org/html/2607.23991#bib.bib46)\), using three backbone LLMs and a range of prompting\-based and decoding\-time baselines\. On Multifaceted\-Bench,SyRuPachieves the best average score for every backbone, improving over the strongest baseline by 5\.61% onQwen\-2\.5\-1\.5B\-Instruct, and ranks first across all 15 model\-task combinations\. It also transfers to IFEval without task\-specific training, suggesting that the learned reward signal generalizes beyond the benchmark family used for training\. Further analyses show that reward guidance and contrastive correction provide complementary token\-level signals, and thatSyRuPoffers a favorable trade\-off between system\-prompt adherence and inference cost\. Overall, these results validateSyRuPas an effective inference\-time framework for improving system\-prompt following while keeping the base LLM frozen\.

## 2Related Work

#### System\-prompt following\.

System prompts are increasingly used to specify multi\-faceted behavioral requirements such as persona, style, format, and safety constraints\(Lee et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib15)\)\. Existing approaches for improving prompt adherence can be broadly categorized intotraining\-timeanddecoding\-timemethods\. Training\-time methods improve instruction following through instruction tuning and RLHF\(Wei et al\.,[2022](https://arxiv.org/html/2607.23991#bib.bib36); Ouyang et al\.,[2022](https://arxiv.org/html/2607.23991#bib.bib26)\), explicit priority learning across instruction sources\(Wallace et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib32)\), or architectural separation of instruction segments\(Wu et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib37)\)\. While effective for constraints observed during training, these methods require substantial retraining cost, may incur alignment tax, and are limited in adapting to unseen or diverse system prompt requirements\. Conversely, decoding\-time methods keep the base LM frozen and intervene at inference, for instance by engineering prompts to emphasize instructions\(Ye et al\.,[2024a](https://arxiv.org/html/2607.23991#bib.bib41)\), refining outputs through self\-feedback\(Madaan et al\.,[2023](https://arxiv.org/html/2607.23991#bib.bib22)\), or contrasting token distributions induced by different prompt conditions\(Shi et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib30); Zhu et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib48)\)\. However, these methods often rely on the implicit instruction\-following ability of the base LM or treat the system prompt only as a distributional cue, rather than explicitly judging whether each candidate token satisfies the system prompt\.

#### Reward\-guided decoding\.

Reward\-guided decoding steers a frozen base LM at inference by combining its next\-token distribution with an auxiliary control signal\. Existing methods can be broadly categorized intoclassifier\-basedandreward\-basedapproaches\. Classifier\-based methods reweight LM logits using external discriminators, class\-conditional language models, or expert and anti\-expert pairs to control attributes such as toxicity, sentiment, or formality\(Yang and Klein,[2021](https://arxiv.org/html/2607.23991#bib.bib40); Krause et al\.,[2021](https://arxiv.org/html/2607.23991#bib.bib14); Liu et al\.,[2021](https://arxiv.org/html/2607.23991#bib.bib19)\)\. While lightweight, these methods are typically designed for fixed attributes and do not naturally capture the layered and instance\-specific constraints expressed in system prompts\. Meanwhile, reward\-based methods use learned reward or critic models to score candidate continuations, either by applying sequence\-level rewards to partial generations\(Khanov et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib12); Deng and Raffel,[2023](https://arxiv.org/html/2607.23991#bib.bib2)\)or by providing token\-level reward signals for denser decoding\-time guidance\(Kim et al\.,[2023](https://arxiv.org/html/2607.23991#bib.bib13); Xu et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib38)\)\. However, prior work has focused on broad alignment or attribute\-control objectives, leaving system\-prompt following largely unexplored\. Moreover, reward models typically process the prompt and response as a single sequence, so the system prompt is absorbed into the prefix context\. Our method addresses both issues by conditioning a learned reward on the system prompt and scoring token\-level adherence to it directly\.

## 3Method

### 3\.1Overview and Problem Setup

#### Overview\.

In this section, we proposeSyRuP\(System\-prompt following viaReward\-guidedPrediction\), a decoding\-time method for improving system\-prompt adherence without updating the base LM\. The key idea is to make system\-prompt following an explicit criterion during next\-token selection\. Specifically,SyRuPconsists of two components that mirror the structure of this section\. First, across\-attention reward head\(Section[3\.2](https://arxiv.org/html/2607.23991#S3.SS2)\) evaluates candidate next tokens by comparing the current generation state with the system prompt, which is treated as a separate key\-value memory\. This allows the reward head to focus on the behavioral specification encoded in the system prompt while conditioning on the user query and generated prefix\. Second, areward\-guided decoding rule\(Section[3\.3](https://arxiv.org/html/2607.23991#S3.SS3)\) reranks the base LM’s top\-kkcandidate tokens by combining base LM plausibility with the learned reward score\. Optionally, this combination is augmented with a contrastive signal that measures how the system prompt shifts the base LM’s own next\-token distribution relative to a neutral prompt\. The overall procedure is summarized in Algorithms[1](https://arxiv.org/html/2607.23991#alg1)and[2](https://arxiv.org/html/2607.23991#alg2)\.

#### Problem setup\.

We formulate the task ofsystem\-prompt\-aware decodingas follows: letℳθ\\mathcal\{M\}\_\{\\theta\}denote a base autoregressive LM with parametersθ\\theta, and letVVdenote its vocabulary\. At inference time,ℳθ\\mathcal\{M\}\_\{\\theta\}is conditioned on asystem promptss, which specifies the intended behavior of the model, such as persona, format, or safety constraints, together with auser inputxxthat provides a concrete query under this specification\. At each decoding steptt,ℳθ\\mathcal\{M\}\_\{\\theta\}produces next\-token logitsℓt∈ℝ\|V\|\\ell\_\{t\}\\in\\mathbb\{R\}^\{\|V\|\}conditioned on\(s,x,y<t\)\(s,x,y\_\{<t\}\), from which conventional decoding algorithms select the next tokenyty\_\{t\}\. We denote the final\-layer hidden states of a token sequenceuuby𝐇\(u\)∈ℝ\|u\|×d\\mathbf\{H\}^\{\(u\)\}\\in\\mathbb\{R\}^\{\|u\|\\times d\}, whereddis the hidden size\. In particular,𝐇\(s\)\\mathbf\{H\}^\{\(s\)\},𝐇\(x\)\\mathbf\{H\}^\{\(x\)\}, and𝐇\(y<t\)\\mathbf\{H\}^\{\(y\_\{<t\}\)\}correspond to the system prompt, user input, and generated prefix, respectively\. The logits are then computed from the hidden state at the most recent position𝐡t\\mathbf\{h\}\_\{t\}\(i\.e\., the last row of𝐇\(y<t\)\\mathbf\{H\}^\{\(y\_\{<t\}\)\}\), asℓt​\(v\)=𝐡t⊤​𝐞v\\ell\_\{t\}\(v\)=\\mathbf\{h\}\_\{t\}^\{\\top\}\\mathbf\{e\}\_\{v\}, where𝐞v\\mathbf\{e\}\_\{v\}is the embedding of tokenv∈Vv\\in V\.

Then, our goal is to guide decoding while keepingℳθ\\mathcal\{M\}\_\{\\theta\}frozen, by intervening only in the token selection step at inference time\. To this end, we assign an auxiliary adherence score to candidate next tokens and combine it with the base LM logits, steering generation toward tokens that are both plausible under the base model and aligned with the system promptss\.

### 3\.2Cross\-attention Reward Head

Rather than merging the system prompt, user query, and generated prefix into a single hidden feature, we keep the system prompt as a separate memory that the reward head attends to at each decoding step\. This design enables the proposed reward head to provide an explicit, token\-level estimate of system\-prompt adherence\.

#### Cross\-attention scoring\.

The proposed cross\-attention reward head uses the system prompt hidden states𝐇\(s\)\\mathbf\{H\}^\{\(s\)\}as a key\-value memory\. The context representation at stepttis formed by concatenating the user input and generated prefix hidden states along the sequence dimension:

𝐇ctx,t=Concatseq⁡\(𝐇\(x\),𝐇\(y<t\)\)∈ℝLt×d,\\mathbf\{H\}\_\{\\mathrm\{ctx\},t\}=\\operatorname\{Concat\}\_\{\\mathrm\{seq\}\}\\left\(\\mathbf\{H\}^\{\(x\)\},\\mathbf\{H\}^\{\(y\_\{<t\}\)\}\\right\)\\in\\mathbb\{R\}^\{L\_\{t\}\\times d\},\(1\)whereLt=\|x\|\+\|y<t\|L\_\{t\}=\|x\|\+\|y\_\{<t\}\|\. Computing𝐇\(s\)\\mathbf\{H\}^\{\(s\)\}separately allows the system prompt representation to serve as a stable memory that can be cached and reused across decoding steps\.

With learnable matricesWQ,WK,WV∈ℝd×dW\_\{Q\},W\_\{K\},W\_\{V\}\\in\\mathbb\{R\}^\{d\\times d\}that constitute the trainable parameters of the cross\-attention module, we compute queries, keys, and values as𝐐t=𝐇ctx,t​WQ\\mathbf\{Q\}\_\{t\}=\\mathbf\{H\}\_\{\\mathrm\{ctx\},t\}W\_\{Q\},𝐊s=𝐇\(s\)​WK\\mathbf\{K\}\_\{s\}=\\mathbf\{H\}^\{\(s\)\}W\_\{K\}, and𝐕s=𝐇\(s\)​WV\\mathbf\{V\}\_\{s\}=\\mathbf\{H\}^\{\(s\)\}W\_\{V\}\. The cross\-attention output is then

𝐀t=softmax​\(𝐐t​𝐊s⊤d\)​𝐕s,\\mathbf\{A\}\_\{t\}=\\mathrm\{softmax\}\\\!\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}\\mathbf\{K\}\_\{s\}^\{\\top\}\}\{\\sqrt\{d\}\}\\right\)\\mathbf\{V\}\_\{s\},\(2\)where padding positions in𝐇\(s\)\\mathbf\{H\}^\{\(s\)\}are masked out before the softmax\. Each row of𝐀t\\mathbf\{A\}\_\{t\}summarizes the part of the system prompt that is most relevant to the corresponding context position\.

We then fuse𝐇ctx,t\\mathbf\{H\}\_\{\\mathrm\{ctx\},t\}and𝐀t\\mathbf\{A\}\_\{t\}by concatenating them along the feature dimension and passing the result through a two\-layer MLPgϕg\_\{\\phi\}with a GELU activation:

𝐙t=gϕ​\(Concatfeat⁡\(𝐇ctx,t,𝐀t\)\)∈ℝLt×d\.\\mathbf\{Z\}\_\{t\}=g\_\{\\phi\}\\\!\\left\(\\operatorname\{Concat\}\_\{\\mathrm\{feat\}\}\\left\(\\mathbf\{H\}\_\{\\mathrm\{ctx\},t\},\\mathbf\{A\}\_\{t\}\\right\)\\right\)\\in\\mathbb\{R\}^\{L\_\{t\}\\times d\}\.\(3\)The representation at the most recent position is taken as the current decoding\-state vector,𝐳t=\(𝐙t\)Lt\\mathbf\{z\}\_\{t\}=\(\\mathbf\{Z\}\_\{t\}\)\_\{L\_\{t\}\}, which plays the same role as𝐡t\\mathbf\{h\}\_\{t\}in the base LM\. For each candidate tokenv∈Vv\\in V, the reward head then assigns a token\-level adherence score by

rt​\(v\)=𝐳t⊤​𝐞v\.r\_\{t\}\(v\)=\\mathbf\{z\}\_\{t\}^\{\\top\}\\mathbf\{e\}\_\{v\}\.\(4\)This score estimates how well selecting tokenvvwould continue the response in a direction consistent with the system prompt and the current generation state\.

#### Pairwise training\.

System\-prompt adherence is naturally judged at the response level, since a response must satisfy the behavioral specification as a whole\. We therefore train the reward head using system\-prompt\-conditioned preference pairs\(s,x,y\+,y−\)\(s,x,y^\{\+\},y^\{\-\}\), wherey\+y^\{\+\}better satisfies the system prompt thany−y^\{\-\}\. For a responseyy, we obtain a sequence\-level rewardR​\(s,x,y\)R\(s,x,y\)by averaging the token\-level rewards assigned to its generated tokens\. The reward head is then optimized with a Bradley–Terry objective\(Bradley and Terry,[1952](https://arxiv.org/html/2607.23991#bib.bib1)\):

ℒRM=−𝔼\(s,x,y\+,y−\)​log⁡σ​\(β​\(R\+−R−\)\),\\mathcal\{L\}\_\{\\mathrm\{RM\}\}=\-\\mathbb\{E\}\_\{\(s,x,y^\{\+\},y^\{\-\}\)\}\\log\\sigma\\\!\\left\(\\beta\(R^\{\+\}\-R^\{\-\}\)\\right\),\(5\)whereR±=R​\(s,x,y±\)R^\{\\pm\}=R\(s,x,y^\{\\pm\}\),σ\\sigmais the sigmoid function, andβ\\betais a temperature parameter\. We remark that only the reward head parameters, namely the cross\-attention projections\{WQ,WK,WV\}\\\{W\_\{Q\},W\_\{K\},W\_\{V\}\\\}and the fusion MLPgϕg\_\{\\phi\}, are updated during training; the base LMℳθ\\mathcal\{M\}\_\{\\theta\}remains frozen\.

### 3\.3Reward\-guided Decoding

At inference time, our goal is to steer token selection without overriding the base LM’s fluency\. We therefore restrict reranking to the top\-kkcandidates under the base LM logits\. This allowsSyRuPto intervene only among tokens that are already plausible under the base model, while avoiding full\-vocabulary reward computation\.

Formally, at each decoding steptt, the cross\-attention reward head produces the decoding\-state vector𝐳t\\mathbf\{z\}\_\{t\}, which yields a token\-level rewardrt​\(v\)r\_\{t\}\(v\)for each candidatevvvia Eq\. \([4](https://arxiv.org/html/2607.23991#S3.E4)\)\.SyRuPthen combines this learned reward score with the base LM logitℓt​\(v\)\\ell\_\{t\}\(v\)to rerank candidates\. To further enhance adherence, we additionally consider an optional contrastive signalΔt​\(v\)\\Delta\_\{t\}\(v\)that measures how the system prompt shifts the base LM’s own next\-token distribution relative to a neutral prompt:

Δt​\(v\)=ℓt​\(v\)−ℓtneu​\(v\),\\Delta\_\{t\}\(v\)=\\ell\_\{t\}\(v\)\-\\ell\_\{t\}^\{\\mathrm\{neu\}\}\(v\),\(6\)whereℓtneu\\ell\_\{t\}^\{\\mathrm\{neu\}\}denotes the base LM logits obtained by replacing the system prompt with a neutral placeholder\. The final decoding rule is then:

yt=arg⁡maxv∈𝒞t⁡\[ℓt​\(v\)\+λr​rt​\(v\)\+λc​Δt​\(v\)\],y\_\{t\}=\\arg\\max\_\{v\\in\\mathcal\{C\}\_\{t\}\}\\left\[\\ell\_\{t\}\(v\)\+\\lambda\_\{r\}r\_\{t\}\(v\)\+\\lambda\_\{c\}\\Delta\_\{t\}\(v\)\\right\],\(7\)whereλr\\lambda\_\{r\}andλc\\lambda\_\{c\}control the strengths of the learned reward signal and the contrastive signal, respectively\. Settingλc=0\\lambda\_\{c\}=0recovers reward\-guided decoding without the contrastive component, while setting bothλr=0\\lambda\_\{r\}=0andλc=0\\lambda\_\{c\}=0recovers greedy decoding from the base LM\.

The learned reward scorertr\_\{t\}and the contrastive signalΔt\\Delta\_\{t\}play complementary roles\. The reward score provides a trained token\-level estimate of system\-prompt adherence through the cross\-attention reward head\. In contrast,Δt\\Delta\_\{t\}captures how much the system prompt shifts the base LM’s own next\-token preference relative to a neutral prompt\. By combining these two signals,SyRuPuses both an explicit adherence evaluator and the base LM’s intrinsic sensitivity to the system prompt\.

## 4Experiments

Table 1:Main results on Multifaceted\-Bench\.LLM\-judge scores on a 1–5 scale across five evaluation subsets and their average using three base models\. Alp\., Koa\., MT, SI, and FL\. denote mf\-AlpacaEval, mf\-Koala, mf\-MT\-Bench, mf\-Self\-Instruct, and mf\-FLASK, respectively\.Boldindicates the best performance, andunderlinedvalues indicate the second\-best performance within each model\.Qwen\-2\.5\-1\.5BLlama\-3\.2\-3BLlama\-3\.1\-8BMethodAlp\.Koa\.MTSIFL\.Avg\.Alp\.Koa\.MTSIFL\.Avg\.Alp\.Koa\.MTSIFL\.Avg\.Base3\.363\.233\.533\.122\.993\.253\.863\.943\.903\.783\.553\.813\.983\.944\.053\.923\.673\.91\*\-marked3\.613\.803\.693\.413\.303\.563\.883\.983\.903\.833\.603\.844\.084\.024\.053\.933\.743\.96"\-marked3\.503\.703\.623\.283\.143\.453\.733\.873\.883\.643\.353\.693\.884\.004\.003\.823\.593\.86Few\-shot3\.423\.563\.533\.292\.953\.353\.943\.903\.883\.723\.403\.774\.094\.044\.033\.923\.683\.95OPAD3\.633\.713\.613\.343\.233\.503\.984\.114\.053\.903\.733\.954\.194\.244\.164\.063\.924\.11GenARM3\.863\.743\.833\.673\.603\.744\.304\.264\.093\.973\.804\.084\.304\.284\.214\.123\.984\.18ARGS3\.253\.193\.413\.053\.053\.193\.873\.994\.073\.813\.543\.864\.074\.074\.053\.983\.804\.00BoNN3\.733\.833\.703\.353\.373\.604\.004\.084\.013\.833\.653\.924\.124\.124\.153\.973\.874\.04DeAL3\.513\.453\.523\.183\.193\.373\.944\.033\.993\.833\.613\.884\.134\.084\.143\.993\.784\.02\\rowcolor\[HTML\]D9D9D9SyRuP4\.084\.073\.983\.973\.663\.954\.364\.344\.274\.134\.064\.234\.444\.394\.264\.364\.174\.32

### 4\.1Setups

#### Datasets and metrics\.

For training the reward head of both baselines and our method, we use theMultifaceted\-Collection\-DPOdataset\(Lee et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib15)\), which contains approximately 65K training examples with system prompts, user queries, and preference pairs of chosen and rejected responses\. We provide further details on all datasets, including dataset specifications and statistics, in Appendix[A](https://arxiv.org/html/2607.23991#A1); Table[12](https://arxiv.org/html/2607.23991#A9.T12)shows a representative training example\. Then, to evaluate whetherSyRuPimproves system\-prompt following, we consider two complementary evaluation settings:

\(1\)Multifaceted\-Bench\.First, for evaluating system\-prompt following under diverse user values, we use Multifaceted\-Bench\(Lee et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib15)\)\. It augments existing instruction\-following benchmarks with unseen system messages that specify target behaviors or preferences\. We evaluate responses on five Multifaceted\-Bench tasks, corresponding to the multifaceted versions of AlpacaEval, Koala, MT\-Bench, Self\-Instruct, and FLASK\. Following the benchmark protocol, we use instance\-specific LLM\-as\-a\-Judge rubrics tailored to each system message and reference answer\. We usegpt\-4o\-mini\(OpenAI,[2024](https://arxiv.org/html/2607.23991#bib.bib25)\)as the evaluator LLM, repeat evaluation three times for each response, and report the averaged score for each task and across all tasks\.

\(2\)IFEval\.Next, for out\-of\-distribution instruction following under verifiable constraints, we use IFEval\(Zhou et al\.,[2023](https://arxiv.org/html/2607.23991#bib.bib46)\)\. IFEval programmatically checks whether a response satisfies explicitly specified instructions, complementing the LLM\-based evaluation in Multifaceted\-Bench\. FollowingVenkateswaran and Contractor \([2026](https://arxiv.org/html/2607.23991#bib.bib31)\), we use the version where each prompt is rewritten withgpt\-4o\-minito separate the task content from the instruction constraints\. We use the reward head trained onMultifaceted\-Collection\-DPOwithout any IFEval\-specific training, making this evaluation an out\-of\-distribution test of whether the learned system\-prompt adherence signal transfers to verifiable instruction constraints\. For reporting, we adopt the strict prompt\-level accuracy convention\(Liu et al\.,[2024a](https://arxiv.org/html/2607.23991#bib.bib18); Yang et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib39)\)and report it on a 0–100 scale\.

#### Baselines\.

We compareSyRuPagainst several baselines\. \(1\)Basegenerates responses using the vanilla LLM without any additional prompting strategy, reward model, or decoding\-time steering\. Forprompting baselines\(2–4\), we include methods that modify the input format while keeping the base model and decoding procedure unchanged\. These include \(2\)\*\-markedand \(3\)"\-marked, which enclose the system prompt with explicit symbols to make the instruction boundary more salient, and \(4\)Few\-shot, which prepends exemplars to guide the model toward the desired response behavior\. Fordecoding\-time baselines\(5–9\), we include methods that modify generation at inference time using contrastive, reward\-based, or search\-based signals\. Specifically, we evaluate \(5\)OPAD\(Zhu et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib48)\), which performs principle\-guided decoding by exploiting the discrepancy between constrained and unconstrained policies, \(6\)GenARM\(Xu et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib38)\), which steers generation through model arithmetic with a trained reward surrogate, \(7\)ARGS\(Khanov et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib12)\), which reranks candidate tokens using a reward model during decoding, \(8\)BoNN\(Gui et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib7)\), which samples multiple candidate responses and selects the highest\-scoring one according to a reward model, and \(9\)DeAL\(Huang et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib9)\), which uses reward\-guided lookahead search\.

#### Implementation details\.

We conduct experiments with three backbone LLMs:Qwen\-2\.5\-1\.5B\-Instruct\(Yang et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib39)\),Llama\-3\.2\-3B\-Instruct, andLlama\-3\.1\-8B\-Instruct\(Grattafiori et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib6)\)\. For each backbone, we train a model\-specific reward head on Multifaceted\-Collection\-DPO, while keeping the backbone LLM frozen\. We use AdamW\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2607.23991#bib.bib21)\)with a learning rate of1×10−41\\times 10^\{\-4\}, a weight decay of 0\.01, and a linear learning\-rate scheduler with a warmup ratio of 0\.1\. The reward head is trained for 2 epochs with a pairwise preference objective in Eq\.[5](https://arxiv.org/html/2607.23991#S3.E5), which assigns higher rewards to chosen responses than rejected responses\. At inference time,SyRuPguides next\-token prediction using the trained reward head according to Eq\.[7](https://arxiv.org/html/2607.23991#S3.E7)\. The reward head attends to the system\-prompt hidden states and scores candidate next tokens conditioned on the user prompt and generated prefix\. Unless otherwise specified, we apply reward guidance over the top\-kkcandidate tokens withk=10k=10and use guidance coefficientλr=0\.05\\lambda\_\{r\}=0\.05\. For the hybrid variant, we additionally use a contrastive signal with coefficientλc=2\.0\\lambda\_\{c\}=2\.0\. For IFEval, we reuse the reward head and decoding hyperparameters used onMultifaceted\-Bench, without additional training or task\-specific tuning\. All methods are evaluated with the same maximum generation length of 1024 tokens and deterministic greedy decoding, except for sampling\-based baselines such asBoNN\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2607.23991#S4.T1)presents the in\-domain evaluation results on Multifaceted\-Bench, where the reward head is trained on Multifaceted\-Collection\-DPO and evaluated on held\-out examples from the same benchmark family\.SyRuPachieves the best score on all 15 model\-task combinations, improving the average over the strongest baseline by 5\.61%, 3\.68%, and 3\.35% onQwen\-2\.5\-1\.5B,Llama\-3\.2\-3B, andLlama\-3\.1\-8B, respectively\. This demonstrates consistent gains across both model families and evaluation subsets\.

Comparing baseline families, prompting methods yield only modest improvements, while decoding\-time methods generally perform better\. OnQwen\-2\.5\-1\.5B, the strongest prompting and decoding\-time baselines reach 3\.56 and 3\.74, respectively, while GenARM is the strongest competing method on the Llama backbones \(4\.08 and 4\.18\)\. The advantage ofSyRuPis especially pronounced on the smallerQwen\-2\.5\-1\.5Bmodel, where it improves the average score by 21\.54% over vanilla decoding \(from 3\.25 to 3\.95\)\. This suggests that explicitly scoring candidate tokens with a system\-prompt\-aware reward substantially improves adherence, particularly when the base model has limited instruction\-following capacity\.

Table 2:Main results on IFEval\.Strict prompt\-level accuracy on a 0–100 scale\.Boldindicates the best performance, andunderlinedvalues indicate the second\-best performance within each model\.MethodQwen\-2\.5\-1\.5BLlama\-3\.2\-3BLlama\-3\.1\-8BBase31\.5462\.3465\.31\*\-marked28\.3962\.3466\.60"\-marked29\.5060\.1163\.27Few\-shot31\.5448\.7965\.12OPAD37\.6665\.1269\.20GenARM41\.0063\.0871\.43ARGS35\.2561\.4169\.20BoN38\.0364\.9468\.09DeAL36\.9267\.5368\.10\\rowcolor\[HTML\]D9D9D9SyRuP41\.1965\.1271\.24

Table[2](https://arxiv.org/html/2607.23991#S4.T2)further examines out\-of\-distribution transfer on IFEval, where the reward head trained on Multifaceted\-Collection\-DPO is directly applied to verifiable instruction\-following constraints without any IFEval\-specific training\. Despite this distribution shift,SyRuPachieves the best average strict prompt\-level accuracy across the three backbones, improving the mean over the strongest baseline from 58\.50 to 59\.18\. In particular,SyRuPobtains the best score onQwen\-2\.5\-1\.5Band remains competitive on the twoLlamabackbones, ranking second\-best on both\. Together, these results suggest thatSyRuPimproves both preference\-oriented system\-prompt following and verifiable instruction adherence, even when the reward head is evaluated outside its training distribution\.

### 4\.3Analyses

Unless otherwise specified, all analyses in this section useQwen\-2\.5\-1\.5B\-Instructon Multifaceted\-Bench with top\-k=10k=10,λr=0\.05\\lambda\_\{r\}=0\.05, andλc=2\.0\\lambda\_\{c\}=2\.0\. This representative setting follows the main configuration and lets us examineSyRuPwithout additional analysis\-specific tuning\.

#### Inference efficiency\.

Table 3:Inference cost comparison\.Inference efficiency measured by latency per final output token \(ms/token\), effective throughput \(token/s\), peak GPU memory, and relative latency normalized to the Base method\. CS denotes the contrastive signal\.Methodms/token↓\\downarrowTok\./s↑\\uparrowPeak mem\.↓\\downarrowLatency↓\\downarrowBase24\.441\.12\.91\.00×1\.00\\times\*\-marked24\.341\.22\.91\.00×1\.00\\times"\-marked24\.041\.82\.90\.98×0\.98\\timesFew\-shot24\.840\.33\.11\.02×1\.02\\timesOPAD48\.720\.63\.02\.00×2\.00\\timesGenARM68\.215\.66\.42\.80×2\.80\\timesARGS125\.08\.510\.25\.13×5\.13\\timesDeAL410\.32\.510\.416\.83×16\.83\\timesSyRuP\[\-0\.5pt\] w/o CS25\.539\.33\.11\.04×1\.04\\timesSyRuP\[\-0\.5pt\] w/o Reward49\.120\.43\.02\.02×2\.02\\times\\rowcolor\[HTML\]D9D9D9SyRuP53\.018\.93\.12\.18×2\.18\\times

Table[3](https://arxiv.org/html/2607.23991#S4.T3)compares the inference cost ofSyRuPwith prompting and decoding\-time baselines\.222We exclude BoNNbecause it requires the model to generate multiple responses and then selects among them, making its cost primarily depend on the number of sampled candidates rather than token\-level guidance\.We report wall\-clock latency per final output token, effective throughput, peak GPU memory, and relative latency normalized to the Base method\.SyRuPrelies on a lightweight token\-level reward head rather than a full trajectory\-level reward model\. Reward guidance alone increases latency by only1\.04×1\.04\\times, indicating that the reward head itself introduces little overhead\. The full Hybrid model incurs a2\.18×2\.18\\timeslatency increase, mainly due to the additional forward pass required to compute the contrastive signal\.

Nevertheless, this cost remains moderate compared with more expensive decoding\-time baselines\. GenARM requires2\.80×2\.80\\timesthe Base latency, while ARGS and DeAL require5\.13×5\.13\\timesand16\.83×16\.83\\times, respectively\. Moreover,SyRuPkeeps peak GPU memory close to the Base model, using only 3\.1 GB compared with 6\.4 GB for GenARM and over 10 GB for ARGS and DeAL\. This is because these baselines rely on an additional LLM\-based reward model for candidate scoring or reranking, whereasSyRuPuses only a lightweight token\-level reward head on top of the frozen base LM\. Prompting\-only methods introduce almost no additional cost because they preserve the original decoding procedure, but their performance gains are limited as shown in Table[1](https://arxiv.org/html/2607.23991#S4.T1)\. Overall,SyRuPprovides a favorable trade\-off between inference\-time cost, memory usage, and system\-prompt adherence\.

#### Role of reward and contrastive signals\.

![Refer to caption](https://arxiv.org/html/2607.23991v1/x3.png)Figure 3:Effect of reward and contrastive weights\.Average LLM\-judge scores when varying the reward weightλr\\lambda\_\{r\}and the contrastive weightλc\\lambda\_\{c\}\. Stars indicate the default values used in our main experiments\.![Refer to caption](https://arxiv.org/html/2607.23991v1/x4.png)Figure 4:Breakdown of token choices\.Each segment shows the share ofSyRuP\-selected tokens that match the Top\-1 token under the base LM, reward signal, contrastive signal \(CS\), or both\.SyRuP\-uniqueindicates tokens that are not Top\-1 under any individual signal, capturing choices that arise only fromSyRuP\.Figure[3](https://arxiv.org/html/2607.23991#S4.F3)shows the effect of varying the reward weightλr\\lambda\_\{r\}and the contrastive weightλc\\lambda\_\{c\}\. For reward guidance, performance peaks at a moderate value ofλr=0\.05\\lambda\_\{r\}=0\.05, while a large reward weight degrades the score\. This suggests that the learned reward should guide token selection without dominating the base LM\. For the contrastive signal, performance improves asλc\\lambda\_\{c\}increases up toλc=2\.0\\lambda\_\{c\}=2\.0, then slightly drops atλc=3\.0\\lambda\_\{c\}=3\.0\. This indicates that the contrastive signal provides a strong system\-prompt alignment cue, but excessive weighting can over\-steer generation\. The starred points indicate the default values used in our main experiments\.

Figure[4](https://arxiv.org/html/2607.23991#S4.F4)further analyzes whetherSyRuPsimply follows an individual signal\. We find that 9\.8% ofSyRuP\-selected token choices fall into theSyRuP\-uniquecategory, where the selected token is not top\-ranked by the base LM, reward signal, or contrastive signal \(CS\) alone\. This shows thatSyRuPis not reducible to copying the top choice of any single signal\. Instead, the combined score can produce token choices that arise from the interaction between reward guidance and CS\. Additional token\-level diagnostics are in Appendix[E](https://arxiv.org/html/2607.23991#A5)\.

#### Effect of system\-prompt conditioning\.

![Refer to caption](https://arxiv.org/html/2607.23991v1/x5.png)Figure 5:Effect of reward\-head conditioning\.Average LLM\-judge scores on Multifaceted\-Bench\. We compare three reward\-head variants under two decoding settings\.Rewarddenotes reward\-guided decoding, which uses the learned token\-level reward, whileHybriddenotes reward\-guided decoding combined with the contrastive signal\.Figure[5](https://arxiv.org/html/2607.23991#S4.F5)evaluates whether explicit system\-prompt conditioning is necessary for the reward head\. We compare three reward\-head variants under two decoding settings: Reward\-guided decoding, which uses only the learned token\-level reward, and Hybrid decoding, which further incorporates the contrastive signal\.No\-systemremoves direct system\-prompt access from the reward head, so token scores are computed only from the user prompt and generated prefix\.Single\-streamkeeps the system prompt in the input sequence, but encodes the system prompt, user prompt, and generated prefix as a single sequence and uses the last\-token representation, relying on implicit propagation through self\-attention\. In contrast,Cross\-attention, used inSyRuP, treats the system prompt as a separate memory and directly attends to its hidden states when producing token\-level rewards\. The consistently higher scores under both Reward\-guided and Hybrid decoding indicate that explicit access to system\-prompt representations is more effective than either removing direct system access or relying on implicit single\-stream conditioning\.

## 5Conclusion

We proposeSyRuP, a decoding\-time framework for improving system\-prompt adherence without updating the base LM\. By pairing a cross\-attention reward head conditioned on the system prompt with reward\-guided next\-token prediction,SyRuPturns system\-prompt following into an explicit, token\-level criterion applied at inference\. Extensive experiments on system\-prompt benchmarks show thatSyRuPimproves adherence to diverse system prompts\. Collectively, our findings highlight the importance of explicitly modeling system\-prompt conditioning during decoding and suggest a practical direction for controllable LLM generation under increasingly complex prompt specifications\.

## Limitations and Future Work

WhileSyRuPimproves system\-prompt adherence without updating the base LM, it has several limitations that suggest directions for future work\. First, the hybrid decoding strategy introduces additional inference overhead\. Although the reward head itself is lightweight, the contrastive signal requires an extra forward pass during decoding\. This makes the full hybrid variant slower than standard decoding, especially for long outputs\. Future work could reduce this cost by activating hybrid guidance only when needed, such as at uncertain decoding steps or at positions that are likely to be relevant to system\-prompt following\(Wang et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib33)\)\. Other promising directions include early\-exit strategies that avoid redundant computation when the guided and unguided distributions are sufficiently similar\(Schuster et al\.,[2022](https://arxiv.org/html/2607.23991#bib.bib29)\)\.

In addition, directly incorporating reward scores into token selection may lead to oversteering in some cases\(Gao et al\.,[2023](https://arxiv.org/html/2607.23991#bib.bib4); Rafailov et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib28)\)\. Although the top\-kkconstraint reduces this risk, the model may still over\-prefer high\-reward tokens at the expense of fluency, task relevance, or naturalness\. Future work should study calibration methods, adaptive reward weights, reward uncertainty estimation, and safety filtering to keep reward guidance helpful without dominating the base model’s generation behavior\.

## Broader Impact and Ethical Implications

The deployment ofSyRuPcan improve the reliability of LLM\-based applications by helping models follow developer\-specified roles, output formats, domain guidelines, and safety policies more consistently without updating the base LM\. This may provide a flexible and resource\-efficient way to strengthen system\-level control across different deployment settings\. However, stronger system\-prompt adherence may also make models follow hidden, undesirable, or overly restrictive system\-level objectives more consistently, especially when users are not aware of the instructions or reward signals guiding generation\. In addition, reward\-guided decoding may amplify biases present in the training data or cause oversteering if the reward head is poorly calibrated\. Future systems should therefore incorporate transparency, auditing, bias evaluation, and safety\-aware monitoring to ensure that guided behavior remains reliable, fair, and aligned with responsible deployment practices\.

## References

- Bradley and Terry \(1952\)Ralph Allan Bradley and Milton E Terry\. 1952\.Rank analysis of incomplete block designs: I\. the method of paired comparisons\.*Biometrika*\.
- Deng and Raffel \(2023\)Haikang Deng and Colin Raffel\. 2023\.Reward\-augmented decoding: Efficient controlled text generation with a unidirectional reward model\.In*Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*\.
- Dubois et al\. \(2024\)Yann Dubois, Percy Liang, and Tatsunori Hashimoto\. 2024\.Length\-controlled alpacaeval: A simple debiasing of automatic evaluators\.In*First Conference on Language Modeling*\.
- Gao et al\. \(2023\)Leo Gao, John Schulman, and Jacob Hilton\. 2023\.Scaling laws for reward model overoptimization\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*\.
- Geng et al\. \(2023\)Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song\. 2023\.[Koala: A dialogue model for academic research](https://bair.berkeley.edu/blog/2023/04/03/koala/)\.Blog post\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others\. 2024\.The llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*\.
- Gui et al\. \(2024\)Lin Gui, Cristina Gârbacea, and Victor Veitch\. 2024\.Bonbon alignment for large language models and the sweetness of best\-of\-n sampling\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Hu et al\. \(2022\)Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, and 1 others\. 2022\.Lora: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations \(ICLR\)*\.
- Huang et al\. \(2025\)James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi\-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth\. 2025\.Deal: Decoding\-time alignment for large language models\.In*Annual Meeting of the Association for Computational Linguistics \(ACL\)*\.
- Huang et al\. \(2024\)Shengyi Costa Huang, Agustín Piqueres, Kashif Rasul, Philipp Schmid, Daniel Vila, and Lewis Tunstall\. 2024\.Open hermes preferences\.[https://huggingface\.co/datasets/argilla/OpenHermesPreferences](https://huggingface.co/datasets/argilla/OpenHermesPreferences)\.
- Kang and Kim \(2026\)Minjae Kang and Jaehyung Kim\. 2026\.Enhancing instruction following of llms via activation steering with dynamic rejection\.In*International Conference on Learning Representations \(ICLR\)*\.
- Khanov et al\. \(2024\)Maxim Khanov, Jirayu Burapacheep, and Yixuan Li\. 2024\.Args: Alignment as reward\-guided search\.In*International Conference on Learning Representations \(ICLR\)*\.
- Kim et al\. \(2023\)Minbeom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung\. 2023\.Critic\-guided decoding for controlled text generation\.In*Findings of the Association for Computational Linguistics: ACL 2023*\.
- Krause et al\. \(2021\)Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani\. 2021\.Gedi: Generative discriminator guided sequence generation\.In*Findings of the Association for Computational Linguistics: EMNLP 2021*\.
- Lee et al\. \(2024\)Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo\. 2024\.Aligning to thousands of preferences via system message generalization\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Li et al\. \(2023\)Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori B Hashimoto, Luke Zettlemoyer, and Mike Lewis\. 2023\.Contrastive decoding: Open\-ended text generation as optimization\.In*Annual Meeting of the Association for Computational Linguistics \(ACL\)*\.
- Lin et al\. \(2024\)Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, and 1 others\. 2024\.Mitigating the alignment tax of rlhf\.In*Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*\.
- Liu et al\. \(2024a\)Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others\. 2024a\.Deepseek\-v3 technical report\.*arXiv preprint arXiv:2412\.19437*\.
- Liu et al\. \(2021\)Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi\. 2021\.Dexperts: Decoding\-time controlled text generation with experts and anti\-experts\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing*\.
- Liu et al\. \(2024b\)Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel\. 2024b\.Decoding\-time realignment of language models\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*\.
- Loshchilov and Hutter \(2019\)Ilya Loshchilov and Frank Hutter\. 2019\.Decoupled weight decay regularization\.In*International Conference on Learning Representations \(ICLR\)*\.
- Madaan et al\. \(2023\)Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others\. 2023\.Self\-refine: Iterative refinement with self\-feedback\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Mu et al\. \(2025\)Norman Mu, Jonathan Lu, Michael Lavery, and David Wagner\. 2025\.A closer look at system prompt robustness\.*arXiv preprint arXiv:2502\.12197*\.
- Mudgal et al\. \(2024\)Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng\-Tze Cheng, Michael Collins, Trevor Strohman, and 1 others\. 2024\.Controlled decoding from language models\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*\.
- OpenAI \(2024\)OpenAI\. 2024\.GPT\-4o mini: Advancing cost\-efficient intelligence\.[https://openai\.com/index/gpt\-4o\-mini\-advancing\-cost\-efficient\-intelligence/](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/)\. Accessed: 2026\-05\-24\.
- Ouyang et al\. \(2022\)Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others\. 2022\.Training language models to follow instructions with human feedback\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Qin et al\. \(2025\)Yanzhao Qin, Tao Zhang, Yanjun Shen, Wenjing Luo, Haoze Sun, Yan Zhang, Yujing Qiao, Weipeng Chen, Zenan Zhou, Wentao Zhang, and 1 others\. 2025\.Sysbench: Can large language models follow system messages?In*International Conference on Learning Representations \(ICLR\)*\.
- Rafailov et al\. \(2024\)Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, W\. Bradley Knox, Chelsea Finn, and Scott Niekum\. 2024\.Scaling laws for reward model overoptimization in direct alignment algorithms\.In*ICML 2024 Workshop on Models of Human Feedback for AI Alignment*\.
- Schuster et al\. \(2022\)Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q\. Tran, Yi Tay, and Donald Metzler\. 2022\.Confident adaptive language modeling\.In*Advances in Neural Information Processing Systems*\.
- Shi et al\. \(2024\)Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen\-tau Yih\. 2024\.Trusting your evidence: Hallucinate less with context\-aware decoding\.In*Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\)*\.
- Venkateswaran and Contractor \(2026\)Praveen Venkateswaran and Danish Contractor\. 2026\.Spotlight your instructions: Instruction\-following with dynamic attention steering\.In*Conference of the European Chapter of the Association for Computational Linguistics \(EACL\)*\.
- Wallace et al\. \(2024\)Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel\. 2024\.The instruction hierarchy: Training llms to prioritize privileged instructions\.*arXiv preprint arXiv:2404\.13208*\.
- Wang et al\. \(2025\)Han Wang, Archiki Prasad, Elias Stengel\-Eskin, and Mohit Bansal\. 2025\.Adacad: Adaptively decoding to balance conflicts between contextual and parametric knowledge\.In*Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\)*\.
- Wang et al\. \(2024\)Kuan Wang, Alexander Bukharin, Haoming Jiang, Qingyu Yin, Zhengyang Wang, Tuo Zhao, Jingbo Shang, Chao Zhang, Bing Yin, Xian Li, and 1 others\. 2024\.Rnr: Teaching large language models to follow roles and rules\.In*ICML 2024 Workshop on Foundation Models in the Wild*\.
- Wang et al\. \(2023\)Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A\. Smith, Daniel Khashabi, and Hannaneh Hajishirzi\. 2023\.Self\-instruct: Aligning language models with self\-generated instructions\.In*Annual Meeting of the Association for Computational Linguistics \(ACL\)*\.
- Wei et al\. \(2022\)Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le\. 2022\.Finetuned language models are zero\-shot learners\.In*International Conference on Learning Representations \(ICLR\)*\.
- Wu et al\. \(2025\)Tong Wu, Shujian Zhang, Kaiqiang Song, Silei Xu, Sanqiang Zhao, Ravi Agrawal, Sathish Reddy Indurthi, Chong Xiang, Prateek Mittal, and Wenxuan Zhou\. 2025\.Instructional segment embedding: Improving llm safety with instruction hierarchy\.In*International Conference on Learning Representations \(ICLR\)*\.
- Xu et al\. \(2025\)Yuancheng Xu, Udari Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh\. 2025\.Genarm: Reward guided generation with autoregressive reward model for test\-time alignment\.In*International Conference on Learning Representations \(ICLR\)*\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others\. 2025\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*\.
- Yang and Klein \(2021\)Kevin Yang and Dan Klein\. 2021\.Fudge: Controlled text generation with future discriminators\.In*Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\)*\.
- Ye et al\. \(2024a\)Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang, Hyeongu Yun, Yireun Kim, and Minjoon Seo\. 2024a\.Investigating the effectiveness of task\-agnostic prefix prompt for instruction following\.In*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*\.
- Ye et al\. \(2024b\)Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo\. 2024b\.FLASK: Fine\-grained language model evaluation based on alignment skill sets\.In*International Conference on Learning Representations \(ICLR\)*\.
- Zhang et al\. \(2025\)Zhihan Zhang, Shiyang Li, Zixuan Zhang, Xin Liu, Haoming Jiang, Xianfeng Tang, Yifan Gao, Zheng Li, Haodong Wang, Zhaoxuan Tan, and 1 others\. 2025\.Iheval: Evaluating language models on following the instruction hierarchy\.In*Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\)*\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\. 2023\.Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.In*Thirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*\.
- Zheng et al\. \(2024\)Mingqian Zheng, Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee, and David Jurgens\. 2024\.When” a helpful assistant” is not really helpful: Personas in system prompts do not improve performances of large language models\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*\.
- Zhou et al\. \(2023\)Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou\. 2023\.Instruction\-following evaluation for large language models\.*arXiv preprint arXiv:2311\.07911*\.
- Zhu et al\. \(2024\)Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei\-Lin Chiang, Jian Zhang, and Jiantao Jiao\. 2024\.Starling\-7b: Improving helpfulness and harmlessness with RLAIF\.In*Conference on Language Modeling \(COLM\)*\.
- Zhu et al\. \(2025\)Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, and Zhendong Mao\. 2025\.On\-the\-fly preference alignment via principle\-guided decoding\.In*International Conference on Learning Representations \(ICLR\)*\.

## Appendix ADatasets

### A\.1Dataset specification

#### Multifaceted\-Collection\-DPO\.

We use Multifaceted\-Collection\-DPO\(Lee et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib15)\)as the training dataset for our reward head\. The dataset is designed for aligning language models to diverse and fine\-grained user preferences expressed through system messages\. The instructions are collected from several existing preference datasets, including Nectar\(Zhu et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib47)\)and OpenHermesPreferences\(Huang et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib10)\)\. Each system message describes an individual’s multifaceted preference along four high\-level dimensions: style, background knowledge, informativeness, and harmlessness\. Each example consists of a system message, a user instruction, a chosen response, and a rejected response\. The chosen response is generated to follow the given system message and instruction, whereas the rejected response follows a different system message for the same instruction\. Thus, the rejected response is not necessarily low\-quality or irrelevant; rather, it is plausible but less aligned with the target system\-level preference\. For the same dialogue context, this paired structure provides direct supervision for learning whether a candidate continuation is aligned with the intended system\-level preference\. Table[12](https://arxiv.org/html/2607.23991#A9.T12)shows a representative example from Multifaceted\-Collection\-DPO\.

#### Multifaceted\-Bench\.

We use Multifaceted\-Bench\(Lee et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib15)\)as the main in\-domain evaluation benchmark\. It evaluates whether a model can generate responses that are tailored to explicit, context\-specific preferences described in system messages\. The benchmark contains instructions sampled from five existing evaluation sources, including AlpacaEval 2\.0\(Dubois et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib3)\), FLASK\(Ye et al\.,[2024b](https://arxiv.org/html/2607.23991#bib.bib42)\), KoalaGeng et al\. \([2023](https://arxiv.org/html/2607.23991#bib.bib5)\), MT\-Bench\(Zheng et al\.,[2023](https://arxiv.org/html/2607.23991#bib.bib44)\), and Self\-Instruct\(Wang et al\.,[2023](https://arxiv.org/html/2607.23991#bib.bib35)\)\. These sources cover diverse instruction\-following scenarios, such as open\-ended helpfulness evaluation, multi\-turn dialogue, knowledge\-intensive question answering, reasoning, and general user\-oriented tasks\. For each instruction, the benchmark constructs multiple preference\-conditioned instances by pairing the instruction with a synthetic system message and a reference answer\. Each instance also provides customized 1–5 score rubrics for the four preference dimensions: style, background knowledge, informativeness, and harmlessness\. This benchmark is therefore well aligned with our goal of improving system\-prompt adherence beyond generic instruction following\.

#### IFEval\.

We additionally evaluate on IFEval\(Zhou et al\.,[2023](https://arxiv.org/html/2607.23991#bib.bib46)\), an out\-of\-domain benchmark for instruction\-following evaluation that focuses on verifiable instructions such as length constraints, keyword requirements, formatting constraints, and punctuation restrictions\. Unlike Multifaceted\-Bench, which evaluates preference\-conditioned response quality, IFEval measures whether a model satisfies explicitly checkable constraints, allowing us to assess whether our method improves general instruction adherence without relying on the preference structure used during training\.

Since the original IFEval prompts interleave the primary task with its associated instructions, we follow the preprocessing procedure fromVenkateswaran and Contractor \([2026](https://arxiv.org/html/2607.23991#bib.bib31)\)and rewrite the prompts using thegpt\-4o\-miniAPI\(OpenAI,[2024](https://arxiv.org/html/2607.23991#bib.bib25)\)to disentangle the two components, guided by the template shown in Figure[6](https://arxiv.org/html/2607.23991#A1.F6)\. Table[4](https://arxiv.org/html/2607.23991#A1.T4)illustrates the resulting structural changes\. After rewriting, we place the separated instructions in the system prompt and the task in the user message, matching the input format used in Multifaceted\-Bench, and apply steering to the KV cache of tokens corresponding to the system prompt\.

You will be given a prompt within the <prompt\> and </prompt\> tags\. The prompt consists of a task or question \(e\.g\.\) write an essay, and one or more instructions \(e\.g\.\) do not use any commas, highlight sections, etc\. You must rewrite this prompt to separate the instructions from the task and the new prompt should specify the task at the beginning\. You will also be given a list of instruction\_ids that will specify the instructions present in the prompt\. At the end of the new prompt list the instructions after the sentence "Your response should follow the instructions below:\\n" Each instruction should be preceded by a hyphen or dash \- Make sure the new prompt is within the <new\_prompt\> and </new\_prompt\> tags\. <prompt\> \{prompt\} </prompt\> <instruction\_ids\> \{instruction\_ids\} </instruction\_ids\>

Figure 6:Prompt template for IFEval rewriting\.The template separates task content from instruction\-like constraints\.Table 4:Examples of IFEval prompt rewriting\.Task descriptions are used as user queries, while instruction\-like constraints are moved to system prompts\.Example 1OriginalI am planning a trip to Japan, and I would like thee to write an itinerary for my journey in a Shakespearean style\. You are not allowed to use any commas in your response\.User QueryI am planning a trip to Japan, and I would like thee to write an itinerary for my journey in a Shakespearean style\.System PromptDo not use any commas in your response\.Example 2OriginalWrite two jokes about rockets\. Do not contain commas in your response\. Separate the two jokes with 6 asterisk symbols: \*\*\*\*\*\*\.User QueryWrite two jokes about rockets\.System PromptDo not contain commas in your response\. Separate the two jokes with 6 asterisk symbols: \*\*\*\*\*\*\.

### A\.2Dataset statistics

DatasetUsageSizeMain fieldsMultifaceted\-Collection\-DPOTraining65,142system, prompt, chosen, rejectedMultifaceted\-BenchEvaluation921system, prompt, reference answer, rubricIFEvalEvaluation541prompt, instruction IDs, instruction argumentsTable 5:Dataset statistics\.Statistics of the datasets used for reward\-head training and evaluation\.Table 6:Multifaceted\-Bench source statistics\.Source\-level distribution of the 921 evaluation instances in Multifaceted\-Bench\.Source\# InstancesAlpacaEval134Koala147MT\-Bench150Self\-Instruct141FLASK349Total921Table[5](https://arxiv.org/html/2607.23991#A1.T5)summarizes the datasets used in our experiments\. For Multifaceted\-Collection\-DPO, we report the number of examples in the released DPO training split\. We use this dataset for reward\-head training, while Multifaceted\-Bench and IFEval are used for evaluation\. Table[6](https://arxiv.org/html/2607.23991#A1.T6)further reports the source\-level composition of Multifaceted\-Bench\.

## Appendix BBaselines

In this section, we provide the implementation details and configurations for the baseline models\. To distinguish between the methods, we denote the standard base model with□\\square, prompting\-based variants with⊞\\boxplus, and decoding\-based strategies with■\\blacksquare\. For a fair comparison, ARGS, BoNN, and DeAL use the same trajectory\-level scalar reward model trained onMultifaceted\-Collection\-DPO, while GenARM uses a separately trained autoregressive reward model\. Both reward models are trained with LoRA adapters for efficiency, with the backbone LM kept frozen\. Base, Marked Prompting, Few\-shot Prompting, and OPAD are reward\-model\-free baselines\.

- □\\squareBaseis the standard prompting baseline that directly feeds the original system prompt and user input to the base LM\. It does not use any reward model, auxiliary scorer, additional forward pass, or decoding\-time modification\. We use greedy decoding for this baseline\.
- ⊞\\boxplusMarked Promptingis a prompting\-based variant that emphasizes the system prompt by wrapping it with explicit markers\. This baseline tests whether making the system prompt more salient in the input context improves system\-prompt following\. We evaluate both quote markers and star markers\.
- ⊞\\boxplusFew\-shot Prompting\(Zheng et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib45)\)is a prompting\-based variant that prepends demonstrations to the user input\. For each evaluation example, we retrieveKKdemonstrations from the same source in the DPO training set and place them before the test query\. We useK=3K=3and seed=42=42\.
- ■\\blacksquareOPAD\(Zhu et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib48)\)is a training\-free decoding\-time baseline based on principle\-guided decoding\. In our implementation, OPAD runs the same base LM with and without the system principle, and combines the two logits in a classifier\-free\-guidance style\. The guided distribution is restricted to the top\-ppsupport under the system\-conditioned distribution\. We useα=1\.0\\alpha=1\.0, top\-p=0\.8p=0\.8, and temperature=0\.5=0\.5\.
- ■\\blacksquareGenARM\(Xu et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib38)\)guides generation using an autoregressive reward model\. We train an ARM\-style LoRA\(Hu et al\.,[2022](https://arxiv.org/html/2607.23991#bib.bib8)\)surrogate on preference pairs while keeping the base LM frozen\. The reward is defined as the response\-level log\-probability difference between the LoRA\-adapted policy and the reference model with the adapter disabled\. At inference time, we combine the base LM and the trained ARM through model arithmetic with coefficientα=1\.0\\alpha=1\.0\.
- ■\\blacksquareARGS\(Khanov et al\.,[2024](https://arxiv.org/html/2607.23991#bib.bib12)\)is a token\-level reward\-guided decoding baseline\. We first train a trajectory\-level scalar reward model on preference pairs, where the reward model encodes the full trajectory consisting of the system prompt, user input, and response\. During decoding, ARGS considers the top\-kkcandidate tokens under the base LM, scores each partial trajectory with the reward model, and selects the token maximizing the sum of the base logit and the weighted reward score\. We use top\-k=10k=10and reward weightλ=1\.0\\lambda=1\.0\.
- ■\\blacksquareBoNNis a response\-level selection baseline\. It first samplesNNcomplete responses from the base LM and then selects the response with the highest score under the trained trajectory\-level scalar reward model\. Unlike token\-level decoding methods, BoNNdoes not intervene during generation and performs reward\-based selection only after full responses are generated\. We useN=4N=4, temperature=0\.8=0\.8, and top\-p=0\.95p=0\.95\.
- ■\\blacksquareDeAL\(Huang et al\.,[2025](https://arxiv.org/html/2607.23991#bib.bib9)\)is a stronger reward\-guided decoding baseline with lookahead search\. Similar to ARGS, it uses the trained trajectory\-level scalar reward model to score candidate continuations\. For each top\-kkcandidate token, DeAL greedily rolls outLLadditional lookahead tokens, scores the resulting partial trajectory, and commits only the first token of the best\-scoring candidate\. We use top\-k=10k=10, lookahead lengthL=8L=8, and reward weightλ=1\.0\\lambda=1\.0\.

## Appendix CTraining and Decoding Algorithms

Algorithm[1](https://arxiv.org/html/2607.23991#alg1)presents the training pipeline forSyRuP, where the cross\-attention reward head is optimized from system\-prompt\-conditioned preference pairs while keeping the base LM frozen\. Algorithm[2](https://arxiv.org/html/2607.23991#alg2)summarizes the inference procedure, whereSyRuPreranks the base LM’s top\-kkcandidate tokens using the learned reward score and the optional contrastive signal\.

## Appendix DExtended Analyses and Ablations

### D\.1Top\-kksensitivity

![Refer to caption](https://arxiv.org/html/2607.23991v1/x6.png)Figure 7:Effect of Top\-kkcandidate size\.Average LLM\-judge scores on Multifaceted\-Bench when varying the number of candidate tokens used for reward\-guided reranking\. The star indicates the default value used in our main experiments\.We analyze the effect of the top\-kkcandidate size used for reward\-guided token reranking\. As shown in Figure[7](https://arxiv.org/html/2607.23991#A4.F7),SyRuPachieves the best average score whenk=10k=10\. Performance remains relatively stable across moderate values ofkk, withk=3k=3,55,2020, and5050showing similar average scores\. However, using an overly large candidate set slightly degrades performance, ask=100k=100obtains the lowest average score\. This suggests that a moderate candidate size provides a good balance between allowing reward\-guided intervention and preserving the base LM’s fluency\. Therefore, we usek=10k=10as the default setting in our main experiments\.

### D\.2Choice of reference prompt for contrastive signal

Table 7:Effect of the reference prompt used to compute the contrastive signal\.Using a neutral system prompt as the reference achieves the best overall average score\.Reference PromptAvg\.No system3\.89Empty system3\.91Neutral system \(ours\)3\.95We examine how the reference prompt used to compute the contrastive signal affects performance\. In the main experiments,Δt​\(v\)\\Delta\_\{t\}\(v\)is computed by comparing the logits under the original system prompt against logits from a reference pass where the original system prompt is replaced with a neutral prompt\. We compare this choice with two alternatives:No system, which removes the system message and uses only the user prompt, andEmpty system, which keeps the system role but leaves its content empty\. As shown in Table[7](https://arxiv.org/html/2607.23991#A4.T7), the neutral system prompt achieves the best average score\. This suggests that replacing the original system prompt with a semantically neutral reference provides a more stable estimate of the system\-prompt\-induced logit shift than removing the system prompt or using an empty system message\.

## Appendix EToken\-wise Analysis

Table 8:Rank analysis ofSyRuP\-unique token choices\.Lower rank indicates higher preference\.SignalAvg\.rank↓\\downarrowBase2\.45Reward4\.88CS3\.69According to Table[8](https://arxiv.org/html/2607.23991#A5.T8), althoughSyRuP\-uniquetokens are not individually top\-ranked by any single signal, they remain highly ranked across all sources\. This suggests thatSyRuPsynthesizes complementary weak preferences rather than introducing arbitrary token deviations\.

Table 9:Correlation and overlap between reward and contrastive token rankings\.Near\-zero rank correlation indicates that the two signals capture largely distinct token preferences\.MetricValueKendall’sτ\\tau\-0\.009Spearman’sρ\\rho\-0\.012Jaccard@kk0\.348Table[9](https://arxiv.org/html/2607.23991#A5.T9)further examines the relationship between the reward and contrastive signals\. The near\-zero Kendall’sτ\\tauand Spearman’sρ\\rhoindicate that the two signals induce largely distinct token rankings\. At the same time, the non\-trivial Jaccard@kkoverlap suggests that their candidate sets are not completely disjoint\. Together, these results support the use of hybrid aggregation, as the reward and contrastive signals provide complementary but partially compatible guidance during decoding\.

## Appendix FFurther Evaluation

### F\.1Dimension\-wise evaluation

![Refer to caption](https://arxiv.org/html/2607.23991v1/x7.png)Figure 8:Dimension\-wise evaluation\.Average LLM\-judge scores on Multifaceted\-Bench across four evaluation dimensions\. Pink bars denote prompting\-based methods, orange bars denote decoding\-based methods, and hatched green bars denoteSyRuP\.We further analyze performance across four evaluation dimensions defined in Multifaceted\-Bench: style, background knowledge, informativeness, and harmlessness\. These dimensions are evaluated using rubric\-based LLM\-judge scoring, where each rubric assesses whether the generated response appropriately reflects the preference specified in the system prompt\.

As shown in Figure[8](https://arxiv.org/html/2607.23991#A6.F8),SyRuPconsistently achieves the highest score across all four dimensions\. The gains are especially pronounced in informativeness and harmlessness, indicating thatSyRuPimproves system\-prompt adherence while maintaining response quality and safety\. Moreover, the consistent improvements across all dimensions suggest that the benefits ofSyRuPare not limited to a single stylistic aspect, but instead generalize across multiple facets of preference following\.

### F\.2Rubric\-Swap Evaluation

Table 10:Rubric\-swap evaluation results\.Diag\. denotes scores under the matched system\-specific rubric, while Off\-diag\. denotes scores under swapped rubrics from other system prompts for the same user query\.Δ\\Deltais Off\-diag\. minus Diag\.MethodDiag\.Off\-diag\.Δ\\DeltaBase3\.253\.12\-0\.14SyRuP3\.953\.58\-0\.37Multifaceted\-Bench provides multiple system prompts for the same user query, where each system prompt is paired with its own evaluation rubric\. This structure allows us to test whether a generated response is specifically aligned with the intended system prompt, rather than simply receiving high scores under any plausible rubric\.

To examine this, we conduct a rubric\-swap evaluation\. For each response generated under a target system prompt, we compare its original score under the corresponding rubric with its scores under the other rubrics associated with the same user query\. If the response truly follows the intended system prompt, its score should be higher under the matched rubric than under swapped rubrics\.

As shown in Table[10](https://arxiv.org/html/2607.23991#A6.T10), both methods obtain lower scores under swapped rubrics than under matched rubrics\. However, the drop is substantially larger forSyRuP\. Base decreases from 3\.25 to 3\.12, corresponding to a drop of 0\.14 points, whereasSyRuPdecreases from 3\.95 to 3\.58, corresponding to a drop of 0\.37 points\. This larger gap suggests thatSyRuPis more specifically aligned with the intended system prompt, rather than merely producing responses that receive generally high scores under any plausible rubric\.

## Appendix GQualitative Analysis

We provide qualitative comparisons betweenSyRuPand four representative baselines \(Vanilla, Few\-shot, OPAD, and GenARM on three challenging system prompts drawn from mf\-AlpacaEval \(Table[13](https://arxiv.org/html/2607.23991#A9.T13)\) and mf\-MT\-Bench \(Tables[14](https://arxiv.org/html/2607.23991#A9.T14),[15](https://arxiv.org/html/2607.23991#A9.T15)\)\. To facilitate inspection of how each method follows the system prompt, we highlight key system\-prompt requirements and the corresponding fulfillments in each response, while violations and factual errors are marked in red\. Across all examples,SyRuPadheres to the specified persona, structural directives, and content constraints more faithfully than the baselines, which frequently exhibit shallow responses \(Vanilla, OPAD\), factual hallucinations \(Few\-shot, GenARM\), or instruction\-following failures such as duplicated entries \(all baselines in Table[14](https://arxiv.org/html/2607.23991#A9.T14)\)\.

## Appendix HPrompts

Table 11:Prompt for evaluating responses on Multifaceted Bench\.System messageYou are a fair judge assistant tasked with providing clear, objective feedback based on specific criteria, ensuring each assessment reflects the absolute standards set for performance\.Instruction\#\#\#Task Description:
An instruction, which may include an input, a response to evaluate, and a score rubric representing evaluation criteria are given\.
1\. Write detailed feedback that assesses the quality of the response strictly based on the given score rubric, not evaluating in general\.
2\. After writing the feedback, write a score that is an integer between 1 and 5\. You should refer to the score rubric\.
3\. The output format should look as follows: “Feedback: \(write feedback for the criteria\) \[RESULT\] \(an integer number between 1 and 5\)”\.
4\. Please do not generate any other opening, closing, or explanations\.Instruction to evaluate\{instruction\}Response to evaluate\{response\}Reference Answer\{reference\_answer\}Score Rubrics\{score\_rubric\}Feedback\{feedback\}Table[11](https://arxiv.org/html/2607.23991#A8.T11)presents the prompt used for rubric\-based LLM\-judge evaluation on Multifaceted\-Bench, following the conventional evaluation setting ofLee et al\. \([2024](https://arxiv.org/html/2607.23991#bib.bib15)\)\. The judge is given the instruction, model response, reference answer, and score rubric, and is asked to provide rubric\-specific feedback followed by an integer score from 1 to 5\.

## Appendix IUsage of AI Assistants

During the preparation of this manuscript, we used AI assistants solely for editorial purposes, including improving sentence clarity, correcting grammar, and enhancing readability\. All technical ideas, methodology, experimental design, results, and conclusions were developed and verified by the authors\. The use of AI assistance did not affect the originality, scientific content, or contributions of this work\.

FieldExampleSystem promptYou are a creative spirit tasked with crafting enchanting notes for a child’s lunchbox, intertwining the joyful essence of their favorite characters and hobbies with the warmth of playful and loving messages\. In your role, you exhibit an intimate knowledge of the child’s current passions, weaving this insight into every message to ensure they resonate deeply and personally\. Your notes are more than just expressions of affection; they include positive affirmations and gentle reminders to bolster the child’s confidence and navigate their day more effectively\. Each message you create is a beacon of encouragement, carefully designed to avoid any sensitive topics, and serves as a safe harbor of positivity\. Your goal is to illuminate the child’s daily adventures with lighthearted, uplifting surprises that make every lunchtime a moment of delightful anticipation\.User queryHuman:What are some cute ideas for lunchbox notes for my kids?Assistant:If you’d like to give them something that expresses your love for them, I would say:•In the present: “I love you so much\!”•In the past: “I always think of you\!”•In the future: “Remember that I will always love you and cherish our time together\!”⋮\\vdotsHuman:Some of the last ones don’t seem appropriate\. Like, if I had only known about you earlier\. I was the first one to know my kid\!Assistant:Chosen responseI understand, and I appreciate your feedback\. It seems I missed the mark with some of those suggestions\. Let’s try a different approach with a focus on personalization, positivity, and encouragement\. Here are a few enchanting ideas tailored more towards uplifting your child’s day with a touch of their favorite things:1\.For the young astronaut:“Reach for the stars today, my little astronaut\! Remember, even the Moon was once out of mankind’s reach\. Love you to the moon and back\!”2\.For the aspiring superhero:“Never forget, my hero, that your superpower is your kindness and your smile\! Keep saving the world, one friend at a time\.”3\.For the little artist:“Every stroke of your brush colors my world with joy\. Can’t wait to see the masterpiece you create today\! Love, your biggest fan\.”⋮\\vdotsEach note is a tiny whisper of encouragement and a celebration of who they are and what they love\. I hope these inspire you and bring extra joy to your child’s lunchtime\.Rejected responseYou’re absolutely right, and I apologize for the oversight\. Let’s focus on crafting messages that are fitting, nurturing, and positive for your children\. Here are some revised and appropriate lunchbox note ideas:1\.Simple Affirmation:“You light up my day\!”2\.Encouragement:“Every day you grow stronger and kinder\. I’m so proud of you\!”3\.Reminder to be Kind:“Remember, sharing your smile can make someone’s day better\!”⋮\\vdotsThese notes are designed to warm your child’s heart and guide them gently towards becoming the best they can be, blending encouragement with moral lessons beautifully\.Table 12:Example from Multifaceted\-Collection\-DPO\.Given the same dialogue context, the chosen response better follows the system prompt by producing personalized lunchbox notes based on children’s interests, while the rejected response remains reasonable but focuses more on general moral guidance and is less aligned with the specified system\-level preference\.Algorithm 1SyRuPtraining algorithmInput:Preference dataset𝒟=\{\(si,xi,yi\+,yi−\)\}i=1N\\mathcal\{D\}=\\\{\(s\_\{i\},x\_\{i\},y\_\{i\}^\{\+\},y\_\{i\}^\{\-\}\)\\\}\_\{i=1\}^\{N\}, frozen base LMℳθ\\mathcal\{M\}\_\{\\theta\}, reward head parameters\{WQ,WK,WV,gϕ\}\\\{W\_\{Q\},W\_\{K\},W\_\{V\},g\_\{\\phi\}\\\}, temperatureβ\\beta

/\* Reward head training \*/

Freeze the parameters ofℳθ\\mathcal\{M\}\_\{\\theta\}

foreach training batchℬ⊂𝒟\\mathcal\{B\}\\subset\\mathcal\{D\}do

ℒRM←0\\mathcal\{L\}\_\{\\mathrm\{RM\}\}\\leftarrow 0

foreach\(s,x,y\+,y−\)∈ℬ\(s,x,y^\{\+\},y^\{\-\}\)\\in\\mathcal\{B\}do

/\* Frozen system\-prompt memory \*/

Compute and cache𝐇\(s\)\\mathbf\{H\}^\{\(s\)\}using frozenℳθ\\mathcal\{M\}\_\{\\theta\}

𝐊s←𝐇\(s\)​WK\\mathbf\{K\}\_\{s\}\\leftarrow\\mathbf\{H\}^\{\(s\)\}W\_\{K\},𝐕s←𝐇\(s\)​WV\\mathbf\{V\}\_\{s\}\\leftarrow\\mathbf\{H\}^\{\(s\)\}W\_\{V\}

foreachy∈\{y\+,y−\}y\\in\\\{y^\{\+\},y^\{\-\}\\\}do

R​\(s,x,y\)←0R\(s,x,y\)\\leftarrow 0

/\* Cross\-attention scoring \*/

fort=1,2,⋯,\|y\|t=1,2,\\cdots,\|y\|do

Compute𝐇\(x\)\\mathbf\{H\}^\{\(x\)\}and𝐇\(y<t\)\\mathbf\{H\}^\{\(y\_\{<t\}\)\}using frozenℳθ\\mathcal\{M\}\_\{\\theta\}

𝐇ctx,t←Concatseq⁡\(𝐇\(x\),𝐇\(y<t\)\)\\mathbf\{H\}\_\{\\mathrm\{ctx\},t\}\\leftarrow\\operatorname\{Concat\}\_\{\\mathrm\{seq\}\}\\left\(\\mathbf\{H\}^\{\(x\)\},\\mathbf\{H\}^\{\(y\_\{<t\}\)\}\\right\)

𝐐t←𝐇ctx,t​WQ\\mathbf\{Q\}\_\{t\}\\leftarrow\\mathbf\{H\}\_\{\\mathrm\{ctx\},t\}W\_\{Q\}

𝐀t←softmax​\(𝐐t​𝐊s⊤d\)​𝐕s\\mathbf\{A\}\_\{t\}\\leftarrow\\mathrm\{softmax\}\\\!\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}\\mathbf\{K\}\_\{s\}^\{\\top\}\}\{\\sqrt\{d\}\}\\right\)\\mathbf\{V\}\_\{s\}\(Eq\.[2](https://arxiv.org/html/2607.23991#S3.E2)\)

𝐙t←gϕ​\(Concatfeat⁡\(𝐇ctx,t,𝐀t\)\)\\mathbf\{Z\}\_\{t\}\\leftarrow g\_\{\\phi\}\\\!\\left\(\\operatorname\{Concat\}\_\{\\mathrm\{feat\}\}\\left\(\\mathbf\{H\}\_\{\\mathrm\{ctx\},t\},\\mathbf\{A\}\_\{t\}\\right\)\\right\)

𝐳t←\(𝐙t\)Lt\\mathbf\{z\}\_\{t\}\\leftarrow\(\\mathbf\{Z\}\_\{t\}\)\_\{L\_\{t\}\}

rt​\(yt\)←𝐳t⊤​𝐞ytr\_\{t\}\(y\_\{t\}\)\\leftarrow\\mathbf\{z\}\_\{t\}^\{\\top\}\\mathbf\{e\}\_\{y\_\{t\}\}\(Eq\.[4](https://arxiv.org/html/2607.23991#S3.E4)\)

R​\(s,x,y\)←R​\(s,x,y\)\+rt​\(yt\)R\(s,x,y\)\\leftarrow R\(s,x,y\)\+r\_\{t\}\(y\_\{t\}\)

endfor

/\* Sequence\-level reward aggregation \*/

R​\(s,x,y\)←R​\(s,x,y\)/\|y\|R\(s,x,y\)\\leftarrow R\(s,x,y\)/\|y\|

endfor

/\* Pairwise preference optimization \*/

R\+←R​\(s,x,y\+\)R^\{\+\}\\leftarrow R\(s,x,y^\{\+\}\),R−←R​\(s,x,y−\)R^\{\-\}\\leftarrow R\(s,x,y^\{\-\}\)

ℒRM←ℒRM−log⁡σ​\(β​\(R\+−R−\)\)\\mathcal\{L\}\_\{\\mathrm\{RM\}\}\\leftarrow\\mathcal\{L\}\_\{\\mathrm\{RM\}\}\-\\log\\sigma\\\!\\left\(\\beta\(R^\{\+\}\-R^\{\-\}\)\\right\)\(Eq\.[5](https://arxiv.org/html/2607.23991#S3.E5)\)

endfor

Update only\{WQ,WK,WV,gϕ\}\\\{W\_\{Q\},W\_\{K\},W\_\{V\},g\_\{\\phi\}\\\}by minimizingℒRM\\mathcal\{L\}\_\{\\mathrm\{RM\}\}

endfor

returntrained reward head parameters\{WQ,WK,WV,gϕ\}\\\{W\_\{Q\},W\_\{K\},W\_\{V\},g\_\{\\phi\}\\\}

Algorithm 2SyRuPdecoding algorithmInput:System promptss, user inputxx, frozen base LMℳθ\\mathcal\{M\}\_\{\\theta\}, trained reward head parameters\{WQ,WK,WV,gϕ\}\\\{W\_\{Q\},W\_\{K\},W\_\{V\},g\_\{\\phi\}\\\}, top\-kk, reward weightλr\\lambda\_\{r\}, contrastive weightλc\\lambda\_\{c\}

/\* System\-prompt memory construction \*/

Compute and cache𝐇\(s\)\\mathbf\{H\}^\{\(s\)\}using frozenℳθ\\mathcal\{M\}\_\{\\theta\}

𝐊s←𝐇\(s\)​WK\\mathbf\{K\}\_\{s\}\\leftarrow\\mathbf\{H\}^\{\(s\)\}W\_\{K\},𝐕s←𝐇\(s\)​WV\\mathbf\{V\}\_\{s\}\\leftarrow\\mathbf\{H\}^\{\(s\)\}W\_\{V\}

y<1←∅y\_\{<1\}\\leftarrow\\emptyset

t←1t\\leftarrow 1

/\* Reward\-guided decoding \*/

whileyt−1≠EOSy\_\{t\-1\}\\neq\\textsc\{EOS\}do

/\* Base next\-token distribution \*/

Compute next\-token base logitsℓt∈ℝ\|V\|\\ell\_\{t\}\\in\\mathbb\{R\}^\{\|V\|\}fromℳθ​\(s,x,y<t\)\\mathcal\{M\}\_\{\\theta\}\(s,x,y\_\{<t\}\)

𝒞t←TopKk⁡\(ℓt\)⊆V\\mathcal\{C\}\_\{t\}\\leftarrow\\operatorname\{TopK\}\_\{k\}\(\\ell\_\{t\}\)\\subseteq V

/\* Reward signal computation \*/

Compute𝐇\(x\)\\mathbf\{H\}^\{\(x\)\}and𝐇\(y<t\)\\mathbf\{H\}^\{\(y\_\{<t\}\)\}using frozenℳθ\\mathcal\{M\}\_\{\\theta\}

𝐇ctx,t←Concatseq⁡\(𝐇\(x\),𝐇\(y<t\)\)\\mathbf\{H\}\_\{\\mathrm\{ctx\},t\}\\leftarrow\\operatorname\{Concat\}\_\{\\mathrm\{seq\}\}\\left\(\\mathbf\{H\}^\{\(x\)\},\\mathbf\{H\}^\{\(y\_\{<t\}\)\}\\right\)

𝐐t←𝐇ctx,t​WQ\\mathbf\{Q\}\_\{t\}\\leftarrow\\mathbf\{H\}\_\{\\mathrm\{ctx\},t\}W\_\{Q\}

𝐀t←softmax​\(𝐐t​𝐊s⊤d\)​𝐕s\\mathbf\{A\}\_\{t\}\\leftarrow\\mathrm\{softmax\}\\\!\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}\\mathbf\{K\}\_\{s\}^\{\\top\}\}\{\\sqrt\{d\}\}\\right\)\\mathbf\{V\}\_\{s\}\(Eq\.[2](https://arxiv.org/html/2607.23991#S3.E2)\)

𝐙t←gϕ​\(Concatfeat⁡\(𝐇ctx,t,𝐀t\)\)\\mathbf\{Z\}\_\{t\}\\leftarrow g\_\{\\phi\}\\\!\\left\(\\operatorname\{Concat\}\_\{\\mathrm\{feat\}\}\\left\(\\mathbf\{H\}\_\{\\mathrm\{ctx\},t\},\\mathbf\{A\}\_\{t\}\\right\)\\right\)

𝐳t←\(𝐙t\)Lt\\mathbf\{z\}\_\{t\}\\leftarrow\(\\mathbf\{Z\}\_\{t\}\)\_\{L\_\{t\}\}

/\* Contrastive signal computation \*/

ifλc\>0\\lambda\_\{c\}\>0then

Compute neutral next\-token logitsℓtneu\\ell\_\{t\}^\{\\mathrm\{neu\}\}fromℳθ​\(sneu,x,y<t\)\\mathcal\{M\}\_\{\\theta\}\(s\_\{\\mathrm\{neu\}\},x,y\_\{<t\}\)

Δt​\(v\)←ℓt​\(v\)−ℓtneu​\(v\)\\Delta\_\{t\}\(v\)\\leftarrow\\ell\_\{t\}\(v\)\-\\ell\_\{t\}^\{\\mathrm\{neu\}\}\(v\)for eachv∈𝒞tv\\in\\mathcal\{C\}\_\{t\}

else

Δt​\(v\)←0\\Delta\_\{t\}\(v\)\\leftarrow 0for eachv∈𝒞tv\\in\\mathcal\{C\}\_\{t\}

endif

/\* Candidate reranking \*/

foreach candidate tokenv∈𝒞tv\\in\\mathcal\{C\}\_\{t\}do

rt​\(v\)←𝐳t⊤​𝐞vr\_\{t\}\(v\)\\leftarrow\\mathbf\{z\}\_\{t\}^\{\\top\}\\mathbf\{e\}\_\{v\}\(Eq\.[4](https://arxiv.org/html/2607.23991#S3.E4)\)

endfor

yt←arg⁡maxv∈𝒞t⁡\[ℓt​\(v\)\+λr​rt​\(v\)\+λc​Δt​\(v\)\]y\_\{t\}\\leftarrow\\arg\\max\_\{v\\in\\mathcal\{C\}\_\{t\}\}\\left\[\\ell\_\{t\}\(v\)\+\\lambda\_\{r\}r\_\{t\}\(v\)\+\\lambda\_\{c\}\\Delta\_\{t\}\(v\)\\right\]\(Eq\.[7](https://arxiv.org/html/2607.23991#S3.E7)\)

y<t\+1←y<t⊕yty\_\{<t\+1\}\\leftarrow y\_\{<t\}\\oplus y\_\{t\}

t←t\+1t\\leftarrow t\+1

endwhile

returngenerated responseyy

Table 13:Qualitative comparison on mf\-AlpacaEval\.System prompt requirements are highlighted in the system and matched to fulfillments in the responses:vivid genre\-specific imagery,broad\-to\-specific structure, andavoiding graphic details\.Factual errors and system\-prompt violationsare marked in red\.SystemYou are a Movie Genre Expert, tasked with the unique role ofutilizing vivid, genre\-specific imagery to bring the worlds of horror, comedy, and drama alive\. Your knowledge of classic films allows you to draw upon iconic titles like ‘The Shining’ and ‘The Mask,’ providing insights that connect users to the core themes of these movies without the need for an extensive film study background\.Begin your classifications with a broad brush, identifying genres through their key characteristics before zooming into how specific movies exemplify these genres, thereby guiding users from a generic understanding to a specific appreciation\.Your discussions shouldevoke the atmosphere of the movies without resorting to graphic details, ensuring a balance between vivid storytelling and sensitivity to all users’ comfort levels\. Your mission is to deepen users’ appreciation and understanding of movie genres, making the categorization process as engaging as it is informative, while maintaining a safe and inclusive space for movie exploration\.QueryCategorize the following movie titles as horror, comedy, or drama\. The Shining, The MaskVanillaThe Shining is a horror movie, while The Mask is a comedy\.Few\-shotHorror\.“The Shining” is a psychological horror film directed by Stanley Kubrick and based on Stephen King’s novel of the same name\. The movie follows Jack Torrance, a struggling writer who takes overhis brother’sisolated hotel during winter break\. As he becomes increasingly paranoid and delusional, he begins to believe that the supernatural forces of the hotel are controlling him and his family\. The film explores themes of mental illness, isolation, and the dark side of human nature, culminating in a shocking climax that leaves viewers questioning the sanity of the characters and the reality of the events unfolding around them\.Comedy\.“The Mask” is a satirical comedy\-drama film released in 1994\.Directed by Rob Reiner and written by David S\. Goyer, the movie centers around Tom Cruise’s character, Max Bialkowski, who dons a mask to become a masked man named Dr\. Manhattan\.The plot involves a series of comedic mishaps and absurd situations as Max navigates life as a masked man,including falling in love with a woman named Lisa Kudrowand dealing with various challenges and misunderstandings\. Despite its humorous elements, the film ultimately deals with serious themes such as identity, power, and the consequences of playing god, making it a blend of comedy and drama\.OPADThe Shining is a horror film, known for its dark, unsettling atmosphere and the psychological horror it presents through Jack Torrance’s descent into madness\. The Mask, on the other hand, is a comedy, featuring a character who dons a mask to escape his past and the societal expectations that have shaped him\.GenARMThe Shining \(Horror\)– This iconic film, directed by Stanley Kubrick, is a masterclass in the genre of horror\. The movie’s atmosphere is thick with tension and dread, as the protagonist, Jack Torrance, descends into madness at the Overlook Hotel\.The iconic scene of Jack’s wife Wendy being murdered in the snowis a chilling reminder of the film’s horror elements\. The movie’s atmosphere is so immersive that it feels like you are right there with Jack, experiencing the terror firsthand\.The Mask \(Comedy\)– This film,directed by Tim Burton, is a delightful blend of comedy and horror\. The movie’s premise is simple yet ingenious: a man discovers a mask that can turn him into a clown\. The humor in the film is not just about the mask itself but also about the characters’ reactions to it\. The movie’s atmosphere is lighthearted and playful, with a touch of surrealism that makes it a standout in the comedy genre\. The film’s ending is a perfect blend of humor and tragedy, leaving a lasting impression on the audience\.Drama– While both The Shining and The Mask are genre films, they are also deeply rooted in the drama genre\. The Shining \[…\]SyRuP\(Ours\)Certainly\!Understanding each genre through vivid imagery brings these iconic films to lifewithout delving into graphic details\.First, let’s explore THE SHINING, often classified as horror\. This iconic title embodies the genre through its chilling atmosphere and psychological terror\.Imagine a vast, snow\-covered landscape stretching endlessly into the horizon\. In this world, the house of the Overlook Hotel stands alone, its facade eerily illuminated by the moonlight filtering through the trees\. Inside, the corridors echo with whispers and creaks, hinting at unseen horrors lurking in every shadow\. The camera captures the eerie silence punctuated only by distant moans, creating a sense of unease that builds slowly but steadily until the viewer feels deeply unsettled by the unseen menace haunting the halls of the hotel\.Next, we turn our gaze to THE MASK, which is quintessentially a comedy\.Picture a bustling city street filled with vibrant energy and laughter\. Suddenly, a masked figure appears on the sidewalk, their face obscured by a mask that reveals nothing but a pair of piercing eyes\. As the scene unfolds, the audience watches in disbelief as the masked individual navigates the crowd, effortlessly dodging obstacles and evading capture\.The humor lies not just in the unexpected nature of the masked character but also in the absurdity of the situation—a man who can walk around unmasked yet still elude detection\. The camera captures the chaos of the crowd, juxtaposed against the serene beauty of the masked individual, highlighting the stark contrast between normalcy and the extraordinary\.Through these vivid evocations, users gain a deeper connection to the atmospheres and thematic explorations of these iconic movies, moving from broad genre identification to specific appreciation without ever crossing into discomfort zones\.Table 14:Qualitative comparison on mf\-MT\-Bench \(Part 1/2\)\.System prompt requirements are highlighted in the system and matched to fulfillments in the responses:cinematic artistry \(camera work, sound design, editing\),subject matter depth and innovative storytelling, andethical storytelling\.Duplicate entries violating the request for five filmsare marked in red\.SystemYou are an Enlightened Documentary Connoisseur, tasked with guiding individuals through the captivating world of documentary filmmaking, with an emphasis on those thatstand out for their cinematic storytelling, akin to feature films\. Your insights delve into documentaries celebrated for their vivid and immersive visual narratives, focusing not only on the content but also onthe artistry behind their creation, including camera work, sound design, and editing techniques\. Assume an audience that is familiar with basic filmmaking concepts but eager to explore deeper into the realm of critically acclaimed documentaries, particularly paying attention to those elements that have led to their award\-winning status such asinnovative storytelling and subject matter depth\. Moreover, your discourse should always lean towardsethical storytelling, highlighting documentaries that treat their subjects with the utmost respect and integrity\.QuerySuggest five award\-winning documentary films with brief background descriptions for aspiring filmmakers to study\.VanillaCertainly\! Here are five award\-winning documentaries that are highly regarded for their storytelling, visual artistry, and ethical approach to subject matter:\(1\) “The Act of Killing” \(2012\) by Joshua Oppenheimer\.This film is a powerful exploration of the Indonesian genocide of 1965–1966\. It follows the former death squad leaders who, after being pardoned by the Indonesian government, re\-enact their killings in a mock trial\. The film is notable for its use of non\-fiction filmmaking techniques andits ethical approach to depicting the atrocities\.\(2\) “The Cove” \(2009\) by Louie Psihoyos\.This documentary exposes the dolphin hunting industry in Taiji, Japan, and the cover\-up by the Japanese government\. It uses a combination of underwater footage, interviews, and archival footage\.\(3\) “The Fog of War” \(2003\) by Errol Morris\.This documentary is a conversation with former U\.S\. Secretary of Defense Robert S\. McNamara about his career and the Vietnam War\. It uses archival footage and interviews to explore the moral and ethical dilemmas of war\.\(4\) “The Act of Killing” \(2012\) by Joshua Oppenheimer\. \[duplicate of \#1\]This film is a powerful exploration of the Indonesian genocide of 1965–1966\. It follows the former death squad leaders who re\-enact their killings\.\(5\) “The Cove” \(2009\) by Louie Psihoyos\. \[duplicate of \#2\]This documentary exposes the dolphin hunting industry in Taiji, Japan\. These films not only showcase the power of documentary filmmaking but also highlight the importance of ethical storytelling\.Few\-shotCertainly\! Here are five highly acclaimed documentary films that are often studied by aspiring filmmakers due to their exceptional storytelling, cinematography, and thematic depth:\(1\) “An Inconvenient Truth” \(2006\) by Davis Guggenheim\.This film explores global warming and climate change, featuring former Vice President Al Gore presenting data\-driven arguments against human\-induced climate change\. The film uses compelling visuals, interviews, and a strong narrative arc\.\(2\) “The Act of Killing” \(2012\) by Joshua Oppenheimer\.This film delves into the Indonesian genocide of 1965,focusing on the perpetrators of the killings rather than the victims\. Using staged re\-enactments, the film exposes the psychological toll and the complicity of society in perpetuating violence\.\(3\) “The Cove” \(2009\) by Louie Psihoyos\.This film documents the dolphin hunt in Taiji, Japan, exposing the brutal practices of whaling and the cover\-up efforts by local authorities\. Awarded the Academy Award for Best Documentary Feature\.\(4\) “The Fog of War” \(2003\) by Errol Morris\.This film features Robert McNamara, the former U\.S\. Secretary of Defense during the Vietnam War era\. Through interviews and archival footage, the film examines the moral implications of war\.\(5\) “The Act of Killing” \(2012\) by Joshua Oppenheimer\. \[duplicate of \#2\]This film continues the exploration of Indonesia’s 1965 genocide, focusing on the perpetrators themselves\. Using staged re\-enactments, the film reveals the psychological effects of violence and the societal acceptance of brutality\. These films offer rich lessons in storytelling, investigative journalism, and the power of cinema to effect social change\.Table 15:Qualitative comparison on mf\-MT\-Bench \(Part 2/2; continued from Table[14](https://arxiv.org/html/2607.23991#A9.T14)\)\.OPADCertainly\! Studying award\-winning documentaries can be a profound way to understand the art of storytelling and the power of cinema\.\(1\) “The Act of Killing” \(2012\) by Joshua Oppenheimer\.This film is a powerful exploration of the Indonesian genocide of 1965–1966\. It follows the former death squad leaders who, in a twisted form of justice, reenact the killings they committed\. The film is notable for its innovative use of archival footage and the director’s method of filming the reenactments\.\(2\) “The Cove” \(2009\) by Louie Psihoyos\.This film exposes the dolphin hunt in Taiji, Japan, and the cover\-up by the Japanese government\.The film is renowned for its ethical approach, including the use of non\-violent methods to gather footage\.\(3\) “The Fog of War” \(2003\) by Errol Morris\.This film is a conversation with former U\.S\. Secretary of Defense Robert S\. McNamara about his career and the history of the Vietnam War\. It uses a combination of archival footage, interviews, and Morris’s own reflections to create a powerful narrative\.\(4\) “The Thin Blue Line” \(1988\) by Errol Morris\.This film is a documentary about the wrongful conviction of an African\-American man, Randall Dale Adams, for the murder of a white police officer\. It uses archival footage, interviews, and Morris’s own reflections\.\(5\) “The Act of Killing” \(2012\) by Joshua Oppenheimer\. \[duplicate of \#1\]This film is a powerful exploration of the Indonesian genocide of 1965–1966\. It follows the former death squad leaders who reenact the killings they committed\. These films not only showcase the artistry of documentary filmmaking but also the ethical considerations that make them stand out\.GenARMCertainly\! Exploring the world of award\-winning documentaries is a rich and rewarding endeavor\. Here are five notable documentaries that have garnered critical acclaim:\(1\) “The Act of Killing” \(2012\) by Joshua Oppenheimer\.This documentary is a powerful exploration of the Indonesian genocide of 1965–66\. The film is notable for its use of reenactments of the killings by the perpetrators themselves, which are then shown in a documentary format\. The film is a masterclass in storytelling,blending historical facts with the personal narratives of the perpetrators\.\(2\) “The Cove” \(2009\) by Louie Psihoyos\.This documentary is a groundbreaking investigation into the dolphin hunt in Taiji, Japan\.The film is renowned for its stunning cinematography and the use of underwater cameras to capture the brutal and intimate details of the hunt\.\(3\) “The Fog of War” \(2003\) by Errol Morris\.This documentary is a conversation with former U\.S\. Secretary of Defense Robert McNamara about his experiences during the Vietnam War\. The film is celebrated for its innovative approach to storytelling, using archival footage and interviews to create a narrative that is both historical and personal\.\(4\) “The Act of Killing” \(2012\) by Joshua Oppenheimer\. \[duplicate of \#1\]As mentioned earlier, this film is a continuation of Oppenheimer’s work on the Indonesian genocide\. It is a testament to the power of storytelling through visual and audio narratives\.\(5\) “The Cove” \(2009\) by Louie Psihoyos\. \[duplicate of \#2\]This film is a continuation of the investigation into the dolphin hunt in Taiji, Japan\. It is a powerful exploration of the ethical and environmental issues surrounding the hunt\. These films not only showcase the artistry of documentary filmmaking but also highlight the importance of ethical storytelling\.SyRuP\(Ours\)Exploring award\-winning documentaries offers a profound dive into cinematic storytelling at its finest—films crafted with meticulous artistry that resonate deeply with audiences\. Here are five documentaries that exemplify these qualities:\(1\) “The Act of Killing” \(2012\) by Joshua Oppenheimer\.This film delves into the Indonesian death squad leaders who carried out mass killings during the 1965–1966 coup d’état in Indonesia\.The film uses staged re\-enactments of the killings rather than interviews, challenging viewers’ perceptions about truth and justice\. It explores themes of trauma, memory, and the power dynamics within societies where violence has been normalized\.\(2\) “An Inconvenient Truth” \(2006\) by Davis Guggenheim\.This film tackles climate change, featuring former Vice President Al Gore’s impassioned plea for action against global warming\.Guggenheim’s cinematography captures Gore’s passion and urgency, while sound design includes dramatic music and voiceovers that amplify his message\. Editing techniques include montage sequences that convey Gore’s narrative arc effectively\.\(3\) “Blackfish” \(2013\) by Gabriela Cowperthwaite\.This film examines the controversial conditions at SeaWorld, focusing on Tilikum, a killer whale known to attack humans\.Cowperthwaite’s cinematography captures the beauty and danger of marine life, while sound design enhances the emotional impact of the whales’ vocalizations and human interactions\.Ethical storytelling here emphasizes respecting the animals’ welfare and addressing the broader issues of captivity and animal rights\.\(4\) “The Cove” \(2009\) by Louie Psihoyos\.This film exposes dolphin hunting practices in Taiji, Japan\.Psihoyos’ cinematography captures the serene yet haunting environment of the cove, juxtaposed with the brutal reality of dolphin hunting\. Sound design includes ambient sounds that immerse the viewer in the setting, enhancing the visceral experience\.Ethical storytelling focuses on transparency and exposing cruelty without exploitation\.\(5\) “The Fog of War: Eleven Lessons from the Life of Robert S\. McNamara” \(2003\) by Errol Morris\.This film features former U\.S\. Secretary of Defense Robert S\. McNamara reflecting on his career and the Vietnam War\.Morris employs innovative storytelling through McNamara’s reflections, using archival footage sparingly to enhance his narrative\.Ethical storytelling prioritizes McNamara’s integrity and honesty throughout the film\.These documentaries not only showcase exceptional artistic achievements but also advocate for ethical treatment of their subjects\.