MIRROR: From Imitation to Internalization in LLM Personalization

# MIRROR: From Imitation to Internalization in LLM Personalization
Source: [https://arxiv.org/html/2610.09795](https://arxiv.org/html/2610.09795)
Jicheng Yang11footnotemark:1Affiliation:Baidu Inc\.Email:[mengchong01@baidu\.com](mailto:)Min YiAffiliation:Baidu Inc\.Chong Meng††thanks:Corresponding author\.Affiliation:Baidu Inc\.

###### Abstract

The demand for personalized LLMs is shifting from style imitation toward content quality\. We investigate whether self\-distillation can bridge this gap in existing fine\-tuning paradigm\. To address this limitation, we introduce MIRROR\(Meta\- personalization by Internalizing Reference\-Revealed On\-policy Reflections\),a novel self\-distillation framework that shifts LLM personalization from imitation toward preference internalization\. First, we replace reference\-token imitation with reference\-revealed on\-policy self\-distillation, aligning the model’s next\-token distributions along its own generation trajectories with those of its reference\-conditioned self, thereby internalizing user preferences rather than reproducing reference wording\. Second, we introduce MIRROR\-F, a focal plug\-in that augments on\-policy distributional alignment with selective supervision over informative reference tokens, thereby strengthening content generation while preserving user\-specific expression\. Across three personalized generation benchmarks, two model scales, and complementary reference\-based and LLM\-based evaluations, MIRROR and MIRROR\-F achieve leading overall personalization performance and superior text quality, while exhibiting less catastrophic forgetting than SFT\-based baselines on three unseen personalized generation tasks\. The gains are consistent across model scales and application scenarios, translating to improved performance in LLM personalization tasks\.

## 1Introduction

Large Language Models \(LLMs\) have exhibited exceptional capabilities across a wide range of domains\([Team et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib1);[Yang et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib2)\), leading to their widespread deployment in real\-world applications such as personalized office assistant\([Lyu et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib41);[Li et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib42)\)and Private medical diagnosis\([Zeng et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib43)\)\. Although various methods enhance the personalization capabilities of general\-purpose LLMs through sophisticated harness designs\([Liu et al\., 2026b](https://arxiv.org/html/2610.09795#bib.bib47)\)and skill injection\([Yang et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib48)\), these approaches largely rely on external mechanisms\. From the perspective of the model intelligence, the core challenge remains to strengthen its intrinsic ability to infer latent user states, capture implicit preferences, and perform preference\-aware reasoning\([Amirizaniani et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib44);[Cao et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib45);[Yan et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib46)\)\.

Mainstream approaches for LLM personalization can be broadly categorized into 3 categories: retrieval\-based,fine\-tuning\-based and user\-based methods\. First, retrieval\-based methods\(as shown in Figure[1](https://arxiv.org/html/2610.09795#S1.F1)\.b\) incorporate relevant user histories into the input context to elicit preference\-aware generation through in\-context learning\([Du et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib5);[Zhang et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib6);[Lee et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib7);[Choi et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib8)\)\. However, their reliance on external contextual information leaves the model’s intrinsic preference\-understanding capabilities largely unchanged and introduces potential privacy and security risks\([Liang et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib9)\)\. Second, fine\-tuning\-based methods\(as shown in Figure[1](https://arxiv.org/html/2610.09795#S1.F1)\.c\) employ parameter\-efficient adaptation with preference\-aware token\- or sample\-level supervision to learn user\-specific linguistic patterns\([Zhang et al\., 2026a](https://arxiv.org/html/2610.09795#bib.bib10);[Zhao et al\., 2026b](https://arxiv.org/html/2610.09795#bib.bib11)\)\. While this method is effective at learning personalized styles and sentence structures, they significantly degrades the text professionalism, and can lead to catastrophic forgetting\([Prashant et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib17)\)and poor out\-of\-distribution generalization\([Karouzos et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib16);[Liu et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib18)\)\. Third, user\-based methods including user\-specific LoRA adapter training\([Tan et al\., 2024](https://arxiv.org/html/2610.09795#bib.bib12);[Tan et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib15);[Hwang et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib50)\),attention\-head pruning\([Zhang et al\., 2026b](https://arxiv.org/html/2610.09795#bib.bib13)\)and steering\-vector injection\([Liu et al\., 2026a](https://arxiv.org/html/2610.09795#bib.bib14)\)\. However, these methods typically require user\-specific parameters or dedicated architectures, leading to prohibitive training and storage costs as the user base scales\([Zhao et al\., 2026b](https://arxiv.org/html/2610.09795#bib.bib11)\), which hinders the deployment of personalized LLMs in real world applications\.

![Refer to caption](https://arxiv.org/html/2610.09795v1/1motivation.png)Figure 1:Overview of personalization paradigms\. \(a\) A case study of the quality defects of existing methods\. \(b\) Retrieval\-based personalization\. \(c\) SFT\-based personalization\. \(d\) MIRROR, which aligns a self\-student with a self\-teacher on the student’s own on\-policy trajectories\.To date, as shown in Figure[1](https://arxiv.org/html/2610.09795#S1.F1)\.a,existing personalization approaches have been predominantly constrained by a style imitation paradigm, where personalization is primarily modeled as the ability to reproduce a user’s linguistic style, expression patterns, and textual distributions\. However, this formulation overlooks a fundamental requirement of personalized generation: the generated content must not only reflect a user’s expressive preferences but also maintain high standards of factual accuracy, completeness, and practical utility while satisfying the user’s underlying needs\. Therefore, this paper investigates the following central question:How can we enhance the modeling of user\-specific expressive styles while maintaining the quality and effectiveness of personalized text generation?

Recent advances in self\-distillation\([Zhao et al\., 2026a](https://arxiv.org/html/2610.09795#bib.bib20);[Shenfeld et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib21)\)introduce a paradigm beyond demonstration imitation\. By leveraging hindsight information, models can internalize preference\-relevant behavioral shifts through self\-supervision along their own generation trajectories\. In LLM personalizaton, user reference information can be viewed as a manifestation of the user’s preferences for stylistic sentence structures and their level of expertise, so we can identify user\-provided references as a natural form of hindsight information for personalized generation\.

Based on this insight, we proposeMIRROR\(Meta\-personalization byInternalizingReference\-RevealedOn\-policyReflections\), a reference\-revealed self\-distillation framework that shifts personalization from imitation toward internalization\. MIRROR \(shown in Figure[1](https://arxiv.org/html/2610.09795#S1.F1)\.d\) employs a single model as both a self\-teacher and a self\-student, the teacher accesses the user\-provided reference to provide preference\-aware guidance, while the student aligns with this guidance along its own on\-policy trajectories instead of fitting the reference sequence token by token\. Building upon this training paradigm, we further introduceMIRROR\-F, a focal personalization framework that selectively concentrates optimization on informative tokens responsible for content quality\. This focal mechanism enhances the expertise, completeness, and practical utility of generated content while preserving user\-specific expressive styles\. Controlled by a continuous focality coefficient\. Extensive experiments on three benchmarks demonstrate that MIRROR achieves strong performance across multiple personalized generation tasks and MIRROR\-F further achieves the best overall performance under comprehensive personalized generation evaluations, demonstrating superior content expertise, completeness, and textual quality while preserving user\-specific expressive preferences\.

The main contribution of this work can be summarized as follows:

- •We introduce a new perspective by formulating personalized generation as reference\-revealed self\-distillation, moving beyond surface\-level imitation toward user preference internalization\.
- •We proposeMIRROR, a self\-distillation framework that transfers preference\-induced behaviors via on\-policy learning, achieving state\-of\-the\-art performance across diverse personalized generation tasks\.
- •We further proposeMIRROR\-F, which selectively enhances informative content generation through focal supervision, improving expertise, completeness, and textual quality while preserving user\-specific expressions\.
- •Extensive experiments demonstrate that MIRROR and MIRROR\-F achieve superior personalization and generation quality across comprehensive personalized generation evaluations\.

## 2Preliminaries

##### Personalized text generation\.

We consider the task of personalized text generation, where the goal is to adapt a language modelπθ\\pi\_\{\\theta\}so that its outputs match the preferences of an individual user\. Each data sample is a triplet\(x,h,y\)\(x,h,y\):xxis the query specifying the task and its content,hhis the user’s historical text that implicitly reflects their preferences, andy=\(y1,…,yT\)y=\(y\_\{1\},\\dots,y\_\{T\}\)is the reference response actually written by the user\. The collection of such samples over all users forms the dataset𝒟\\mathcal\{D\}, with\(x,h,y\)∈𝒟\(x,h,y\)\\in\\mathcal\{D\}\. A non\-personalized model producesy^=πθ​\(x\)\\hat\{y\}=\\pi\_\{\\theta\}\(x\), whereas a personalized model additionally conditions on the user history,y^=πθ​\(x,h\)\\hat\{y\}=\\pi\_\{\\theta\}\(x,h\)\. Our goal is to improveπθ\\pi\_\{\\theta\}such thaty^\\hat\{y\}both reflects the user’s expressive preferences and remains accurate and complete as a response toxx\.

##### Autoregressive factorization and contexts\.

We writeπθ\(⋅∣c,y<t\)∈Δ\|𝒱\|\\pi\_\{\\theta\}\(\\cdot\\mid c,y\_\{<t\}\)\\in\\Delta^\{\|\\mathcal\{V\}\|\}for the next\-token distribution over the vocabulary𝒱\\mathcal\{V\}at steptt, given a contextccand a prefixy<ty\_\{<t\}\. Two contexts are used throughout this paper\. The*deployable context*cS=\(h,x\)c\_\{\\mathrm\{S\}\}=\(h,x\)contains only what is genuinely available at inference time, and the*reference\-revealed context*cT=\(h,x,y\)c\_\{\\mathrm\{T\}\}=\(h,x,y\)additionally exposes the user\-written reference\. Since the reference is unavailable at deployment, a model conditioned oncTc\_\{\\mathrm\{T\}\}can serve only as a source of supervision during training, never as a deployed policy\. We usesg⁡\[⋅\]\\mathrm\{sg\}\\\!\\left\[\\cdot\\right\]to denote the stop\-gradient operator, so thatπ¯θ\(⋅∣c,y<t\):=sg\[πθ\(⋅∣c,y<t\)\]\\bar\{\\pi\}\_\{\\theta\}\(\\cdot\\mid c,y\_\{<t\}\):=\\mathrm\{sg\}\\\!\\left\[\\pi\_\{\\theta\}\(\\cdot\\mid c,y\_\{<t\}\)\\right\]denotes a detached copy of the current model evaluated on contextcc\.

##### Supervised personalization objective\.

The standard way to personalizeπθ\\pi\_\{\\theta\}is to fine\-tune it on the reference response under teacher forcing, minimizing the token\-level negative log\-likelihood

ℒSFT\(θ\)=−1T∑t=1Tωtlogπθ\(yt∣cS,y<t\),\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(\\theta\)=\-\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\omega\_\{t\}\\log\\pi\_\{\\theta\}\\\!\\left\(y\_\{t\}\\mid c\_\{\\mathrm\{S\}\},y\_\{<t\}\\right\),\(1\)whereωt\\omega\_\{t\}is a token\-level weight\. Settingωt=1\\omega\_\{t\}=1for allttrecovers plain supervised fine\-tuning, while recent personalization methods instantiateωt\\omega\_\{t\}with estimates of how strongly tokenyty\_\{t\}reflects the user’s preferences\. Two properties of Equation[1](https://arxiv.org/html/2610.09795#S2.E1)are central to this paper: the supervision target is the reference tokenyty\_\{t\}itself, and it is evaluated on the reference prefixy<ty\_\{<t\}rather than on a prefix the model would actually produce\.

##### On\-policy rollouts\.

Given a deployable contextcSc\_\{\\mathrm\{S\}\}, we writey^∼πθ\(⋅∣cS\)\\hat\{y\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid c\_\{\\mathrm\{S\}\}\)for a response sampled from the current model, and refer to the resulting prefixesy^<t\\hat\{y\}\_\{<t\}as*on\-policy states*\. These are the states the model visits at inference time, and they generally differ from the reference prefixesy<ty\_\{<t\}used in Equation[1](https://arxiv.org/html/2610.09795#S2.E1)\.

##### Divergences between next\-token distributions\.

LetPPandQQbe two distributions over𝒱\\mathcal\{V\}\. We use the generalized Jensen–Shannon divergence with mixing coefficientβ∈\[0,1\]\\beta\\in\[0,1\],

JSDβ\(P∥Q\)=βDKL\(P∥M\)\+\(1−β\)DKL\(Q∥M\),M=βP\+\(1−β\)Q,\\JSD\_\{\\beta\}\\\!\\left\(P\\,\\\|\\,Q\\right\)=\\beta\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(P\\,\\\|\\,M\\right\)\+\(1\-\\beta\)\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(Q\\,\\\|\\,M\\right\),\\qquad M=\\beta P\+\(1\-\\beta\)Q,\(2\)which measures the discrepancy betweenPPandQQthrough their mixture distributionMM\. The coefficientβ\\betacontrols the relative contribution of the two KL\-to\-mixture terms, rather than recovering the forward or reverse KL divergence at the endpoints\. In particular,β=0\.5\\beta=0\.5yields the standard symmetric Jensen–Shannon divergence and treats the two distributions equally\. The divergence is finite and bounded for allP,QP,Qwhenβ∈\[0,1\]\\beta\\in\[0,1\]; under natural logarithms, its upper bound is the binary entropyh⁡\(β\)h\(\\beta\)and is at mostlog⁡2\\log 2\. Throughout our experiments, we fixβ=0\.5\\beta=0\.5to avoid introducing an additional asymmetry between the reference\-revealed teacher and the deployable student\. We do not claim thatβ=0\.5\\beta=0\.5is optimal; rather, it is adopted as a neutral symmetric setting\. A systematic study of alternative mixing coefficients is beyond the scope of this work\.

## 3Methodology

![Refer to caption](https://arxiv.org/html/2610.09795v1/2methods.png)Figure 2:Overview of MIRROR\-F\. A frozen reference\-conditioned teacher \(h,x,yh,x,y\) supervises the trainable student \(h,xh,x\) on the student’s own rollouts viaℒMIRROR\\mathcal\{L\}\_\{\\text\{MIRROR\}\}, whileℒANC\\mathcal\{L\}\_\{\\text\{ANC\}\}anchors only the selected informative tokens of the referenceyy\.We present MIRROR, which supervises a model with its own reference\-conditioned predictions over states it visits itself \(Section[3\.1](https://arxiv.org/html/2610.09795#S3.SS1)\), and MIRROR\-F, which adds a sparse anchoring term whose strength is exposed as a single coefficient \(Section[3\.2](https://arxiv.org/html/2610.09795#S3.SS2)\)\.

### 3\.1MIRROR: Internalizing Reference\-Revealed Reflections

##### Self\-teacher and self\-student\.

Rather than treating the reference responseyyas a string to be reproduced, we treat it as hindsight information: conditioning onyymakes it substantially easier to judge which continuations suit this user\. We therefore evaluate the same model under two contexts, the reference\-revealedcT=\(h,x,y\)c\_\{\\mathrm\{T\}\}=\(h,x,y\)and the deployablecS=\(h,x\)c\_\{\\mathrm\{S\}\}=\(h,x\), on a shared prefixy^<t\\hat\{y\}\_\{<t\}:

ptT=π¯θ\(⋅∣cT,y^<t\),ptS=πθ\(⋅∣cS,y^<t\),p^\{\\mathrm\{T\}\}\_\{t\}=\\bar\{\\pi\}\_\{\\theta\}\\\!\\left\(\\cdot\\mid c\_\{\\mathrm\{T\}\},\\hat\{y\}\_\{<t\}\\right\),\\qquad p^\{\\mathrm\{S\}\}\_\{t\}=\\pi\_\{\\theta\}\\\!\\left\(\\cdot\\mid c\_\{\\mathrm\{S\}\},\\hat\{y\}\_\{<t\}\\right\),\(3\)whereπ¯θ=sg⁡\[πθ\]\\bar\{\\pi\}\_\{\\theta\}=\\mathrm\{sg\}\\\!\\left\[\\pi\_\{\\theta\}\\right\]is the detached model\. Since both distributions come from the same parameters and the same prefix, their difference isolates the behavioral shift induced by the reference alone; matching them transfers that shift, rather than the wording ofyy, into the model, without an external teacher or reward model\.

##### On\-policy alignment\.

At each optimization step, a response prefix is sampled from the current student under stop\-gradient sampling,y^∼sg\[πθ\(⋅∣cS\)\]\\hat\{y\}\\sim\\operatorname\{sg\}\[\\pi\_\{\\theta\}\(\\cdot\\mid c\_\{\\mathrm\{S\}\}\)\], and the student is supervised on the states visited by this rollout\. MIRROR optimizes the following stopped\-gradient surrogate objective:

ℒ~MIRROR\(θ\)=𝔼\(x,h,y\)∼𝒟𝔼y^∼sg\[πθ\(⋅∣cS\)\]\[1\|y^\|∑t=1\|y^\|JSDβ\(ptT∥ptS\)\],\\widetilde\{\\mathcal\{L\}\}\_\{\\mathrm\{MIRROR\}\}\(\\theta\)=\\mathbb\{E\}\_\{\(x,h,y\)\\sim\\mathcal\{D\}\}\\,\\mathbb\{E\}\_\{\\hat\{y\}\\sim\\operatorname\{sg\}\[\\pi\_\{\\theta\}\(\\cdot\\mid c\_\{\\mathrm\{S\}\}\)\]\}\\left\[\\frac\{1\}\{\|\\hat\{y\}\|\}\\sum\_\{t=1\}^\{\|\\hat\{y\}\|\}\\JSD\_\{\\beta\}\\\!\\left\(p^\{\\mathrm\{T\}\}\_\{t\}\\,\\middle\\\|\\,p^\{\\mathrm\{S\}\}\_\{t\}\\right\)\\right\],\(4\)withβ=0\.5\\beta=0\.5\. The sampled prefixes and the teacher distribution are detached during backpropagation, so gradients are propagated only through the student distribution evaluated on the fixed on\-policy states\. We use a mixed divergence because personalization admits multiple valid realizations for each query: the symmetric objective transfers preference\-relevant probability mass from the reference\-revealed teacher while retaining support for alternative valid continuations\.

##### Why supervision must be on\-policy\.

Equation[4](https://arxiv.org/html/2610.09795#S3.E4)departs from the supervised objective of Equation[1](https://arxiv.org/html/2610.09795#S2.E1)in two respects that jointly determine what the model learns\. Its target is a distribution rather than a single reference token, so the model is taught how probability mass should be redistributed instead of which word to emit; and it is evaluated on prefixes the model itself produces, so training and inference operate over the same states\. Reweightingωt\\omega\_\{t\}in Equation[1](https://arxiv.org/html/2610.09795#S2.E1)alters only which reference tokens are emphasized and leaves both properties intact\. What MIRROR acquires is therefore a preference\-reasoning behavior that transfers to queries for which no reference exists—the only regime that matters at deployment\. Check Appendix[A\.7](https://arxiv.org/html/2610.09795#A1.SS7)for more details\.

### 3\.2MIRROR\-F: Focal MIRROR with Selective Grounded Anchoring

##### Selective grounded anchoring\.

Distributional alignment governs generation as a whole, whereas professional usability often depends on a small set of content\-bearing tokens, such as technical terms, named entities, and quantities\. MIRROR\-F therefore applies an auxiliary anchoring loss only to selected reference positions\. A reference word is selected when it is*informative*according to the deterministic rules in Appendix[A\.4](https://arxiv.org/html/2610.09795#A1.SS4)and*grounded*, meaning that its normalized form occurs as a complete word in the student contextcSc\_\{\\mathrm\{S\}\}\. Selection is performed at the word level and expanded to all subword units of the selected word\. Let𝒯sel\\mathcal\{T\}\_\{\\mathrm\{sel\}\}denote the resulting reference\-token indices\. The anchoring loss is

ℒanc​\(θ\)=\{−1\|𝒯sel\|∑t∈𝒯sellogπθ\(yt∣cS,y<t\),\|𝒯sel\|\>0,0,\|𝒯sel\|=0,\\mathcal\{L\}\_\{\\mathrm\{anc\}\}\(\\theta\)=\\begin\{cases\}\-\\dfrac\{1\}\{\|\\mathcal\{T\}\_\{\\mathrm\{sel\}\}\|\}\\displaystyle\\sum\_\{t\\in\\mathcal\{T\}\_\{\\mathrm\{sel\}\}\}\\log\\pi\_\{\\theta\}\\\!\\left\(y\_\{t\}\\mid c\_\{\\mathrm\{S\}\},y\_\{<t\}\\right\),&\|\\mathcal\{T\}\_\{\\mathrm\{sel\}\}\|\>0,\\\\\[10\.0pt\] 0,&\|\\mathcal\{T\}\_\{\\mathrm\{sel\}\}\|=0,\\end\{cases\}\(5\)whereyty\_\{t\}is thett\-th reference token\. Normalization by\|𝒯sel\|\|\\mathcal\{T\}\_\{\\mathrm\{sel\}\}\|prevents terminology\-rich references from dominating the gradient, while the empty\-set convention ensures that the objective reduces to MIRROR when no token is selected\. Complete selection criteria are provided in Appendix[A\.4](https://arxiv.org/html/2610.09795#A1.SS4)\.

##### Focality coefficient\.

As shown in Figure[2](https://arxiv.org/html/2610.09795#S3.F2), because the two terms differ in scale and evolve at different rates, we divide each by a running average of its own magnitude,ℒ~k=ℒk/\(mk\+ϵ\)\\tilde\{\\mathcal\{L\}\}\_\{k\}=\\mathcal\{L\}\_\{k\}/\(m\_\{k\}\+\\epsilon\)withmk←γ​mk\+\(1−γ\)​sg​\[ℒk\]m\_\{k\}\\leftarrow\\gamma m\_\{k\}\+\(1\-\\gamma\)\\mathrm\{sg\}\\\!\\left\[\\mathcal\{L\}\_\{k\}\\right\]fork∈\{MIRROR,anc\}k\\in\\\{\\mathrm\{MIRROR\},\\mathrm\{anc\}\\\}, and combine them through one coefficientα∈\[0,1\]\\alpha\\in\[0,1\]:

ℒMIRROR​\-​F​\(θ\)=\(1−α\)​ℒ~MIRROR​\(θ\)\+α​ℒ~anc​\(θ\)\.\\mathcal\{L\}\_\{\\mathrm\{MIRROR\\text\{\-\}F\}\}\(\\theta\)=\(1\-\\alpha\)\\,\\tilde\{\\mathcal\{L\}\}\_\{\\mathrm\{MIRROR\}\}\(\\theta\)\+\\alpha\\,\\tilde\{\\mathcal\{L\}\}\_\{\\mathrm\{anc\}\}\(\\theta\)\.\(6\)Equation[6](https://arxiv.org/html/2610.09795#S3.E6)reduces exactly to MIRROR atα=0\\alpha=0, and by construction for any sample whose selected set is empty, so the objective is well defined for allα\\alphaand all samples\. Since personalized style is carried by the distributional term over the whole trajectory,α\\alphadoes not trade style against content: it sets how sharply supervision is focused on content\-bearing positions on top of an already internalized behavior\. Increasingα\\alphathus selects an operating point along the specialization axis, and moving along it requires no change to the objective, the data, or the pipeline\.

## 4Experiments

### 4\.1Experimental Settings

##### Datasets\.

Experiments are conducted on three personalized generation tasks drawn from two widely used benchmarks for LLM personalization, LaMP\([Salemi et al\., 2024](https://arxiv.org/html/2610.09795#bib.bib23)\)and LongLaMP\([Kumar et al\., 2024](https://arxiv.org/html/2610.09795#bib.bib24)\)\. Specifically,Abstract Generationis taken from LongLaMP, whileNews Headline Generation\(LaMP\-4\) andPaper Topic Generation\(LaMP\-5\) are taken from LaMP\. To ensure that training and evaluation operate over an identical context distribution, we performed context cropping and dataset filtering based on methods from existing papers\([Zhang et al\., 2026a](https://arxiv.org/html/2610.09795#bib.bib10);[Zhao et al\., 2026b](https://arxiv.org/html/2610.09795#bib.bib11)\)\. For evaluating out\-of\-distribution \(OOD\) performance, we further use three benchmarks in Amazon Reviews Dataset\([Ni et al\., 2019](https://arxiv.org/html/2610.09795#bib.bib49)\), and Music Review, which are not included during training and are reserved exclusively for assessing cross\-task transfer and catastrophic forgetting\.

##### Baselines\.

Our methods are compared against three categories of baselines\.\(1\) Base Model\([Yang et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib2)\): the backbone LLM is prompted without any personalization and Thinking mode is disabled by default\(check Appendix[A\.9](https://arxiv.org/html/2610.09795#A1.SS9)for thinking performance ablation\)\.\(2\) Retrieval\-based Methods: RAG\([Lei et al\., 2023](https://arxiv.org/html/2610.09795#bib.bib28)\), in which relevant user history is injected into the prompt without updating the model, LatestK\([Liu et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib18)\)selects the most recentKKentries by timestamp, and LLM TRSR\([Zheng et al\., 2024](https://arxiv.org/html/2610.09795#bib.bib27)\)compresses the history into a structured summary via recurrent summarization\.\(3\) PEFT\-based methods, in which the model is fine\-tuned on user\-specific data: SFT\([Hu et al\., 2021](https://arxiv.org/html/2610.09795#bib.bib33)\)is trained on history\-augmented inputs, OPPU\([Tan et al\., 2024](https://arxiv.org/html/2610.09795#bib.bib12)\)learns a dedicated adapter per user, and PerCE\([Zhang et al\., 2026a](https://arxiv.org/html/2610.09795#bib.bib10)\)and NextQuill\([Zhao et al\., 2026b](https://arxiv.org/html/2610.09795#bib.bib11)\)additionally reweight the supervision according to token\-level preference signals\. PerCE and NextQuill are the two most recent state\-of\-the\-art training\-based methods and are therefore adopted as the primary points of comparison\.

##### Evaluation Metrics\.

Personalization performance is evaluated from two complementary perspectives\. First,for textual similarity against the gold reference, three reference\-based metrics are reported: ROUGE\-1\([Lin, 2004](https://arxiv.org/html/2610.09795#bib.bib37)\), METEOR\([Banerjee and Lavie, 2005](https://arxiv.org/html/2610.09795#bib.bib38)\), and BERTScore\([Zhang et al\., 2019](https://arxiv.org/html/2610.09795#bib.bib39)\), where BERTScore is computed with theroberta\-largemodel\. Second, for comprehensive personalization evaluation, we further adopt an LLM\-as\-judge protocol to assess aspects beyond surface overlap\. Textual quality and professionalism are evaluated using two established frameworks: the reference\-free G\-Eval protocol\([Liu et al\., 2023](https://arxiv.org/html/2610.09795#bib.bib34)\)combined with the SummEval dimensions\([Fabbri et al\., 2021](https://arxiv.org/html/2610.09795#bib.bib35)\), under which each response is rated on Coherence, Consistency, Fluency, and Relevance; and the reference\-based ExPerT framework\([Salemi et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib36)\), which decomposes both the reference and the candidate into atomic content and style aspects and reports content\- and style\-level agreement\. The judge model used in all LLM\-as\-judge evaluations isQwen3\-30B\-A3B\. For each reference\-based evaluation and LLM\-as\-judge evaluation, we conduct three independent training runs with different random seeds and evaluate each result under the same protocol\. We report the arithmetic mean across the three runs\. The judge decoding temperature is fixed to00throughout to ensure reproducible and deterministic scoring for each model output\.

##### Implementation Details

We use Qwen3\-1\.7B and Qwen3\-4B as the backbone LLMs for training and evaluating all methods\. For most model training procedures, we employ low\-rank adaptation \(LoRA\)\([Hu et al\., 2021](https://arxiv.org/html/2610.09795#bib.bib33)\)\. During evaluation, vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.09795#bib.bib40)\)is used to accelerate inference\. Since different methods use different trained models, we select the best\-performing configuration for training and evaluation\. For the MIRROR\-based methods, each Qwen3\-1\.7B model is trained for 200 steps, while each Qwen3\-4B model is trained for 250 steps\. For the main comparison between MIRROR and MIRROR\-F, we set the focality coefficientα\\alphato 0\.5 to ensure a consistent comparison\. In the ablation study, we further evaluateα∈\{0\.1,0\.3,0\.5,0\.7,0\.9\}\\alpha\\in\\\{0\.1,0\.3,0\.5,0\.7,0\.9\\\}to investigate how the focal supervision in MIRROR\-F affects personalized styles and syntactic patterns\.

Table 1:Main results of MIRROR and MIRROR\-F on three personalized generation benchmarksBoldnumbers denote the best performance, whileunderlinednumbers denote the second best\.DatasetsMethods \(→\\rightarrow\)BaseRetrieval\-based MethodsPEFT\-based MethodsOursMetrics \(↓\\downarrow\)Qwen3RAGLatestKLLM\-TRSRSFTOPPUPerCENextQuillMIRRORMIRROR\-FQwen3\-1\.7BAbstractGenerationROUGE\-10\.37170\.38110\.36850\.37580\.37550\.30820\.36700\.36980\.39670\.4015METEOR0\.21250\.22620\.21450\.23110\.23650\.20730\.22400\.24370\.25060\.2529BERTScore0\.85770\.86750\.86570\.86560\.86680\.84430\.86210\.86940\.85870\.8701News HeadlineGenerationROUGE\-10\.14220\.14950\.14020\.13870\.14680\.15590\.15270\.15940\.16740\.1763METEOR0\.09760\.10420\.09560\.09570\.09220\.10380\.09760\.10050\.10200\.1156BERTScore0\.84650\.84970\.84750\.84610\.85440\.85640\.85590\.85580\.85210\.8925Paper TopicGenerationROUGE\-10\.38420\.40520\.36590\.38880\.41040\.40730\.40030\.40950\.41330\.4368METEOR0\.32470\.35160\.31430\.34660\.33830\.33720\.33090\.33500\.35540\.3721BERTScore0\.88450\.88870\.88410\.88590\.89230\.89170\.89120\.89170\.88940\.8922Qwen3\-4BAbstractGenerationROUGE\-10\.39490\.39840\.38840\.38690\.40610\.36970\.36940\.38440\.40270\.4195METEOR0\.25750\.25970\.24960\.25000\.24750\.22250\.23010\.23910\.26590\.2662BERTScore0\.86890\.86980\.86800\.86690\.87170\.86350\.85650\.86630\.87300\.8695News HeadlineGenerationROUGE\-10\.15780\.15170\.14620\.14540\.15600\.15930\.16930\.15930\.16910\.1747METEOR0\.11050\.10690\.10140\.10190\.10310\.10220\.11300\.10510\.11420\.1209BERTScore0\.85440\.85370\.85190\.85020\.85610\.85610\.85980\.85650\.85670\.8579Paper TopicGenerationROUGE\-10\.41180\.42820\.42110\.42240\.41650\.39750\.42570\.45150\.43760\.4486METEOR0\.39540\.39450\.38240\.39370\.34520\.32330\.35480\.34780\.38690\.3949BERTScore0\.89170\.89460\.89030\.88930\.80320\.89040\.89420\.89270\.89170\.8949

### 4\.2Experiment Results

##### Reference\-based Evaluation

MIRROR and MIRROR\-F achieve consistently strong performance on reference\-based evaluation metrics, demonstrating that our method possesses superior capabilities for learning personalized styles and sentence structures\. As shown in Table 1, the two methods outperform or remain highly competitive with retrieval\-based and PEFT\-based baselines across all three personalized generation tasks, two model backbones, and the ROUGE\-1, METEOR, and BERTScore metrics\. The advantage is particularly consistent on ROUGE\-1 and METEOR, where MIRROR\-F achieves the best results in most settings, including all three tasks with the Qwen3\-1\.7B backbone, while MIRROR also frequently ranks first or second\. The two methods further maintain strong BERTScore performance, indicating that their gains are not limited to lexical overlap but also extend to semantic similarity\. These results show that the proposed on\-policy internalization paradigm provides a more reliable approach to reference\-aligned personalization than conventional retrieval augmentation and token\-level preference reweighting\.

##### LLM\-based Evaluation

MIRROR and MIRROR\-F achieve leading performance under the LLM\-based evaluation framework, showing that their advantages extend beyond reference\-based similarity to broader aspects of personalized generation quality\. Personalized generation is inherently underdetermined, as multiple responses can satisfy the same user preferences and task requirements\. Therefore, metrics based on similarity to a single reference may fail to recognize valid alternatives and may not fully reflect content quality or stylistic alignment\. As shown in Table 2, MIRROR\-F achieves the highest Content\-and\-Style \(C&S\) score comparing to three trained methods across all three tasks, while MIRROR also consistently outperforms or remains competitive with SFT, PerCE, and NextQuill\. MIRROR\-F further obtains most of the best C&S scores among the trained methods on 3 benchmarks, indicating that its improvements are stable across tasks and model scales\. The consistent gains in both content and style suggest that the proposed methods internalize preference\-relevant information while preserving the content adequacy and stylistic characteristics required for high\-quality personalized generation\.

Figure 3:Catastrophic forgetting on held\-out OOD personalized generation tasks from the Amazon Review dataset\. Performance degradation is measured relative to the corresponding base model under the same evaluation protocol; smaller values indicate better retention\.Boldnumbers denote the smallest degradation, whileunderlinednumbers denote the second\-smallest\.
##### OOD Performance

MIRROR and MIRROR\-F exhibit the strongest resistance to catastrophic forgetting on OOD personalized generation tasks, demonstrating superior generalization and scalability compared with existing SFT\-based methods\. As shown in Figure[3](https://arxiv.org/html/2610.09795#S4.F3), conventional SFT methods, including PerCE and NextQuill, experience substantial performance degradation on unseen tasks relative to their in\-distribution performance, indicating that their strong reliance on reference\-specific token\-level supervision can lead to over\-specialization and impaired transferability\. In contrast, MIRROR and MIRROR\-F preserve performance much more effectively on the OOD test set and achieve the smallest degradation across the evaluated similarity metrics\.

MIRROR and MIRROR\-F consistently outperform existing SFT\-based methods across all four dimensions of the third\-party text quality evaluation framework\. As shown in Figure[4](https://arxiv.org/html/2610.09795#S4.F4), both methods achieve higher scores on Relevance, Coherence, Consistency, and Fluency than SFT, PerCE, and NextQuill, demonstrating that their improvements extend beyond reference\-based similarity to the intrinsic quality of generated responses\. MIRROR\-F generally obtains the strongest performance, benefiting from its focal supervision of informative and context\-grounded content tokens\.

Table 2:ExPert evaluation of personalized generation quality on three benchmarks\. The judge model scores each generated response for*Content*,*Style*, and their combination*C&S*\(Content & Style\); higher is better\.Boldnumbers denote the best performance among trained methods, whileunderlinednumbers denote the second best among trained methods\.Datasets \(→\\rightarrow\)Abstract GenerationNews Headline GenerationPaper Topic GenerationMethods \(↓\\downarrow\)ContentStyleC&SContentStyleC&SContentStyleC&SQwen3\-1\.7BBase0\.6698\.3607\.5230\.0774\.5202\.6440\.3538\.0005\.766SFT0\.5637\.9906\.8080\.0804\.7302\.7650\.4178\.3606\.267PerCE0\.4927\.8106\.3650\.0684\.8102\.7450\.4138\.1006\.116NextQuill0\.5428\.0006\.7080\.0784\.6702\.7250\.3928\.1406\.028MIRROR \(Ours\)0\.6718\.3307\.5220\.0454\.6902\.5700\.4588\.2906\.436MIRROR\-F \(Ours\)0\.6858\.3907\.6200\.0954\.8902\.9200\.4838\.4406\.633Qwen3\-4BBase0\.6958\.5007\.7240\.1145\.4103\.2730\.4708\.4406\.571SFT0\.6008\.0807\.0420\.0784\.9702\.8760\.4388\.3106\.345PerCE0\.5367\.9606\.6610\.0914\.8902\.8980\.3978\.1906\.078NextQuill0\.5638\.0906\.8590\.0864\.9302\.8960\.3858\.1005\.972MIRROR \(Ours\)0\.7048\.4607\.7500\.0945\.1003\.0200\.4408\.3406\.372MIRROR\-F \(Ours\)0\.7148\.5207\.8320\.0985\.2703\.1250\.4808\.3506\.573

Figure 4:Comparison of model performance scores based on the G\-eval framework\.
##### Content Quality Performance

##### Ablation Study

The personalization performance of MIRROR\-F remains stable across different values of the focality coefficientα\\alpha\. As shown in Figure[5](https://arxiv.org/html/2610.09795#S5.F5), varyingα\\alphafrom0\.10\.1to0\.90\.9does not lead to a substantial degradation in the evaluated personalization metrics, indicating that the model consistently preserves its ability to generate user\-specific styles and sentence structures\. This stability suggests that the focal supervision mainly adjusts the emphasis on informative and grounded content tokens without disrupting the underlying preference\-relevant behavior learned by MIRROR\.

## 5Related Work

##### LLM Personalization\.

LLM personalization has been extensively studied across text generation\([Lai et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib3);[Wang et al\., 2024](https://arxiv.org/html/2610.09795#bib.bib4)\), conversational systems\([Wang et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib22);[Li et al\., 2024](https://arxiv.org/html/2610.09795#bib.bib26)\), and multimodal applications\([Shen et al\., 2024](https://arxiv.org/html/2610.09795#bib.bib25)\)\. Recent benchmarks, including LaMP\([Salemi et al\., 2024](https://arxiv.org/html/2610.09795#bib.bib23)\)and LongLaMP\([Kumar et al\., 2024](https://arxiv.org/html/2610.09795#bib.bib24)\), have provided standardized frameworks for evaluating personalized text generation\. Existing methods for LLM personalization can be broadly categorized into three groups: retrieval\-based, fine\-tuning\-based, and user\-centric approaches\. Retrieval\-based methods incorporate user histories into the context through retrieval or augmentation strategies\([Lei et al\., 2023](https://arxiv.org/html/2610.09795#bib.bib28);[Du et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib5);[Zhang et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib6);[Liu et al\., 2026b](https://arxiv.org/html/2610.09795#bib.bib47);[Yang et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib48);[Li et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib42);[Chen et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib52)\), enabling training\-free adaptation but relying on external user information without enhancing the model’s internal preference understanding\. PEFT \-based methods adapt model parameters using user\-specific data, with recent approaches introducing preference\-aware supervision reweighting\([Zhang et al\., 2026a](https://arxiv.org/html/2610.09795#bib.bib10);[Zhao et al\., 2026b](https://arxiv.org/html/2610.09795#bib.bib11)\)\. Although effective under reference\-based similarity metrics, they mainly optimize toward reproducing user\-specific expressions and suffer from over\-adaptation\. User\-centric methods explicitly model users through dedicated adapters\([Tan et al\., 2024](https://arxiv.org/html/2610.09795#bib.bib12);[Hwang et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib50)\), embeddings, or internal interventions\([Liu et al\., 2025](https://arxiv.org/html/2610.09795#bib.bib18);[Zhang et al\., 2026b](https://arxiv.org/html/2610.09795#bib.bib13);[Wang et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib51)\), but their scalability is limited because of high costs\. Existing methods prioritize reference imitation or external preference conditioning, often under\-optimizing the intrinsic quality of generated content\. In contrast MIRROR introduces a reference\-revealed self\-distillation training paradigm for personalized generation, while MIRROR\-F further focuses optimization on informative content, enhancing the professionalism and overall textual quality of personalized generation\.

##### LLM Self\-Distillation\.

A recent line of work removes the need for an external, stronger teacher by letting a model conditioned on hindsight information supervise its own weaker self: the teacher sees privileged signals—environment feedback, demonstrations, or verified reasoning traces—while the student sees only the query, and training matches their per\-token distributions over the student’s own rollouts\([Hübotter et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib19);[Hwang et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib50);[Shenfeld et al\., 2026](https://arxiv.org/html/2610.09795#bib.bib21)\)\. Follow\-up work extends this paradigm to new settings and objectives, including long\-trace compaction\([Kim and Lee, 2026](https://arxiv.org/html/2610.09795#bib.bib29)\), cross\-lingual and long\-context reasoning, privacy\-constrained training\([Zhang et al\., 2026c](https://arxiv.org/html/2610.09795#bib.bib31)\), UI generation\([Dong, 2026](https://arxiv.org/html/2610.09795#bib.bib32)\)and policy\-gradient reformulations\([Liu et al\., 2026c](https://arxiv.org/html/2610.09795#bib.bib30)\)\. MIRROR apply on\-policy self\-distillation to LLM personalization\. We further propose a focal plug\-in\(MIRROR\-F\) mechanism for personalized generation, which improves the professionalism and textual quality of generated content without compromising personalization performance\.

Figure 5:The impact of fluctuations in the value ofα\\alphaon the performance of the model’s stylistic and syntactic personalization assessment\.

## 6Conclusion

We introduced MIRROR, an on\-policy self\-distillation framework that shifts LLM personalization from reference imitation toward preference internalization\. We further proposed MIRROR\-F, which selectively enhances informative content while preserving user\-specific expression\. Across multiple tasks, model scales, and evaluation protocols, both methods achieve strong personalization, improved text quality, and less catastrophic forgetting than existing SFT\-based approaches, demonstrating the effectiveness of on\-policy preference internalization for robust and transferable personalized generation\.

### Reproducibility statement

We provide detailed implementation and experimental settings, hyperparameters, training configurations in appendix[A\.2](https://arxiv.org/html/2610.09795#A1.SS2)\.

### AI use statement

Generative AI tools were used to assist with code debugging, LaTeX formatting, writing and polishing portions of the manuscript, and literature search for identifying potentially relevant prior work\. All AI\-assisted outputs, including text, code suggestions, and literature recommendations, were reviewed and verified by the authors before being incorporated into the final work\. Generative AI tools were not used to formulate the main research hypotheses, design the proposed methodology or experiments, derive mathematical claims or proofs, generate synthetic data, or interpret the reported experimental results\. The authors take full responsibility for the accuracy, validity, and final content of the paper\.

## References

- Amirizanianiet al\.\(2026\)M\. Amirizaniani, B\. C\. G\. Lee, J\. West, and N\. WeberTraining llms with reinforcement learning for intent\-aware personalized question answering\.arXiv preprint arXiv:2605\.12645\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p1.1)\.
- Banerjee and Lavie \(2005\)S\. Banerjee and A\. LavieMETEOR: an automatic metric for mt evaluation with improved correlation with human judgments\.InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization,pp\. 65–72\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px3.p1.1)\.
- Caoet al\.\(2026\)H\. Cao, F\. Zhang, J\. Yao, Y\. Zheng, C\. Zhao, L\. Hao, Y\. Wei, W\. Ni, H\. Fu, Y\. Sun,et al\.Beyond retrieval: learning compact user representations for scalable llm personalization\.arXiv preprint arXiv:2606\.04547\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p1.1)\.
- Chenet al\.\(2025\)Y\. Chen, X\. Liu, R\. Wang, Z\. Li, P\. Chen, C\. Yu, Q\. Yin, P\. Nigam, M\. Jiang, and B\. YinPOPI: personalizing llms via optimized natural language preference inference\.arXiv preprint arXiv:2510\.17881\.Cited by:[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Choiet al\.\(2026\)W\. Choi, Y\. Kim, Y\. Lee, and S\. YoonPGMem: tightly coupled persona\-memory graph for lifelong personalized agents\.arXiv preprint arXiv:2608\.01708\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1)\.
- Dong \(2026\)H\. DongSelf\-distillation policy optimization via visual feedback: bridging code and visual artifacts\.arXiv preprint arXiv:2606\.10334\.Cited by:[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px2.p1.1)\.
- Duet al\.\(2026\)L\. Du, Y\. Yuan, Z\. Zhao, F\. Lyu, E\. Penaloza, X\. Chen, Z\. Sun, J\. Kang, L\. Charlin, X\. Liu,et al\.Optimizing user profiles via contextual bandits for retrieval\-augmented llm personalization\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 31803–31817\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Fabbriet al\.\(2021\)A\. R\. Fabbri, W\. Kryściński, B\. McCann, C\. Xiong, R\. Socher, and D\. RadevSummeval: re\-evaluating summarization evaluation\.Transactions of the Association for Computational Linguistics9,pp\. 391–409\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px3.p1.1)\.
- Huet al\.\(2021\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLora: low\-rank adaptation of large language models\.arXiv preprint arXiv:2106\.09685\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px4.p1.1)\.
- Hübotteret al\.\(2026\)J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. K\. Buening, C\. Guestrin,et al\.Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.Cited by:[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px2.p1.1)\.
- Hwanget al\.\(2026\)E\. Hwang, K\. Mitra, D\. Zhang, H\. Kim, and E\. HruschkaHypotheses\-guided self distillation for continual personalization\.arXiv preprint arXiv:2609\.00251\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px2.p1.1)\.
- Karouzoset al\.\(2026\)C\. Karouzos, X\. Tan, and N\. AletrasAn empirical study on preference tuning generalization and diversity under domain shift\.arXiv preprint arXiv:2601\.05882\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1)\.
- Kim and Lee \(2026\)J\. Kim and D\. LeeOPSD compresses what rlvr teaches: a post\-rl compaction stage for reasoning models\.arXiv preprint arXiv:2605\.06188\.Cited by:[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px2.p1.1)\.
- Kumaret al\.\(2024\)I\. Kumar, S\. Viswanathan, S\. Yerra, A\. Salemi, R\. A\. Rossi, F\. Dernoncourt, H\. Deilamsalehy, X\. Chen, R\. Zhang, S\. Agarwal,et al\.Longlamp: a benchmark for personalized long\-form text generation\.arXiv preprint arXiv:2407\.11016\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px4.p1.1)\.
- Laiet al\.\(2026\)H\. Lai, S\. Song, S\. Niu, H\. Wang, J\. Yang, Z\. Wang, Z\. Yin, and X\. LiangRoleCDE: benchmarking and mitigating role–alignment trade\-offs in role\-playing agents\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 2226–2248\.Cited by:[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Leeet al\.\(2026\)C\. Lee, J\. Kim, and T\. GongFrom volume to value: preference\-aligned memory construction for on\-device rag\.arXiv preprint arXiv:2605\.18271\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1)\.
- Leiet al\.\(2023\)Y\. Lei, L\. Ding, Y\. Cao, C\. Zan, A\. Yates, and D\. TaoUnsupervised dense retrieval with relevance\-aware contrastive pre\-training\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 10932–10940\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2024\)X\. Li, R\. Zhou, Z\. C\. Lipton, and L\. LeqiPersonalized language modeling from personalized human feedback\.arXiv preprint arXiv:2402\.05133\.Cited by:[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2025\)Z\. Li, C\. Xi, C\. Li, D\. Chen, B\. Chen, S\. Song, S\. Niu, H\. Wang, J\. Yang, C\. Tang,et al\.Memos: a memory os for ai system\.arXiv preprint arXiv:2507\.03724\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p1.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Lianget al\.\(2025\)X\. Liang, S\. Niu, Z\. Li, S\. Zhang, H\. Wang, F\. Xiong, Z\. Fan, B\. Tang, J\. Zhao, J\. Yang,et al\.Saferag: benchmarking security in retrieval\-augmented generation of large language model\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4609–4631\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1)\.
- Lin \(2004\)C\. LinRouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px3.p1.1)\.
- Liuet al\.\(2026a\)J\. Liu, W\. Yu, Q\. Dai, Z\. Li, J\. Zhu, M\. Yang, T\. Chua, and I\. KingPerfit: exploring personalization shifts in representation space of llms\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1)\.
- Liuet al\.\(2025\)J\. Liu, Y\. Zhu, S\. Wang, X\. Wei, E\. Min, Y\. Lu, S\. Wang, D\. Yin, and Z\. DouLlms\+ persona\-plug= personalized llms\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9373–9385\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2026b\)J\. Liu, Y\. Sun, W\. Cheng, H\. Lei, Y\. Li, Y\. Chen, and D\. WangHierarchical memory orchestration for personalized persistent agents\.arXiv preprint arXiv:2604\.01670\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p1.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-eval: nlg evaluation using gpt\-4 with better human alignment\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 2511–2522\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px3.p1.1)\.
- Liuet al\.\(2026c\)Y\. Liu, S\. Zhang, Y\. Zhang, and Q\. GuSelf\-distilled policy gradient\.arXiv preprint arXiv:2606\.04036\.Cited by:[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px2.p1.1)\.
- Lyuet al\.\(2026\)Y\. Lyu, G\. Chen, R\. Shao, W\. Guan, and L\. NiePersonalalign: hierarchical implicit intent alignment for personalized gui agent with long\-term user\-centric records\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 36074–36089\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p1.1)\.
- Niet al\.\(2019\)J\. Ni, J\. Li, and J\. McAuleyJustifying recommendations using distantly\-labeled reviews and fine\-grained aspects\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\),pp\. 188–197\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px1.p1.1)\.
- Prashantet al\.\(2026\)P\. P\. Prashant, J\. Zhu, A\. Creo, and B\. SalimiFine\-tuning without forgetting via loss\-adaptive learning rates\.arXiv preprint arXiv:2605\.20005\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1)\.
- Salemiet al\.\(2025\)A\. Salemi, J\. Killingback, and H\. ZamaniExpert: effective and explainable evaluation of personalized long\-form text generation\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 17516–17532\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px3.p1.1)\.
- Salemiet al\.\(2024\)A\. Salemi, S\. Mysore, M\. Bendersky, and H\. ZamaniLamp: when large language models meet personalization\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7370–7392\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Shenet al\.\(2024\)X\. Shen, R\. Zhang, X\. Zhao, J\. Zhu, and X\. XiaoPmg: personalized multimodal generation with large language models\.InProceedings of the ACM Web Conference 2024,pp\. 3833–3843\.Cited by:[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Shenfeldet al\.\(2026\)I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. AgrawalSelf\-distillation enables continual learning\.arXiv preprint arXiv:2601\.19897\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p4.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px2.p1.1)\.
- Tanet al\.\(2024\)Z\. Tan, Q\. Zeng, Y\. Tian, Z\. Liu, B\. Yin, and M\. JiangDemocratizing large language models via personalized parameter\-efficient fine\-tuning\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 6476–6491\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Tanet al\.\(2026\)Z\. Tan, Z\. Zhang, H\. Wen, Z\. Li, R\. Zhang, P\. Chen, F\. Mo, Z\. Liu, Q\. Zeng, Q\. Yin,et al\.Instant personalized large language model adaptation via hypernetwork\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 23557–23580\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1)\.
- Teamet al\.\(2026\)K\. Team, T\. Bai, Y\. Bai, Y\. Bao, J\. Cai, X\. Cai, P\. Cao, Y\. Cao, Z\. Chai, Y\. Charles,et al\.Kimi k3: open frontier intelligence\.arXiv preprint arXiv:2607\.24653\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p1.1)\.
- Wanget al\.\(2026\)B\. Wang, Y\. Du, Y\. Pei, and Z\. WangFrom explicit to implicit: a theoretical framework and transfer method for preference internalization in language models\.TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS14,pp\. 1074–1095\.Cited by:[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)N\. Wang, Z\. Peng, H\. Que, J\. Liu, W\. Zhou, Y\. Wu, H\. Guo, R\. Gan, Z\. Ni, J\. Yang,et al\.Rolellm: benchmarking, eliciting, and enhancing role\-playing abilities of large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 14743–14777\.Cited by:[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)X\. Wang, H\. Wang, Y\. Zhang, X\. Yuan, R\. Xu, J\. Huang, S\. Yuan, H\. Guo, J\. Chen, S\. Zhou,et al\.CoSER: coordinating llm\-based persona simulation of established roles\.\.InICML,Cited by:[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Yanet al\.\(2026\)Y\. Yan, S\. Wu, and T\. LeLocating and controlling implicit personalization in large language models\.arXiv preprint arXiv:2608\.11735\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px2.p1.1)\.
- Yanget al\.\(2026\)Y\. Yang, J\. Li, Q\. Pan, B\. Zhan, Y\. Cai, L\. Du, J\. Zhou, K\. Chen, Q\. Chen, X\. Li,et al\.Autoskill: experience\-driven lifelong learning via skill self\-evolution\.arXiv preprint arXiv:2603\.01145\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p1.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Zenget al\.\(2026\)L\. Zeng, S\. Zhang, and X\. ZhangEviDx: evidence\-aware active diagnosis with scaffolded llm agents\.arXiv preprint arXiv:2608\.24570\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p1.1)\.
- Zhanget al\.\(2026a\)C\. Zhang, Y\. Lu, L\. Fang, C\. Zheng, J\. Chai, X\. Wang, G\. Yin, W\. Lin, Y\. Wang, and Z\. LinRethinking personalization in large language models at the token level\.arXiv preprint arXiv:2603\.06595\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2019\)T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. ArtziBertscore: evaluating text generation with bert\.arXiv preprint arXiv:1904\.09675\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2026b\)W\. Zhang, Y\. Yuan, C\. Han, Y\. Tian, Z\. Sun, L\. Du, J\. Kang, H\. Kang, X\. Liu, and H\. WuPreference heads in large language models: a mechanistic framework for interpretable personalization\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 47742–47754\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025\)W\. Zhang, X\. Zhang, C\. Zhang, L\. Yang, J\. Shang, Z\. Wei, H\. P\. Zou, Z\. Huang, Z\. Wang, Y\. Gao,et al\.Personaagent: when large language model agents meet personalization at test time\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026c\)X\. Zhang, Z\. Ding, T\. Pan, R\. Yang, C\. Kang, X\. Xiong, and J\. GuOpsdl: on\-policy self\-distillation for long\-context language models\.arXiv preprint arXiv:2604\.17535\.Cited by:[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2026a\)S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. GroverSelf\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p4.1)\.
- Zhaoet al\.\(2026b\)X\. Zhao, J\. Jun, Y\. Zhang, W\. Wang, H\. Cheng, F\. Feng, S\. Ng, and T\. ChuaNextquill: causal preference modeling for enhancing llm personalization\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 118745–118770\.Cited by:[§1](https://arxiv.org/html/2610.09795#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2610.09795#S5.SS0.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2024\)Z\. Zheng, W\. Chao, Z\. Qiu, H\. Zhu, and H\. XiongHarnessing large language models for text\-rich sequential recommendation\.InProceedings of the ACM Web Conference 2024,pp\. 3207–3216\.Cited by:[§4\.1](https://arxiv.org/html/2610.09795#S4.SS1.SSS0.Px2.p1.1)\.

## Appendix AAppendix

### A\.1More Case Study

To provide a more detailed qualitative analysis, we present one complete question–answer case for each personalization task\. Each case includes the task input, the gold answer, the responses generated by different methods, and a task\-specific comparison\. The selected examples are drawn from the evaluation set and are intended to illustrate how MIRROR and MIRROR\-F balance reference alignment, user\-specific expression, and intrinsic content quality\.

#### A\.1\.1Abstract Generation

##### Question\.

The input asks the model to generate an abstract for the title “Learning the Language: The Importance of Studying Written Directions in Designing Navigational Technologies for the Blind\.” The user\-provided key information consists of five aspects: independent navigation, text\-to\-speech directions, language and cues, qualitative analysis, and landmarks and alternate routes\.

##### Analysis\.

The responses exhibit clear differences in both content fidelity and unsupported detail\. The base model covers most of the requested concepts, including text\-to\-speech directions, language, landmarks, and alternate routes, but its discussion remains relatively generic\. NextQuill produces a fluent abstract and explicitly mentions landmarks and alternate routes\. However, it introduces the unsupported detail that the study involved “11 blind participants” and adds a preference for a conversational tone, neither of which is included in the task input or the gold answer\.

Table 3:Question–answer case for Abstract Generation\.ItemContentQuestionGenerate an abstract for the title “Learning the Language: The Importance of Studying Written Directions in Designing Navigational Technologies for the Blind” using the following items: \(1\) Independent navigation; \(2\) Text\-to\-Speech directions; \(3\) Language and cues; \(4\) Qualitative analysis; and \(5\) Landmarks and alternate routes\.Gold answerIndependent navigation is important to individuals who are blind and visually impaired \(VI\)\. Researchers have long explored how blind and VI people navigate to inform the design of more useful, accessible wayfinding devices\. However, there has been little research on the role language plays in providing effective text\-to\-speech directions for this population\. Through qualitative analysis, this work examines the language and cues expressed in written navigational directions, including how distances are represented, how direction is indicated, and what landmarks are referenced\. The analysis further identifies warnings about overshooting a destination, alternative routes that are easier to navigate, and the accessibility of destinations for people with disabilities\.Table 4:Different model answers for the Abstract Generation case\.MethodAnswerBaseThis paper explores the importance of studying written directions in the design of navigational technologies for the blind, focusing on how independent navigation can be enhanced through effective use of text\-to\-speech directions\. We conducted a qualitative analysis of how language and cues are used in written directions to guide users, emphasizing the need for clear, concise, and contextually relevant information\. The study also examines the role of landmarks and alternate routes in providing users with multiple options for navigation, ensuring that they can make informed decisions in complex environments\. Our findings highlight the significance of incorporating user\-centered design principles that consider the unique needs and experiences of blind individuals when developing navigational technologies\.NextQuillIndependent navigation is a challenge for many people who are blind\. One promising solution is to use text\-to\-speech directions to guide them\. However, there has been little inquiry into how people who are blind use and interpret directions, and how technologies can be designed to better support them\. We conducted a qualitative analysis of 11 blind participants’ experiences with and suggestions for text\-to\-speech directions\. We found that users rely on language and cues to interpret directions, and that they had strong preferences for how these directions should be presented\. We present two findings that can be used to guide the design of future text\-to\-speech directions: \(1\) the use of landmarks and alternate routes can be beneficial to users, and \(2\) users prefer directions that are presented in a conversational tone\.MIRRORIndependent navigation is a critical skill for individuals who are blind, enabling them to move confidently and safely in their environment\. However, the effectiveness of navigational technologies for the blind often hinges on the clarity and usability of the directions provided\. This paper explores the importance of studying written directions in the design of navigational technologies for the blind, focusing on how language and cues influence user experience\. Through qualitative analysis of user interactions, we examine how text\-to\-speech directions are perceived and understood by users, highlighting the role of language in conveying information about landmarks and alternate routes\. Our findings reveal that the way directions are structured and the cues used can significantly impact the user’s ability to navigate independently\. We argue that a deeper understanding of language and cues in written directions is essential for designing more effective navigational technologies that support the unique needs of blind users\.MIRROR\-FIndependent navigation for the blind is significantly influenced by the clarity and effectiveness of navigational technologies\. This study emphasizes the importance of studying written directions in the design of navigational technologies\. Through qualitative analysis, we explore how text\-to\-speech directions, language, and cues impact the user’s ability to navigate independently\. The study highlights the role of language in conveying information and the necessity of clear, concise cues that aid in understanding the environment\. We examine how landmarks and alternate routes are described and how these elements can be integrated into navigational systems to enhance usability\. The findings underscore the need for a deep understanding of language in the design process, ensuring that navigational technologies are not only functional but also intuitive and accessible for users with visual impairments\.MIRROR retains the main research focus while avoiding the unsupported participant count\. It connects independent navigation with the clarity of directions and explicitly relates language and cues to landmarks and alternative routes\. This indicates that the model is not merely reproducing isolated reference phrases, but is reconstructing the central research argument\. MIRROR\-F further strengthens the content\-oriented organization\. It explicitly distinguishes text\-to\-speech directions, language, cues, landmarks, and alternate routes, and connects these elements to the usability and accessibility of navigation systems\. The output is also more concise and avoids the speculative participant\-level details introduced by NextQuill\. The case therefore demonstrates the ability of MIRROR\-F to improve content focus while preserving the central semantic structure of the personalized reference\.

#### A\.1\.2News Headline Generation

##### Question\.

The input asks the model to generate a headline for a fashion article describing H&M’s updated classics and the appeal of clothing for readers whose “leggy limbs” have not appeared in fashion magazines\.

Table 5:Question–answer case for News Headline Generation\.ItemContentQuestionGenerate a headline for the following article: “Um, OK\. For those of us whose leggy limbs haven’t been immortalized in print, we can turn to H&M’s updated classics, like …”Gold answerH&M Model Collection Inspired By Joan Smalls, Daphne Groeneveld & More \(PHOTOS\)Table 6:Different model answers for the News Headline Generation case\.MethodAnswerPerCE7 Ways To Make Your Pantyhose Look Like A Model \(PHOTOS\)NextQuillJenna Holmes’ ’90s Style Is Too Retro For A 2013 Cover \(PHOTOS\)MIRRORH&M’s Updated Classics: Leggy Limbs For Every Body Type \(PHOTOS\)MIRROR\-FH&M’s Updated Classics: Leggy Limbs For Every Style \(PHOTOS\)
##### Analysis\.

This case highlights the difference between topical relevance and personalized headline generation\. PerCE produces a fluent fashion headline, but it changes the article topic from an H&M collection to general advice about pantyhose\. NextQuill preserves the magazine\-style “\(PHOTOS\)” suffix, but introduces a different person, a different time period, and a different editorial topic\. Both outputs therefore reproduce the broad fashion domain without accurately identifying the article\-specific subject\.

MIRROR correctly retains “H&M” and “updated classics,” which are the most important topical cues in the article\. It also preserves the compact tabloid\-style structure and the “\(PHOTOS\)” suffix\. The phrase “leggy limbs” further maintains the distinctive expressive pattern of the input\. MIRROR\-F retains all of these properties while replacing the narrower “body type” expression with the more general “style\.” This produces a headline that remains faithful to the fashion context and the user’s preferred editorial format while being less restrictive in its description of the target audience\. The example demonstrates that the proposed methods can preserve both the publication\-specific form and the article\-specific content, whereas the SFT\-based baselines tend to drift toward memorized or superficially related fashion expressions\.

#### A\.1\.3Paper Topic Generation

##### Question\.

The input asks the model to generate a title for a paper on converting discrete\-time single\-input single\-output piecewise affine state\-space models into equivalent input\-output representations\. The abstract emphasizes necessary and sufficient conditions, a constructive conversion procedure, the growth in the number of modes and parameters, and numerical examples\.

Table 7:Question–answer case for Paper Topic Generation\.ItemContentQuestionGenerate a title for the following abstract: “This paper addresses the conversion of discrete\-time single input\-single output PieceWise Affine \(PWA\) models from state space to input\-output form\. Necessary and sufficient conditions are given for a PWA state space model to admit equivalent input\-output representations\. When an equivalent input\-output model exists, a constructive procedure is presented to derive both its parameters and the partition of the regressors domain\. It is shown that the number of modes and the number of parameters may grow considerably when converting a PWA state space model into an equivalent input\-output representation\. Numerical examples highlight the role of the derived necessary and sufficient conditions for input\-output realization of PWA state space models\.”Gold answerInput\-Output Realization Of Piecewise Affine State Space ModelsTable 8:Different model answers for the Paper Topic Generation case\.MethodAnswerPerCEOn the Input\-Output Representation of Piecewise Affine State Space ModelsNextQuillInput\-output realization of piecewise affine state space modelsMIRRORConversion of Discrete\-Time Piecewise Affine Models to Input\-Output FormMIRROR\-FConversion of Piecewise Affine State Space Models to Input\-Output Form
##### Analysis\.

All methods identify the main technical topic, but they differ in how they balance semantic coverage, readability, and title style\. PerCE produces a valid title, although “representation” is less specific than the “realization” terminology used in the reference\. NextQuill closely follows the gold title and provides a concise answer, but it mainly reproduces the surface form of the reference\.

MIRROR reformulates the title using the operation “conversion,” which directly reflects the central procedure described in the abstract\. It also preserves the important pair of technical concepts, namely piecewise affine models and input\-output form\. MIRROR\-F produces the most compact formulation among the two proposed methods while retaining the essential technical relation\. It removes the unnecessary “discrete\-time” qualifier for a title\-level summary but preserves the central transformation from state\-space models to input\-output form\. The output is therefore concise, technically informative, and consistent with the scholarly title style expressed in the user’s history\. This example illustrates how MIRROR\-F can focus supervision on content\-bearing terminology without forcing the model to copy the reference title verbatim\.

##### Overall observation\.

The three cases reveal a consistent distinction between reference imitation and preference internalization\. In Abstract Generation, MIRROR and MIRROR\-F avoid unsupported details while preserving the research motivation and findings\. In News Headline Generation, they maintain the article\-specific topic and the user’s preferred headline format instead of producing broadly related fashion headlines\. In Paper Topic Generation, they preserve the central technical relation while generating concise titles with appropriate scholarly phrasing\. MIRROR\-F further strengthens the content\-bearing portions of the outputs, especially when the task requires the model to identify informative entities, technical concepts, or article\-specific themes\. These qualitative examples support the view that on\-policy reference\-revealed self\-distillation can improve personalized generation without reducing the factual and practical quality of the generated text\.

### A\.2Training Hyperparameters

MIRROR and MIRROR\-F were trained under the same experimental configuration for a controlled comparison\. Unless otherwise stated, both methods used the same backbone, training data, context construction procedure, optimizer, learning rate, batch size, training steps, rollout strategy, and random seed\. The only difference was the training objective: MIRROR used on\-policy distributional alignment alone, whereas MIRROR\-F additionally applied the selective grounded anchoring loss\. The shared configuration used for the comparison is summarized in Table[9](https://arxiv.org/html/2610.09795#A1.T9)\.

Table 9:Training hyperparameters used for the MIRROR and MIRROR\-F comparison\.CategoryHyperparameterValueModel and dataBackboneQwen3\-1\.7B/4BDatasetLongLaMP,LaMP\-4,LaMP\-5OptimizationOptimizerAdamWLearning rate1×10−41\\times 10^\{\-4\}Per\-device batch size1Gradient accumulation1Warm\-up ratio0\.05Maximum gradient norm1\.0LoRARankrr16Scaling factorαLoRA\\alpha\_\{\\mathrm\{LoRA\}\}32Dropout0\.05Trainable parametersLoRA parameters onlyOn\-policy rolloutMaximum training steps200 \(Qwen3\-1\.7B\) / 250 \(Qwen3\-4B\)Sampling temperature1\.0Top\-pp1\.0Rollout length ratio1\.3×\\timesreference lengthMinimum / maximum rollout length64 / 400 tokensMIRRORDivergenceGeneralized JSDMixing coefficientβ\\beta0\.5Anchoring lossNot usedMIRROR\-FDivergenceGeneralized JSDMixing coefficientβ\\beta0\.5Focal coefficientα\\alpha0\.5ReproducibilityRandom seed456166,9874,1123Thinking modeDisabledFor both methods, the student generated on\-policy rollouts using the same sampling configuration, and the teacher used the same reference\-revealed context\. The user history, task questions, context truncation strategy, and reference\-answer budget were also kept identical\. Therefore, the comparison was conducted under one shared training setup, with the selective grounded anchoring term being the only method\-specific addition in MIRROR\-F\.

### A\.3Rule\-Based Information Completeness Analysis

To complement the LLM\-based quality evaluation, we additionally measure information completeness using a deterministic, rule\-based information recall metric\. This metric is designed to quantify whether a model preserves specific information\-bearing words from the reference output, rather than whether the generated text is generally fluent or semantically coherent\. Therefore, it should be interpreted as a targeted content\-coverage measure and is complementary to the G\-Eval professional\-quality scores\.

For each exampleii, letgig\_\{i\}denote the gold reference,cic\_\{i\}denote the student input context, andyi,my\_\{i,m\}denote the output generated by modelmm\. We first extract word\-like units using the regular expression

𝒲⁡\(x\)=\{w∣w​matches​\[A\-Za\-z0\-9\]​\[A\-Za\-z0\-9’\\\-\]∗​in​x\}\.\\mathcal\{W\}\(x\)=\\left\\\{w\\mid w\\text\{ matches \}\[\\texttt\{A\-Za\-z0\-9\}\]\[\\texttt\{A\-Za\-z0\-9'\\textbackslash\-\}\]^\{\*\}\\text\{ in \}x\\right\\\}\.This extractor treats sequences containing letters, digits, apostrophes, and hyphens as word\-level units and avoids substring matching\. For a wordww, we define its alphanumeric core as

core⁡\(w\)=filteralnum⁡\(w\),\\operatorname\{core\}\(w\)=\\operatorname\{filter\}\_\{\\mathrm\{alnum\}\}\(w\),and use lowercase forms for case\-insensitive matching\.

Following the focal\-token selection rules used by MIRROR\-F, a word is considered informative if it contains explicit lexical signals of information\-bearing content\. Formally,

Informative⁡\(w\)=𝕀⁡\[Digit⁡\(w\)∨InnerUpper⁡\(w\)∨HeadUpper⁡\(w\)\],\\operatorname\{Informative\}\(w\)=\\mathbb\{I\}\\left\[\\operatorname\{Digit\}\(w\)\\lor\\operatorname\{InnerUpper\}\(w\)\\lor\\operatorname\{HeadUpper\}\(w\)\\right\],where

Digit\(w\)=𝕀\[∃j:core\(w\)j∈\{0,…,9\}\],\\operatorname\{Digit\}\(w\)=\\mathbb\{I\}\\left\[\\exists j:\\operatorname\{core\}\(w\)\_\{j\}\\in\\\{0,\\ldots,9\\\}\\right\],InnerUpper\(w\)=𝕀\[∃j≥2:core\(w\)jis uppercase\],\\operatorname\{InnerUpper\}\(w\)=\\mathbb\{I\}\\left\[\\exists j\\geq 2:\\operatorname\{core\}\(w\)\_\{j\}\\text\{ is uppercase\}\\right\],and

HeadUpper\(w\)=𝕀\[\|core\(w\)\|≥3∧core\(w\)1is uppercase∧lower\(core\(w\)\)∉𝒮stop\]\.\\operatorname\{HeadUpper\}\(w\)=\\mathbb\{I\}\\left\[\|\\operatorname\{core\}\(w\)\|\\geq 3\\land\\operatorname\{core\}\(w\)\_\{1\}\\text\{ is uppercase\}\\land\\operatorname\{lower\}\(\\operatorname\{core\}\(w\)\)\\notin\\mathcal\{S\}\_\{\\mathrm\{stop\}\}\\right\]\.Here,𝒮stop\\mathcal\{S\}\_\{\\mathrm\{stop\}\}is the fixed stopword set used by the training\-time focal selector\. Words whose alphanumeric core has fewer than two characters are not treated as informative\.

We further require a candidate reference word to be grounded in the available student context\. Let

𝒞i=\{lower⁡\(w\)∣w∈𝒲⁡\(ci\)\}\\mathcal\{C\}\_\{i\}=\\left\\\{\\operatorname\{lower\}\(w\)\\mid w\\in\\mathcal\{W\}\(c\_\{i\}\)\\right\\\}denote the set of lowercase word units extracted from the input context\. Then the grounding condition is

Grounded\(w,ci\)=𝕀\[lower\(core\(w\)\)∈𝒞i\]\.\\operatorname\{Grounded\}\(w,c\_\{i\}\)=\\mathbb\{I\}\\left\[\\operatorname\{lower\}\(\\operatorname\{core\}\(w\)\)\\in\\mathcal\{C\}\_\{i\}\\right\]\.Thus, a reference word is counted only when it is both informative and explicitly recoverable as a complete word from the student input\. In particular, the criterion does not count arbitrary substring matches\.

The target set of information\-bearing words for exampleiiis therefore defined as

𝒮i=\{lower\(core\(w\)\)\|w∈𝒲\(gi\),Informative\(w\)=1,Grounded\(w,ci\)=1\}\.\\mathcal\{S\}\_\{i\}=\\left\\\{\\operatorname\{lower\}\(\\operatorname\{core\}\(w\)\)\\;\\middle\|\\;w\\in\\mathcal\{W\}\(g\_\{i\}\),\\;\\operatorname\{Informative\}\(w\)=1,\\;\\operatorname\{Grounded\}\(w,c\_\{i\}\)=1\\right\\\}\.The corresponding output set for modelmmis

𝒪i,m=\{lower⁡\(w\)∣w∈𝒲⁡\(yi,m\)\}\.\\mathcal\{O\}\_\{i,m\}=\\left\\\{\\operatorname\{lower\}\(w\)\\mid w\\in\\mathcal\{W\}\(y\_\{i,m\}\)\\right\\\}\.Both sets are implemented as sets rather than multisets\. Consequently, repeating the same word multiple times does not increase the score\.

For examples with a non\-empty target set, the information recall is computed as

Ri,m=\|𝒮i∩𝒪i,m\|\|𝒮i\|\.R\_\{i,m\}=\\frac\{\\left\|\\mathcal\{S\}\_\{i\}\\cap\\mathcal\{O\}\_\{i,m\}\\right\|\}\{\\left\|\\mathcal\{S\}\_\{i\}\\right\|\}\.Examples for which\|𝒮i\|=0\\left\|\\mathcal\{S\}\_\{i\}\\right\|=0are excluded because they contain no eligible grounded information\-bearing word and therefore do not define a meaningful recall denominator\. Let

ℐm,t=\{i∈𝒟t∣\|𝒮i\|\>0\}\\mathcal\{I\}\_\{m,t\}=\\left\\\{i\\in\\mathcal\{D\}\_\{t\}\\mid\|\\mathcal\{S\}\_\{i\}\|\>0\\right\\\}be the valid examples for modelmmon tasktt\. The reported information\-completeness score is the arithmetic mean of the per\-example recalls:

ICm,t=1\|ℐm,t\|​∑i∈ℐm,tRi,m\.\\operatorname\{IC\}\_\{m,t\}=\\frac\{1\}\{\|\\mathcal\{I\}\_\{m,t\}\|\}\\sum\_\{i\\in\\mathcal\{I\}\_\{m,t\}\}R\_\{i,m\}\.This aggregation first computes recall independently for each example and then averages across examples; it is not obtained by pooling all words from the dataset into a single global numerator and denominator\.

Table[10](https://arxiv.org/html/2610.09795#A1.T10)reports the resulting scores\. In the table, OPD corresponds to MIRROR, while OPD\-sftel corresponds to MIRROR\-F\. Bold numbers indicate the best performance among trained methods, and underlined numbers indicate the second\-best trained method\. The Base model is included as a non\-personalized reference but is excluded from the ranking\.

Table 10:Rule\-based information completeness evaluation\. Higher values indicate better coverage of information\-bearing content\. Bold numbers denote the best performance among trained methods, while underlined numbers denote the second\-best\. The Base model is reported as a non\-personalized reference and is excluded from the ranking\.BackboneTaskBaseContextSFTPerCENextQuillMIRROR\(Ours\)MIRROR\-F\(Ours\)Qwen3\-1\.7BAbstract0\.9330\.8950\.7850\.8390\.9490\.952News0\.7740\.4470\.4040\.3580\.3870\.631Paper0\.5440\.5660\.5270\.5210\.5880\.596Qwen3\-4BAbstract0\.9510\.9190\.8540\.8190\.9570\.963News0\.6860\.4580\.4160\.4610\.6760\.640Paper0\.6450\.5640\.5570\.5430\.6150\.624Several observations emerge from Table[10](https://arxiv.org/html/2610.09795#A1.T10)\. First, MIRROR\-F achieves the highest information\-completeness score in five of the six backbone–task combinations\. Its advantage is particularly pronounced on Qwen3\-1\.7B News, where it reaches0\.6310\.631, compared with0\.4470\.447for ContextSFT and0\.3870\.387for MIRROR\. This result suggests that selective grounded anchoring helps preserve concrete information\-bearing content in a setting where generic supervised adaptation substantially reduces coverage\.

Second, MIRROR and MIRROR\-F consistently outperform the other trained personalization methods on the Abstract and Paper Topic tasks\. On Abstract, the two methods achieve0\.9490\.949and0\.9520\.952for Qwen3\-1\.7B and0\.9570\.957and0\.9630\.963for Qwen3\-4B, respectively\. The same pattern holds on Paper Topic, where MIRROR\-F obtains0\.5960\.596and0\.6240\.624on the two backbones\. These results are consistent with the intended role of selective anchoring: the model is encouraged to retain grounded, information\-bearing content while the reference\-revealed on\-policy objective provides broader personalization guidance\.

The only exception is Qwen3\-4B News, where MIRROR obtains0\.6760\.676and outperforms MIRROR\-F at0\.6400\.640\. This difference indicates that selective anchoring is not uniformly beneficial for every task and backbone\. In this case, the unanchored MIRROR objective may provide a better trade\-off between content retention and flexible headline generation\. Overall, the rule\-based metric supports the conclusion that MIRROR\-family methods generally improve the preservation of targeted contextual information, while also showing that the effect of focal anchoring remains task\-dependent\.

Finally, this metric should not be interpreted as a replacement for semantic\-quality evaluation\. It measures exact word\-level coverage of a predefined set of grounded information\-bearing terms and does not capture paraphrases, factual entailment, fluency, coherence, or professional style\. Accordingly, we use it as a transparent diagnostic of information preservation and interpret it jointly with the G\-Eval quality results\.

### A\.4Rules for Focal Token Selection

The focal selector in MIRROR\-F provides a lightweight mechanism for identifying reference tokens that may carry important, context\-accessible information\. Rather than applying uniform supervision to the entire reference, additional supervision is concentrated on positions that satisfy two conditions: an informative surface form and lexical presence in the student’s input\. This design provides an initial approximation to key\-information extraction without requiring an external annotator, a named\-entity recognition model, or an additional LLM\. Importantly, the selector is modular: its rules can be edited to reflect different requirements for text quality while retaining the same on\-policy distillation framework\.

##### Selection inputs and outputs\.

LetcSc\_\{S\}denote the student’s input content, consisting of the retained user history and the current question, and lety=\(y1,…,yT\)y=\(y\_\{1\},\\ldots,y\_\{T\}\)denote the tokenized reference\. The selector produces a binary mask

𝐦=\(m1,…,mT\),mt∈\{0,1\},\\mathbf\{m\}=\(m\_\{1\},\\ldots,m\_\{T\}\),\\qquad m\_\{t\}\\in\\\{0,1\\\},\(7\)wheremt=1m\_\{t\}=1indicates that reference tokenyty\_\{t\}receives the additional anchoring supervision\. The mask is computed from the reference and the student\-side context, not from the teacher’s reference\-revealed prompt\. Consequently, the presence of a word in the reference alone is insufficient for its selection\.

##### Subword grouping\.

Selection is performed over word\-like groups before being mapped back to token positions\. In the implementation, a new group begins at the first reference token or at a token whose vocabulary representation starts with the whitespace markerĠ\. Consecutive tokens are collected until the next such boundary\. For each groupgjg\_\{j\}, the token strings are concatenated, the whitespace markers are removed, and only alphanumeric characters are retained to construct a core stringwjw\_\{j\}\.

This grouping prevents a multi\-subword term from receiving supervision on only an arbitrary fragment\. Once a group is selected, all token positions in that group are marked\. Since grouping follows tokenizer whitespace markers rather than a linguistic parser, attached punctuation can also belong to a selected group\.

##### Rule 1: informative surface forms\.

Three inexpensive surface\-form tests are used to identify candidate information\-bearing groups:

1. 1\.Numeric content\.The core contains at least one digit\. This test can identify quantities, years, and alphanumeric identifiers\.
2. 2\.Internal capitalization\.At least one character after the first character is uppercase\. This test can identify abbreviations and mixed\-case names, such asPWA,DBT, andiPhone\.
3. 3\.Initial capitalization\.The core begins with an uppercase character, contains at least three characters, and is not included in a predefined stopword set\. This test can identify names and capitalized topic terms\.

Groups whose cores contain fewer than two characters are excluded before these tests are applied\. In particular, the current rule does not select an isolated single\-digit core merely because it is numeric\. Formally,

I\(w\)=𝕀\[\|w\|≥2\]𝕀\[D\(w\)∨Uinner\(w\)∨\(\|w\|≥3∧Ufirst\(w\)∧lower\(w\)∉ℬ\)\],I\(w\)=\\mathbb\{I\}\[\|w\|\\geq 2\]\\,\\mathbb\{I\}\\\!\\left\[D\(w\)\\lor U\_\{\\mathrm\{inner\}\}\(w\)\\lor\\left\(\|w\|\\geq 3\\land U\_\{\\mathrm\{first\}\}\(w\)\\land\\operatorname\{lower\}\(w\)\\notin\\mathcal\{B\}\\right\)\\right\],\(8\)whereD⁡\(w\)D\(w\)indicates the presence of a digit,Uinner​\(w\)U\_\{\\mathrm\{inner\}\}\(w\)indicates an uppercase character after the first position,Ufirst​\(w\)U\_\{\\mathrm\{first\}\}\(w\)indicates an uppercase initial character, andℬ\\mathcal\{B\}is the stopword set\.

The stopword filter is applied only to the initial\-capitalization branch\. It suppresses common sentence\-initial words such asThe,This, andWe, while preserving the numeric and internal\-capitalization tests\. The implementation uses the following set:

Stopwords for the initial\-capitalization rulethe, a, an, this, that, these, those, it, its, he, she, they, we, you, i, in, on, at, of, to, for, and, or, but, if, then, so, as, is, are, was, were, be, been, being, have, has, had, do, does, did, will, would, can, could, may, might, must, should, i’m, it’s, there, here, what, when, where, who, why, how, not, no, yes, with, without, from, by, about, into, over, under, after, before, my, your, his, her, our, their\.

These tests are surface\-form heuristics rather than semantic entity classification\. Their purpose is to obtain a transparent initial set of potentially important terms, not to identify every content\-bearing expression\.

##### Rule 2: lexical grounding in the student context\.

An informative group is retained only if its lowercased core is present in the student\-side context vocabulary\. This vocabulary is constructed using the regular expression

`\[A\-Za\-z0\-9\]\[A\-Za\-z0\-9'\\\-\]\*`

and lowercasing the extracted strings\. Let𝒱⁡\(cS\)\\mathcal\{V\}\(c\_\{S\}\)denote the resulting set\. The grounding condition is

G\(w,cS\)=𝕀\[lower\(w\)∈𝒱\(cS\)\]\.G\(w,c\_\{S\}\)=\\mathbb\{I\}\\\!\\left\[\\operatorname\{lower\}\(w\)\\in\\mathcal\{V\}\(c\_\{S\}\)\\right\]\.\(9\)
Membership is checked against complete extracted strings rather than arbitrary substrings\. For example, the coreIDis not considered present merely because the context containsvideooridentify\. The lookup is case\-insensitive, so a capitalized reference term can match its lowercase occurrence in the input\.

This condition makes the auxiliary target depend on information that is lexically accessible to the student\. Here,*grounding*specifically denotes lexical presence; it does not constitute a semantic entailment test\. The implementation also uses different normalization procedures for the reference core and the context vocabulary: punctuation is removed from reference cores, whereas internal apostrophes and hyphens are retained in context matches\. Thus, expressions such asInput\-Outputare not automatically treated as identical to the reference coreInputOutput\. No stemming, synonym matching, or entity linking is applied\.

##### Rule 3: expansion to token positions\.

The final selection criterion is the conjunction of informativeness and lexical grounding:

A⁡\(wj,cS\)=I⁡\(wj\)​G​\(wj,cS\)\.A\(w\_\{j\},c\_\{S\}\)=I\(w\_\{j\}\)G\(w\_\{j\},c\_\{S\}\)\.\(10\)The selected token set is then defined as

𝒯sel=⋃j:A⁡\(wj,cS\)=1gj,mt=𝕀\[t∈𝒯sel\]\.\\mathcal\{T\}\_\{\\mathrm\{sel\}\}=\\bigcup\_\{j:A\(w\_\{j\},c\_\{S\}\)=1\}g\_\{j\},\\qquad m\_\{t\}=\\mathbb\{I\}\[t\\in\\mathcal\{T\}\_\{\\mathrm\{sel\}\}\]\.\(11\)All subword positions within an accepted group are included\. The end\-of\-sequence token is explicitly excluded from the selective mask\.

##### Illustrative cases\.

The following controlled examples explain the decisions made by the implemented rules\. They are constructed illustrations of the selector, rather than additional empirical generation results\. The context excerpts shown in each box are treated as the complete context for the corresponding lexical check; no additional occurrences in user history are assumed\. Boldface in the reference identifies the word\-like groups selected by the rules\.

Case 1: Abstract generation—method names and numerical informationStudent context\.Generate an abstract for a study of DBT using 120 participants recruited in London\.Reference illustration\.This study evaluatesDBTwith120participants inLondonand compares outcomes with a Paris cohort\.Selection decisions\.•DBT: selected because it contains internal uppercase characters and occurs in the context\.•120: selected because it contains digits, has at least two characters, and occurs in the context\.•London: selected because it is initially capitalized, is not a stopword, and occurs in the context\.•Paris: rejected by the grounding condition, despite satisfying the capitalization rule\.•This: rejected by the stopword filter\.•participants: not selected because its lowercase form does not satisfy an informativeness test\.

This example illustrates how a simple selector can identify several details that are important for scientific reporting: a method abbreviation, a sample size, and a study location\. The reference\-only location is excluded from the auxiliary target because it is absent from the student’s context\. The remaining reference positions are not uniformly anchored\. Thus, additional supervision is concentrated on selected details without requiring reproduction of the complete reference wording\.

Case 2: News headline generation—entities and mixed\-case namesStudent context\.Generate a headline for the following article: Apple introduced an iPhone update in London\.Reference illustration\.AppleUnveilsiPhoneUpdate inLondonSelection decisions\.•Apple: selected by initial capitalization and context membership\.•iPhone: selected by internal capitalization and context membership, although its first character is lowercase\.•Update: selected because the reference form is capitalized and its lowercase form matchesupdatein the article\.•London: selected by initial capitalization and context membership\.•Unveils: not selected because that exact lexical form is absent from the context\.•in: not selected\.

Here, the selected groups recover the organization, product name, event topic, and location\. These elements form a useful preliminary description of the news content\. The example also shows that the rule is not restricted to formally recognized named entities: title capitalization allows an ordinary topic word such asUpdateto be selected\. This behavior is useful for a lightweight content selector, while making its dependence on the reference’s surface form explicit\.

Case 3: Paper title generation—technical terminologyStudent context\.Generate a title for the following abstract of a paper: This paper studies piecewise affine models and gives conditions for their realization\.Reference illustration\.RealizationofPiecewiseAffineModelsSelection decisions\.•Realization,Piecewise,Affine, andModels: selected because they are capitalized in the reference and their lowercase forms occur in the abstract\.•of: not selected\.•IfPiecewiseis represented by multiple subword tokens, all tokens in its word\-like group receive mask value 1\.

The selected positions jointly describe the central technical object and operation of the paper\. Although no phrase\-level parser is used, adjacent accepted groups can retain a multiword expression such as*piecewise affine models*\. This illustrates the intended role of the selector: a small set of explicit rules can already identify useful information\-bearing positions, providing a starting point for more specialized annotation policies\.

##### Use in the anchoring objective\.

The mask controls the reference\-token loss, not the contents of an additional prompt\. In particular, the selected words are not presented to the model as a separate instruction or annotation list\. The student is evaluated under its ordinary context and the reference prefix, and the token\-level negative log\-likelihood is averaged only over selected positions:

ℒanc​\(θ\)=−∑t=1Tmt​log⁡πθ​\(yt∣cS,y<t\)max⁡\(1,∑t=1Tmt\)\.\\mathcal\{L\}\_\{\\mathrm\{anc\}\}\(\\theta\)=\-\\frac\{\\sum\_\{t=1\}^\{T\}m\_\{t\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid c\_\{S\},y\_\{<t\}\)\}\{\\max\\\!\\left\(1,\\sum\_\{t=1\}^\{T\}m\_\{t\}\\right\)\}\.\(12\)
Normalizing by the number of selected tokens prevents the auxiliary signal from being diluted by the length of the complete reference\. Unselected reference tokens remain part of the teacher\-forced prefix but do not receive a direct anchoring loss at their own prediction positions\. Meanwhile, on\-policy distributional alignment continues to operate over the student’s generated trajectory\.

When no token is selected, the anchoring loss is zero\. In the implementation, the anchoring\-loss moving average is not updated for that sample, and the effective focal coefficient is set to zero\. The update then uses the normalized on\-policy alignment term alone\. This makes selection optional on a per\-sample basis and avoids forcing an arbitrary reference token into the target set\.

##### Customizable selection according to quality requirements\.

The current rules are intentionally simple, interpretable, and inexpensive\. Their role is to establish a first\-stage extraction mechanism for potentially important information, rather than to prescribe a universal definition of text quality\. Different users and applications may require different information to be preserved\. For scientific writing, terminology, measurements, and methodological conditions may be prioritized\. For news writing, organizations, locations, dates, and event participants may be emphasized\. For review generation, product attributes and experience\-related expressions may be more relevant\.

These requirements can be incorporated by editing the selection policy\. Possible extensions include domain\-specific terminology lists, explicit handling of units and numerical expressions, phrase\-level matching, and user\-specified inclusion or exclusion rules\. Such changes modify the mask construction while leaving the student prompt and on\-policy distillation mechanism unchanged\. They also allow lowercase technical expressions to be selected when capitalization is not a useful cue\.

##### Extension to small\-LLM\-assisted annotation\.

A further direction is to replace or augment the surface\-form rules with a small language model that identifies important reference spans under explicit quality requirements\. Letrur\_\{u\}encode the user’s desired quality criteria\. A generalized selector can be written as

𝒯sel=𝒮ϕ​\(cS,y,ru\),\\mathcal\{T\}\_\{\\mathrm\{sel\}\}=\\mathcal\{S\}\_\{\\phi\}\(c\_\{S\},y,r\_\{u\}\),\(13\)where𝒮ϕ\\mathcal\{S\}\_\{\\phi\}may be a rule system, a compact sequence\-labeling model, or a small LLM\.

For example, a small LLM could identify technical phrases, distinguish central entities from incidental mentions, and annotate expressions related to numerical accuracy or content completeness\. Returned spans could be validated against the reference and mapped to tokenizer positions before training\. A separate context\-support check could be retained or strengthened to prevent importance annotations from automatically admitting reference\-only information\. These annotations could be generated offline, so that no additional annotator would be required at inference time\.

This learned selector is a prospective extension, not a component of the current implementation\. The present rule\-based mechanism already supplies the necessary interface: important reference spans are converted into a token mask and consumed by the same anchoring objective\. Consequently, improved information identification, user\-editable quality criteria, and small\-model annotation can be explored within a common framework, providing a concrete foundation for further development of MIRROR\-F\.

### A\.5Effect of the Anchoring Coefficientα\\alphaon Professional Quality

We further study how the anchoring coefficientα\\alphaaffects the professional quality of generated outputs\. Here,α\\alphacontrols the relative contribution of the selective grounded anchoring loss in MIRROR\-F\. We use the same G\-Eval protocol described in Section, where the judge model scores each output along four dimensions, including coherence, consistency, fluency, and relevance\. The overall score is computed as the arithmetic mean of these four dimensions\.

Table[11](https://arxiv.org/html/2610.09795#A1.T11)reports representative results on the Scholarly Title task under four different values ofα\\alpha\. The results show a non\-monotonic relationship between the anchoring strength and professional quality\. The overall score is4\.7134\.713atα=0\.1\\alpha=0\.1and slightly decreases to4\.7054\.705atα=0\.3\\alpha=0\.3\. It then increases to the highest value of4\.7354\.735atα=0\.7\\alpha=0\.7, before decreasing substantially to4\.6504\.650atα=0\.9\\alpha=0\.9\. The consistency score follows a similar pattern: it reaches its maximum of4\.8804\.880atα=0\.7\\alpha=0\.7, compared with4\.8104\.810,4\.8304\.830, and4\.6904\.690atα=0\.1\\alpha=0\.1,0\.30\.3, and0\.90\.9, respectively\. These results indicate that a moderate anchoring coefficient provides the most favorable trade\-off for professional\-quality generation\.

Table 11:Representative G\-Eval results under different values ofα\\alphaon the Scholarly Title task\. Moderate anchoring achieves the highest overall professional\-quality score, whereas an excessively large anchoring coefficient leads to performance degradation\. Bold numbers denote the best result in each column\.α\\alphaCoherenceConsistencyFluencyRelevanceOverall0\.14\.4804\.8104\.9804\.5804\.7130\.34\.4404\.8305\.0004\.5504\.7050\.74\.5004\.8804\.9904\.5704\.7350\.94\.4404\.6905\.0004\.4704\.650This behavior suggests that the selective grounded anchoring loss is most effective when assigned a moderate weight\. Whenα\\alphais relatively small, the anchoring signal may be insufficient to consistently stabilize grounded, information\-bearing content\. Increasingα\\alphato a moderate level improves the balance between preserving relevant content and maintaining flexible generation, resulting in the highest coherence, consistency, and overall quality atα=0\.7\\alpha=0\.7\. However, whenα\\alphabecomes excessively large, the objective may over\-emphasize word\-level anchoring and constrain the model’s ability to organize the output naturally\. This can explain the noticeable decline in consistency, relevance, and overall professional quality atα=0\.9\\alpha=0\.9\.

Overall, the results demonstrate that the effect of selective grounded anchoring is not monotonic\. Moderate anchoring is more beneficial than either weak anchoring or overly strong anchoring, suggesting that the anchoring coefficient should be carefully controlled to avoid over\-constraining the generation process\.

For consistency and fairness, all reported evaluation results in the main experiments use a fixed anchoring coefficient ofα=0\.5\\alpha=0\.5\.

### A\.6More Information about LLM\-as\-Judge Methods

We employ two complementary LLM\-as\-judge protocols to evaluate personalized generation quality\. The first protocol is a reference\-free G\-Eval procedure based on the four SummEval dimensions, which measures the intrinsic quality of each generated response with respect to the task input\. The second protocol is a reference\-based, aspect\-level evaluation inspired by ExPerT, which measures the alignment between the generated response and the user\-specific reference in terms of content and style\. The two protocols are complementary: the former evaluates whether an answer is coherent, faithful, fluent, and relevant, whereas the latter evaluates whether the answer preserves the information and expressive characteristics conveyed by the reference\.

#### A\.6\.1Judge Model and General Evaluation Settings

All LLM\-as\-judge evaluations are conducted using the locally deployedQwen3\-30B\-A3Bmodel\. The judge is served with vLLM using tensor parallelism over eight GPUs\. To improve reproducibility, the decoding temperature is set to zero, and the judge is instructed not to activate its internal thinking mode\. The maximum number of newly generated tokens is set to 512\. The same judge model and decoding configuration are used for all compared methods\.

For each task, we identify the intersection of sample IDs available for all compared methods\. This procedure ensures that every method is evaluated on exactly the same questions\. If the number of common samples exceeds the predefined evaluation budget, a fixed random seed is used to select the evaluation subset\. Unless otherwise specified, 100 common samples are evaluated for each task\. No method\-specific sample filtering or method\-specific evaluation prompt is applied\.

#### A\.6\.2Reference\-Free G\-Eval with SummEval Dimensions

##### Evaluation objective\.

The reference\-free protocol evaluates the intrinsic quality of a candidate response without exposing the gold reference to the judge\. For each sample, the judge receives only the original task input and one candidate response generated by one method\. The judge does not compare two methods and does not use the gold answer when assigning scores\. This design avoids reducing content quality to lexical similarity with a single reference, which is particularly important for personalized generation because multiple responses may satisfy the same user preference\.

The evaluation dimensions are adopted from G\-Eval and SummEval:

- •Coherence: whether the response is logically organized and forms a unified answer rather than a collection of loosely related statements\.
- •Consistency: whether the response is factually supported by the source input and avoids hallucinated or unsupported claims\.
- •Fluency: whether the response is grammatical, readable, and appropriately written in terms of spelling, word choice, and punctuation\.
- •Relevance: whether the response preserves the important information from the source input while avoiding redundant or off\-topic content\.

Each dimension is scored from 1 to 5, where 1 denotes the lowest quality and 5 denotes the highest quality\. The judge is also asked to provide a short justification before returning the four scores in a fixed machine\-readable format\.

##### Task\-specific framing\.

The scoring dimensions are identical across tasks\. To account for differences in output form, the judge receives one neutral task\-form description:

- •For Abstract Generation, the candidate is described as a research\-paper abstract written for a given title\.
- •For News Headline Generation, the candidate is described as a short headline written for a given article\.
- •For Paper Topic Generation, the candidate is described as the title of an academic paper written for a given abstract\.
- •For other personalization tasks, the corresponding task form is specified without changing the scoring criteria\.

This task\-specific framing clarifies the expected genre while keeping the evaluation rubric shared by all methods\.

##### G\-Eval scoring prompt\.

The following box presents the prompt template used by the judge\. The task description, source input, and candidate output are inserted into the placeholders\{TASK\_FORM\},\{SOURCE\_INPUT\}, and\{CANDIDATE\}, respectively\.

Reference\-free G\-Eval promptSystem:You are a meticulous, impartial NLG evaluation judge\.User:You will be given one CANDIDATE text written for a task\. Your job is to rate the candidate on four metrics\. Read these instructions carefully and keep them open while reviewing\.\{TASK\_FORM\}Evaluation Criteria:\- Coherence \(1\-\-5\): the collective quality of the response\. It should be well\-structured and well\-organized, not just a heap of related sentences\.\- Consistency \(1\-\-5\): factual alignment between the response and the given source input\. A consistent response contains only statements that are entailed by or faithful to the source; penalize hallucinated or unsupported facts\.\- Fluency \(1\-\-5\): the quality of the response in terms of grammar, spelling, word choice, punctuation, and overall readability\.\- Relevance \(1\-\-5\): selection of the important, on\-topic content from the source\. The response should include salient and pertinent information and exclude redundant or off\-topic content\.Evaluation Steps:1\. Read the SOURCE INPUT carefully\.2\. Read the CANDIDATE and compare it against the source input\.3\. Assign an integer score from 1 \(worst\) to 5 \(best\) for each metric\.SOURCE INPUT:""" \{SOURCE\_INPUT\} """CANDIDATE:""" \{CANDIDATE\} """Do not use any external reference answer\. Give a one or two sentence justification, then end with exactly four lines:COHERENCE: <1\-\-5\>CONSISTENCY: <1\-\-5\>FLUENCY: <1\-\-5\>RELEVANCE: <1\-\-5\>

##### Per\-sample and per\-task computation\.

Letci,mc\_\{i,m\},si,ms\_\{i,m\},fi,mf\_\{i,m\}, andri,mr\_\{i,m\}denote the Coherence, Consistency, Fluency, and Relevance scores assigned to sampleiigenerated by methodmm\. The overall quality score for one sample is calculated as the arithmetic mean of the four dimensions:

qi,m=ci,m\+si,m\+fi,m\+ri,m4\.q\_\{i,m\}=\\frac\{c\_\{i,m\}\+s\_\{i,m\}\+f\_\{i,m\}\+r\_\{i,m\}\}\{4\}\.\(14\)
For a taskttcontainingNtN\_\{t\}successfully parsed samples, the dimension\-level score for methodmmis calculated as:

d¯m,t=1Nt​∑i=1Ntdi,m,d∈\{coh,cons,flu,rel\}\.\\overline\{d\}\_\{m,t\}=\\frac\{1\}\{N\_\{t\}\}\\sum\_\{i=1\}^\{N\_\{t\}\}d\_\{i,m\},\\qquad d\\in\\\{\\mathrm\{coh\},\\mathrm\{cons\},\\mathrm\{flu\},\\mathrm\{rel\}\\\}\.\(15\)
The task\-level G\-Eval score is computed as the mean of the sample\-level overall scores:

q¯m,t=1Nt​∑i=1Ntqi,m\.\\overline\{q\}\_\{m,t\}=\\frac\{1\}\{N\_\{t\}\}\\sum\_\{i=1\}^\{N\_\{t\}\}q\_\{i,m\}\.\(16\)
If multiple tasks are evaluated, the final model\-level score is obtained using an unweighted macro\-average over tasks:

q¯m=1\|𝒯m\|​∑t∈𝒯mq¯m,t,\\overline\{q\}\_\{m\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{m\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\_\{m\}\}\\overline\{q\}\_\{m,t\},\(17\)
where𝒯m\\mathcal\{T\}\_\{m\}denotes the set of tasks for which methodmmhas valid evaluation results\. Thus, each task contributes equally to the final score, regardless of the number of samples available for that task\. Scores are retained on the original 1–5 scale and are not rescaled to a 0–1 or 0–10 range\.

#### A\.6\.3Reference\-Based Aspect\-Level Evaluation

##### Evaluation objective\.

The reference\-based protocol evaluates whether a candidate preserves the information and expressive behavior represented by the gold reference\. Unlike the reference\-free G\-Eval protocol, this evaluation explicitly uses the reference answer\. However, it does not compare candidate and reference through token\-level overlap\. Instead, the judge first decomposes both texts into atomic content and style aspects and then evaluates their alignment\.

This aspect\-based design is suitable for personalized generation because the candidate may express the same preference in a different wording\. A response is therefore rewarded when it preserves the underlying content and style aspects, even when it does not reproduce the exact reference sequence\.

##### Aspect extraction\.

For each reference and candidate, the judge extracts two lists:

- •Content aspects: atomic factual or semantic points explicitly expressed by the text\.
- •Style aspects: writing characteristics such as tone, structure, formality, voice, and length preference\.

The extracted aspects are represented as strict JSON objects\. The aspect extraction stage is performed before the matching stage\. The reference aspects are extracted once for each\(task,sample ID\)\(\\text\{task\},\\text\{sample ID\}\)pair and are reused for all methods evaluated on that sample\.

Aspect extraction promptSystem:You extract atomic aspects from text as strict JSON\.User:Extract the aspects of the following TEXT\.Return STRICT JSON with exactly two keys:"content": list of short strings, each an atomic factual/content point the text actually makes;"style": list of short strings, each a writing\-style attribute \(tone, structure, formality, voice, length habit\)\.Return ONLY the JSON object, no prose\.TEXT:""" \{TEXT\} """

##### Aspect matching and scoring\.

After aspect extraction, the judge receives the content and style aspects of the reference and candidate\. It then estimates three quantities:

- •Content recall: the fraction of reference content aspects expressed in the candidate\.
- •Content precision: the fraction of candidate content aspects supported by the reference\.
- •Style alignment: the degree to which the candidate’s style matches the style of the reference\.

Content recall measures whether the candidate preserves the information conveyed by the reference\. Content precision penalizes unsupported or ungrounded content introduced by the candidate\. Style alignment measures whether the candidate maintains user\-specific expressive properties without requiring exact lexical reproduction\.

Aspect matching promptSystem:You are a meticulous aspect\-alignment judge for personalized text\.User:Compare a CANDIDATE against a REFERENCE using their pre\-extracted aspects\.REFERENCE content aspects: \{REFERENCE\_CONTENT\_ASPECTS\}REFERENCE style aspects: \{REFERENCE\_STYLE\_ASPECTS\}CANDIDATE content aspects: \{CANDIDATE\_CONTENT\_ASPECTS\}CANDIDATE style aspects: \{CANDIDATE\_STYLE\_ASPECTS\}Assess alignment:\- CONTENT\_RECALL: fraction 0\-\-1 of REFERENCE content aspects expressed in CANDIDATE;\- CONTENT\_PRECISION: fraction 0\-\-1 of CANDIDATE content aspects supported by REFERENCE;\- STYLE: integer 1\-\-10 for how well CANDIDATE style matches REFERENCE style\.End with exactly three lines:CONTENT\_RECALL: <0\-\-1\>CONTENT\_PRECISION: <0\-\-1\>STYLE: <1\-\-10\>

##### Reference\-based score computation\.

For each sample, content precision and content recall are combined using the harmonic mean:

F1,i,mcontent=2​Pi,m​Ri,mPi,m\+Ri,m,F^\{\\mathrm\{content\}\}\_\{1,i,m\}=\\frac\{2P\_\{i,m\}R\_\{i,m\}\}\{P\_\{i,m\}\+R\_\{i,m\}\},\(18\)
wherePi,mP\_\{i,m\}andRi,mR\_\{i,m\}denote content precision and content recall, respectively\. If both quantities are zero, the content F1 score is defined as zero\.

Because content F1 is defined on the range\[0,1\]\[0,1\]while style is scored on the range\[1,10\]\[1,10\], content F1 is first converted to a ten\-point scale\. The sample\-level overall score is then calculated as:

qi,maspect=10​F1,i,mcontent\+Si,m2,q^\{\\mathrm\{aspect\}\}\_\{i,m\}=\\frac\{10F^\{\\mathrm\{content\}\}\_\{1,i,m\}\+S\_\{i,m\}\}\{2\},\(19\)
whereSi,m∈\[1,10\]S\_\{i,m\}\\in\[1,10\]is the style score\. The resulting score jointly reflects content preservation and stylistic alignment\. Content recall, content precision, content F1, style, and the combined score are all averaged arithmetically over samples within each task:

z¯m,t=1Nt​∑i=1Ntzi,m,\\overline\{z\}\_\{m,t\}=\\frac\{1\}\{N\_\{t\}\}\\sum\_\{i=1\}^\{N\_\{t\}\}z\_\{i,m\},\(20\)
wherezzcan be any of the five aspect\-level quantities\. As in the G\-Eval protocol, the final score across multiple tasks is computed using an unweighted macro\-average over task\-level scores\.

#### A\.6\.4Relationship between the Two Evaluation Protocols

The two protocols measure different properties and are therefore reported separately\. The reference\-free G\-Eval score measures the quality of the generated text as a response to the task input:

Task Input⟶Candidate Quality\.\\text\{Task Input\}\\longrightarrow\\text\{Candidate Quality\}\.\(21\)
The reference\-based aspect\-level score measures how well the candidate preserves reference\-revealed content and style:

Reference⟶Content and Style Alignment\.\\text\{Reference\}\\longrightarrow\\text\{Content and Style Alignment\}\.\(22\)
Consequently, a high G\-Eval score does not require lexical similarity to the gold reference, and a high aspect\-level content score does not require the candidate to reproduce the reference verbatim\. This distinction is important for personalized generation, where several different outputs may be valid\. The combined evaluation framework therefore distinguishes between producing a high\-quality response and reproducing the exact wording of a single reference\.

In the main experiments, the reference\-free G\-Eval results are used to assess intrinsic content quality through Coherence, Consistency, Fluency, and Relevance\. The reference\-based aspect\-level results are used to complement the reference\-based automatic metrics, such as ROUGE\-1, METEOR, and BERTScore, by evaluating semantic content preservation and user\-specific style alignment at the aspect level\.

### A\.7On\-policy Rollout and Gradient Handling

The objective in Equation[4](https://arxiv.org/html/2610.09795#S3.E4)is implemented as a stopped\-gradient on\-policy distillation objective\. At each optimization step, the current student model first generates one response under the deployable contextcSc\_\{\\mathrm\{S\}\}\. The generated token sequence is then treated as a fixed discrete rollout when the distributional alignment loss is calculated\. Therefore, the implementation does not differentiate through the discrete sampling operation and does not use a score\-function estimator, REINFORCE term, importance\-weighted policy\-gradient estimator, or a separate lagged behavior policy\.

More precisely, the rollout is generated as

y^∼sg\[πθ\(⋅∣cS\)\],\\hat\{y\}\\sim\\operatorname\{sg\}\\\!\\left\[\\pi\_\{\\theta\}\(\\cdot\\mid c\_\{\\mathrm\{S\}\}\)\\right\],\(23\)wheresg⁡\[⋅\]\\operatorname\{sg\}\[\\cdot\]denotes stop\-gradient\. For each sampled prefixy^<t\\hat\{y\}\_\{<t\}, the student distribution is computed as

ptS=πθ\(⋅∣cS,y^<t\),p\_\{t\}^\{\\mathrm\{S\}\}=\\pi\_\{\\theta\}\\left\(\\cdot\\mid c\_\{\\mathrm\{S\}\},\\hat\{y\}\_\{<t\}\\right\),\(24\)while the reference\-revealed teacher distribution is detached:

ptT=sg\[πθ\(⋅∣cT,y^<t\)\]\.p\_\{t\}^\{\\mathrm\{T\}\}=\\operatorname\{sg\}\\\!\\left\[\\pi\_\{\\theta\}\\left\(\\cdot\\mid c\_\{\\mathrm\{T\}\},\\hat\{y\}\_\{<t\}\\right\)\\right\]\.\(25\)Consequently, the gradient is propagated throughptSp\_\{t\}^\{\\mathrm\{S\}\}only\. The sampled tokens, the rollout probabilities, andptTp\_\{t\}^\{\\mathrm\{T\}\}do not receive gradients\. The student model is updated after the loss computation, and a new rollout is generated from the updated student at the next optimization step\.

The rollout uses stochastic sampling with temperature1\.01\.0andtop​\-​p=1\.0\\mathrm\{top\}\\text\{\-\}p=1\.0\. The maximum generation length is determined on a per\-example basis from the reference length and the task\-specific cap\. Generation terminates when either the EOS token is produced or the maximum length is reached\. If EOS is produced before the length limit, the EOS token is retained in the aligned completion and tokens after EOS are discarded\. If the length limit is reached without EOS, the capped completion is used and the event is recorded for diagnostic purposes\. Samples with fewer than two completion tokens are skipped because they do not provide a meaningful sequence\-level alignment signal\.

The implementation therefore optimizes a surrogate objective rather than the full derivative of an expectation whose sampling distribution depends onθ\\theta\. In particular, the gradient does not contain the additional score\-function term

𝔼y^∼πθ​\[ℒ⁡\(y^,θ\)​∇θ​log⁡πθ​\(y^∣cS\)\]\.\\mathbb\{E\}\_\{\\hat\{y\}\\sim\\pi\_\{\\theta\}\}\\left\[\\mathcal\{L\}\(\\hat\{y\},\\theta\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(\\hat\{y\}\\mid c\_\{\\mathrm\{S\}\}\)\\right\]\.\(26\)This choice avoids the high variance associated with policy\-gradient estimation and directly trains the student to match the teacher distribution on states that the deployable student actually visits\. The term “on\-policy” therefore refers to the state distribution used to construct the training prefixes, whereas the distributional alignment itself is optimized by ordinary backpropagation on the detached sampled trajectories\.

### A\.8Other Prompts

This section describes the prompts used for model inference, user\-history conditioning, and MIRROR training\. The prompts used for LLM\-as\-judge evaluation are described separately in Appendix[A\.6](https://arxiv.org/html/2610.09795#A1.SS6)\. All prompts are converted into the backbone model’s native chat format using the tokenizer’s chat template\. Unless otherwise stated, the system message is shared across training and evaluation\.

#### A\.8\.1Common Generation Prompt

For all generation\-based evaluations, the model receives a system instruction followed by a user message\. The system instruction constrains the model to output only the requested text and prevents additional explanations or formatting from being included in the prediction\.

Common generation promptSystem:You are a helpful assistant\. Write only the requested text itself, with no preamble, no explanation, and no surrounding quotation marks\.User:\{USER\_CONDITIONING\}\{TASK\_QUESTION\}

Here,\{TASK\_QUESTION\}denotes the original task instruction and input, while\{USER\_CONDITIONING\}is either empty, a retrieved user profile, or the full user history, depending on the evaluation protocol\. The model is always evaluated using the same system instruction, regardless of whether it is a base model, an SFT\-based model, or a MIRROR\-based model\.

For example, when no user history is supplied, the user message consists only of the task question:

Non\-personalized generation promptSystem:You are a helpful assistant\. Write only the requested text itself, with no preamble, no explanation, and no surrounding quotation marks\.User:\{TASK\_QUESTION\}

The tokenizer\-specific chat template is applied after the system and user messages have been constructed\. For Qwen3 models, the evaluation configuration disables thinking mode by default\. Thus, the model directly generates the requested answer without producing an additional<think\>segment\.

#### A\.8\.2Task Question Templates

The original task question is preserved during evaluation\. We use the following task\-specific forms\.

##### Abstract Generation\.

For Abstract Generation, the model is asked to generate an abstract conditioned on a paper title and a set of key points:

Abstract Generation question templateGenerate an abstract for the title ‘‘\{TITLE\}’’ using the following items:1\. \{KEY\_POINT\_1\}2\. \{KEY\_POINT\_2\}…\{K\}\. \{KEY\_POINT\_K\}

The output is expected to be a self\-contained research\-paper abstract\. The model is not explicitly asked to copy the reference abstract\. Instead, the key points and the user history provide the information and stylistic context needed to generate a new abstract\.

##### News Headline Generation\.

For News Headline Generation, the model receives a news article and is asked to generate a headline:

News Headline Generation question templateGenerate a headline for the following article:\{ARTICLE\_TEXT\}

The generated answer is expected to be a short headline rather than a paragraph\. The task therefore evaluates whether the model can identify the article\-specific topic while preserving the user’s preferred headline format, vocabulary, and editorial style\.

##### Paper Topic Generation\.

For Paper Topic Generation, the model receives an academic abstract and is asked to produce a paper title:

Paper Topic Generation question templateGenerate a title for the following abstract of a paper:\{PAPER\_ABSTRACT\}

The output is expected to be a concise academic title\. The model should preserve the central technical concepts of the abstract while matching the user’s preferred terminology and title structure\.

##### Other LongLaMP tasks\.

For product review generation, the question specifies the product and the requested review context\. For topic writing, the question provides the title or prompt for a Reddit\-style post body\. The same common generation prompt is used, while only the task question changes\.

#### A\.8\.3User\-History Conditioning

Personalization is introduced by appending user\-specific historical examples before the current task question\. The historical examples are formatted as demonstrations that associate an input with the corresponding user\-written output\. The formatting depends on the task because different tasks contain different input–output structures\.

##### Full\-history conditioning\.

In the full\-history setting, historical examples are retained in their original temporal order\. When the complete history exceeds the input budget, the most recent entries are preferentially retained\. The current question is always preserved in full whenever possible\. The resulting user message is:

Full\-history conditioning template\{HISTORY\_EXAMPLE\_1\}, and\{HISTORY\_EXAMPLE\_2\}, and…, and\{HISTORY\_EXAMPLE\_M\}\. Following the given patterns, \{TASK\_QUESTION\}

The phrase “Following the given patterns” indicates that the preceding examples describe the user’s writing behavior\. It does not instruct the model to copy any particular historical answer verbatim\.

##### Task\-specific history formatting\.

For Abstract Generation and Paper Topic Generation, each historical item is rendered as:

Academic history example‘‘\{TITLE\}’’ is a title for the abstract ‘‘\{ABSTRACT\}’’

For News Headline Generation, each historical item is rendered as:

News history example‘‘\{HEADLINE\}’’ is a headline for the article ‘‘\{ARTICLE\_TEXT\}’’

For product review generation, a historical item contains the review summary, rating, product description, and user\-written review:

Product\-review history examplethe review with summary ‘‘\{SUMMARY\}’’ and rating \{RATING\} for the product ‘‘\{DESCRIPTION\}’’ is ‘‘\{REVIEW\}’’

For Amazon review tasks, the historical item is rendered as:

Amazon\-review history examplethe review with rating \{RATING\} and title ‘‘\{REVIEW\_TITLE\}’’ for the item ‘‘\{ITEM\_TITLE\}’’ \(\{ITEM\_DESCRIPTION\}\) is ‘‘\{REVIEW\_TEXT\}’’

For topic writing, historical examples are rendered as:

Topic\-writing history examplethe reddit post with summary ‘‘\{SUMMARY\}’’ is ‘‘\{POST\_BODY\}’’

These task\-specific templates preserve the semantic relation between a historical input and the corresponding user\-written output\. Consequently, the model can infer both the user’s preferred content selection and the user’s preferred expression pattern\.

#### A\.8\.4BM25 Retrieval Prompt

For retrieval\-based baselines, the complete user history is not appended to every query\. Instead, BM25 is applied to the user profile, and the two most relevant historical examples are selected\. The retrieval corpus is constructed from both the input and output fields of each historical item\. For example, the corpus for Abstract Generation contains the paper title and abstract, while the corpus for News Headline Generation contains the headline and article text\.

The selected examples are rendered using the task\-specific history templates above and concatenated with the current task question:

BM25 retrieval\-conditioned prompt\{RETRIEVED\_EXAMPLE\_1\}, and \{RETRIEVED\_EXAMPLE\_2\}\.Following the given patterns, \{TASK\_QUESTION\}

The retrieval operation is performed independently for each test question\. It affects only the prompt provided to the model and does not modify the model parameters\. The gold answer is never used to construct the retrieval prompt\.

#### A\.8\.5MIRROR Student Prompt

The MIRROR student is trained to operate under the same information available during deployment\. Therefore, the student receives the user history and the current task question, but not the gold reference answer\. The student prompt is identical to the full\-history inference prompt:

MIRROR student promptSystem:You are a helpful assistant\. Write only the requested text itself, with no preamble, no explanation, and no surrounding quotation marks\.User:\{FULL\_USER\_HISTORY\}Following the given patterns, \{TASK\_QUESTION\}

During training, the current student model samples an on\-policy response from this prompt\. Lety^\\hat\{y\}denote the sampled response andy^<t\\hat\{y\}\_\{<t\}denote its prefix at steptt\. The student distribution is evaluated as

pt=πθ\(⋅∣cS,y^<t\),p\_\{t\}=\\pi\_\{\\theta\}\\left\(\\cdot\\mid c\_\{S\},\\hat\{y\}\_\{<t\}\\right\),\(27\)
wherecSc\_\{S\}contains only the information available to the student at deployment time\.

#### A\.8\.6MIRROR Teacher Prompt

The MIRROR teacher receives the same user history and task question as the student, together with the reference answer\. The reference is treated as privileged hindsight information used only during training\. It is not provided to the deployed model\.

The teacher prompt is constructed by placing the reference answer after the task question and adding an instruction that asks the teacher to solve the task independently after understanding the reference:

MIRROR teacher promptSystem:You are a helpful assistant\. Write only the requested text itself, with no preamble, no explanation, and no surrounding quotation marks\.User:\{FULL\_USER\_HISTORY\}\{TASK\_QUESTION\}Here is a reference solution:\{GOLD\_REFERENCE\}After understanding the reference solution, please try to solve this problem using your own approach below:

The teacher is evaluated on the student’s sampled prefix rather than on the reference prefix\. Specifically, the teacher distribution is computed as

qt=π¯θ\(⋅∣cT,y^<t\),q\_\{t\}=\\overline\{\\pi\}\_\{\\theta\}\\left\(\\cdot\\mid c\_\{T\},\\hat\{y\}\_\{<t\}\\right\),\(28\)
wherecTc\_\{T\}is the reference\-revealed context andπ¯θ\\overline\{\\pi\}\_\{\\theta\}denotes a detached copy of the current model\. The reference answer therefore changes the teacher’s distribution, but it is not used as a token\-level target sequence\.

The student and teacher prompts are intentionally different in information content:

cS=\(user history,task question\),c\_\{S\}=\(\\text\{user history\},\\text\{task question\}\),\(29\)
cT=\(user history,task question,reference answer,reference hint\)\.c\_\{T\}=\(\\text\{user history\},\\text\{task question\},\\text\{reference answer\},\\text\{reference hint\}\)\.\(30\)
The teacher and student are nevertheless evaluated on the same on\-policy prefixy^<t\\hat\{y\}\_\{<t\}\. This construction transfers the behavior induced by the reference into states that the student actually visits during inference\.

#### A\.8\.7MIRROR\-F Teacher Prompt and Focal Anchoring

MIRROR\-F uses the same student and teacher prompts as MIRROR\. The difference lies in the training objective rather than in the basic prompt structure\. In addition to aligning the student and teacher distributions over the student’s on\-policy trajectory, MIRROR\-F applies a selective anchoring loss to informative reference tokens\.

The selected token set contains content\-bearing and context\-grounded tokens\. In particular, the selection procedure favors terminology, named entities, quantities, and other tokens that carry task\-specific information, while excluding purely functional or stylistic tokens\. The selected word\-level positions are expanded to their complete subword spans\.

The focal anchoring prompt therefore does not expose any additional information to the deployed model\. It only determines which reference tokens receive the auxiliary training signal:

MIRROR\-F selective anchoring templateReference solution:\{GOLD\_REFERENCE\}Selected informative and context\-grounded positions:\{SELECTED\_REFERENCE\_TOKENS\}

The selective anchoring term is applied only during training\. At inference time, MIRROR\-F receives exactly the same deployable prompt as MIRROR:

Inference inputMIRROR−F=Inference inputMIRROR\.\\text\{Inference input\}\_\{\\mathrm\{MIRROR\-F\}\}=\\text\{Inference input\}\_\{\\mathrm\{MIRROR\}\}\.\(31\)
This separation is essential\. The gold reference and the selected reference tokens are privileged training information and are never supplied to the deployed model\.

#### A\.8\.8User Profile Summary Prompt

For the summary\-based ablation, a compact user profile is generated from the user’s historical texts\. The profile summary is generated without access to the current gold answer\. It is therefore intended to compress writing preferences rather than reveal the target output\.

The summary prompt is:

User profile summary promptSystem:You are a helpful assistant\. Write only the requested text itself, with no preamble, no explanation, and no surrounding quotation marks\.User:Here is the writing history of a single user \(their past examples\):\{USER\_HISTORY\}Based only on the writing history above, write a complete user profile report describing the user’s preferences for content selection, organization, tone, terminology, sentence structure, and level of detail for \{TASK\_DESCRIPTION\}\. Be specific and evidence\-based\. Do not quote whole sentences from the history and do not mention that you were given a history\.

The generated summary is inserted only into the privileged teacher context in the summary\-based variant\. The student still receives the original user history and task question\. The summary is truncated to a fixed token budget before being inserted into the teacher prompt\.

### A\.9Thinking\-Mode Ablation

We further examine whether enabling the model’s internal thinking mode substantially changes personalized generation performance\. Table[12](https://arxiv.org/html/2610.09795#A1.T12)reports the results of the same Qwen3 backbones under thinking\-on and thinking\-off configurations\. The reported values are macro\-averaged across the three LongLaMP tasks, while the task\-level results are provided to show the variation across generation settings\.

Table 12:Thinking\-mode ablation on LongLaMP\. Values are macro\-averaged across Abstract Generation, Product Review, and Topic Writing\.Δ\\Deltadenotes the change from thinking off to thinking on; positive values indicate improvement\.Qwen3\-1\.7BQwen3\-4BModeROUGE\-1METEORBERTScoreROUGE\-1METEORBERTScoreThinking off0\.26610\.14370\.83780\.33340\.19850\.8467Thinking on0\.29080\.16160\.83510\.32200\.18470\.8340Δ\\Delta\+0\.0247\+0\.0179−0\.0027\-0\.0027−0\.0114\-0\.0114−0\.0138\-0\.0138−0\.0127\-0\.0127The results show that enabling thinking does not induce a consistent performance trend in personalized generation\. For Qwen3\-1\.7B, thinking mode improves the macro\-averaged ROUGE\-1 and METEOR scores, while producing a slightly lower BERTScore\. For Qwen3\-4B, thinking mode decreases all three macro\-averaged metrics, although it improves ROUGE\-1 and METEOR on the Topic Writing task\. These results indicate that the effect of thinking is task\- and scale\-dependent rather than uniformly beneficial or harmful\. Overall, the main performance trend of personalized generation remains largely preserved across the two thinking configurations\.

This observation suggests that personalized generation and long\-horizon reasoning should be regarded as related but distinct capability dimensions\. Thinking mode is designed to allocate additional computation to intermediate reasoning, whereas personalization requires the model to infer and apply user\-specific preferences in content selection, organization, terminology, and expression\. Improvements in complex reasoning ability alone therefore do not necessarily translate into improvements in personalized generation\. Likewise, the effectiveness of RL\- or OPD\-based training for long\-horizon problem solving should not be assumed to imply a corresponding gain in personalization\. The two capabilities may rely on different forms of internal representation and different training signals\.

Accordingly, these results motivate the study of LLM personalization as an independent research problem rather than treating it as an automatic consequence of general reasoning improvement\. Methods that explicitly internalize user preferences, such as MIRROR and MIRROR\-F, address this separate dimension by introducing preference\-revealed supervision and personalization\-specific objectives\. Further investigation is needed to understand how reasoning ability and personalization ability interact and whether they can be improved jointly without assuming that progress in one dimension automatically transfers to the other\.