TIAO: Token Importance-Aware Policy Optimization for Text Summarization

# TIAO: Token Importance-Aware Policy Optimization for Text Summarization
Source: [https://arxiv.org/html/2609.16748](https://arxiv.org/html/2609.16748)
Qixiu Li ​ Chenlong Bao ​ Xiang Zhu ​ Xiaoyong Li ​ Ruixin Cao ​ Shukai Chen ​ Zhenxiong Zhou††thanks:$†$ Equal Contribution\. \*Corresponding Author\.

###### Abstract

Text summarization requires models to condense content while preserving key qualities such as consistency and coherence\. Large language models \(LLMs\) have shown strong performance on this task and can be further improved through reinforcement learning \(RL\)\. However, most existing methods apply reward signals directly to undifferentiated token sequences, overlooking the varying importance of individual tokens to word and sentence level quality in summarization\. In this paper, we propose Token Importance\-Aware Policy Optimization \(TIAO\), a novel reinforcement learning strategy that explicitly leverages token\-importance awareness\. Specifically, TIAO identifies core tokens based on token dependency and reweights a trajectory’s advantage according to its overall dependencies\. Experiments on the real world dataset show that our TIAO achieves highly competitive results, and that a 7B foundation model enhanced by TIAO performs comparably to GPT\-4 and GPT\-5\-nano\. Code is available at[https://github\.com/TechCloud\-x/TIAO](https://github.com/TechCloud-x/TIAO)\.

###### Index Terms:

Reinforcement Learning, Text Summarization

††address:1National University of Defense Technology## 1Introduction

Text summarization is a selective information compression problem: a system must shorten a source document while preserving salient entities, numbers, events, and relations, and while maintaining coherence, relevance, fluency, and factual consistency\. Pre\-trained sequence\-to\-sequence models and recent large language models \(LLMs\) have substantially improved abstractive summarization quality\[[3](https://arxiv.org/html/2609.16748#bib.bib21),[7](https://arxiv.org/html/2609.16748#bib.bib22),[22](https://arxiv.org/html/2609.16748#bib.bib10)\]\. Yet factuality studies show that fluent summaries may still hallucinate or distort source\-supported content\[[18](https://arxiv.org/html/2609.16748#bib.bib23),[11](https://arxiv.org/html/2609.16748#bib.bib11),[13](https://arxiv.org/html/2609.16748#bib.bib12)\]; once key evidence is dropped during compression, fluent surface realization cannot recover it\. This has motivated multi\-dimensional evaluators for consistency, coherence, relevance, and fluency\[[23](https://arxiv.org/html/2609.16748#bib.bib13),[5](https://arxiv.org/html/2609.16748#bib.bib14),[9](https://arxiv.org/html/2609.16748#bib.bib15)\]\. Reinforcement learning \(RL\) offers a direct way to optimize such non\-differentiable quality signals, from human\-feedback summarization\[[16](https://arxiv.org/html/2609.16748#bib.bib16)\]to recent multi\-objective reward balancing policy HVO\[[15](https://arxiv.org/html/2609.16748#bib.bib4)\]\. These methods mainly improve what reward should define a better summary\.

However, summarization policy optimization also faces a credit\-resolution mismatch\. Autoregressive generation is a token\-level sequence decision process, where the policy selects each token conditioned on the source and the generated prefix\. In contrast, summary quality is usually evaluated only after the whole sequence is completed\. In a GRPO\-style objective\[[14](https://arxiv.org/html/2609.16748#bib.bib3)\], the same trajectory advantage is commonly multiplied by token\-wise policy ratios across positions\. This distinguishes better and worse summary trajectories, but cannot identify which positions actually depend on the source document or determine factual coverage\. The resulting reward broadcast can dilute useful gradients over many low\-information tokens\.

Recent fine\-grained RL methods partially address this limitation\. Process reward models and step\-wise verifiers provide denser feedback for reasoning traces\[[8](https://arxiv.org/html/2609.16748#bib.bib17),[17](https://arxiv.org/html/2609.16748#bib.bib18)\], while reward redistribution decomposes holistic feedback into token\-level rewards\[[6](https://arxiv.org/html/2609.16748#bib.bib19)\]\. Entropy\-based analysis further shows that a small fraction of uncertain tokens can dominate RL gains in reasoning models\[[19](https://arxiv.org/html/2609.16748#bib.bib20)\]\. Optimization\-side advances such as DAPO\[[20](https://arxiv.org/html/2609.16748#bib.bib2)\]and SAPO\[[4](https://arxiv.org/html/2609.16748#bib.bib7)\]improve large\-scale LLM RL through token\-level design, dynamic sampling, and token\-adaptive update control\. Nevertheless, these approaches often rely on auxiliary reward models, process annotations, or output\-side proxy signals, and are largely developed for mathematical or general reasoning\. For summarization, the central question is different: whether an output token is genuinely supported by source evidence\.

![Refer to caption](https://arxiv.org/html/2609.16748v1/dependency_grouped_figure.png)Figure 1:TIAO token dependency analysis under source masking\. \(a\) compares full\-source and masked\-source token probabilities and highlights source\-dependent update positions\. \(b\) summarizes trajectory\-level dependency patterns across token categories\.This paper targets this missing link\. Under multi\-dimensional summarization rewards, existing algorithms still lack source\-dependency\-aware credit assignment that can align sequence\-level quality feedback with token\-level evidence usage\. Such a mechanism should distinguish source\-grounded trajectories from language\-prior shortcuts and, within the same trajectory, assign stronger learning pressure to tokens whose probabilities are sensitive to the source document\. Importantly \(Fig\.[1](https://arxiv.org/html/2609.16748#S1.F1)\), this should be achieved without training additional token critic\. Better rewards specify what to learn; our goal is to decide where and how strongly the policy should learn\.

To this end, we proposeTokenImportance\-Aware PolicyOptimization \(TIAO\), a source\-sensitive policy optimization method for text summarization\. TIAO forms a dual\-scale optimization loop: source perturbation estimates token dependency; token dependencies are aggregated into trajectory\-level source dependency; trajectory advantages are scaled by this dependency; and token updates are focused on source\-sensitive positions\. Our contributions are threefold\. ❶ We define a source\-dependency measure for summary output tokens based on probability changes under counterfactual source masking\. ❷ We introduce a trajectory\-token credit assignment strategy that jointly reweights trajectory advantages and concentrates token\-level policy updates\. ❸ We validate TIAO on CNN/DailyMail\[[12](https://arxiv.org/html/2609.16748#bib.bib5)\], demonstrating improvements in summary quality, training stability\.

![Refer to caption](https://arxiv.org/html/2609.16748v1/fig-main.png)Figure 2:The logic implementation of our proposed TIAO\.
## 2Methodology

### 2\.1Overview

We propose Token Importance\-Aware Policy Optimization \(TIAO\), a source\-sensitive reinforcement learning framework for abstractive summarization\. As shown in Fig\.[2](https://arxiv.org/html/2609.16748#S1.F2), TIAO keeps the standard group\-relative rollout and multi\-dimensional summarization rewards, but changes how the resulting learning signal is assigned\. For each sampled summary, we perturb the source document and re\-score the same output tokens under the original and perturbed sources\. The induced probability shift estimates how strongly each output token depends on source evidence\. TIAO then uses this signal at two levels: it reshapes the trajectory advantage according to the summary’s overall source dependency, and it filters token\-level policy gradients toward the most source\-sensitive output positions\. Thus, the reward still defines what summary is better, while TIAO determines where the policy should learn from it\.

### 2\.2Problem Formulation

Given a documentxxand a reference summaryy⋆y^\{\\star\}, a policyπθ\\pi\_\{\\theta\}generates an abstractive summaryy=\(y1,…,yT\)y=\(y\_\{1\},\\ldots,y\_\{T\}\)autoregressively:

yt∼πθ\(⋅∣x,y<t\),t=1,…,T\.y\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,y\_\{<t\}\),\\quad t=1,\\ldots,T\.\(1\)Text summarization policy optimization is therefore a sequence decision problem with token\-level actions and sequence\-level feedback\. Since generated summaries have variable lengths, we define a valid\-token maskci,t∈\{0,1\}c\_\{i,t\}\\in\\\{0,1\\\}for theii\-th sampled trajectory, where tokens after the first end\-of\-sequence marker are excluded andTi=∑tci,tT\_\{i\}=\\sum\_\{t\}c\_\{i,t\}\. Let𝐫⁡\(x,y\)∈ℝD\\mathbf\{r\}\(x,y\)\\in\\mathbb\{R\}^\{D\}denote a multi\-dimensional evaluator, where the dimensions correspond to coherence, consistency, fluency, and relevance in our implementation\. A scalar trajectory reward is obtained through an external aggregation functionΦ\\Phi, calculated as:

R⁡\(x,y\)=Φ⁡\(𝐫⁡\(x,y\)\)\.R\(x,y\)=\\Phi\(\\mathbf\{r\}\(x,y\)\)\.\(2\)TIAO is independent of the specific evaluator and aggregation rule\. Its role is to convert the resulting trajectory\-level signal into source\-aware trajectory and token credits without an additional token critic\.

### 2\.3Multi\-Dimensional Group\-Relative Policy Optimization

For each documentxx, the old policyπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}samples a group ofGGsummaries\{yi\}i=1G\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}\. Each summary first receives a reward vector𝐫i=\[ri,1,…,ri,D\]\\mathbf\{r\}\_\{i\}=\[r\_\{i,1\},\\ldots,r\_\{i,D\}\]\. To make different dimensions comparable before scalarization, we write a group\-wise standardized score as:

r^i,d=ri,d−μdσd\+ϵd,μd=1G​∑j=1Grj,d\.\\hat\{r\}\_\{i,d\}=\\frac\{r\_\{i,d\}\-\\mu\_\{d\}\}\{\\sigma\_\{d\}\+\\epsilon\_\{d\}\},\\quad\\mu\_\{d\}=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}r\_\{j,d\}\.\(3\)The scalar reward is then obtained asRi=Φ⁡\(𝐫^i,𝝀\)R\_\{i\}=\\Phi\(\\hat\{\\mathbf\{r\}\}\_\{i\};\\bm\{\\lambda\}\), where𝝀∈ΔD−1\\bm\{\\lambda\}\\in\\Delta^\{D\-1\}denotes non\-negative dimension preferences andΦ\\Phimay represent a linear or non\-linear aggregation\. TIAO treats this reward construction as an external evaluator and modifies only the credit assigned to generated tokens\. Following the group\-relative policy optimization paradigm, we normalize scalar rewards within the group, which can be mathematically formulated as:

Ai=Ri−μRσR\+ϵ,μR=1G​∑j=1GRj,A\_\{i\}=\\frac\{R\_\{i\}\-\\mu\_\{R\}\}\{\\sigma\_\{R\}\+\\epsilon\},\\quad\\mu\_\{R\}=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}R\_\{j\},\(4\)whereσR\\sigma\_\{R\}is the within\-group reward standard deviation andϵ\\epsilonis a small constant\. The token\-wise policy ratio is calculated as:

wi,t​\(θ\)=πθ​\(yi,t∣x,yi,<t\)πθold​\(yi,t∣x,yi,<t\)\.w\_\{i,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\.\(5\)The corresponding unclipped token contribution can be written asgi,t=ci,t​wi,t​\(θ\)​Aig\_\{i,t\}=c\_\{i,t\}w\_\{i,t\}\(\\theta\)A\_\{i\}, which shows the credit\-resolution issue explicitly: all valid positions inherit the same trajectory\-level scalar\. In conventional GRPO\-style training, this is efficient, but it treats factual content tokens, discourse markers, and low\-information function words as equally responsible for the sequence\-level reward\.

### 2\.4Source\-Dependency Estimation for Output Tokens

TIAO estimates token importance by measuring whether the probability of an already generated token changes when source evidence is partially removed\. Let the source bex=\(x1,…,xN\)x=\(x\_\{1\},\\ldots,x\_\{N\}\)\. For each sampled trajectory, we draw an independent source mask as:

bi,n∼Bernoulli⁡\(1−ρ\),x~i,n=\{xn,bi,n=1,\[𝙼𝙰𝚂𝙺\],bi,n=0,b\_\{i,n\}\\sim\\operatorname\{Bernoulli\}\(1\-\\rho\),\\quad\\tilde\{x\}\_\{i,n\}=\\begin\{cases\}x\_\{n\},&b\_\{i,n\}=1,\\\\ \\mathtt\{\[MASK\]\},&b\_\{i,n\}=0,\\end\{cases\}\(6\)whereρ∈\(0,1\)\\rho\\in\(0,1\)controls the perturbation strength\. This defines the counterfactual sourcex~i=ℳρ​\(x,𝐛i\)\\tilde\{x\}\_\{i\}=\\mathcal\{M\}\_\{\\rho\}\(x;\\mathbf\{b\}\_\{i\}\)while keeping the generated prefixyi,<ty\_\{i,<t\}unchanged\. We then compute teacher\-forced log probabilities under the rollout policy\. The process is as follow:

ℓi,tF=log⁡πθold​\(yi,t∣x,yi,<t\),ℓi,tM=log⁡πθold​\(yi,t∣x~i,yi,<t\)\.\\ell^\{F\}\_\{i,t\}=\\log\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\),\\quad\\ell^\{M\}\_\{i,t\}=\\log\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{i,t\}\\mid\\tilde\{x\}\_\{i\},y\_\{i,<t\}\)\.\(7\)We define the token source dependency as a sampled\-token low\-variance estimate of the conditional divergence between the full\-source and masked\-source predictions, which can be expressed as:

di,t=clip⁡\(ℓi,tM−ℓi,tF,−τ,τ\),Ii,t=ci,t​\[exp⁡\(di,t\)−di,t−1\]\.d\_\{i,t\}=\\operatorname\{clip\}\(\\ell^\{M\}\_\{i,t\}\-\\ell^\{F\}\_\{i,t\},\-\\tau,\\tau\),\\quad I\_\{i,t\}=c\_\{i,t\}\\left\[\\exp\(d\_\{i,t\}\)\-d\_\{i,t\}\-1\\right\]\.\(8\)A largerIi,tI\_\{i,t\}indicates thatyi,ty\_\{i,t\}is more sensitive to the source document\. The clipping thresholdτ\\tauonly stabilizes extreme log\-probability shifts and does not introduce any trainable critic\. This formulation focuses on source support rather than output\-side uncertainty alone: a token is important only when removing source evidence changes the model’s confidence in producing it\.

### 2\.5Trajectory–Token Credit Assignment

Based on the dependency scores, TIAO reshapes the learning signal at both macro and micro levels\.

Macro\-level: trajectory advantage shaping\.For each generated summary, we aggregate token dependencies into a trajectory dependency score\. This operation can be mathematically formulated as:

Si=∑tci,t​Ii,t∑tci,t\+δT,S\_\{i\}=\\frac\{\\sum\_\{t\}c\_\{i,t\}I\_\{i,t\}\}\{\\sum\_\{t\}c\_\{i,t\}\+\\delta\_\{T\}\},\(9\)whereδT\\delta\_\{T\}prevents division by zero for degenerate completions\. We then compute a mean\-preserving positive scale over the rollout batchℬ\\mathcal\{B\}, this operation can be formulated as:

αi=SiS¯\+δ,S¯=1\|ℬ\|​∑j∈ℬSj,\\alpha\_\{i\}=\\frac\{S\_\{i\}\}\{\\bar\{S\}\+\\delta\},\\quad\\bar\{S\}=\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{j\\in\\mathcal\{B\}\}S\_\{j\},\(10\)whereδ\\deltaprevents numerical instability\. The shaped advantage is:

A~i=αi​Ai\.\\tilde\{A\}\_\{i\}=\\alpha\_\{i\}A\_\{i\}\.\(11\)This scaling amplifies updates for high\-reward summaries that are strongly grounded in the source, while also applying stronger corrective pressure to low\-reward summaries whose errors occur in source\-dependent regions\.

Micro\-level: token update focusing\.Within each trajectory, we select the top\-κ\\kappaproportion of valid output tokens according toIi,tI\_\{i,t\}, whereκ∈\(0,1\)\\kappa\\in\(0,1\)\. Letrank↓⁡\(Ii,t\)\\operatorname\{rank\}\_\{\\downarrow\}\(I\_\{i,t\}\)be the descending rank of tokenttamong valid tokens in the same trajectory, with invalid positions assigned infinite rank\. The selected index set and binary token gate are computed mathematically as:

𝒦i=\{t∣ci,t=1,rank↓\(Ii,t\)≤⌈κTi⌉\},mi,t=ci,t𝕀\(t∈𝒦i\)\.\\mathcal\{K\}\_\{i\}=\\\{t\\mid c\_\{i,t\}=1,\\operatorname\{rank\}\_\{\\downarrow\}\(I\_\{i,t\}\)\\leq\\lceil\\kappa T\_\{i\}\\rceil\\\},\\quad m\_\{i,t\}=c\_\{i,t\}\\mathbb\{I\}\(t\\in\\mathcal\{K\}\_\{i\}\)\.\(12\)The gate is detached from policy optimization and serves only as a credit\-assignment mask\. By concentrating the policy\-gradient term on source\-sensitive tokens, TIAO reduces the diffusion of sequence\-level rewards over generic or weakly grounded positions\.

### 2\.6Training Objective

Integrating trajectory advantage shaping and token update focusing gives the final TIAO objective, which can be formulated as:

ℒTIAO​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{TIAO\}\}\(\\theta\)=𝔼x∼𝒟,𝐲∼πθoldG\(⋅∣x\)\[1G∑i=1G1Ti∑t\\displaystyle=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},\\,\\mathbf\{y\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}^\{G\}\(\\cdot\\mid x\)\}\\Bigg\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{T\_\{i\}\}\\sum\_\{t\}\(13\)mi,tmin\(wi,t\(θ\)A~i,clip\(wi,t\(θ\),1−ε,1\+ε\)A~i\)\]\.\\displaystyle m\_\{i,t\}\\min\\big\(w\_\{i,t\}\(\\theta\)\\tilde\{A\}\_\{i\},\\operatorname\{clip\}\(w\_\{i,t\}\(\\theta\),1\-\\varepsilon,1\+\\varepsilon\)\\tilde\{A\}\_\{i\}\\big\)\\Bigg\]\.Equivalently, TIAO optimizes an active surrogate as follow:

ψi,t​\(θ\)=mi,t​min⁡\(wi,t​\(θ\)​A~i,clip⁡\(wi,t​\(θ\),1−ε,1\+ε\)​A~i\),\\psi\_\{i,t\}\(\\theta\)=m\_\{i,t\}\\min\\left\(w\_\{i,t\}\(\\theta\)\\tilde\{A\}\_\{i\},\\operatorname\{clip\}\(w\_\{i,t\}\(\\theta\),1\-\\varepsilon,1\+\\varepsilon\)\\tilde\{A\}\_\{i\}\\right\),\(14\)whose support is the source\-sensitive set𝒜i=\{t∣mi,t=1\}\\mathcal\{A\}\_\{i\}=\\\{t\\mid m\_\{i,t\}=1\\\}\. Thus, the gradient estimator excludes inactive positions before summation while preserving the trajectory\-level reward ordering\. The expectation emphasizes that dependency estimation, reward normalization, and token gating are recomputed for every rollout batch\. The denominator remains the full completion lengthTiT\_\{i\}, matching the original GRPO normalization; unselected tokens therefore reduce the effective policy\-gradient mass instead of changing the reduction rule\. Consequently, TIAO changes the credit resolution of policy optimization: sequence\-level summarization rewards are preserved, but their learning pressure is routed toward source\-dependent trajectories and tokens\.

Table 1:The results of multi\-dimensional evaluation on the CNN/DailyMail dataset\. Top three results for each metric are highlighted asbest,second, andthird, respectively\.DatasetModelMethodCoherence↑\\uparrowConsistency↑\\uparrowFluency↑\\uparrowRelevance↑\\uparrowOverall↑\\uparrowSTD↓\\downarrowCNN/DailyMail\[[12](https://arxiv.org/html/2609.16748#bib.bib5)\]PEGASUSSFT0\.9360\.9390\.8150\.6840\.8430\.121Scaling ModelsQwen2\.5 1\.5BZero\-shot0\.8710\.8190\.9360\.8610\.8720\.048Qwen2\.5 7BZero\-shot0\.8900\.8200\.9320\.8740\.8790\.046Qwen2\.5 14BZero\-shot0\.9310\.8260\.8590\.9070\.8810\.047Qwen2\.5 32BZero\-shot0\.9180\.8430\.9330\.8930\.8970\.040Proprietary ModelsGPT\-4Zero\-shot0\.9670\.8400\.9450\.9340\.9210\.056GPT\-5\-nanoZero\-shot0\.8350\.7350\.8130\.7940\.7940\.084Reinforcement Learning MethodsQwen2\.5 7BGRPO\[[14](https://arxiv.org/html/2609.16748#bib.bib3)\]0\.9080\.9030\.9220\.9540\.9220\.023Qwen2\.5 7BHVO\[[15](https://arxiv.org/html/2609.16748#bib.bib4)\]0\.9610\.9260\.9510\.9340\.9430\.016Qwen2\.5 7BDAPO\[[20](https://arxiv.org/html/2609.16748#bib.bib2)\]0\.9370\.8640\.9060\.9290\.9090\.036Qwen2\.5 7BSAPO\[[4](https://arxiv.org/html/2609.16748#bib.bib7)\]0\.9310\.8690\.9140\.9270\.9100\.037Our Method and Ablation StudiesQwen2\.5 7BTIAO \(Ours\)\-mask 20%0\.9430\.9430\.9800\.9690\.9590\.059Qwen2\.5 7BTIAO \(Ours\)\-mask 80%0\.9420\.9400\.9790\.9680\.9570\.063Qwen2\.5 7BTIAO \(Ours\)0\.9610\.9460\.9670\.9660\.9600\.020

## 3Experiments

### 3\.1Experimental Settings

Datasets\. Following\[[15](https://arxiv.org/html/2609.16748#bib.bib4)\], we evaluate TIAO on the CNN/DailyMail abstractive summarization benchmark\*\*\*[https://huggingface\.co/google/pegasus\-cnn\_dailymail](https://huggingface.co/google/pegasus-cnn_dailymail)\[[12](https://arxiv.org/html/2609.16748#bib.bib5)\]\. CNN/DailyMail contains online news articles paired with ordered multi\-sentence highlights, and is widely used to test whether a system can compress long news documents while preserving salient events and entities\.

Baselines and Evaluation Metrics\. We compare TIAO with three groups of representative baselines\. The first group contains PEGASUS\[[21](https://arxiv.org/html/2609.16748#bib.bib9)\], a supervised abstractive summarization model\. Following HVO\[[15](https://arxiv.org/html/2609.16748#bib.bib4)\], we report its SFT result and the zero\-shot result of GPT\-4\[[1](https://arxiv.org/html/2609.16748#bib.bib6)\]\. The second group includes zero\-shot open\-source LLMs at different scales, including Qwen2\.5 1\.5B, 7B, 14B, and 32B\[[2](https://arxiv.org/html/2609.16748#bib.bib1)\], which allows us to separate the effect of model scale from that of policy optimization\. We further evaluate GPT\-5\-nano through the Poe API†††[https://poe\.com/api](https://poe.com/api)\. The third group includes RL\-based methods, including GRPO\[[14](https://arxiv.org/html/2609.16748#bib.bib3)\], HVO\[[15](https://arxiv.org/html/2609.16748#bib.bib4)\], DAPO\[[20](https://arxiv.org/html/2609.16748#bib.bib2)\], and SAPO\[[4](https://arxiv.org/html/2609.16748#bib.bib7)\]\. For evaluation, we use UniEval\[[23](https://arxiv.org/html/2609.16748#bib.bib13)\]\. We report four standard summarization dimensions: coherence, consistency, fluency, and relevance\. The overall score is the arithmetic mean of the four dimensions\. We also report STD, the standard deviation over the four dimension scores, to measure whether a method improves summary quality in a balanced manner\. Higher values indicate better performance for all UniEval scores, while lower values are better for STD\.

Implementation Details\. All RL methods use Qwen2\.5\-7B\-Instruct as the initial policy for a fair comparison\. We use the same prompt template for training and inference: “Summarize the Text without any Explanation\.” The maximum prompt length and completion length are set to 2048 and 512 tokens, respectively\. We use UniEval\-sum to compute the four reward dimensions, and normalize rewards within each generation group\. The group size is set to 8\. We train for 4 epochs with AdamW\[[10](https://arxiv.org/html/2609.16748#bib.bib8)\], using a learning rate of5×10−75\\times 10^\{\-7\}, betas of\(0\.9,0\.999\)\(0\.9,0\.999\), weight decay of0\.10\.1, a cosine learning\-rate schedule, a warmup ratio of0\.10\.1, and a maximum gradient norm of0\.40\.4\. The rollout temperature is set to 1\.0\. For TIAO, we randomly mask 50% of source tokens and compare the token probabilities of the same generated summary under the original and masked sources to estimate source\-dependent token importance\. The token\-importance signal is used only for credit assignment\. We reweight each trajectory advantage by its mean token importance normalized by the rollout mean, and apply policy\-gradient updates only to the top 40% most important valid output tokens in each trajectory\. At inference time, all trained and zero\-shot open\-source models use greedy decoding with a maximum of 512 new tokens\. Experiments are conducted on 32×\\times4 NVIDIA A100 GPUs with 40GB memory\.

### 3\.2Performance Comparison

Table[1](https://arxiv.org/html/2609.16748#S2.T1)reports the main evaluation results\.1\)Overall, TIAO achieves the best overall score of 0\.960 and ranks first on consistency, fluency, and relevance, reaching 0\.946, 0\.967, and 0\.966, respectively\. It also obtains the second\-best coherence score of 0\.961 and the second\-lowest STD of 0\.020\. Compared with the Qwen2\.5 7B zero\-shot, TIAO improves the overall score by 0\.081 and consistency by 0\.126, showing that the gain is not merely inherited from the foundation model but comes from policy optimization\. Compared with the larger Qwen2\.5 32B zero\-shot model, the 7B policy optimized by TIAO still improves the overall score by 0\.063, indicating that source\-aware RL can be more effective than simply increasing model scale under the same evaluation setting\.2\)Scaling open\-source LLMs improves summarization quality only moderately\. The overall score increases from 0\.872 for Qwen2\.5 1\.5B to 0\.897 for Qwen2\.5 32B, but the improvements are uneven across dimensions\. In contrast, TIAO substantially improves all dimensions over the Qwen2\.5 7B zero\-shot model\. Against proprietary models, GPT\-4 obtains the highest coherence score of 0\.967, but TIAO surpasses it on consistency, fluency, relevance, and overall score by 0\.106, 0\.022, 0\.032, and 0\.039, respectively\. These results suggest that a 7B model can reach highly competitive summarization quality when the optimization signal is assigned to source\-dependent tokens\.3\)Among RL\-based baselines, TIAO also shows clear advantages\. It improves the overall score over GRPO, HVO, DAPO, and SAPO by 0\.038, 0\.017, 0\.051, and 0\.050\. Compared with HVO, the strongest previous RL baseline in overall performance, TIAO matches its coherence score and further improves consistency, fluency, and relevance by 0\.020, 0\.016, and 0\.032\. Although HVO obtains a slightly lower STD, TIAO maintains a similarly balanced profile while achieving much higher absolute quality\. This supports our central hypothesis: under sequence\-level summarization rewards, explicitly resolving trajectory\-token credit through source dependency leads to more reliable improvements than broadcasting the same advantage to all generated tokens\.

### 3\.3Ablation Study

Table[1](https://arxiv.org/html/2609.16748#S2.T1)also evaluates the masking ratio used in source\-dependency estimation\. Both 20% and 80% masking remain strong, obtaining overall scores of 0\.959 and 0\.957, respectively, which confirms that the token\-importance mechanism is robust to perturbation strength\. However, the default 50% setting achieves the best overall score \(0\.960\), the highest consistency \(0\.946\), and a much lower STD\. This indicates that moderate perturbation provides more balanced source\-dependency signals than too weak or too aggressive masking\.

### 3\.4Case Study

Figure 3:Case study on effective gradient dynamics and overall reward\.Fig\.[3](https://arxiv.org/html/2609.16748#S3.F3)illustrates why TIAO behaves differently from vanilla GRPO\. In GRPO, the group\-relative advantage is broadcast to all valid tokens in a completion\. TIAO first estimates how much each sampled token depends on the source document by comparing token probabilities under the original and masked sources, and then uses this dependency signal to reshape trajectory advantages and select the most source\-sensitive output tokens for policy\-gradient updates\. GRPO improves the overall reward rapidly in the early stage\. As training proceeds, TIAO shows a stronger pre\-clip gradient signal and reaches a higher overall reward\. This late\-stage separation suggests that concentrating optimization on source\-dependent tokens helps the policy keep improving evidence\-related decisions instead of repeatedly reinforcing low\-information positions\.

## 4Conclusion

In this paper, we address the credit\-resolution mismatch in RL\-based text summarization, where sequence\-level rewards are broadcast to undifferentiated tokens\. We propose TIAO, a token importance\-aware policy optimization method that estimates source\-dependent token importance through source masking, reweights trajectory advantages, and focuses updates on important output tokens\. Experiments on CNN/DailyMail show that the 7B policy achieves competitive quality against strong open\-source and proprietary baselines\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774,pp\. 1–100\.Cited by:[§3\.1](https://arxiv.org/html/2609.16748#S3.SS1.p2.1)\.
- \[2\]\(2025\)Qwen2\.5\-vl technical report\.arXiv preprint arXiv:2502\.13923,pp\. 1–23\.Cited by:[§3\.1](https://arxiv.org/html/2609.16748#S3.SS1.p2.1)\.
- \[3\]Y\. Fan, Y\. Wan, X\. Zhong, H\. Cheng, N\. Ding, and B\. Zhou\(2026\)Eva\-score: evaluating abstractive long\-form summarization on informativeness through extraction and validation\.InIEEE International Conference on Acoustics, Speech and Signal Processing,pp\. 16782–16786\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p1.1)\.
- \[4\]C\. Gao, C\. Zheng, X\. Chen, K\. Dang, S\. Liu, B\. Yu, A\. Yang, S\. Bai, J\. Zhou, and J\. Lin\(2025\)Soft adaptive policy optimization\.arXiv preprint arXiv:2511\.20347,pp\. 1–9\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p3.1),[Table 1](https://arxiv.org/html/2609.16748#S2.T1.8.1.15.2),[§3\.1](https://arxiv.org/html/2609.16748#S3.SS1.p2.1)\.
- \[5\]P\. Laban, T\. Schnabel, P\. N\. Bennett, and M\. A\. Hearst\(2022\)SummaC: re\-visiting nli\-based models for inconsistency detection in summarization\.Transactions of the Association for Computational Linguistics10,pp\. 163–177\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p1.1)\.
- \[6\]J\. Li, L\. Li, T\. Chang, K\. Kuang, L\. Chen, J\. Zhou, and C\. Yang\(2025\)RED: unleashing token\-level rewards from holistic feedback via reward redistribution\.InProceedings of the Conference on Empirical Methods in Natural Language Processing,pp\. 4993–5022\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p3.1)\.
- \[7\]W\. Liang, Y\. Zhang, Z\. Wu, H\. Lepp, W\. Ji, X\. Zhao, H\. Cao, S\. Liu, S\. He, Z\. Huang,et al\.\(2025\)Quantifying large language model usage in scientific papers\.Nature Human Behaviour9\(12\),pp\. 2599–2609\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p1.1)\.
- \[8\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 39578–39601\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p3.1)\.
- \[9\]Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu\(2023\)G\-eval: nlg evaluation using gpt\-4 with better human alignment\.InProceedings of the conference on empirical methods in natural language processing,pp\. 2511–2522\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p1.1)\.
- \[10\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,pp\. 4061–4078\.Cited by:[§3\.1](https://arxiv.org/html/2609.16748#S3.SS1.p3.1)\.
- \[11\]J\. Maynez, S\. Narayan, B\. Bohnet, and R\. McDonald\(2020\)On faithfulness and factuality in abstractive summarization\.InProceedings of the annual meeting of the association for computational linguistics,pp\. 1906–1919\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p1.1)\.
- \[12\]R\. Nallapati, B\. Zhou, C\. Dos Santos, Ç\. Gulçehre, and B\. Xiang\(2016\)Abstractive text summarization using sequence\-to\-sequence rnns and beyond\.InProceedings of the SIGNLL conference on computational natural language learning,pp\. 280–290\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p5.1),[Table 1](https://arxiv.org/html/2609.16748#S2.T1.8.1.2.1.1.1.1.1),[§3\.1](https://arxiv.org/html/2609.16748#S3.SS1.p1.1)\.
- \[13\]A\. Pagnoni, V\. Balachandran, and Y\. Tsvetkov\(2021\)Understanding factuality in abstractive summarization with frank: a benchmark for factuality metrics\.InProceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 4812–4829\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p1.1)\.
- \[14\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300,pp\. 1–30\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p2.1),[Table 1](https://arxiv.org/html/2609.16748#S2.T1.8.1.12.2),[§3\.1](https://arxiv.org/html/2609.16748#S3.SS1.p2.1)\.
- \[15\]J\. Song, Y\. Liu, D\. Li, Y\. Sun, S\. Fu, S\. Chen, and Y\. Cao\(2026\)Balancing rewards in text summarization: multi\-objective reinforcement learning via hypervolume optimization\.InIEEE International Conference on Acoustics, Speech and Signal Processing,pp\. 16777–16781\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p1.1),[Table 1](https://arxiv.org/html/2609.16748#S2.T1.8.1.13.2),[§3\.1](https://arxiv.org/html/2609.16748#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2609.16748#S3.SS1.p2.1)\.
- \[16\]N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. Christiano\(2020\)Learning to summarize with human feedback\.Advances in neural information processing systems33,pp\. 3008–3021\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p1.1)\.
- \[17\]P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui\(2024\)Math\-shepherd: verify and reinforce llms step\-by\-step without human annotations\.InProceedings of the Annual Meeting of the Association for Computational Linguistics,pp\. 9426–9439\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p3.1)\.
- \[18\]R\. Wang, Z\. Li, S\. Sun, G\. Sun, H\. Huang, J\. Yu, Y\. Zhou, and P\. Torr\(2026\)RAPTM: retrieval\-augmented prompting for short\-text topic modeling\.InIEEE International Conference on Acoustics, Speech and Signal Processing,pp\. 17497–17501\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p1.1)\.
- \[19\]S\. Wang, L\. Yu, C\. Gao, C\. Zheng, S\. Liu, R\. Lu, K\. Dang, X\. Chen, J\. Yang, Z\. Zhang,et al\.\(2026\)Beyond the 80/20 rule: high\-entropy minority tokens drive effective reinforcement learning for llm reasoning\.Advances in Neural Information Processing Systems38,pp\. 115452–115486\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p3.1)\.
- \[20\]Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.\(2026\)Dapo: an open\-source llm reinforcement learning system at scale\.Advances in Neural Information Processing Systems38,pp\. 113222–113244\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p3.1),[Table 1](https://arxiv.org/html/2609.16748#S2.T1.8.1.14.2),[§3\.1](https://arxiv.org/html/2609.16748#S3.SS1.p2.1)\.
- \[21\]J\. Zhang, Y\. Zhao, M\. Saleh, and P\. Liu\(2020\)Pegasus: pre\-training with extracted gap\-sentences for abstractive summarization\.InInternational conference on machine learning,pp\. 11328–11339\.Cited by:[§3\.1](https://arxiv.org/html/2609.16748#S3.SS1.p2.1)\.
- \[22\]T\. Zhang, F\. Ladhak, E\. Durmus, P\. Liang, K\. McKeown, and T\. B\. Hashimoto\(2024\)Benchmarking large language models for news summarization\.Transactions of the Association for Computational Linguistics12,pp\. 39–57\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p1.1)\.
- \[23\]M\. Zhong, Y\. Liu, D\. Yin, Y\. Mao, Y\. Jiao, P\. Liu, C\. Zhu, H\. Ji, and J\. Han\(2022\)Towards a unified multi\-dimensional evaluator for text generation\.InProceedings of the Conference on Empirical Methods in Natural Language Processing,pp\. 2023–2038\.Cited by:[§1](https://arxiv.org/html/2609.16748#S1.p1.1),[§3\.1](https://arxiv.org/html/2609.16748#S3.SS1.p2.1)\.