Multiscale Policy Optimization for Long-Horizon LLM Agents
Summary
本文提出 Multiscale Policy Optimization（MPO），通过在轮次尺度上对齐信用分配与策略更新（结合多尺度优势估计 MAE 和轮次级重要性采样）来改进长时程 LLM 智能体的训练，在搜索、工具使用和自动化任务上对 4B 和 30B-A3B 模型均取得一致增益。
View Cached Full Text
Cached at: 10/28/26, 01:26 PM
Paper page - Multiscale Policy Optimization for Long-Horizon LLM Agents
Source: https://huggingface.co/papers/2610.35013 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Multiscale Policy Optimization improves long-horizon agent training by aligning credit assignment and policy updates at the turn scale.
Long-horizon LLM agentsinteract with their environments through complete assistant turns while generating each action token by token. We observe thatcritic valuesvary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens. We proposeMultiscale Policy Optimization(MPO), combiningMultiscale Advantage Estimation(MAE) andturn-level importance samplingto align credit assignment and policy updates at the turn scale. Extensive experiments across search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent gains over strong RL baselines.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2610.35013 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2610.35013 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2610.35013 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents
Introduces Belief Entropy and Metacognitive Memory Policy Optimization (MMPO) to improve memory quality in long-horizon LLM agents, outperforming existing methods and maintaining performance over long contexts.
TEMPO: Temporal Enforcement via Mode-Separated Policy Optimization for Trustworthy LLM Backtesting
Proposes TEMPO, a policy optimization method that trains LLMs to reason exclusively from pre-cutoff information by using a two-mode reward and GRPO-based training, reducing knowledge leakage by 2–13% while improving task performance by 6–13%.
Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target
提出了Cross-Behavior Policy Optimization (CBPO)，通过共享的固定参考回归目标，将离线示范数据与在线/离线策略rollout统一到单一损失函数中进行LLM推理后训练，在数学推理基准上显著优于SFT→DAPO两阶段管线。
Competence-Aware Orchestration for LLM-Based Multi-Agent Systems
该论文提出了一种两阶段的 competence-aware 框架，用于 LLM 多智能体系统的自动编排，根据模型在当前查询上的能力动态调整角色激活与通信图，在六个基准上平均达到 93.81%，同时在高能力骨干上降低 36.26% 的 token 成本。
PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
PlanPO is a reinforcement learning method that introduces coarse-to-fine advantage signals for multi-turn agentic LLMs, improving performance over GRPO by 27.2% on benchmarks like ALFWorld, WebShop, and SciWorld.