Player-1: Learning to Play from Synthetic Experience

Hugging Face Daily Papers Papers
reinforcement-learning game-agents synthetic-experience computer-use post-training ppo

Summary

Tencent ARC 的论文提出 Player-1 一套面向游戏控制的后训练框架，通过扩展动作空间、在 6,044 条游戏轨迹上进行 SFT 初始化，并利用 16 个可程序化验证的合成环境进行 PPO 多环境强化学习，显著超越监督基线与现有最强 baseline。

Computer-use agents have demonstrated strong capabilities in interpreting screens and executing keyboard and mouse actions. However, games, with their visually rich worlds and diverse mechanics, remain challenging and offer a demanding next step toward general digital control. Current game-agent learning pipelines often rely on large-scale demonstrations with limited coverage of dynamic game states, while scalable environments for interactive training remain difficult to obtain. We introduce Player-1, a post-training framework for learning game control beyond demonstrations through synthetic experience. We extend the computer-use action space with relative pointer motion, sustained inputs, and composite actions. For supervised initialization, we fine-tune computer-use models on 6,044 curated gameplay trajectories that pair recorded actions with task instructions and visual reasoning. We construct Player-1-Gym, sixteen synthetic 2D and 3D environments designed around control demands shared across games. Programmatically verifiable rewards and randomized observations and control dynamics support multi-environment reinforcement learning with PPO. The resulting Player-1-9B and Player-1-35B-A3B improve MineDojo-300 success over their SFT policies by 27.8 and 17.7 percentage points, respectively. Player-1-9B reaches 38.33% success compared with 12.63% for the strongest evaluated baseline, with further gains over SFT on ViZDoom-10 and GameWorld. Further analyses show that broader synthetic environment coverage improves transfer, while the use of composite actions that combine multiple keyboard and mouse inputs increases during RL. We demonstrate that learning from synthetic experience substantially improves game control beyond supervised initialization, with gains that generalize beyond the RL training suite.
Original Article
View Cached Full Text

Cached at: 10/15/26, 07:04 AM

Paper page - Player-1: Learning to Play from Synthetic Experience

Source: https://huggingface.co/papers/2610.20584 Published on Oct 14

#1 Paper of the day Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

Computer-useagentshavedemonstratedstrongcapabilitiesininterpretingscreensandexecutingkeyboardandmouseactions.However,games,withtheirvisuallyrichworldsanddiversemechanics,remainchallengingandofferademandingnextsteptowardgeneraldigitalcontrol.Currentgame-agentlearningpipelinesoftenrelyonlarge-scaledemonstrationswithlimitedcoverageofdynamicgamestates,whilescalableenvironmentsforinteractivetrainingremaindifficulttoobtain.WeintroducePlayer-1,apost-trainingframeworkforlearninggamecontrolbeyonddemonstrationsthroughsyntheticexperience.Weextendthecomputer-useactionspacewithrelativepointermotion,sustainedinputs,andcompositeactions.Forsupervisedinitialization,wefine-tunecomputer-usemodelson6,044curatedgameplaytrajectoriesthatpairrecordedactionswithtaskinstructionsandvisualreasoning.WeconstructPlayer-1-Gym,sixteensynthetic2Dand3Denvironmentsdesignedaroundcontroldemandssharedacrossgames.Programmaticallyverifiablerewardsandrandomizedobservationsandcontroldynamicssupportmulti-environmentreinforcementlearningwithPPO.TheresultingPlayer-1-9BandPlayer-1-35B-A3BimproveMineDojo-300successovertheirSFTpoliciesby27.8and17.7percentagepoints,respectively.Player-1-9Breaches38.33%successcomparedwith12.63%forthestrongestevaluatedbaseline,withfurthergainsoverSFTonViZDoom-10andGameWorld.Furtheranalysesshowthatbroadersyntheticenvironmentcoverageimprovestransfer,whiletheuseofcompositeactionsthatcombinemultiplekeyboardandmouseinputsincreasesduringRL.Wedemonstratethatlearningfromsyntheticexperiencesubstantiallyimprovesgamecontrolbeyondsupervisedinitialization,withgainsthatgeneralizebeyondtheRLtrainingsuite.

View arXiv pageView PDFProject pageGitHub2Add to collection

Get this paper in your agent:

hf papers read 2610\.20584

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.20584 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.20584 in a dataset README.md to link it from this page.

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models

Hugging Face Daily Papers

ACCRUE proposes a continual off-policy reinforcement learning method for robot foundation models. By freezing the foundation model and only updating lightweight actor–critic networks while inheriting experience across tasks, robots can learn new skills within 4–12 minutes of real-world interaction while retaining 98.3% of success rates on previously acquired tasks.

SPADE: Self-Play in Adaptive Synthetic Executable Environments

Hugging Face Daily Papers

SPADE introduces a self-play reinforcement learning framework for language models that generates adaptive executable training environments to enhance reasoning and tool-use capabilities, demonstrating significant performance gains across multiple benchmarks.

@0xLogicrw: Former OpenAI post-training core member Jiayi Weng proposed a new reinforcement learning paradigm called "Heuristic Learning" in his personal capacity and open-sourced all experimental code. He used Codex (GPT-5.4) to repeatedly play the Atari game Breakout, but GPT-5.4 was never retrained...

X AI KOLs Timeline

Former OpenAI researcher Jiayi Weng proposed a new paradigm called "Heuristic Learning", which uses large language models to generate and iteratively modify Python code to solve reinforcement learning tasks. Knowledge is stored in interpretable code rather than neural network parameters, effectively avoiding catastrophic forgetting. It has achieved excellent results on Atari and MuJoCo benchmarks and the code has been open-sourced.

PEARL: Process-Experience Aware Reinforcement Learning for Multi-Turn Agents

arXiv cs.AI

PEARL提出了一种面向多轮智能体的强化学习方法，通过分支轨迹的首次分歧点构建局部决策特征，并利用成功/失败记忆聚类识别重复模式，从而在不引入额外critic的情况下改进信用分配。在AppWorld和ALFWorld基准上，PEARL在4B和8B规模模型上均超越了ARPO等基线，ALFWorld 4B达到97.9%成功率。