Tech report now live https://alphaxiv.org/abs/2609.reinforcing-agents-collective-skills…

Reinforcing Agents with Collective Skills

Source: https://www.alphaxiv.org/abs/2609.reinforcing-agents-collective-skills Imagine you are trying to teach an AI agent to do the kinds of jobs people actually delegate: writing a literature review, auditing code ownership, generating a marketing brief. To improve through reinforcement learning—where the model tries tasks, receives a score, and adjusts—you need two things: thousands of self-contained practice environments the agent can run in isolation, and a way to grade whether the agent did the job right. Real workloads are often private. Synthesized tasks drawn from software repositories skew heavily toward coding. Exercises generated from hand-built topic lists may cover more ground but can miss how practitioners actually work or evaluate quality. The authors of this paper ask whether publicly posted “Agent Skills”—practitioner-written folders containing instructions, scripts, references, and notes on what typically goes wrong—can be compiled into reliable training environments that keep both the job and its standards. After 300 rounds of reinforcement learning on tasks derived from about 3,400 such skills, a 27-billion-parameter model improved by 4.7 points on a held-out public terminal benchmark, closing part of the gap between local and larger hosted systems.

After 300 updates of outcome-only reinforcement learning on Skill-derived environments, the trained model (pale green, 54.1) improved over its base (purple, 49.4) on this held-out public terminal benchmark. A dashed line shows that a gap to larger hosted frontier models remains.Scaling reinforcement learning for agents requires a sufficient supply of training tasks and environments. Existing approaches mine tasks from software repositories or prompt a model to invent them from a hand-designed topic list. Both strategies produce large corpora, buttheir coverage depends on source selection, and many terminal datasets concentrate on technical workflows such as system administration, file manipulation, and coding.

What these approaches also miss is thehow. A test that checks whether a delivered file is correct cannot distinguish a careful process from one that deleted evidence of failure and reported success.Outcome tests indicate only whether a task was completed, not whether it was done through an optimal process valued by human society, and RL on outcome-only rewards has limited means to favor a careful literature review over a fabricated one.

An Agent Skill is a folder rooted in aSKILL\.mdfile with structured frontmatter—name, description, trigger conditions—followed by free-form instructions, and optionally scripts, references, and assets. Anthropic introduced the formatto package recurring agent workflowsthe way an onboarding guide packages institutional know-how.Since the format was introduced in late 2025, practitioners have published thousands of Skills covering software, data, research, design, business operations, and more. Each one is a small record of a job someone cared enough about to document: what tools it needs, what typically goes wrong, and what a good result looks like. TheSkill2Env GitHub repositoryprovides the full pipeline, SkillHub, and a quick-start guide.

To keep the source pool usable, the authors filtered crawled Skills: a Skill is dropped if it requires an external account, login, or live service; physical hardware; a persona-only prompt with no executable workflow; or harmful activity. Near-duplicates—common because popular Skills are forked and lightly edited—are removed by comparing text similarity across theSKILL\.mdfiles. The result is SkillHub, a snapshot of about 3,400 Skills.

The Skill2Env pipeline turns public Agent Skills into verified RL environments in six stages. Planning decomposes each Skill into distinct workflows and identifies real-world assets; diversification samples six axes including archetype, persona, and complexity; creation produces a containerized world, deterministic tests, and a Skill-derived behavioral rubric; verification gates out tasks where the reference solution fails or a no-operation agent passes.The Skill2Env pipeline turns one Skill into several independent practice tasks through four stages: plan, diversify, create, and verify.

A planning agent—aCodex CLIinstance running inside a Docker container with network access—reads the complete Skill bundle. It extracts distinct, verifiable workflows the Skill teaches and ranks them by three criteria: the outcome should be checkable from the container’s final state, the task should be well-specified, and the difficulty should come from the work itself rather than from ambiguity. A literature-review Skill might yield a screening-and-synthesis workflow and a separate citation-reconciliation workflow.

For each workflow the planner also proposes grounded external assets—real repositories at pinned commits, versioned documents, official API specifications—and a set of task axes: archetype (for example, audit report or bug fix), verifier type (for example, evidence traceability or functional correctness), complexity level, persona, instruction tone, and requester expertise. These axes are conditioned on the workflow so that a research workflow gets paired with evidence-traceability verification rather than an arbitrary random draw.

A fresh creator agent then builds the task in a fixed order: it constructs the initial world insideenvironment/, writes the instruction, implements the tests, writes the rubric, and only then writes the reference solution.Tests and rubric are frozen before the solution exists, so the solution must satisfy the grading contract rather than the other way round; a failing reference solution is treated as a solution bug, not a reason to weaken the verifier. Live-service dependencies are replaced by local stand-ins—a stub server with deterministic responses, record-and-replay fixtures, or a seeded local database—so the environment never depends on the network when the agent is solving it.

The rubric is not a generic checklist. It carries bullet points for Must-do, Must-avoid, and Best-practice derived directly from the source Skill’s own methodology. For a security-ownership audit Skill, the rubric inherits the Skill’s default exclusions—merge commits, bots, bulk release snapshots—as explicit Must-avoid clauses.The appendix traces how the source Skill’s ownership-risk workflows became the generated task’s world, instruction, and audit-report rubric.

Before a task is accepted, the host runs two checks in fresh containers:the reference solution must earn the full reward on every metric, and a no-operation agent must earn zero on every metric; candidates that fail either trial are rejected. Base images are pinned to a content digest so the world stays reproducible.

The resulting corpus contains 7,971 tasks spanning thirteen domains. A retrieval probe confirms that the pipeline preserved source-Skill content:the true source Skill is the top-1 match for 73.2% of tasks when the task instruction and rubric are used as a query, and within the top 10 for 94.6%, compared with a 0.03% chance baseline.

The RL experiment uses Qwen3.8-27B as the base model. A 2,400-task subset of Skill2Env serves as the training distribution, each task graded by itstest\.shscript, which writes up to six named metrics in ([0,1])—each covering a distinct facet of the workflow—and averages them into one programmatic score per episode. The training algorithm is a variant of DPPO: it normalizes rewards within groups of eight rollouts, then masks individual tokens whose update would stray too far from the distribution recorded at sampling time, without requiring a separate reference model. Groups where all rollouts score the same produce no gradient, which implicitly filters tasks that are trivially solved or currently unsolvable by the on-policy model.

After 300 updates of outcome-only RL, pass@1 on Terminal-Bench 2.1—the fraction of tasks where every test passes on a single attempt, measured across five independent trials—rose from 49.4% to 54.1%. On the hand-verified S2EBench (79 tasks, held out from the same SkillHub source), the all-tests-pass rate rose from 33.4% to 37.7%, and the mean partial-credit score rose from 56.6 to 75.1.The gain transfers to Terminal-Bench 2.1, which shares no task above the stated text-similarity threshold with the training corpus, and the authors interpret the transfer as improved planning, tool use, and task completion rather than memorization of a task family.

That interpretation is reasonable but not uniquely established by these results. The training subset covers thirteen domains while Terminal-Bench skews toward technical workflows; whether the breadth, the Skill provenance, the verifier quality, or simply the volume of RL training drives the transfer is not separated here. S2EBench is private, contains 79 tasks, and is drawn from the same SkillHub distribution as training even though no individual task overlaps.

The rubric-calibrated run uses the same RL algorithm but adds a judge score to each episode. The judge—a separate large language model—receives the task instruction, a fixed meta-rubric penalizing aimless loops and reward hacking, the per-task rubric, and the agent’s recorded messages. It returns an integer from −5 to +5. That score is scaled and added to the programmatic reward with weight 0.2, so the judge can shift the total reward by at most ±0.2. A score of −5, reserved for test tampering or fabricated evidence, can pull a partially passing episode into negative territory.

Total training reward over 300 steps. The outcome-only run (blue) generally sits above the rubric-calibrated run (green). Because the two lines optimize different objectives, the gap reflects a different balance of signals rather than a clean performance comparison.The held-out benchmarks show the rubric run trailing: 50.1 versus 54.1 on Terminal-Bench 2.1, 34.7 versus 37.7 on S2EBench pass, and 63.6 versus 75.1 on S2EBench mean score. During training,the rubric run’s programmatic reward stayed between 0.5 and 0.6 for most of training, below the outcome-only run, while the judge term remained flat from the first update to the last. The policy appears to have traded some test passes for behavior the rubric rewards, while the judge signal provided no upward gradient to climb.

The authors name two likely causes without claiming to have verified them: the single fixed weight λ=0.2 and additive form were not tuned, and the methodology Skills prescribe may not maximize benchmark pass rates. The judge itself is discriminating:it assigned −2 or lower to 21% of traces and graded evidenced reward hacking at −5.

Yet the separate behavioral alignment probe points in a different direction. In a pairwise comparison using 200 tasks, a judge received the original source Skill alongside two anonymized trajectories and chose which followed the Skill’s described method more closely. Against the base model, the judge preferred the outcome-only checkpoint on 54.5% of tasks and the rubric checkpoint on 73.0%.The rubric checkpoint is preferred more often, supporting the assumption that a rubric-based reward dimension adds behavioral alignment beyond what outcome tests alone reward. This comparison is suggestive but not decisive: each checkpoint was evaluated against the base model separately, not head-to-head, and the same judge family supplied both the rubric training rewards and the preference ratings.

The opening tension was that realistic agent training needs environments that capture both what a job requires and how practitioners judge it done well. The Skill2Env pipeline resolves the first part by compiling practitioner-authored folders into offline containers with deterministic tests, and the second by freezing each Skill’s methodology as a per-task rubric before the reference solution is written. The outcome-only RL result—a 4.7-point gain on a held-out public benchmark with no task-family overlap—suggests that Skill-derived environments teach transferable terminal-task capabilities, not just task-specific patterns. The rubric result establishes that injecting the procedural half of a Skill’s content into the reward signal can shift model behavior toward Skill-described methods, but at the cost of benchmark-measurable task completion, leaving the question of how to balance the two channels open for future work.

Binfeng Xu (@billxbf): Today we give Superintelligence back to its owners. Introducing Skill2Env 🎨 , the most aligned and diverse dataset to fuel modern Agentic RL . 🧵