@qingke_ai: When building RL environments for agents, the most expensive rework often isn't the model — it's that the environment runs and scores get computed, but they're computing the wrong thing. CoworkEnv is a practice write-up on backtracking from "what capability do we want to train" to designing the environment. A task package must clearly specify four things, all consistent with each other: the user request, the initial state, the tool interface...

X AI KOLs Timeline Papers
agent-rl rl-environment reward-design sft ppo evaluation

Summary

A practical write-up on CoworkEnv, discussing how to design agent RL environments by backtracking from "what capability do we want to train." It distills 11 lessons covering environment consistency, scorer flaws, trajectory persistence, and training interruptions. An 8B LoRA SFT model improved Claw-Eval Overall from 46.6 to 64.1, but still trails Claude Opus 4.6's 80.4.

When building RL environments for agents, the most expensive rework often isn't the model — it's that the environment runs end-to-end and scores get computed, but what gets computed is wrong. CoworkEnv is a practice write-up on backtracking from "what capability do we want to train" to the environment design. A task package must clearly specify four things, all consistent with each other: the user request, the initial state, the tool interface, and the grading checks. Two real-world pitfalls. Five tasks pointed contact operations to /contacts/send, while the service actually exposes /contacts/send_message. Another task required creating 3 to-dos, but the scorer only checked for 2 successful creation calls. Counting only successful calls can mistake three duplicate creations for three distinct items. Trajectory persistence isn't safe either: overly long tool observations were truncated to the first 1,000 characters, and entire samples exceeding a 40,000-character budget were dropped outright. Even after a sample passes filtering, you should go back and check that the key evidence is still there. The most painful training incident: PPO crashed at step 122 out of a planned 187 due to a decoding failure, with the checkpoint interval set to one million updates — and nothing was left behind in the end. In terms of results, an 8B model with LoRA SFT on Claw-Eval went from Overall 46.6 → 64.1, multi-turn 45.9 → 77.8, and general 46.7 → 60.9. In the same table, Claude Opus 4.6 scores 80.4 — still a gap from the frontier. The most worth copying is their closing approach: first identify the most important question that needs answering right now, then design the minimal experiment capable of answering it. Limitations are stated clearly as well: the RL portion of the repo currently consists mainly of data processing and reward preprocessing; the full distributed RL stack and datasets haven't been packaged yet. The GDPO comparison group also changed batch size at the same time. If you work on agents, environment synthesis, or RL post-training, this 14-page, 11-lesson practical write-up is worth bookmarking first.
Original Article
View Cached Full Text

Cached at: 10/28/26, 01:38 PM

When building RL environments for agents, the most expensive rework rarely comes from the model itself — it comes from cases where the environment runs fine and the scores are computed, but the scores are wrong.

CoworkEnv is a practice write-up that reverse-engineers environments from the question of “what capability are you trying to train?” A task package needs to state four things clearly, and they all need to be consistent with each other: the user request, the initial state, the tool interface, and the grading check.

Two real pitfalls. Five tasks pointed contact operations at /contacts/send, while the service actually exposes /contacts/send_message. Another task asked for 3 todos to be created, but the grader only required 2 successful creation calls. Counting only the number of successful calls can treat three repeated creations of the same item as three distinct items.

Trajectory persistence isn’t safe either. Overly long tool observations are truncated to the first 1,000 characters, and any sample exceeding a 40,000-character budget is dropped entirely. Even after samples pass filtering, you still need to go back and check whether the key evidence survived.

The most painful training-side incident: PPO crashed at step 122 due to a decoding failure, out of a planned 187 steps, with the checkpoint interval set to one million updates — so nothing was left behind at all.

On results, an 8B LoRA SFT run improved Claw-Eval Overall from 46.6 → 64.1, multi-turn from 45.9 → 77.8, and general from 46.7 → 60.9. In the same table, Claude Opus 4.6 scores 80.4, so there’s still a gap to the frontier.

The most worth copying is how they wrap things up: first pin down the question that most urgently needs answering, then design the minimal experiment capable of answering it.

The limitations are stated clearly as well: the RL portion of the repo is currently mostly data processing and reward preprocessing — the full distributed RL stack and datasets haven’t been packaged yet. The GDPO comparison also changed batch size at the same time.

For anyone working on agents, environment synthesis, or RL post-training, this 14-page, 11-lessons practice note is worth bookmarking first.

Similar Articles

@Xudong07452910: A classic challenge in RL training of LLM agents: after a long task fails, where should the model start learning? The final reward can usually only tell the agent 'success' or 'failure', but it's hard to pinpoint which intermediate judgments are worth keeping and which actions led the entire trajectory astray. This paper proposes SEED, using 'self-evolving online distillation...'

X AI KOLs Timeline

This paper proposes SEED, a method that internalizes post-hoc skills from trajectories into model parameters through self-evolving online distillation, solving the reward sparsity problem in long-horizon RL training, achieving significant improvements on benchmarks such as ALFWorld.

@NFT_Chen: This is incredible! Agent interactive RL is finally out of "engineering hell"! HarborRL open-sources a full framework enabling real coding agents to do stable RL! The hardest part of training agents with RL was never the algorithm — it was the engineering nightmares: trajectory collection, token alignment, multi-turn tool calls, policy version consistency. HarborRL solves all of them as a reproducible pipeline.

X AI KOLs Timeline

HarborRL has open-sourced a lightweight agent reinforcement learning framework that integrates model serving, trajectory generation, tool execution, verifier-based rewards, and policy updates into a reproducible pipeline. It supports flexible combinations of Qwen/GLM models × GRPO/DAPO algorithms × 200+ sandbox tasks × 40+ agent harnesses (such as Claude Code and SWE-agent).

@FeitengLi: Built a ReAct agent system by hand: Doing agent systems with LLMs. While walking this evening, I was thinking about how to train an LLM's agentic capabilities, data preparation, model training, constructing RL training with agent trajectory actions, and also about Claude's progress over the past year…

X AI KOLs Following

The author shares their experience building a ReAct agent system and introduces the GLM-5 technical report released by Zhipu AI, which achieves breakthroughs in agentic, reasoning, and coding capabilities.