@rohanpaul_ai: New Shanghai AI Laboratory paper finds that training on verified runs of human-written agent skills makes a model a bet…
Summary
A Shanghai AI Laboratory paper introduces SkillGym, which turns human-written agent skill files into sandboxed pass-or-fail tasks and fine-tunes models on verified runs, improving agent performance (e.g., Terminal-Bench 2.1 +19.10 points in Claude Code) even without the skill files at prompt time.
View Cached Full Text
Cached at: 10/06/26, 05:37 AM
New Shanghai AI Laboratory paper finds that training on verified runs of human-written agent skills makes a model a better agent, even without the skill files.
Prompt-time skills depend on retrieval and instruction following, and fine-tuning on verified skill runs reduces that dependence.
Turning each skill file into a sandboxed task with a pass-or-fail checker produced training data that lifted Terminal-Bench 2.1 success by 19.10 points in Claude Code.
Skill files usually sit in the prompt, so they only help if the agent finds and follows them.
SkillGym turns each skill into a sandboxed task with a code checker, then trains on the runs that pass.
In Claude Code, Qwen3.5-35B-A3B jumped from 39.33% to 58.43% on Terminal-Bench 2.1. With no skill files, it scored 26.81% on SkillsBench, beating the base model with skills at 23.34%.
Loading the skills on top still helps, lifting it to 51.47%.
Similar Articles
@dair_ai: An interesting idea is to continually train specialized models on skills. This paper explores that idea. They propose S…
SkillGym is a framework that converts human-written skills into training environments for LLMs, enabling fine-tuning that boosts performance on benchmarks like Terminal-Bench and SkillsBench, surpassing scores from models such as Claude Sonnet 4.6 and GPT-5.4 Mini.
SkillGym: Training Skill-Use Agents with Automatic Verifiable Environment Generation
SkillGym proposes an automatic pipeline that crawls reproducible skills from the internet, builds verifiable difficulty-controlled environments, and collects 19k verified trajectories to train skill-use agents. Fine-tuning Qwen3.5 models (2B to 122B) on these trajectories improves performance across four skill-use benchmarks, with the 9B SFT model outperforming a 397B untrained model on two of them.
@rohanpaul_ai: New Tencent paper letting AI auto-improve your agent's instructions works better and costs far less if it remembers pas…
Tencent's paper SkillAdam improves agent skill-evolution by having the rewriting AI keep a memory of past fixes and make smaller edits when results are mixed, achieving 28.3% accuracy vs. SkillOpt's 21.7% while using about a third of the tokens on long shopping and travel-planning tasks.
@AlphaSignalAI: https://x.com/AlphaSignalAI/status/2069064122218717387
This article explores how AI agents can automatically write and optimize their skill files using techniques like SkillOpt from Microsoft Research, which treats skill documents as trainable state and delivers significant performance improvements. It addresses the challenge of manual skill tuning and presents frameworks like GEPA and EvoSkill as evolutionary approaches.
@dair_ai: Finally, a good paper testing whether Agent Skills actually help. Worth reading if you are maintaining a skill library …
A benchmark study shows that injecting Agent Skills in Web Development tasks often reduces performance and increases token cost, with failure modes like length-distracted and content-misled models, highlighting the need for per-deployment evaluation.