Building agent skills by demonstration instead of hand-writing them: a record-and-compile approach

Reddit r/ArtificialInteligence Tools

Summary

A record-and-compile tool lets users demonstrate desktop tasks, then an LLM converts the session into structured skill files for agents to replay, aiming to reduce brittleness and lower the barrier to creating reusable agent capabilities.

I was automating a repetitive desktop task and got tired of hand-writing agent skill files, so I went looking for a better approach and found a tool that takes an interesting angle - sharing it here for discussion. Instead of scripting the automation step by step, you demonstrate the task once on screen. It records the session, then an LLM compiles the event log plus screen context into a structured SKILL.json and a readable SKILL.md the agent can replay later. It runs as an MCP server exposing record, stop, compile, and list, so it plugs into any MCP client. What I found worth discussing is the "show, don't script" idea: reading native UI events plus a screen recording makes the resulting skill far less brittle than a rigid macro, and it lowers the barrier to turning everyday workflows into reusable agent capabilities. Open source, and I'm not affiliated with it - just a useful find from a fellow builder. I'll drop the link in the comments. Do you think demonstration-based skill creation is a viable path for agents, or does it break down on complex, branching tasks?
Original Article

Similar Articles

Formal Skill: Programmable Runtime Skills for Efficient and Accurate LLM Agents

arXiv cs.AI

This paper introduces Formal Skill, a runtime-native abstraction for LLM agents that encodes reusable procedures as executable state machines with JSON metadata, Python executors, and hook-governed control logic. An open-source implementation called FairyClaw is presented, showing competitive performance on Harness-Bench with reduced token usage.

Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems

arXiv cs.CL

This paper presents a method for compiling repeated standard operating procedure steps into validated, versioned tools before deployment, replacing inference-time code generation. In a fulfillment center alarm-triage system, this approach reduces p50 latency by 42% and end-to-end error rate by up to 53%.

Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction

arXiv cs.AI

Work2Skill 提出一个框架,通过 WorkRecorder 收集人类 GUI 操作演示,构建 WorkDemo-350 多模态数据集,并用智能体式技能归纳方法从人类经验中提炼可复用的智能体技能,在 MyPCBench 和 OSWorld-Verified 上显著提升了多种执行模型的任务表现。