SkillHarness: Harnessing Safe Skills for Computer-Use Agents

Hugging Face Daily Papers Papers
safe-ai skill-learning computer-use-agents safety-constraints dynamic-environments adversarial-interactions agent-safety

Summary

SkillHarness is a framework that enables computer-use agents to safely learn and execute skills in dynamic environments by incorporating safety constraints and adaptive skill selection mechanisms, reducing unsafe rates by 57.1%.

Computer-Use Agents (CUAs) are increasingly deployed in dynamic interactive environments, creating a growing need for continual skill learning during interaction. Recent approaches address this challenge by learning reusable skills from successful trajectories. However, these skill learning methods largely assume static and safe environments, overlooking risks from adversarial interactions (e.g., prompt injections) and environmental dynamics (e.g., pop-ups). In dynamic settings, such assumptions can lead to risky skill learning and brittle execution, undermining the reliability of CUAs. This raises the question: how can CUAs learn and use skills safely in dynamic environments? To address this problem, we propose SkillHarness, a framework for safe skill harnessing in dynamic environments. SkillHarness moves beyond static skill abstractions by modeling skill learning and utilization as a safety-constrained interaction process. Specifically, we introduce the skill boundary that leverages multi-source supervision signals to identify safe skills from interaction trajectories, and construct self-improving safety constraints throughout the skill lifecycle. In addition, SkillHarness introduces selective skill reuse, where tasks are guided to decompose according to context and completed through the selective activation of skill subsets. Our experiments demonstrate that SkillHarness significantly reduces the unsafe rate of learned skills by 57.1% and consistently improves execution stability under dynamic environmental changes, outperforming existing baselines.
Original Article
View Cached Full Text

Cached at: 06/23/26, 09:41 AM

Paper page - SkillHarness: Harnessing Safe Skills for Computer-Use Agents

Source: https://huggingface.co/papers/2606.20636

Abstract

SkillHarness is a framework that enables computer-use agents to safely learn and execute skills in dynamic environments by incorporating safety constraints and adaptive skill selection mechanisms.

Computer-Use Agents(CUAs) are increasingly deployed in dynamic interactive environments, creating a growing need forcontinual skill learningduring interaction. Recent approaches address this challenge by learning reusable skills from successful trajectories. However, theseskill learningmethods largely assume static and safe environments, overlooking risks fromadversarial interactions(e.g., prompt injections) andenvironmental dynamics(e.g., pop-ups). In dynamic settings, such assumptions can lead to riskyskill learningand brittle execution, undermining the reliability of CUAs. This raises the question: how can CUAs learn and use skills safely indynamic environments? To address this problem, we propose SkillHarness, a framework for safe skill harnessing indynamic environments. SkillHarness moves beyond static skill abstractions by modelingskill learningand utilization as asafety-constrained interactionprocess. Specifically, we introduce theskill boundarythat leveragesmulti-source supervision signalsto identify safe skills from interaction trajectories, and constructself-improving safety constraintsthroughout theskill lifecycle. In addition, SkillHarness introducesselective skill reuse, where tasks are guided to decompose according to context and completed through the selective activation of skill subsets. Our experiments demonstrate that SkillHarness significantly reduces the unsafe rate of learned skills by 57.1% and consistently improves execution stability under dynamic environmental changes, outperforming existing baselines.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2606\.20636

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.20636 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.20636 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.20636 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Self-Harness: Harnesses That Improve Themselves

Hacker News Top

Self-Harness introduces a new paradigm where LLM-based agents iteratively improve their own operating harness by mining model-specific weaknesses, proposing harness modifications, and validating them through regression testing, achieving substantial performance gains on Terminal-Bench-2.0 across multiple base models.