Paper page - CLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation Pipeline

Source: https://huggingface.co/papers/2610.19128

Abstract

CLASHarness introduces a controllable framework for generating verifiable multi-agent conflicts in computer-use tasks, revealing that frontier vision-language models struggle with verifying ground truth, resolving conflicts, and coordinating shared resources.

Vision-language models(VLMs) are increasingly deployed inmulti-agent systemsto automate complex computer-use workflows, yet existing benchmarks mostly assume consistent, cooperative environments. In contrast, real-world collaboration rarely runs on consistent information: peer or sub-agents frequently report outdated states, issue contradictory instructions, or compete over shared permissions and files, forcing agents to check others’ outputs and verify ground truth in the environment. Evaluating suchconflict resolutionremains challenging: manual authoring cannot scale across complex systems, while unconstrained LLM generation produces narrative drift, answer leakage, and unsatisfiable rubrics. We introduceCLASHarness(Conflict-Loaded Agent Scenario Harness), a controllable framework for generating verifiable multi-agent conflicts in computer use, with a benchmark of 600 multi-round tasks. By combining hierarchical scenario planning with an eight-step verified micro-pipeline,CLASHarnesspre-commits decisive visual facts before rendering and compiles hybrid code and semantic graders directly from executed reference trajectories, guaranteeing every task is solvable by construction. Evaluating 15 frontier models onCLASHarnessreveals three core bottlenecks. First, when sub-agent outputs conflict with the environment, weaker models blindly follow misleading information, whereas stronger models inspect visual evidence to verify ground truth. Second, resolving conflicts is much harder than finding them: while agents can spot inconsistencies, actively fixing system state drops task success significantly. Third,coordination protocolsremain a universal bottleneck: even frontier models fail over 58% of closing actions, often forgetting to releaseshared locksor hand off tasks.CLASHarnessprovides a scalable, verifiable foundation for building reliablemulti-agent systems.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2610\.19128

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.19128 in a model README.md to link it from this page.

Datasets citing this paper1

#### yuxianglai117/CLASHarness Updatedabout 13 hours ago • 3

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.19128 in a Space README.md to link it from this page.

Collections including this paper2