@0xLogicrw: Alibaba Tongyi Lab launches Agent Evaluation Benchmark PawBench v1.0, for the first time integrating base models and runtime frameworks into a unified evaluation system. The evaluation cross-tests 9 large models with three frameworks: Hermes, OpenClaw, and QwenPaw, covering 150 real-world tasks and 4050 ...

Alibaba Tongyi Lab launches Agent Evaluation Benchmark PawBench v1.0, for the first time integrating base models and runtime frameworks into a unified evaluation system. The evaluation cross-tests 9 large models with three frameworks: Hermes, OpenClaw, and QwenPaw, covering 150 real-world tasks and 4050 test units. The results show that the design of the runtime framework directly determines whether agent capabilities can be stably deployed. With the same model, the three frameworks exhibit significant performance gaps. QwenPaw scores 76.4, OpenClaw scores 75.4, while Hermes only scores 70.4. The 6.4-point gap is comparable to a major model version upgrade. Excellent design can even enable smaller models to achieve 'upset victories': in the Hermes framework, GLM 5.1 only scores 68.2, while in the QwenPaw framework, the smaller-scale Qwen3.6-35b-a3b scores 70.4. By analyzing runtime trajectories, the performance differences stem from a lack of substantive verification of workspace artifacts, loose tool path constraints, and an oversized tool table that increases model decision burden. Most frameworks also exhibit clear deficiencies in active discovery of local specialized skills (Skill) and zero-configuration availability of web search. The evaluation team proposed four fundamental principles for framework design: First, Inform Fully: clearly communicate environmental context such as cwd and workspace; Second, Equip on Demand: control the number of tools and ensure key tools like password-free search are available by default; Third, Monitor Actively: verify that task artifacts such as file writes are actually realized; Fourth, Recover Gracefully: provide opportunities for error correction and continuation when tools are abnormal or artifacts are missing.