CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments

arXiv cs.AI Papers
multimodal-agents benchmark behavioral-science ai-evaluation llm-agents data-contamination

Summary

CogArena は社会科学・行動科学の10の実験パラダイムをブラウザ経由でエージェントに実行させるベンチマークを提案し、マルチモーダルモデル4種を評価した結果、実験の完了と行動特性の再現が別物であることを示した。

arXiv:2610.36484v1 Announce Type: new Abstract: Claims about the cognitive capabilities of large language models are typically tested by translating behavioral paradigms into text prompts, bypassing the visual interfaces and time pressure through which human participants actually engage with experiments. At the same time, AI agents can now complete online behavioral tasks and produce human-like data, which raises concerns that online datasets in the social and behavioral sciences are increasingly open to contamination. We introduce CogArena, a benchmark of ten experimental paradigms from the social and behavioral sciences. Agents complete the experiments through a standard web browser, working with the same visual stimuli and response interfaces that online participants use. A three-level scoring procedure evaluates whether the agent completed the task, how accurately it performed, and whether its data show the behavioral signatures each paradigm reliably produces in people. Across four multimodal models, each run ten times per task, we find that finishing an experiment and reproducing its signatures are different things. Many sessions did not finish, and those that did reproduced some signatures reliably and others not at all. CogArena provides the tasks, scoring code, and a public leaderboard for new submissions.
Original Article

Similar Articles

Cowork-Gym: Synthesizing Interactive Work Environments at Scale

Hugging Face Daily Papers

Cowork-Gym 通过 build-execute-revise 循环与有状态用户模拟器大规模合成协作式 agent 环境（64.9K 个、14 个应用场景），并用于后训练 9B/27B/35B-A3B Qwen 模型，在六个 agentic 基准上分别提升 8.7、6.4、4.1 分，其中 27B 版本在 PawBench 和 ClawEval 上取得最高平均分。

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction

Hugging Face Daily Papers

WorldMemArena is a new benchmark with 400 multi-session multimodal tasks for evaluating multimodal agent memory, comparing long-context, RAG, and harness-based memory approaches, revealing that better memory writing does not guarantee better performance and that systems struggle with visual evidence.

Multi-Paradigm Agent Interaction in Practice:A Systematic Analysis of Generator-Evaluator, ReAct Loop,and Adversarial Evaluation in the buddyMe Framework

arXiv cs.AI

This paper presents a systematic analysis of three agent interaction paradigms (Generator-Evaluator, ReAct Loop, and Adversarial Evaluation) implemented in the buddyMe framework, with empirical case studies from real-world deployments. It formalizes a five-stage pipeline and a six-dimensional evaluation schema, offering practical design guidelines for multi-paradigm agent systems.

Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

Hugging Face Daily Papers

Co-Design Bench is a benchmark of 273 real-user multimodal coding tasks with 2,331 expert acceptance criteria, evaluating agents via a judge-then-refute protocol across 10 model endpoints, where the best endpoint (gpt-6-astra) passes 69.9% of criteria but fully satisfies only 26.4% of tasks.