real-world-tasks

ByteDance Seed released EdgeBench, a benchmark that tests whether AI agents can improve through experience by performing real-world tasks over 12+ hours, shifting evaluation from static knowledge to dynamic learning.

Explores the trend of AI agents autonomously spending money to complete real-world tasks like purchasing services, booking resources, and running ads without human approval.

This article discusses the key requirements for AI agents to successfully complete real-world tasks: a real phone number, email address, and payment method, highlighting products like AgentLine, Agent Mail, and Agent Card that provide these capabilities.

Discusses a limitation of current agent benchmarks that assume the world changes only when the agent acts, whereas many real-world tasks require the agent to wait for external events before acting.

WeaveBench is a new benchmark for evaluating computer-use agents across multiple interfaces (GUI, CLI, code) in long-horizon real-world tasks. It reveals that current models achieve only 41.2% PassRate and that outcome-only grading overestimates performance, highlighting significant gaps in evaluation.

SpatialWorld is a unified benchmark for evaluating interactive spatial reasoning in multimodal agents across diverse real-world tasks, revealing that even the strongest models achieve low task success rates.

Introduces Agents' Last Exam (ALE), a benchmark for evaluating AI agents on long-horizon, economically valuable real-world tasks across 13 industry clusters with over 1000 tasks, revealing a large gap between benchmark performance and practical deployment.

Discusses the persistent challenges that prevent AI agents from reliably handling real-world tasks, such as changing websites and inconsistent workflows, despite progress in task execution.

RLDX-1 is a general-purpose robotic policy for dexterous manipulation that uses a Multi-Stream Action Transformer architecture to integrate heterogeneous modalities, outperforming existing VLA models in real-world tasks.

SkillLearnBench introduces the first benchmark for evaluating continual skill learning in LLM agents across 20 real-world tasks, revealing that no method dominates and scaling LLMs does not guarantee better skills.

GTA-2 introduces a hierarchical benchmark for evaluating general tool agents across atomic tool-use and open-ended workflows, revealing a significant capability cliff where frontier models achieve only 14.39% success on complex tasks despite reasonable atomic performance.

OpenAI introduces GDPval, a new evaluation framework measuring AI model performance on economically valuable, real-world tasks across 44 occupations in the top 9 US GDP-contributing industries. The benchmark includes 1,320 specialized tasks based on actual professional work products, representing a progression from academic benchmarks to more realistic occupational assessments.

OpenAI introduces SWE-Lancer, a benchmark of over 1,400 real-world freelance software engineering tasks from Upwork valued at $1 million USD, designed to evaluate AI model performance on practical engineering work and map model capabilities to economic value.