@rohanpaul_ai: Most agent benchmarks end after one task, but running a store doesn't, and that's where these agents come apart. Mercha…
Summary
MerchantBench is a benchmark that assesses AI agents by having them manage a simulated online store for a year, revealing challenges with sustained performance and continuous action.
View Cached Full Text
Cached at: 08/24/26, 09:48 AM
Most agent benchmarks end after one task, but running a store doesn’t, and that’s where these agents come apart.
MerchantBench hands an agent a small online store and lets it run for a simulated year, sourcing products, setting prices, managing cash. It also scores something most evals skip: whether the agent is still acting at all.
The best agent finished a simulated year of shopkeeping with about a quarter of what humans earned, mostly by going quiet, so track how often yours still acts.
So log your agent’s actions per window and watch that curve. A decent final score can hide an agent that stopped working months ago.
– arxiv. org/abs/2607.28956
Title: “MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations”
Similar Articles
@ApexAIHighlight: Most AI benchmarks test whether a model can give the right answer. @Accio_official is testing something far harder: Can…
CommerceAgentBench is a new benchmark with 107 real-world e-commerce tasks designed to test whether AI agents can actually complete jobs, moving beyond traditional answer-based AI benchmarks.
@rohanpaul_ai: Current agent benchmarks may be ending before the real failures start. FM-Bench shows that a model can look strong afte…
FM-Bench is a benchmark for evaluating long-horizon AI agents, revealing that short-term performance does not guarantee long-term success in simulated management tasks over 20 years.
StoreBench: A Live-Commerce Environment for Evaluating and Training Autonomous Operator Agents
StoreBench introduces a live-commerce RL environment where autonomous agents run an online apparel store with 29 merchant tools, testing long-horizon planning under uncertainty. No frontier LLM matches a scripted smart-triage baseline (best, DeepSeek-V4-Pro, passes 49% vs 97%), while GRPO post-training on Qwen3.5-27B substantially improves held-out performance.
EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
Introduces EComAgentBench, a benchmark for evaluating LLM-based shopping agents on long-horizon tasks with hidden intents distributed across queries, profiles, and clarifications. The benchmark uses real Amazon products and automated scoring, revealing that even the best model achieves only 57.1% accuracy.
StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows
StartupBench introduces a benchmark for evaluating general-purpose AI agents on real-world startup workflows, revealing that top models complete only about 30% of tasks due to gaps in complex instruction following and domain-specific expertise.