WorkBench Revisited: Workplace Agents Two Years On

# WorkBench Revisited: Workplace Agents Two Years On
Source: [https://arxiv.org/html/2606.13715](https://arxiv.org/html/2606.13715)
\[ Extension = \.otf, UprightFont = \*\-regular, BoldFont = \*\-bold, ItalicFont = \*\-italic, BoldItalicFont = \*\-bolditalic, \]

###### Abstract

The best agent on WorkBench in March 2024, GPT\-4, completed 43% of tasks and took an unintended harmful action, such as emailing the wrong person, on 26% of them\. We re\-visit the benchmark in June 2026 and find that the best agent to date, Claude Opus 4\.8, completes 89% and takes an unintended harmful action on 2\.5%\. Aside from this considerable progress in frontier agent performance, three things stand out\. First, capability and safety go together on WorkBench rather than trade off, so the models that finish the most tasks also do the least unintended damage\. Second, while several classes of error have been totally eliminated, frontier models still make some basic mistakes that occasionally result in irreversible harm, such as sending an email to the wrong person\. Third, the rise of open\-weight models has drastically lowered costs for a performance level that was previously only accessible to proprietary models, while frontier costs have stayed relatively stable\. We release an updated version of the benchmark with data and code quality improvements, new model scores, and analysis of agent progress on WorkBench since 2024\.

![[Uncaptioned image]](https://arxiv.org/html/2606.13715v1/x1.png)

Figure 1:Outcome composition by model\.Each model’s 690 WorkBench tasks split into correct, failed\-but\-harmless, and harmful side effect, ordered by task completion \(best at bottom\)\. GPT\-4 is the original 2024 result \(43% completion, 26% side effects, scored with a ReAct loop on the pre\-revision benchmark\); the other seven are 2026 runs from Table[1](https://arxiv.org/html/2606.13715#S2.T1)\.
## 1Introduction

Large language models are now increasingly deployed as agents that take actions on a user’s behalf: updating customer records, managing email, scheduling meetings\. Most agent benchmarks measure something adjacent to this, such as web navigation\(Zhou et al\.,[2024](https://arxiv.org/html/2606.13715#bib.bib6)\), general assistance\(Mialon et al\.,[2023](https://arxiv.org/html/2606.13715#bib.bib3)\), or broad tool use\(Liu et al\.,[2023](https://arxiv.org/html/2606.13715#bib.bib2)\)\. WorkBench\(Styles et al\.,[2024](https://arxiv.org/html/2606.13715#bib.bib4)\)was built to measure the workplace case directly\. It provides a sandbox of five databases \(a calendar of 300 events, an inbox of 500 emails, 500 website\-analytics records, a customer relationship manager with 200 customers, and a project\-management board of 300 tasks\), 26 tools to read and write them, and 690 tasks generated from templates \(10 tasks per template\) that span single\-domain and multi\-domain work\. A task is graded by comparing the sandbox’s final state against the ground truth, so the agent can take any path it likes, recover from its own mistakes, and there’s no second model sitting in judgment\. This is what sets WorkBench apart from action\-matching benchmarks and from those that lean on an LLM evaluator, such as API\-Bank\(Li et al\.,[2023](https://arxiv.org/html/2606.13715#bib.bib1)\)\.

At release in 2024 the benchmark was far from solved\. The strongest agent, a ReAct\(Yao et al\.,[2022](https://arxiv.org/html/2606.13715#bib.bib5)\)loop around GPT\-4, completed 43% of tasks; the weakest open model evaluated, Llama2\-70B, managed 3%\. Two years later the set of available models is almost entirely new\. So the question is simple: how much has changed, and in what ways?

We do three things:

- •We re\-run WorkBench on 21 models released between 2023 and 2026, spanning four vendors and both proprietary and open\-weight models, under a single modern agent harness that uses native tool\-calling rather than text\-parsed ReAct \(Section[2\.1](https://arxiv.org/html/2606.13715#S2.SS1)\)\.
- •We report task completion alongside two axes the original paper did not: the rate of harmful side effects, and an estimate of the dollar cost of running the benchmark once \(Section[2](https://arxiv.org/html/2606.13715#S2)\)\.
- •We correct a set of scoring, ground\-truth, and prompt issues in the original benchmark, make engineering improvements to its tools, and quantify which tasks they affect, so that results on the 2026 benchmark are comparable from here on \(Section[4](https://arxiv.org/html/2606.13715#S4)\)\.

In short: the field has improved sharply but unevenly, and a newer release date buys surprisingly little\. Completion on the corrected benchmark has roughly doubled, the best models now fail almost entirely in harmless ways, and the spread in capability and cost across same\-era models is wide enough that release date tells you little about either\. The model results come first; we close with the updates to the benchmark since 2024, the corrections and engineering improvements that underlie those numbers\.

## 2Results

### 2\.1Experimental Setup

Agent harness\.The original paper ran each model as a ReAct\(Yao et al\.,[2022](https://arxiv.org/html/2606.13715#bib.bib5)\)loop that parsed a tool call out of free\-form text\. We instead use the native tool\-calling \(structured output\) interface that every current model provider exposes, so the model emits a typed call against the tool schema and the harness never has to recover a malformed action from prose\. The loop is otherwise unchanged: the agent is given the task, all 26 tools on every task, and up to 20 steps to reach a final state, with temperature set to zero where the model permits it\. This single change removes the format\-adherence failures that dominated the 2024 results \(Section[3](https://arxiv.org/html/2606.13715#S3)\), and we adopt it for every model so that the comparison is like\-for\-like\.

Models\.We evaluate 21 models released between March 2023 and May 2026: the GPT line from GPT\-3\.5\-turbo through GPT\-5\.5, three Claude models \(Opus 4\.8, Sonnet 4\.6, and Haiku 4\.5\), two Gemini models \(3\.1\-pro and 3\.5\-flash\), and four open\-weight models \(Qwen, DeepSeek, Kimi, GLM\)\.

Cost estimation\.We estimate the cost of one full benchmark run from the logged prompts and completions\. Input and output token counts are approximated from string lengths at four characters per token, and the fixed per\-call overhead of the system prompt and the full tool schema \(about 7,000 tokens, re\-sent on every call\) is added back\. We price these at each provider’s published standard per\-token rate, without caching\. The resulting figure is therefore an upper bound: a provider that caches the repeated system prompt and schema would bill materially less\.

### 2\.2Findings

Table[1](https://arxiv.org/html/2606.13715#S2.T1)reports, for the 21 models, successful task completion, the rate of harmful side effects \(a wrong action taken, such as an email sent to the wrong person, where lower is better\), and the estimated cost per task\.

ModelSuccessful taskcompletion \(↑\\uparrow\)Side effects \(↓\\downarrow\)Cost per task \(↓\\downarrow\)Claude Opus 4\.888\.8%2\.5%$0\.182GPT\-5\.587\.7%3\.9%$0\.206Gemini\-3\.1\-pro87\.7%3\.0%$0\.076Gemini\-3\.5\-flash84\.2%3\.0%$0\.067Claude Sonnet 4\.680\.7%9\.7%$0\.105Kimi\-K2\.680\.6%6\.8%$0\.022DeepSeek\-V4\-pro77\.8%12\.8%$0\.017GPT\-577\.7%13\.0%$0\.050GPT\-5\.471\.2%16\.8%$0\.087o371\.0%17\.5%$0\.072GLM\-4\.670\.7%17\.1%$0\.017GPT\-4\.170\.0%19\.4%$0\.065Claude Haiku 4\.567\.5%16\.7%$0\.034GPT\-5\.263\.3%18\.8%$0\.055Qwen3\.563\.2%21\.4%$0\.003GPT\-4o62\.9%15\.1%$0\.068GPT\-4\-turbo56\.7%22\.3%$0\.307GPT\-5\.4\-mini53\.9%30\.3%$0\.027GPT\-5\.152\.5%18\.1%$0\.036GPT\-5\.4\-nano44\.2%28\.6%$0\.007GPT\-3\.5\-turbo25\.8%38\.7%$0\.016Table 1:Task completion, side effects, and cost per task\.Completion and side effects are over all 690 tasks; cost per task is the total benchmark spend divided by 690\. Rows are ordered by task completion\.↑\\uparrowmeans higher is better and↓\\downarrowmeans lower is better\.Completion has roughly doubled\.The best agent in 2024 completed 43% of tasks; the best in 2026, Claude Opus 4\.8, completes 88\.8%\. Six models from four different providers now clear 80%\. The strongest model still fails one task in nine, but the headroom is thinner than that suggests: the failures it has left are the harder reasoning and multi\-step retrieval cases, plus a residue of tasks where model reasoning is defensible despite being scored as incorrect \(Section[3](https://arxiv.org/html/2606.13715#S3)\)\.

Progress has been smooth since 2024\.Figure[2](https://arxiv.org/html/2606.13715#S2.F2)plots task completion against each model’s public release date across all vendors\. The grey line traces the frontier, the best completion reached by any model up to that date, and it climbs steadily from GPT\-3\.5\-turbo’s 25\.8% in early 2023 to Opus 4\.8’s 88\.8% in mid\-2026\. Many later releases sit well below that frontier: the smaller GPT\-5\.4\-mini and \-nano tiers, and even some flagships such as GPT\-5\.1, land below older models\.

![Refer to caption](https://arxiv.org/html/2606.13715v1/x2.png)Figure 2:Task completion on WorkBench by release date\.Successful task completion for every evaluated model against its public release date, coloured by vendor\. The grey line is the frontier of best completion over time: it only moves upward, so models that fall below the running best are not joined to it\.![Refer to caption](https://arxiv.org/html/2606.13715v1/x3.png)Figure 3:Cost per task versus task completion on WorkBench\.Cost per task is the total spend to run the benchmark once divided by 690 tasks; the horizontal axis is logarithmic\. Points are coloured by whether the model has open weights or is proprietary\. The grey line is the efficient frontier \(the most capable model at each price\); only frontier models and a few notable outliers are labelled\.Cost spans two orders of magnitude\.Figure[3](https://arxiv.org/html/2606.13715#S2.F3)plots cost per task against task completion on a log cost axis\. The grey line is the efficient frontier: its cheap end is entirely open weights, and every one of those models comes from a Chinese lab: Qwen3\.5, DeepSeek\-V4\-pro, and Kimi\-K2\.6\. The expensive, high\-completion end is the mirror image, entirely Western and proprietary: the two Gemini models and Opus 4\.8\. What falls off the frontier in between is mostly the Western budget tier\. Claude Haiku 4\.5 and the GPT\-5\.4 mini and nano models are neither the cheapest nor the most capable, and each is dominated by a cheaper, more capable open\-weight model: GPT\-5\.4\-nano \($0\.007, 44%\) loses outright to Qwen3\.5 \($0\.003, 63%\), and Haiku 4\.5 \($0\.034, 68%\) is beaten by DeepSeek, Kimi, and GLM at once\. The cheapest capable agent today is a Chinese open\-weight model and the most capable is a Western proprietary one, and the Western budget options are squeezed from both sides\. The cost reduction over time is stark: Qwen3\.5 exceeds GPT\-4’s performance, the 2024 state\-of\-the\-art, at approximately one\-hundredth the cost\.

## 3Revisiting the Original Error Categories

We revisit the seven largest sources of error from the original paper with current models\. The two largest sources for the frontier model of early 2024 \(GPT\-4\) have been eliminated by today’s best models\.

- •Failing to follow ReAct\.Eliminated\. This is due to more tool\-use\-specific training and to the ability to use constrained decoding \(also known as structured outputs\) to guarantee schema adherence\.
- •Sending information to the wrong email address\.Almost eliminated\. Models no longer treat thename@example\.comin the docstring as a pattern to imitate, and instead resolve the correct address using the provided tool\. We did, however, observe a single instance of GPT\-5\.5 sending an email to an@exampleaddress\.
- •Plotting future data\.Not improved\. The system prompt states that today’s date is Thursday2023\-11\-30and the current time is00:00:00\. Models consistently plot with2023\-11\-30as the end date, for which there is no data\. This is arguably defensible behaviour, but in practice it produces a null day in the plot, so we count it as incorrect\.
- •Failing to identify an available calendar slot\.Eliminated\. Frontier LLMs consistently use the calendar search tool correctly and book the correct slot\.
- •Misinterpreting retrieved data\.Reduced, though models still make basic errors\. For example, given the task“please check the percent growth of engaged users since Friday\. If it grew by more than average session duration…”, Opus 4\.8 compared the percentage growth of engaged users against the most recent raw value of average session duration\.
- •Updating the wrong event\.Eliminated\. Frontier LLMs consistently use the correct search tools to identify which event to update\.
- •Using search incorrectly\.Reduced\. Many search tools in WorkBench cap results at five per query, so the agent is expected to make multiple calls when more than five results are needed to complete a task\. This still occasionally trips up frontier LLMs\.

## 4Updates Since 2024

The numbers above are scored on a benchmark that has changed since the 2024 release\. We have made two kinds of change: corrections that fix cases where the benchmark was unfair to the agent or simply scored the wrong thing, and engineering improvements to the tools and task design\. We document both here, and quantify how much the corrections move scores\.

### 4\.1Benchmark Corrections

These aren’t cosmetic\. They remove cases where the benchmark was unfair to the agent or simply scored the wrong thing\. A few representative examples:

- •Off\-by\-one in “lastNNdays” ground truth\.The date cutoff was computed astoday−N\-Nrather thantoday−N\+1\-N\{\+\}1, which shifted the answer key for a handful of email and calendar tasks\. Fixed and regenerated\.
- •Prompt and answer\-key mismatches\.Several tasks asked one thing but were graded against another\. A placeholder always displayed “more than” while the answer was sometimes computed for “less than”, and one email task had a graded subject that was unreachable from the wording\. The agent was being marked wrong for following the instructions correctly\. Prompts now match the branch that is graded\.
- •A silent\-zero aggregation bug\.A “fewest overdue tasks” task used anidxminthat dropped people with zero overdue tasks, so the correct answer could never be a zero\-count person\. Reworded and recomputed\.
- •Push\-back tasks now respect working hours \(1 task\)\.A “push back my first meeting by 2 hours” task moved a meeting to end at 18:30, breaking the prompt’s own “no meeting ends after 6pm” rule, so a model that correctly refused was scored wrong\. The delay is now clamped to keep the meeting ending by 6pm, changing only that one task\.
- •Email bodies show real newlines \(10 tasks\)\.Send\-email prompts displayed the body with escaped\\nwhile the answer key parsed to real newlines, so a faithful copy stored literal backslash\-n and failed all 10 send\-email tasks\. The displayed body now contains real line breaks, leaving a single canonical answer \(the gold column is unchanged\)\.
- •Clearer tool descriptions\.Tool docstrings now enumerate the allowed values for enum parameters and document result limits, so the agent is told the rules of the sandbox up front rather than discovering them by trial\.
- •Counting queries made solvable\.“Assign to the person with the fewest or most tasks” was unwinnable whilesearch\_taskscapped results at five and no tool enumerates people\. The cap onsearch\_tasksalone is raised to 200 so the agent can enumerate a board and aggregate; other searches keep the cap of five\.

#### 4\.1\.1Scope of impact

The cleanest way to see how much the benchmark changed is to hold the model fixed and re\-score it on both versions\. GPT\-4, the 2024 frontier model, went from 49% on the old benchmark to 57% on the corrected one, resampled the same way on both\. That eight\-point jump is the benchmark getting fairer, not the model getting smarter, and it is the number to keep in mind whenever a 2026 result is set next to a 2024 one\.

Under the hood the fixes land on a minority of tasks\. The hard floor, tasks whose ground truth or wording literally changed, is 56 tasks \(8%\), and that count is deterministic\. Widen it to include the fixes that change whether a correct attempt is allowed to pass, the CRM read\-data corrections and the move to order\-independent grading, and it is roughly 90 to 95 tasks \(about 14%\)\. Everything else scores the same task the same way it always did\. Either way the takeaway is the same: a model’s 2026 score isn’t directly comparable to its 2024 score, so the comparison we trust is between models run under the 2026 benchmark\.

## 5Specific Failure Cases

A collection of agent failures from the 2026 runs, each showing a task and how a model went wrong on it\.

### 5\.1Over\-zealous: acts when the condition is false

Task\.“I think dmitri might have some overdue tasks\. Can you check and if so, book a half hour meeting with them called ‘Catch up on overdue tasks’ at the earliest time I’m free tomorrow\.” Failure\.Strictly, Dmitri has no*past\-due*tasks: every task with a due date before today \(2023\-11\-30\) is already marked “Completed”\. Two tasks are due*today*and still incomplete, so they are about to become overdue but are not overdue yet\. The conditional \(“if so”\) should evaluate to false and the correct outcome is to take no action, but the model treats “about to be overdue” as overdue and books the meeting anyway\.

### 5\.2Reasoning failure: comparing a percentage to a raw value

Task\.“please check the percent growth of engaged users since Friday\. If it grew by more than average session duration make a front\-end backlog task called ‘Improve average session duration’ for kofi that’s due next Friday and schedule a 30 minute meeting called ‘Discuss engaged users’ for us at the earliest slot i’m free tomorrow\.” Failure\.The task compares two percent\-growth figures and acts only if one exceeds the other\. The model instead compares a percentage against a raw value \(or misjudges the sign\) and skips the action the gold answer requires\. It concludes:“The engaged users grew by 0%, which is not more than the average session duration\.”

### 5\.3Trusting a truncated search result

Failure\.Search tools return*at most five*results, and the docstring says so\. When five calendar events come back, the model assumes it has seen them all and stops, rather than re\-querying with tighter parameters to narrow the set\. It then answers from an incomplete view of the data\.

### 5\.4Plotting a day that has not happened

Failure\.The model calls the plotting tool for a date beyond the sandbox’s current day, producing a chart for a day that has no data yet\.

## 6Discussion

Two years ago, nothing cleared WorkBench\. Today the best agents largely pass it\. It still discriminates, though: completion runs from 26% to 89%, cost over two orders of magnitude, and side effects from 2% to 39%\.

Several limitations bound these conclusions\. The sandbox is simpler than a real workplace: a real inbox has thousands of historic messages and spam, and a real calendar has years of events, so absolute completion here is an optimistic estimate of deployed reliability\. The cost figures are no\-caching upper bounds derived from token\-length estimates rather than billed spend, and providers that cache the repeated system prompt and tool schema would pay less, so cost should be read as an ordering rather than an invoice\. Outcome\-centric evaluation, the feature that makes WorkBench cheap and reproducible to score, can’t grade pure\-retrieval question answering that leaves the sandbox unchanged, and it forces a judgement call on cases like plotting a day with no data, which we score as incorrect\. Finally, the release\-date analysis mixes vendors with different training objectives and data, so the frontier in Figure[2](https://arxiv.org/html/2606.13715#S2.F2)is descriptive, not a claim about any single lab’s rate of progress\.

WorkBench has been public on GitHub since 2024, and we keep no private hold\-out set, so every task and its answer key is fair game for a web crawl\. Any model trained after the 2024 release may well have seen the benchmark during pre\-training\. We have no clean way to tell genuine capability gains from memorisation\. A held\-out split, refreshed each year and never published, is the obvious fix for the next iteration, and until there is one the over\-time results should be read as an upper bound on real progress\.

WorkBench in 2026 is largely solved by frontier models: task completion on the corrected benchmark has roughly doubled since 2024, the strongest agents now fail almost entirely in harmless ways, and capability and safety improve together rather than trading off\. Costs span two orders of magnitude, however, and cheaper models lag behind the frontier\. We release the updated benchmark, the updated harness, and the per\-model cost estimates so that the next generation of models can be placed on the same axes\.

## References

- Li et al\. \(2023\)Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li\.Api\-bank: A comprehensive benchmark for tool\-augmented llms, 2023\.
- Liu et al\. \(2023\)Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang\.Agentbench: Evaluating llms as agents, 2023\.
- Mialon et al\. \(2023\)Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom\.Gaia: a benchmark for general ai assistants, 2023\.
- Styles et al\. \(2024\)Olly Styles, Sam Miller, Patricio Cerda\-Mardini, Tanaya Guha, Victor Sanchez, and Bertie Vidgen\.Workbench: a benchmark dataset for agents in a realistic workplace setting\.In*Conference on Language Modeling \(COLM\)*, 2024\.arXiv:2405\.00823\.
- Yao et al\. \(2022\)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao\.React: Synergizing reasoning and acting in language models, 2022\.
- Zhou et al\. \(2024\)Shuyan Zhou, Frank F\. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig\.Webarena: A realistic web environment for building autonomous agents\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=oKn9c6ytLx](https://openreview.net/forum?id=oKn9c6ytLx)\.