OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

Hugging Face Daily Papers Papers
benchmark llm-agents office-automation economic-grounding long-horizon-tasks evaluation

Summary

OmegaUse-OfficeVal is a benchmark for evaluating LLM agents on long-horizon office-suite tasks with economic grounding, comparing human costs and LLM inference costs. It includes 100 tasks requiring ~2.3 hours of human labor each, and finds that frontier LLMs are cheaper and faster but still below human quality.

Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.
Original Article
View Cached Full Text

Cached at: 07/30/26, 05:46 AM

Paper page - OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

Source: https://huggingface.co/papers/2607.27155 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Largelanguagemodel(LLM)agentsareincreasinglyexpectedtoassistusersincompletingtasks.However,existingbenchmarksprovidelimitedsupportforevaluatingwhetheragentscancarryoutoffice-suiteworkflowsatareasonablecost.WeintroduceOmegaUse-OfficeVal,abenchmarkforevaluatingLLMagentsonlong-horizonoffice-suitetaskswithtask-leveleconomicgrounding.Thebenchmarkcomprises100tasksderivedfromoffice-suiterequestsproposedbypractitionersandadaptedthroughaprivacy-preservingprocess.Onaverage,thesetasksrequire2.32hoursofhumanlabortocomplete.Animportantfeatureofthebenchmarkisthateachtaskispairedwithtwoeconomicsignals:humanlabortimeandtaskpriceproxy.ThesesignalsenabledirectcomparisonsbetweenhumancostsandLLMinferencecosts,aswellasvalue-weightedevaluation.Tosupportstableevaluation,wedevelopcode-basedverifiersfromfine-grainedrubrics.WeevaluateseveralfrontierLLMstogetherwithahumanbaseline.AlthoughallevaluatedLLMsaresubstantiallycheaperandfasterthanhumanworkers,theyhavenotyetapproachedhuman-leveldeliverablequality.Thecodeanddatasetarefullyopen-sourced,andmoreinformationisavailableonourprojectwebsite:https://omegause-officeval.github.io.

View arXiv pageView PDFProject pageGitHub3Add to collection

Get this paper in your agent:

hf papers read 2607\.27155

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.27155 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.27155 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.27155 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles