Back to the Future: A workbook time machine for spread sheet creation benchmarks

# A workbook time machine for spreadsheet creation benchmarks
Source: [https://arxiv.org/html/2608.07873](https://arxiv.org/html/2608.07873)
## ↺\\circlearrowleftBack to the Future: A*workbook time machine*for spreadsheet creation benchmarks

Mukul SinghGust VerbruggenVu LeSumit GulwaniMicrosoftEmail in order:\{mansiuniyal, t\-agasingh, ananyasingha, priyansgupta, singhmukul, gverbruggen, levu, sumitg\}@microsoft\.com

###### Abstract

We introduce the*workbook time machine*, a pipeline that automatically creates benchmarks evaluating the ability of language models to create derived objects in spreadsheets \(formulas, charts, pivot tables, and conditional formatting\)\. Applied to public workbook corpora, it producesWTM\-Corpus–a collection of \(input workbook, output workbook, query\) triples spanning four artifact types and varying complexity\. From this corpus we curateWTM\-Bench, a 150\-task evaluation benchmark with queries at three levels of specificity\. We evaluate existing spreadsheet manipulation agents and baselines onWTM\-Benchacross artifact types, step complexity, and instruction granularity\. Our evaluations show that query specificity, agent orchestration, and interface API used to control spreadsheets play a big role in LLM performance on Excel tasks\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.07873v1/x1.png)Figure 1:Example of creating a benchmark with the*workbook time machine*, which removes objects from a workbook \(left\) and then generates an instruction that describes the removed objects\.Spreadsheets are the world’s most widely used low\-code platform, with over 750 million users\(Bendreet al\.,[2019](https://arxiv.org/html/2608.07873#bib.bib26); Microsoft,[2024a](https://arxiv.org/html/2608.07873#bib.bib27)\), serving as the primary computational tool for analysts, accountants, and domain experts who are not professional developers\(Hermans,[2016](https://arxiv.org/html/2608.07873#bib.bib28)\)\. Users routinely go beyond data entry, building formulas, charts, pivot tables, and conditional formatting rules–among other derived artifacts–that transform raw entries into structured analyses\.

The recent success of coding agents in boosting developer productivity\(Penget al\.,[2023](https://arxiv.org/html/2608.07873#bib.bib29)\)suggests that similar gains are within reach for spreadsheet users, if models can reliably create the full range of*derived artifacts*that real workbooks contain\. This has motivated enterprise agents for spreadsheet environments\(Microsoft,[2025](https://arxiv.org/html/2608.07873#bib.bib30); Anthropic,[2026](https://arxiv.org/html/2608.07873#bib.bib31); OpenAI,[2026](https://arxiv.org/html/2608.07873#bib.bib32)\)\. Yet progress has been hard to measure: existing benchmarks each cover a different slice of the problem–some use realistic workbooks but restrict tasks to formulas and data entry\(Maet al\.,[2024](https://arxiv.org/html/2608.07873#bib.bib2)\); others support diverse artifact types but operate on simple, hand\-crafted files\(Liet al\.,[2023](https://arxiv.org/html/2608.07873#bib.bib1)\); and still others target table reasoning rather than artifact creation\(Donget al\.,[2024](https://arxiv.org/html/2608.07873#bib.bib8)\)or focus on conversational data analysis\(Duttaet al\.,[2025](https://arxiv.org/html/2608.07873#bib.bib15)\)\(see Section[2](https://arxiv.org/html/2608.07873#S2)for a detailed comparison\)\. To the best of our knowledge, no existing benchmark jointly evaluates on \(i\) realistic, user\-authored workbooks with complex structure \(multiple sheets, non\-standard layouts, cross\-sheet dependencies\); \(ii\) multi\-step creation of derived artifacts–formulas, charts, pivot tables, and conditional formatting; and \(iii\) instructions at controllable levels of specificity\.

We address this gap by*reverse\-engineering*real user work\. Taking inspiration from reverse curriculum generation approaches in reinforcement learning\(Florensaet al\.,[2017](https://arxiv.org/html/2608.07873#bib.bib33); Andrychowiczet al\.,[2017](https://arxiv.org/html/2608.07873#bib.bib34)\), which construct training distributions by working backward from goal states, we start from finished, user\-authored spreadsheets and automatically reconstruct candidate edit histories–the orderings in which derived artifacts could plausibly have been created\. From these histories, we generate natural language instructions at multiple specificity levels for the same transformation, enabling systematic evaluation of how agents handle varying instructional detail\. We call this process the*workbook time machine*\(Figure[1](https://arxiv.org/html/2608.07873#S1.F1)\)\.

###### Example 1

Consider a spreadsheet with formulas computing BMI from height and weight columns, a conditional formatting rule highlighting high values, and a scatter chart plotting BMI against age\. The backward step strips these artifacts to recover the raw data\. The forward step generates instructions such as “Calculate BMI and plot it against age” \(abstract\) or “In cell D2, enter=10000\*C2/\(B2\*B2\), drag to D7, then create an XY scatter chart from A2:A7 vs D2:D7” \(fully specified\)\.

This process yields three dimensions of controlled variation: \(1\)*artifact type*–which derived object must be created; \(2\)*step complexity*–how many intermediate artifacts the transformation requires; and \(3\)*instruction specificity*–how much detail the query provides\. Applied to the Enron\(Hermans and Murphy\-Hill,[2015](https://arxiv.org/html/2608.07873#bib.bib4)\)and FUSE\(Bariket al\.,[2015](https://arxiv.org/html/2608.07873#bib.bib5)\)corpora, the pipeline producesWTM\-Corpus: 8,931 queries over 2,977 unique tasks covering the major categories of Excel derived artifacts\. From this we curateWTM\-Bench, a balanced 150\-task evaluation subset with near\-uniform artifact distribution across three specificity levels\.

We make the following contributions:

- •We introduce the*workbook time machine*, a pipeline that reverse\-engineers real user\-authored spreadsheets into benchmark triples \(input workbook, output workbook, query\) by modeling candidate edit histories through dependency\-aware DAG construction\. Applying it to public corpora yieldsWTM\-Corpus\.
- •FromWTM\-Corpus, we curateWTM\-Bench, a 150\-task evaluation benchmark with controlled variation across artifact types, complexity, & query specificity\.
- •We evaluate multiple agentic configurations across 6frontiermodels onWTM\-Bench, finding that \(a\) API choice fundamentally shapes performance–OfficeJS provides richer Excel feature coverage while OpenPyXL struggles with charts and pivot tables; \(b\) instruction specificity affects models asymmetrically–direct code generation excels with detailed instructions while agentic approaches better handle abstract queries; and \(c\) artifact difficulty varies sharply–formulas are most tractable while pivot tables remain nearly unsolved\.

## 2Related Works

Table 1:Comparison of benchmark properties∗The paper mentions, yes, it is not found in practice in the dataset\.#### Spreadsheet benchmarks\.

Existing spreadsheet benchmarks each cover a different slice of the problem space \(Table[1](https://arxiv.org/html/2608.07873#S2.T1)\)\. SheetCopilotBench\(Liet al\.,[2023](https://arxiv.org/html/2608.07873#bib.bib1)\)supports diverse artifact types \(charts, pivot tables, conditional formatting\) but operates on hand\-crafted workbooks that lack the structural complexity–multiple sheets, non\-standard layouts, cross\-sheet references–of real user files\. InstructExcel\(Payanet al\.,[2023](https://arxiv.org/html/2608.07873#bib.bib9)\)scales to thousands of instruction\-code pairs and uses real workbooks, yet each task targets a single isolated operation; multi\-step workflows and varying instruction granularity are not supported\. SpreadsheetBench\(Maet al\.,[2024](https://arxiv.org/html/2608.07873#bib.bib2)\)grounds evaluation in realistic, user\-authored workbooks sourced from forums, but its task scope is limited to data entry and formula manipulation–charts and pivot tables are absent in practice despite being mentioned\. SheetRM\(Chenet al\.,[2025](https://arxiv.org/html/2608.07873#bib.bib3)\)introduces a reward model for spreadsheet agents trained on synthetic workbooks, which limits transferability to the messy, multi\-table environments encountered in practice\. ConDABench\(Duttaet al\.,[2025](https://arxiv.org/html/2608.07873#bib.bib15)\)evaluates LLMs on*conversational*data analysis tasks requiring multi\-turn interaction and disambiguation of under\-specified goals, but it targets analytical insights over tabular data rather than the creation of spreadsheet artifacts \(formulas, charts, pivot tables\) thatWTM\-Benchfocuses on and avoids low\-value operation like manual cell\-manipulations\. In contrast,WTM\-Benchcombines real workbooks with multi\-artifact creation tasks and controllable instruction specificity\.

#### Spreadsheet understanding and code generation\.

SpreadsheetLLM\(Donget al\.,[2024](https://arxiv.org/html/2608.07873#bib.bib8)\)develops encoding schemes that preserve spatial relationships for spreadsheet question answering, while TableTalk\(Lianget al\.,[2025](https://arxiv.org/html/2608.07873#bib.bib13)\)enables natural language interaction with structured tables–both are read\-only and do not modify workbook content\. SheetMind\(Zhuet al\.,[2025](https://arxiv.org/html/2608.07873#bib.bib12)\)reasons over cell dependencies and formula relationships, an ability we leverage in our dependency\-aware pruning \(Section[4\.2](https://arxiv.org/html/2608.07873#S4.SS2)\)\. On the code generation side, approaches for programmatic spreadsheet control\(Payanet al\.,[2023](https://arxiv.org/html/2608.07873#bib.bib9); Zhuet al\.,[2025](https://arxiv.org/html/2608.07873#bib.bib12)\)and multi\-step task planning\(Liet al\.,[2023](https://arxiv.org/html/2608.07873#bib.bib1); Chenet al\.,[2025](https://arxiv.org/html/2608.07873#bib.bib3)\)address orchestration of complex workflows–but all assume clean starting states rather than the artifact\-rich environments of real workbooks\. More broadly, code\-generating LLM agents\(Yanget al\.,[2024](https://arxiv.org/html/2608.07873#bib.bib21)\)have shown promise for automating multi\-step tasks, yet spreadsheet\-specific challenges \(non\-standard layouts, cross\-sheet dependencies, API heterogeneity\) remain underexplored\.

#### Backward generation and self\-improvement\.

Our reverse\-engineering view is also related to methods that construct learning problems by working backward from known goal states\. Reverse curriculum learning and hindsight experience replay\(Florensaet al\.,[2017](https://arxiv.org/html/2608.07873#bib.bib33); Andrychowiczet al\.,[2017](https://arxiv.org/html/2608.07873#bib.bib34)\)use reachable goals to densify supervision, while agentic self\-debugging and action\-observation loops\(Chenet al\.,[2023](https://arxiv.org/html/2608.07873#bib.bib19); Yaoet al\.,[2023](https://arxiv.org/html/2608.07873#bib.bib20)\)use intermediate feedback to refine multi\-step solutions\. The Edit DAG differs in purpose: it is not a training\-time search policy, but a data\-construction mechanism that enumerates reachable spreadsheet transformations from real final workbooks\.

## 3Problem Formulation

Let𝒲=\{W1,W2,…,Wm\}\\mathcal\{W\}=\\\{W\_\{1\},W\_\{2\},\\ldots,W\_\{m\}\\\}be a corpus of Excel workbooks\. Each workbookW∈𝒲W\\in\\mathcal\{W\}consists of raw dataDDand a set of*derived artifacts*C=\{c1,…,cn\}C=\\\{c\_\{1\},\\ldots,c\_\{n\}\\\}–formulas, charts, pivot tables, conditional formatting rules–built on top ofDD\. We writeW=\{D\}∪CW=\\\{D\\\}\\cup C\.

Given𝒲\\mathcal\{W\}, our goal is to produce a benchmark

ℬ=\{\(Wkin,Wkout,qk\)\}k=1K,\\mathcal\{B\}=\\bigl\\\{\(W^\{\\text\{in\}\}\_\{k\},\\;W^\{\\text\{out\}\}\_\{k\},\\;q\_\{k\}\)\\bigr\\\}\_\{k=1\}^\{K\},whereWkin⊂Wkout⊆WW^\{\\text\{in\}\}\_\{k\}\\subset W^\{\\text\{out\}\}\_\{k\}\\subseteq Ware intermediate workbook states, that differ by≥\\geq1 artifacts, andqkq\_\{k\}is a natural language instructions describing the transformation fromWkinW^\{\\text\{in\}\}\_\{k\}toWkoutW^\{\\text\{out\}\}\_\{k\}\. At evaluation time, a model receives\(Wkin,qk\)\(W^\{\\text\{in\}\}\_\{k\},q\_\{k\}\)and must produce a state matchingWkoutW^\{\\text\{out\}\}\_\{k\}\.

Given a final workbook, our method operates in two passes\. In the*backward step*, we decompose the workbook by stripping its derived artifacts and estimating the various*candidate edit histories*–timelines of edits that could have produced the final workbook\. In the*forward step*, we sample transformations from these candidate histories and generate natural language instructions at varying levels of specificity\.

![Refer to caption](https://arxiv.org/html/2608.07873v1/x2.png)Figure 2:Generation pipeline flowchart### 4\.1Component extraction\.

Recall from Section[3](https://arxiv.org/html/2608.07873#S3)that a workbookW=\{D\}∪CW=\\\{D\\\}\\cup Ccomprises raw dataDDand derived artifactsC=\{c1,…,cn\}C=\\\{c\_\{1\},\\ldots,c\_\{n\}\\\}\. Starting from the final workbookWW, the backward step strips away derived artifacts to recover the raw stateW0=\{D\}W^\{0\}=\\\{D\\\}, and then reconstructs the candidate edit histories–all semantically valid orderings in which the artifacts could have been added back\. We constructW0W^\{0\}by stripping all derived artifacts fromWW\. Beyond straightforward removal, we apply two heuristics to ensure clean extraction\. First, we perform*formula grouping*: spreadsheet features such as FlashFill\(Gulwani,[2011](https://arxiv.org/html/2608.07873#bib.bib16)\)and formula drag allow users to replicate a formula across contiguous ranges, so we anonymize cell references and group adjacent formulas that share the same template into a single*formula group*, substantially reducing the number of artifacts\. Second, we perform*semantic descriptor mapping*: label cells that describe an adjacent formula–for example, a “Total” cell next to=SUM\(\)–leak information about the target transformation and must be associated with the source state and removed if the component is removed\. We use an LLM to identify semantic table ranges and mark row and column headers that serve as descriptors for formula groups\.

### 4\.2Backward Step: Decomposition

#### Edit DAG construction\.

We represent these candidate histories compactly as a Directed Acyclic Graph \(DAG\)\. Let the true edit history of the workbook be𝒯=\(W0,W1,W2,…,W\)\\mathcal\{T\}=\(W^\{0\},W^\{1\},W^\{2\},\\ldots,W\)\. Since𝒯\\mathcal\{T\}is not available, we model all possible candidate edit histories via the*Edit DAG*𝒢=\(V,E\)\\mathcal\{G\}=\(V,E\), whereVVcomprises all workbook states obtainable by adding subsets of\{c1,…,cn\}\\\{c\_\{1\},\\ldots,c\_\{n\}\\\}toW0W^\{0\}, and a directed edge\(Wi,Wj\)∈E\(W^\{i\},W^\{j\}\)\\in Eexists iffWj=Wi∪\{ck\}W^\{j\}=W^\{i\}\\cup\\\{c\_\{k\}\\\}for some artifactckc\_\{k\}\. Each path fromW0W^\{0\}toWWin𝒢\\mathcal\{G\}corresponds to one candidate edit history\.

#### Dependency\-aware pruning\.

Naïvely, a workbook withnnartifacts admits up ton\!n\!orderings\. However, not all orderings are semantically valid–a chart built on a derived column cannot precede the creation of that column\. Building on prior work on cell dependency analysis\(Tanget al\.,[2023](https://arxiv.org/html/2608.07873#bib.bib14); Zhuet al\.,[2025](https://arxiv.org/html/2608.07873#bib.bib12)\), we capture such constraints through a*artifact dependency graph*𝒢D=\(\{c1,…,cn\},ED\)\\mathcal\{G\}\_\{D\}=\(\\\{c\_\{1\},\\ldots,c\_\{n\}\\\},E\_\{D\}\), where\(ci,cj\)∈ED\(c\_\{i\},c\_\{j\}\)\\in E\_\{D\}when the input range ofcjc\_\{j\}overlaps the output range ofcic\_\{i\}\. We prune every edge\(Wi,Wi∪\{cj\}\)∈E\(W^\{i\},W^\{i\}\\cup\\\{c\_\{j\}\\\}\)\\in Efor which at least one prerequisite ofcjc\_\{j\}is absent fromWiW^\{i\}\. If the pruned graph is no longer fully connected, we retain the connected artifact containingWW, preferentially selecting artifact\-dense states\. We call the result the*Dependency\-Pruned DAG*𝒢P\\mathcal\{G\}\_\{P\}\.

#### Enumerating candidate histories\.

We enumerate all sub\-paths in𝒢P\\mathcal\{G\}\_\{P\}as candidate edit histories\. Crucially, these sub\-paths need not originate atW0W^\{0\}: a candidate history may begin at any intermediate state that already contains a subset of derived artifacts\. This allows the benchmark to include the realistic scenario in which a user continues building on a partially constructed workbook rather than starting from raw data\.

Table 2:Different levels of instructions with increasing levels of abstractness\.

### 4\.3Forward Step: Query Generation

Given the candidate edit histories enumerated from𝒢P\\mathcal\{G\}\_\{P\}, the forward step selects endpoint pairs as input\-output tuples\(Win,Wout\)\(W^\{\\text\{in\}\},W^\{\\text\{out\}\}\)and generates natural language instructions for each transformation\.

#### Action extraction and necessity scoring\.

For each transformation, we extract a structured*action*representation consisting of key\-value pairs that describe the relevant parameters \(e\.g\., chart type, data range, axis labels, formula expression\)\. An LLM assigns a*necessity score*s∈\{0,0\.5,1\}s\\in\\\{0,0\.5,1\\\}to each parameter, wheres=1s\{=\}1denotes parameters essential for identifying the task ands=0s\{=\}0denotes incidental details\. This yields action representations at three levels of specificity: fully specified \(all parameters\), moderately specified \(s\>0s\>0\), and minimal \(s=1s\{=\}1\)\.

#### Natural language instruction generation\.

Each filtered action variant is provided as conditioning context to an LLM, which generates a corresponding natural language instruction\. This produces instructions ranging from fully specified to abstract, as illustrated in Table[2](https://arxiv.org/html/2608.07873#S4.T2)\.

## 5Benchmark Construction

The workbook time machine generates benchmark instances from real\-world Excel workbooks, producing 8,931 natural\-language queries over 2,977 unique tasks from the Enron\(Hermans and Murphy\-Hill,[2015](https://arxiv.org/html/2608.07873#bib.bib4)\)and FUSE\(Bariket al\.,[2015](https://arxiv.org/html/2608.07873#bib.bib5)\)corpora \(WTM\-Corpus\)\. However, this distribution suffers from severe class imbalance \(67\.5% formulas vs\. 0\.8% pivot tables\), making LLM evaluation both computationally prohibitive and methodologically problematic\. We therefore curateWTM\-Bench, a 150\-task evaluation subset \(450 queries across three specificity levels\) via principled lexicographic sampling that achieves near\-uniform task\-type distribution while preserving semantic properties\.

![Refer to caption](https://arxiv.org/html/2608.07873v1/x3.png)Figure 3:WTM\-Benchachieves better artifact distribution while preserving other attributes\.Bluebars forWTM\-Corpusandredbars forWTM\-Benchdataset\.#### Addressing distribution imbalance\.

Our primary design challenge was the extreme skew in the original distribution where formulas dominate \(67\.5%\) while pivot tables represent only 0\.8% of tasks\. This imbalance would render aggregate performance metrics meaningless, as they would primarily reflect formula\-writing capability rather than comprehensive Excel automation skills\. Our lexicographic sampling strategy achieves dramatic rebalancing: 41\.3% formulas, 23\.3% charts, 20\.0% conditional formatting, and 15\.3% pivot tables\.

#### Multi\-level query design rationale\.

We hypothesized that sequential generation \(Level 1→2→3\) sequentially in 1 LLM call, would provide superior information retention compared to direct Level 3 generation\. This design choice was validated through systematic comparison on 75 randomly sampled tasks, confirming that the multi\-level approach achieves better information preservation with reduced data leakage \(detailed analysis in Appendix[C](https://arxiv.org/html/2608.07873#A3)\)\.

#### Computational efficiency considerations\.

Pipeline construction scales predictably with workbook complexity:LLMtotal=LLMextract\+LLMannotate\\text\{LLM\}\_\{\\text\{total\}\}=\\text\{LLM\}\_\{\\text\{extract\}\}\+\\text\{LLM\}\_\{\\text\{annotate\}\}, where annotation cost grows as2×∑\# permutations per trajectory2\\times\\sum\\text\{\\\# permutations per trajectory\}\. This ensures richer workbooks amortize annotation costs across proportionally more training examples\. More details are in Appendix[D](https://arxiv.org/html/2608.07873#A4)\.

#### Quality validation methodology\.

We conducted systematic quality evaluation using 3 human annotators across 3 dimensions with inter\-annotator reliability analysis to characterize query naturalness and completeness \(details in Appendix[B](https://arxiv.org/html/2608.07873#A2)\)\. Our sampling strategy maintains semantic equivalence across the complete dataset, with intent distribution remaining stable across query levels, validating that the 3 query variants target the same underlying transformations despite varying specificity\.

## 6Benchmark Analysis:WTM\-Bench

#### Task distribution insights\.

Figure[4](https://arxiv.org/html/2608.07873#S6.F4)shows how our rebalanced benchmark provides diagnostic capability across Excel’s functional spectrum rather than primarily testing formula proficiency\. The complexity distribution reveals authentic real\-world patterns: while most tasks \(51\.3%\) involve simple 1\-2 artifact transformations, a substantial tail extends to 14\-artifact workflows, enabling analysis of where reasoning capabilities break down as complexity increases\. We define this expected number of artifact transformation as*Task Complexity*of any data pair\.

Diversity with respect to workbook selection is also captured in the industry domain\. Where top\-5 major sectors serve as a natural regularizer against domain\-specific overfitting\. This taxonomy analysis uses sector categorization from NAICS\(NAICS,[2022](https://arxiv.org/html/2608.07873#bib.bib35)\)and uses an LLM for classification\.

#### Multi\-level design as capability probe\.

Figure[5](https://arxiv.org/html/2608.07873#S6.F5)plots the 3\-level specificity gradient, creating a controlled experimental setting that isolates instruction\-following from task execution capabilities\. The systematic degradation of specificity from detailed \(Level 1\) to concise \(Level 3\) instructions reveals whether models fail due to task complexity or insufficient context\. This diagnostic dimension also gets missed in single\-level benchmarks\. Thus enabling precise identification of model limitations: strong Level 1 performance with poor Level 3 results indicates gap\-filling rather than reasoning deficits\.

The annotation study suggests a trade\-off between brevity and completeness: shorter queries tend to receive higher human\-likeness scores and lower completeness scores, although these dimensions exhibit only moderate agreement and should be interpreted as indicative rather than definitive\. This tension helps explain why Level 3 queries pose particular challenges, as they reflect realistic human communication patterns that rely on workbook context and spreadsheet conventions\.

![Refer to caption](https://arxiv.org/html/2608.07873v1/x4.png)Figure 4:Task distribution analysis overWTM\-Bench\.![Refer to caption](https://arxiv.org/html/2608.07873v1/x5.png)\(a\)Specificity gradient across levels
![Refer to caption](https://arxiv.org/html/2608.07873v1/x6.png)\(b\)Query length distributions

Figure 5:Query characteristic analysis\.

## 7Experiment Setup

#### Models\.

We evaluate 6 state\-of\-the\-art language models spanning two major families: Anthropic \(Claude Haiku 4\.5, Claude Sonnet 4\.5, Claude Opus 4\.6\) and OpenAI \(GPT\-4\.1 Mini, GPT\-5\.2 Reasoning, GPT\-5\.4 Reasoning\)\.

#### Orchestration\.

We evaluate popular Python\-based Excel LLM orchestration frameworks–Sheet Agent, SheetCopilot, and SpreadsheetBench–with GPT 5\.4 Reasoning\. However, motivated by advances in tool\-execution agents\(Jimenezet al\.,[2024](https://arxiv.org/html/2608.07873#bib.bib38); Merrillet al\.,[2026](https://arxiv.org/html/2608.07873#bib.bib39)\), we propose a custom orchestration framework adopting modern tool\-calling best practices in which the model iteratively emits structured code calls, observes execution results, and self\-debugs across up to 10 turns\. We evaluate across three Excel interface APIs: Python \(openpyxl\), VBA, and Office\.js\. Full implementation details are provided in Appendix[F](https://arxiv.org/html/2608.07873#A6)\.

#### Sampling parameters\.

Generation uses GPT\-5\.4 Reasoning to author the natural\-language query variants and necessity scores\. During evaluation, GPT reasoning models use their default sampling configuration and Claude models use temperature 1\.0, following Anthropic’s benchmark setting\. The turn cap is 10 for all multi\-turn configurations\.

#### Evaluation

We evaluate performance onWTM\-Benchusing two complementary metrics\. TheSoft Scoreis a continuous reward in\[0,1\]\[0,1\]measuring partial task completion, computed as a weighted combination of component\-level similarity scores across cells, formulas, charts, pivot tables, and conditional formatting \(see Appendix[H](https://arxiv.org/html/2608.07873#A8)for details\)\. TheHard Scoreis the fraction of tasks where the agent achieves a perfect Soft Score of 1\.0, capturing complete and exact task reconstruction\. All grading is deterministic and programmatic: no LLM is used to judge model outputs, and the query generator is never consulted during evaluation\. We present both metrics as % scores \(scaled 0–100\)\. Additionally,\# Turnscounts the code execution turns, with execution feedback, used by the model to complete a task\.

## 8Results and Analysis

Table 3:Results for popular frontier models and orchestration frameworks on level 3 queries\. The best\-performing scores are marked inbold, and the second best inunderline\.We evaluate 6 frontier models of various sizes onWTM\-Benchusing our tool\-calling orchestration across three API tooling, and compare against three established agent frameworks\. Our analysis reveals three key findings:\(1\)agent design matters more than model scale—our orchestration with structured state observation outperforms existing frameworks by 7–17 pp in Hard Score using the same backbone model;\(2\)all models fail catastrophically on pivot tables \(≤\\leq10% Soft Score\), exposing a fundamental reasoning gap; and\(3\)performance degrades monotonically with query abstraction and task complexity, validatingWTM\-Benchas a multi\-dimensional diagnostic benchmark\.

#### Agent Design and Orchestration

Table[3](https://arxiv.org/html/2608.07873#S8.T3)reports results on Level 3 queries\. Using GPT\-5\.4 Reasoning as a common backbone, our simple orchestration achieves 20\.0% Hard Score and 46\.3% Soft Score \(Python API\), outperforming existing specialized agents such as SheetCopilot \(12\.7% Hard, 34\.9% Soft\), SheetAgent \(12\.0% Hard, 32\.6% Soft\), and SpreadsheetBench \(3\.3% Hard, 8\.5% Soft\)\. These gaps highlight the importance of agent architecture over raw model capability\. Notably, SpreadsheetBench’s single\-turn, no\-feedback approach yields the lowest performance\. This highlights that models require structured state observation: our orchestration surfaces the post\-execution workbook state at each turn, enabling the model to ground its next action in the actual spreadsheet rather than relying on just prior code generation\.

#### Model and API Analysis

The best\-performing configuration overall is Claude Opus 4\.6 with VBA \(33\.7% Hard, 54\.0% Soft\)\. Across APIs, we observe that VBA and OfficeJS consistently outperform Python for the same model, which we attribute to richer native Excel primitives\. For instance, creating a pivot table in VBA requires a singlePivotTablecall, whereas Python requires orchestrating Pandas for the pivot operation and OpenPyXL for writing the result—a multi\-step workflow that compounds errors\.

This effect is model\-dependent: Claude models benefit substantially from richer APIs \(Opus: \+15\.0 pp Hard Score from Python to VBA; Sonnet: \+16\.0 pp\), whereas GPT\-5\.4 Reasoning shows only a modest gain \(\+2\.8 pp\)\. This suggests that Claude models are better at leveraging unfamiliar API surfaces, possibly due to stronger instruction\-following capabilities\.

Turn efficiency reveals an additional pattern: smaller models \(GPT\-4\.1 Mini, Claude Haiku\) use fewer turns \(2\.2–3\.2\) but score lower, while flagship models use 3–5 turns more productively\. Independent Python/OpenPyXL re\-runs preserve model ordering, with Hard Score standard deviations at or below 3\.5 percentage points \(Appendix[G](https://arxiv.org/html/2608.07873#A7)\), suggesting that the headline differences are not single\-run artifacts\.

![Refer to caption](https://arxiv.org/html/2608.07873v1/x7.png)\(a\)Soft Score for different query levels
![Refer to caption](https://arxiv.org/html/2608.07873v1/x8.png)\(b\)Soft Scores for different task complexities

Figure 6:Soft scores decrease with complexity; Haiku 4\.5 outperforming GPT\-4\.1 Mini\.
#### Effect of Query Specificity

Figure[6\(a\)](https://arxiv.org/html/2608.07873#S8.F6.sf1)shows Soft Scores across the three query levels\. Performance degrades monotonically from Level 1 \(verbose, detailed\) to Level 3 \(concise, abstract\)\. This aligns with our dataset analysis: the proportion of “Very Well\-Specified” queries drops 4\.4×\\timesfrom Level 1 to Level 3, and mean query length shrinks from 357 to 139 characters\. The multi\-level design also provides a natural curriculum for future fine\-tuning—models can be trained on Level 1 queries first, then progressively exposed to more abstract formulations, we leave this exploration to future works\.

#### Impact of Task Complexity

Figure[6\(b\)](https://arxiv.org/html/2608.07873#S8.F6.sf2)plots Soft Score against task complexity \(the number of artifact transformations required\)\. Performance declines with increasing complexity: single\-artifact tasks are handled reasonably by most models, but tasks requiring 5\+ transformations see scores collapse\. This gap reveals that current frontier models struggle with long\-horizon, multi\-step spreadsheet workflows—a class of tasks that, while less frequent in theWTM\-Corpusdistribution \(most tasks involve 1–2 transformations\), represents precisely the high\-value scenarios where automation would be most impactful\.

#### Artifact\-Type Performance Breakdown

Table[4](https://arxiv.org/html/2608.07873#S8.T4)presents the Soft Score breakdown by primary artifact type across models using VBA\. Formulas and charts are handled competitively by most models, while conditional formatting and pivot tables remain challenging\. Pivot tables in particular yield near\-zero scores across all models, reflecting the multi\-step coordination required \(source data identification, field selection, aggregation specification, and output placement\)\.

Table 4:Soft Score \(%\) results broken down by task artifact type\.
#### Failure modes\.

A rollout\-level analysis over 4,910 deduplicated rollouts \(Appendix[G](https://arxiv.org/html/2608.07873#A7)\) shows that failures \(Soft Score – failure < 0\.5; strict failure < 0\.10\) are overwhelmingly structural rather than numerical: among 1,858 failed Level 3 Python rollouts, 88\.5% either omit the artifact entirely \(62\.3%, wrong shape\) or place it at the wrong coordinate \(26\.2%, wrong placement\), while only 11\.6% produce a correctly shaped artifact with wrong values\. Tool\-call failure is not the bottleneck, with frontier models siting at 3–5%, and its correlation with score is weak \(ρ=−0\.11\\rho=\-0\.11\)\. Code volume is the strongest negative predictor \(ρ=−0\.21\\rho=\-0\.21\), suggesting that long generations often mark unsuccessful repair attempts\. Most consequentially for evaluation design, models frequently claim completion despite strict failure: hallucinated success reaches 80\.7% for Claude rollouts and 43\.2% for GPT\.

## 9Conclusion

We present the*workbook time machine*, a pipeline for generating spreadsheet automation benchmarks from real\-world Excel workbooks\. Applied to public corpora, it producesWTM\-Corpus\(8,931 queries over 2,977 tasks\), from which we curateWTM\-Bench, a balanced 150\-task evaluation benchmark with near\-uniform distribution across major Excel artifact types\. Evaluation across 18 model\-API combinations and 4 sheet\-based orchestration reveals significant performance variations, demonstrating how API choice fundamentally impacts automation success, while our multi\-level query design shows that frontier models achieve higher completion rates but face substantial challenges with ambiguous instructions and complex multi\-step workflows, establishing robust evaluation standards that provide actionable insights for spreadsheet automation system improvement\.

#### Future Work

The multi\-level query design enables curriculum learning approaches, training models progressively from detailed Level 1 instructions to concise Level 3 queries to improve robustness to ambiguous instructions\. The pipeline can be extended beyond creation tasks to support comprehensive CRUD operations, including deletion \(by reversing input/output\) and modification operations\. Advanced workbook preprocessing techniques could further minimize structural information leakage through intelligent content repositioning and formula descriptor detection\. Finally, the language\-agnostic pipeline design allows straightforward adaptation to multilingual contexts, broadening applicability for international enterprise environments\.

## References

- M\. Andrychowicz, F\. Wolski, A\. Ray, J\. Schneider, R\. Fong, P\. Welinder, B\. McGrew, J\. Tobin, P\. Abbeel, and W\. Zaremba \(2017\)Hindsight experience replay\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p3.1),[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px3.p1.1)\.
- Anthropic \(2026\)Claude for excel\.External Links:[Link](https://claude.com/claude-for-excel)Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p2.1)\.
- T\. Barik, K\. Lubick, J\. Smith, J\. Slankas, and E\. Murphy\-Hill \(2015\)Fuse: a reproducible, extendable, internet\-scale corpus of spreadsheets\.In2015 IEEE/ACM 12th Working Conference on Mining Software Repositories,pp\. 486–489\.Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p4.1),[§5](https://arxiv.org/html/2608.07873#S5.p1.1)\.
- M\. Bendre, T\. Wattanawaroon, S\. Rahman, K\. Mack, Y\. Liu, S\. Zhu, Y\. Lu, P\. Yang, X\. Zhou, K\. C\. Chang,et al\.\(2019\)Faster, higher, stronger: redesigning spreadsheets for scale\.In2019 IEEE 35th International Conference on Data Engineering \(ICDE\),pp\. 1972–1975\.Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p1.1)\.
- X\. Chen, M\. Lin, N\. Schärli, and D\. Zhou \(2023\)Teaching large language models to self\-debug\.arXiv preprint arXiv:2304\.05128\.Cited by:[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Chen, Y\. Yuan, Z\. Zhang, Y\. Zheng, J\. Liu, F\. Ni, J\. Hao, H\. Mao, and F\. Zhang \(2025\)SheetAgent: towards a generalist agent for spreadsheet reasoning and manipulation via large language models\.InProceedings of the ACM on Web Conference 2025,pp\. 158–177\.Cited by:[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Dong, J\. Zhao, Y\. Tian, J\. Xiong, S\. Xia, M\. Zhou, Y\. Lin, J\. Cambronero, Y\. He, S\. Han,et al\.\(2024\)Spreadsheetllm: encoding spreadsheets for large language models\.arXiv preprint arXiv:2407\.09025\.Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p2.1),[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Dutta, P\. Gupta, H\. Hasanbeig, R\. P\. Singh, H\. Nigam, S\. Gulwani, A\. Radhakrishna, G\. Soares, and A\. Tiwari \(2025\)ConDABench: interactive evaluation of language models for data analysis\.External Links:2510\.13835Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p2.1),[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Florensa, D\. Held, M\. Wulfmeier, M\. Zhang, and P\. Abbeel \(2017\)Reverse curriculum generation for reinforcement learning\.InProceedings of the 1st Conference on Robot Learning \(CoRL\),Vol\.78,pp\. 482–495\.Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p3.1),[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px3.p1.1)\.
- E\. Gazoni and C\. Clark \(2024\)Openpyxl: a python library to read/write excel 2010 xlsx/xlsm files\.External Links:[Link](https://openpyxl.readthedocs.io/)Cited by:[§F\.2](https://arxiv.org/html/2608.07873#A6.SS2.SSS0.Px1.p1.1)\.
- S\. Gulwani \(2011\)Automating string processing in spreadsheets using input\-output examples\.InProceedings of the 38th Annual ACM SIGPLAN\-SIGACT Symposium on Principles of Programming Languages \(POPL\),pp\. 317–330\.External Links:[Document](https://dx.doi.org/10.1145/1926385.1926423)Cited by:[§4\.1](https://arxiv.org/html/2608.07873#S4.SS1.p1.7)\.
- F\. Hermans and E\. Murphy\-Hill \(2015\)Enron’s spreadsheets and related emails: a dataset and analysis\.In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering,Vol\.2,pp\. 7–16\.Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p4.1),[§5](https://arxiv.org/html/2608.07873#S5.p1.1)\.
- F\. Hermans \(2016\)Spreadsheets are code\.In2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering \(SANER\),External Links:[Document](https://dx.doi.org/10.1109/SANER.2016.40)Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§7](https://arxiv.org/html/2608.07873#S7.SS0.SSS0.Px2.p1.1)\.
- H\. Li, J\. Su, Y\. Chen, Q\. Li, and Z\. ZHANG \(2023\)SheetCopilot: bringing software productivity to the next level through large language models\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 4952–4984\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/0ff30c4bf31db0119a6219e0d250e037-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p2.1),[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px2.p1.1)\.
- J\. T\. Liang, A\. Kumar, Y\. Bajpai, S\. Gulwani, V\. Le, C\. Parnin, A\. Radhakrishna, A\. Tiwari, E\. Murphy\-Hill, and G\. Soares \(2025\)TableTalk: scaffolding spreadsheet development with a language agent\.ACM Trans\. Comput\.\-Hum\. Interact\.32\(6\)\.External Links:[Document](https://dx.doi.org/10.1145/3765286)Cited by:[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Ma, B\. Zhang, J\. Zhang, J\. Yu, X\. Zhang, X\. Zhang, S\. Luo, X\. Wang, and J\. Tang \(2024\)Spreadsheetbench: towards challenging real world spreadsheet manipulation\.Advances in Neural Information Processing Systems37,pp\. 94871–94908\.Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p2.1),[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Maruta, N\. Tagai, and M\. P\. Kato \(2025\)Investigating information needs during spreadsheet data analysis\.Journal of Information Processing33,pp\. 507–521\.External Links:[Document](https://dx.doi.org/10.2197/ipsjjip.33.507)Cited by:[§E\.1](https://arxiv.org/html/2608.07873#A5.SS1.SSS0.Px2.p1.1)\.
- M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan, J\. Shen, G\. Ye, H\. Lin, J\. Poulos, M\. Wang, M\. Nezhurina, J\. Jitsev, D\. Lu, O\. M\. Mastromichalakis, Z\. Xu, Z\. Chen, Y\. Liu, R\. Zhang, L\. L\. Chen, A\. Kashyap, J\. Uslu, J\. Li, J\. Wu, M\. Yan, S\. Bian, V\. Sharma, K\. Sun, S\. Dillmann, A\. Anand, A\. Lanpouthakoun, B\. Koopah, C\. Hu, E\. Guha, G\. H\. S\. Dreiman, J\. Zhu, K\. Krauth, L\. Zhong, N\. Muennighoff, R\. Amanfu, S\. Tan, S\. Pimpalgaonkar, T\. Aggarwal, X\. Lin, X\. Lan, X\. Zhao, Y\. Liang, Y\. Wang, Z\. Wang, C\. Zhou, D\. Heineman, H\. Liu, H\. Trivedi, J\. Yang, J\. Lin, M\. Shetty, M\. Yang, N\. Omi, N\. Raoof, S\. Li, T\. Y\. Zhuo, W\. Lin, Y\. Dai, Y\. Wang, W\. Chai, S\. Zhou, D\. Wahdany, Z\. She, J\. Hu, Z\. Dong, Y\. Zhu, S\. Cui, A\. Saiyed, A\. Kolbeinsson, J\. Hu, C\. M\. Rytting, R\. Marten, Y\. Wang, A\. Dimakis, A\. Konwinski, and L\. Schmidt \(2026\)Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.External Links:2601\.11868,[Link](https://arxiv.org/abs/2601.11868)Cited by:[§7](https://arxiv.org/html/2608.07873#S7.SS0.SSS0.Px2.p1.1)\.
- Microsoft \(2024a\)Earnings release FY24 Q4\.Note:Microsoft 365 Consumer subscribers grew to 82\.5 million\. July 30, 2024\.External Links:[Link](https://www.microsoft.com/en-us/investor/earnings/fy-2024-q4/press-release-webcast)Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p1.1)\.
- Microsoft \(2024b\)Office add\-ins documentation: office javascript api\.External Links:[Link](https://learn.microsoft.com/en-us/office/dev/add-ins/reference/javascript-api-for-office)Cited by:[§F\.2](https://arxiv.org/html/2608.07873#A6.SS2.SSS0.Px3)\.
- Microsoft \(2025\)Agent mode in excel is now generally available on desktop\.External Links:[Link](https://techcommunity.microsoft.com/blog/excelblog/agent-mode-in-excel-is-now-generally-available-on-desktop/4457408)Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p2.1)\.
- NAICS \(2022\)North american industry classification system, 2022\.Note:Federal Register, Vol\. 86, No\. 245External Links:[Link](https://www.census.gov/naics/)Cited by:[§E\.2](https://arxiv.org/html/2608.07873#A5.SS2.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.07873#S6.SS0.SSS0.Px1.p2.1)\.
- OpenAI \(2023\)GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§F\.1](https://arxiv.org/html/2608.07873#A6.SS1.SSS0.Px2.p1.1)\.
- OpenAI \(2026\)ChatGPT for spreadsheets\.External Links:[Link](https://chatgpt.com/apps/spreadsheets)Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p2.1)\.
- J\. Payan, S\. Mishra, M\. Singh, C\. Negreanu, C\. Poelitz, C\. Baral, S\. Roy, R\. Chakravarthy, B\. Van Durme, and E\. Nouri \(2023\)Instructexcel: a benchmark for natural language instruction in excel\.arXiv preprint arXiv:2310\.14495\.Cited by:[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Peng, E\. Kalliamvakou, P\. Cihon, and M\. Demirer \(2023\)The impact of ai on developer productivity: evidence from github copilot\.arXiv preprint arXiv:2302\.06590\.Cited by:[§1](https://arxiv.org/html/2608.07873#S1.p2.1)\.
- C\. Shah, R\. White, R\. Andersen, G\. Buscher, S\. Counts, S\. S\. S\. Das, A\. Montazer, S\. Manivannan, J\. Neville, N\. Rangan, T\. Safavi, S\. Suri, M\. Wan, L\. Wang, and L\. Yang \(2025\)Using large language models to generate, validate, and apply user intent taxonomies\.ACM Transactions on the Web19\(3\),pp\. 34:1–34:29\.External Links:[Document](https://dx.doi.org/10.1145/3732294)Cited by:[§E\.1](https://arxiv.org/html/2608.07873#A5.SS1.SSS0.Px2.p1.1)\.
- D\. Tang, F\. Chen, C\. De Leon, T\. Wattanawaroon, J\. Yun, S\. Seshadri, and A\. G\. Parameswaran \(2023\)Efficient and compact spreadsheet formula graphs\.Proceedings of the VLDB Endowment\.External Links:2302\.05482Cited by:[§4\.2](https://arxiv.org/html/2608.07873#S4.SS2.SSS0.Px2.p1.11)\.
- K\. Yang, J\. Liu, J\. Wu, C\. Yang, Y\. Fung, S\. Li, Z\. Huang, X\. N\. Cao, X\. Wang, Y\. Wang,et al\.\(2024\)If llm is the wizard, then code is the wand: a survey on how code empowers large language models to serve as intelligent agents\.arXiv preprint arXiv:2401\.00812\.Cited by:[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px3.p1.1)\.
- R\. Zhu, X\. Cheng, K\. Liu, B\. Zhu, D\. Jin, N\. Parihar, Z\. Xu, and O\. Gao \(2025\)SheetMind: an end\-to\-end llm\-powered multi\-agent framework for spreadsheet automation\.External Links:2506\.12339Cited by:[§2](https://arxiv.org/html/2608.07873#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2608.07873#S4.SS2.SSS0.Px2.p1.11)\.

## Appendix ALimitations and Scope

The current version ofWTM\-Benchfocuses on derived analytical artifacts: formulas, charts, conditional formatting, and pivot tables\. This scope is taxonomic rather than purely frequency\-based\. These artifacts transform source data into new analytical output, whereas data validation, filtering, sorting, tables, freezing panes, and related operations primarily control views, input constraints, or worksheet interaction state\. Table[5](https://arxiv.org/html/2608.07873#A1.T5)reports the prevalence audit that informed this decision\. The pipeline can be extended to these interaction\-oriented operations, but they require different extraction and grading logic\.

Table 5:Prevalence of candidate Excel features in the source corpora\. Sparklines were not observed in the audited FUSE workbooks\.WTM\-Benchcurrently handles derivable objects with explicit formulas, while other derived content \(such as manually entered lookup values\) represents a distinct class of spreadsheet operations that could be addressed through extended formula descriptor analysis\.

## Appendix BQuery Quality Evaluation

![Refer to caption](https://arxiv.org/html/2608.07873v1/x9.png)\(a\)Quality score distributions across query levels
![Refer to caption](https://arxiv.org/html/2608.07873v1/x10.png)\(b\)Inter\-annotator agreement

Figure 7:Query quality assessment: score distributions and annotation reliability analysis\.#### Multi\-annotator quality assessment\.

We conducted a systematic quality evaluation using 3 human annotators, all paper authors with ML research backgrounds and spreadsheet\-domain experience, across 75 sampled queries\. Annotators independently rated completeness, complexity, and human\-like quality on a 1–5 scale\. Figure[7\(a\)](https://arxiv.org/html/2608.07873#A2.F7.sf1)presents score distributions across levels: completeness decreases with query level \(3\.29 to 2\.71\), complexity remains stable \(2\.89 to 2\.70\), while human\-like scores increase \(2\.76 to 3\.31\)\. We interpret this as an indicative brevity–completeness trade\-off, not as a correctness criterion for the benchmark\.

#### Inter\-annotator reliability\.

To ensure annotation quality, we assessed inter\-annotator agreement across the 3 quality dimensions \(Figure[7\(b\)](https://arxiv.org/html/2608.07873#A2.F7.sf2)\)\. Complexity shows the highest agreement \(average correlation: 0\.888\), indicating it is the most objectively assessable\. Completeness shows moderate agreement \(0\.702\), while human\-like assessment has lower agreement \(0\.494\), particularly for Level 1 queries \(0\.398\), suggesting this dimension is more subjective\. Completeness was scored as whether the query explicitly communicates details a human reader would consider necessary to execute the task, not whether the query uniquely determines the output workbook\. Annotators may reasonably differ on whether axis titles, header names, or placements are essential or inferable from workbook context; therefore, this study characterizes query variants rather than validating task correctness\. Correctness is instead determined by the programmatic grader in Appendix[H](https://arxiv.org/html/2608.07873#A8)\.

![Refer to caption](https://arxiv.org/html/2608.07873v1/x11.png)Figure 8:Information retention comparison between sequential multi\-level generation \(left\) and direct Level 3 generation \(right\)\. Multi\-level generation shows superior performance on essential information with fewer data leaks across 75 test cases\.

## Appendix CQuery Generation Pipeline: Design Decision Analysis

#### Motivation for multi\-level generation\.

Given that Level 3 queries are the most under\-specified, complex, and human\-like in our taxonomy, our initial hypothesis was that generating all three levels \(1 to 2 to 3\) sequentially would provide better information retention and query quality than directly generating Level 3 queries\. The rationale was that starting with verbose, well\-specified Level 1 queries would preserve essential task details that could be progressively condensed while maintaining semantic fidelity\.

#### Empirical comparison methodology\.

To validate this design decision, we conducted a systematic comparison of two generation approaches using 75 randomly sampled workbook tasks: \(1\) sequential generation of all three levels starting from detailed Level 1 queries and \(2\) direct generation of Level 3 queries without intermediate steps\. We evaluated information retention using data\-pair weighted metrics, tracking missing essential information \(necessity=1\.0\) and leaked contextual details \(necessity=0\.5 and 0\.0\)\.

#### Findings\.

Figure[8](https://arxiv.org/html/2608.07873#A2.F8)presents the information retention analysis results\. The multi\-level approach performs better than direct Level 3 generation across the key retention metrics\. Direct generation has slightly more data pairs with missing critical information and a higher net retention score because it leaks more non\-essential information\. Ideally, each score should be close to 0; the multi\-level method is closer to that target and yields better query quality\.

We ran a small annotation over 75 samples to compare human\-like scores for both these query generations: multi\-level \(3\.44\) vs\. single \(3\.14\), which clearly shows how multi\-level generation is more aligned\.

![Refer to caption](https://arxiv.org/html/2608.07873v1/x12.png)\(a\)Quality score distributions across query levels
![Refer to caption](https://arxiv.org/html/2608.07873v1/x13.png)\(b\)Inter\-annotator agreement

Figure 9:Comparing query quality assessment over a multi\-level vs\. single\-generation pass of 75 queries\.

## Appendix DPipeline Construction Efficiency

#### Construction time analysis\.

The mean processing time per workbook is 401 s \(median 192 s\) across our sample of 40 workbooks\. Runtime is dominated by the Edit DAG construction stage \(mean 254 s, 63% of total wall time\), which enumerates and materializes all intermediate workbook states; Component extraction accounts for 25% \(mean 102 s\) and Query generation for 11% \(mean 44 s\)\. Figure[10](https://arxiv.org/html/2608.07873#A4.F10)demonstrates how total runtime scales with transformation complexity and data pair generation, confirming that richer workbooks amortize annotation cost across more training examples\.

#### LLM usage scaling\.

For data generation, we use GPT\-5\.4\-Reasoning due to its instruction\-following capabilities, maintaining consistency throughout the pipeline\. LLM usage scales directly with workbook complexity through the number of artifacts and worksheets requiring annotation\. The decompositionLLMextract=\# worksheets\\text\{LLM\}\_\{\\text\{extract\}\}=\\text\{\\\# worksheets\}for table identification andLLMannotate=2×\# data pairs=2×∑\# permutations of steps per trajectory\\text\{LLM\}\_\{\\text\{annotate\}\}=2\\times\\text\{\\\# data pairs\}=2\\times\\sum\\text\{\\\# permutations of steps per trajectory\}ensures that workbooks with richer artifact structures generate proportionally more training data while maintaining consistent annotation coverage\.

![Refer to caption](https://arxiv.org/html/2608.07873v1/x14.png)\(a\)Computational time distribution across 3 pipeline stages and \# artifacts per workbook
![Refer to caption](https://arxiv.org/html/2608.07873v1/x15.png)\(b\)Logarithmic distribution of \# artifacts against the generated number of data pairs

Figure 10:Pipeline construction efficiency: time distribution across stages \(left\) and scaling relationship between number of artifacts in workbook and generated data pairs \(right\)\.

## Appendix EWTM\-CorpusDistribution

The workbook time machine generates benchmark instances from a large pool of real\-world Excel workbooks, producing 8,931 natural\-language queries over 2,977 unique tasks from the Enron and Fuse corpora\. The full dataset pairs each task with multi\-query variants at Levels 1, 2, and 3, enabling systematic evaluation across a verbosity\-to\-specificity spectrum while holding the underlying spreadsheet transformation fixed\.

### E\.1Task\-Type and Capability Distribution

#### Task type composition\.

Figure[11\(a\)](https://arxiv.org/html/2608.07873#A5.F11.sf1)shows the distribution of the four primary task types across the 2,957 classified tasks \(20 tasks could not be unambiguously categorized and are excluded from type\-level analyses\)\. Formulas constitute the largest category at 67\.5%, followed by Conditional Formatting \(22\.7%\), Charts \(8\.3%\), and Pivot Tables \(0\.8%\)\. This skewed distribution reflects the ecological reality of enterprise spreadsheet work, where formula\-writing is the dominant activity and chart or pivot\-table construction are specialist tasks performed comparatively rarely\.

#### Capability structure\.

Each query is annotated with a set of*capability*labels drawn from a structured ontology covering computation, data management, visualization, and analysis; most tasks require a combination \(e\.g\.use\_formulas\+\+style\_data\)\. Following established intent taxonomy methodologies\(Marutaet al\.,[2025](https://arxiv.org/html/2608.07873#bib.bib36); Shahet al\.,[2025](https://arxiv.org/html/2608.07873#bib.bib37)\), we categorize each query’s functional intent\. Figure[11\(b\)](https://arxiv.org/html/2608.07873#A5.F11.sf2)\(left\) shows the frequency of individual artifact capability, and \(right\) the distribution of intent categories broken down by query level\. The intent distribution is stable across levels: the proportional breakdown of Aggregation & Summarization, Transformation, Formulas & Logic, Visualization, and related intents remains consistent from Level 1 to Level 3, validating that the three query variants are semantically equivalent\.

![Refer to caption](https://arxiv.org/html/2608.07873v1/figures/corp_dist/fig1_task_type_pie.png)\(a\)Task\-type distribution \(N=2,957N\{=\}2\{,\}957\)\. Formulas account for 67\.5% of the full dataset\.
![Refer to caption](https://arxiv.org/html/2608.07873v1/figures/corp_dist/fig4_capability_intent.png)\(b\)Artifact capability \(left\) and intent distribution by query level \(right\)\. Intent proportions are stable across Level 1–3, confirming semantic equivalence of query variants\.

Figure 11:Task\-type and capability analysis of all query levels in the dataset \(8,931 queries\)\.

### E\.2Industry and Scenario Coverage

#### NAICS sector coverage\.

The complete benchmark spans 20 distinct industry sectors coded under the North American Industry Classification System\(NAICS,[2022](https://arxiv.org/html/2608.07873#bib.bib35)\), providing cross\-industry generalisation that prevents models from over\-fitting to domain\-specific vocabulary or conventions\. The top five sectors–Utilities \(33\.2%\), Educational Services \(19\.7%\), Mining & Oil/Gas \(13\.1%\), Finance & Insurance \(12\.4%\), and Professional & Technical Services \(5\.3%\)–account for approximately 84% of tasks\. The remaining 16% spans 15 additional sectors including Healthcare, Manufacturing, Real Estate, Public Administration, and Transportation\.

#### Financial workflow scenarios\.

Each task is labelled with a*financial scenario*from a ten\-category vocabulary that captures functional spreadsheet intent\. The most frequent scenarios are Revenue Analysis \(14\.3%\), Financial Modelling \(12\.8%\), Treasury & Cash Flow \(7\.3%\), Budgeting & Forecasting \(7\.2%\), and Expense Tracking \(4\.5%\)\. Formulas are distributed nearly uniformly across all scenario types, consistent with their role as a general\-purpose tool; Charts concentrate in Basic Analysis and Visualisation scenarios; and Pivot Tables concentrate in Advanced Analysis\.

![Refer to caption](https://arxiv.org/html/2608.07873v1/figures/corp_dist/fig2_industry_distribution.png)Figure 12:NAICS sector and financial scenario distributions across the complete 3k dataset\.

### E\.3Query\-Level Specificity Gradient

The complete dataset implements a three\-level query design where each of the 2,977 unique tasks is described by three semantically equivalent natural language queries at distinct abstraction levels: Level 1 \(verbose, mean 357 characters\), Level 2 \(moderate, mean 220 characters\), and Level 3 \(concise, mean 139 characters\)\. The proportion of*Very Well\-Specified*queries drops from 77\.9% at Level 1 to 17\.8% at Level 3–a 4\.4×\\timesreduction–while Ambiguous or Reasonably Specified queries increase from 0% to 3\.1%\. Query length contracts monotonically across levels, and Level 1 queries imply more steps on average \(mean=3\.96=3\.96\) than Level 3 \(mean=2\.99=2\.99\) despite requiring the same underlying action\.

![Refer to caption](https://arxiv.org/html/2608.07873v1/figures/corp_dist/fig3_level_analysis.png)Figure 13:Level analysis across the complete dataset: specificity \(left\), query length \(centre\), implied steps \(right\) by query level\.

## Appendix FDetailed Experimental Specifications

### F\.1Model Specifications

We evaluate 6 state\-of\-the\-art language models from two major families:

#### Anthropic models:

Claude Haiku 4\.5is a fast, lightweight model optimized for simple tasks with efficient processing capabilities\.Claude Sonnet 4\.5represents a balanced performance model designed for general use cases, offering good trade\-offs between capability and efficiency\.Claude Opus 4\.6serves as the most capable model in the family, specifically designed for complex reasoning tasks requiring sophisticated analytical capabilities\.

#### OpenAI models:

\(OpenAI,[2023](https://arxiv.org/html/2608.07873#bib.bib22)\)GPT\-4\.1 Miniprovides an efficient model architecture tailored for straightforward tasks where computational efficiency is prioritized\.GPT\-5\.2 Reasoningincorporates advanced reasoning capabilities designed to handle more complex logical operations and multi\-step problem\-solving\.GPT\-5\.4 Reasoningrepresents the state\-of\-the\-art reasoning model, offering the highest level of analytical and inferential capabilities in the evaluated model suite\.

### F\.2API Details

We evaluate 3 API interfaces for Excel manipulation:

#### Python

OpenPyXL\(Gazoni and Clark,[2024](https://arxiv.org/html/2608.07873#bib.bib17)\)is a Python library providing direct programmatic access to Excel files\. It offers comprehensive support for reading and writing Excel 2010 xlsx/xlsm files, including formulas, charts, and conditional formatting\. However, limitations include lack of VBA macro support, missing Excel object primitives and reduced performance with very large files\. We also provide the pandas library in the environment, to help with Pivot Tasks, which OpenPyXL cannot natively perform\.

#### VBA

serves as Excel’s native macro language with full feature access to all Excel functionality\. It provides complete control over Excel objects, methods, and properties, enabling sophisticated automation scenarios\. The primary advantages include comprehensive feature coverage and direct Excel integration without external dependencies\. Limitations encompass Windows\-only operation and a steeper learning curve compared to modern programming interfaces\.

#### OfficeJS\(Microsoft,[2024b](https://arxiv.org/html/2608.07873#bib.bib18)\)

represents Microsoft’s modern JavaScript API for web\-based interaction with Office applications\. Designed for cross\-platform compatibility and modern web development practices, it enables spreadsheet automation through web\-based interfaces\. Advantages include robust cross\-platform support and modern API design principles\. However, limitations include reduced functionality compared to VBA and dependency on web\-based execution environments\.

### F\.3Orchestration Framework

#### Multi\-Turn Agent Loop\.

The core evaluation driver implements a multi\-turn tool\-calling loop\. At each turn, the model receives the conversation history and may either produce a final text response \(terminating the task rollout\) or emit a structured tool call containing executable code\. Each of these outputs can be accompanied with an optional <think\>…</think\> paragraph\. Tool call outputs–including execution results and the sheet state \(post code execution\)–are appended to the conversation as tool\-role messages, and the loop continues for up to a maximum number of 10 turns\.

#### Tool Execution Backends\.

The framework supports three interchangeable code execution backends, each defined by a distinct system prompt, function\-calling schema, and executor\.

- •Python \(OpenPyXl\):Code is executed in\-process viaexec\(\)in a sandboxed daemon thread with a timeout\.
- •OfficeJS:Code is executed through an Excel runtime\.
- •VBA:Code is executed through COM automation \(win32com\)\.

Each backend returns a structured result containing an execution status flag and the post\-execution sheet state\.

## Appendix GRobustness and Failure Analysis

#### Independent re\-runs\.

To test whether the main results are stable under API nondeterminism, we ran 3 additional independent trials on the most evaluated configuration, Python/OpenPyXL Level 3, across 7 models\. Table[6](https://arxiv.org/html/2608.07873#A7.T6)reports mean±\\pmstandard deviation\. Model ordering is preserved across runs and Hard Score standard deviations are at most 0\.035\.

Table 6:Python/OpenPyXL Level 3 runs over 150 tasks per run with 0–1 scale\.
#### Turn budget\.

The 10\-turn budget is generally permissive rather than binding\. Mean turns range from 1\.66 to 5\.50 in the repeated Python setting and from 2\.31 to 4\.61 for the reported OfficeJS/VBA runs\. The two smaller Claude models hit the cap on a minority of hard Python tasks \(Haiku: 24\.2%; Sonnet: 17\.8%\), while GPT models rarely do so\. Thus, the single\-turn SpreadsheetBench gap is not primarily a turn\-budget artifact\.

Table 7:Mean turns by API; parentheses show fraction of rollouts that reached 10\-turn cap\.
#### Failure taxonomy\.

We analyzed 4,910 deduplicated rollouts covering Python Level 3 \(150 tasks×\\times3 runs×\\times7 models\), Python Level 1, and OfficeJS\. Failure is defined as Soft Score below 0\.5; strict failure is below 0\.10\. Among 1,858 failed Level 3 Python rollouts, 88\.5% are structural: either the artifact is effectively absent \(wrong shape\) or present at the wrong coordinate \(wrong placement\)\. Wrong values account for only 11\.6% \(Table[8](https://arxiv.org/html/2608.07873#A7.T8)\)\.

Table 8:Grader\-derived error verticals for failed Level 3 Python rollouts\.Both model families have similar error\-vertical splits; their main difference is hallucinated success, where the final message claims completion despite strict failure\. As Table[9](https://arxiv.org/html/2608.07873#A7.T9)shows, Claude rollouts have higher hallucinated success and more turns on average, while GPT rollouts terminate earlier\.

Table 9:Family\-level failure behavior in the rollout analysis\.Execution\-surface correlations \(Table[10](https://arxiv.org/html/2608.07873#A7.T10)\) show that tool failure is not the main bottleneck: frontier models sit at 3–5% tool\-call failure, and score correlation with tool\-call failure is weak\. Code volume is the strongest negative predictor, suggesting that long code generations often mark unsuccessful repair attempts\. Exploration polarity differs by family: read\-only exploration correlates negatively with score for Claude and positively for GPT\.

Table 10:Spearman correlations between execution surfaces and score\.Across query levels, Soft Score drops modestly from Level 1 to Level 3, but Hard Score drops more sharply \(for example, Opus falls from 27% to 18%, and GPT\-5\.4 Reasoning from 21% to 13%\)\. Thus Level 3 primarily reduces fully correct rollouts rather than only lowering partial credit\. Across APIs, OfficeJS improves Soft Score slightly for all models, but GPT tool\-call failure increases by 17 percentage points while Claude remains stable, exposing an execution weakness that aggregate Soft Score alone masks\.

## Appendix HDetailed Grading Schema

All components compare only changes from the pre\-task baseline \(initial→\\rightarrowground truth vs\. initial→\\rightarrowgeneration\); pre\-existing content is ignored\. Each component is active only when the ground truth introduces that artifact type, and raw weights are normalized to sum to 1\.0\. Supporting signals \(worksheet creation and tables\) carry half the weight of primary signals\.

#### Cell Values & Formulas:

Cell values are evaluated by computing formula results \(via the Formulas library\), so equivalent expressions that produce the same output receive full credit\.

#### Charts:

Each ground\-truth chart is matched to the best available generated chart\. Chart similarity score is computed across three criteria: chart type \(40%: full credit for exact match, 20% for same family, e\.g\.BarChartvs\.BarChart3D\), series count \(20%: proportional partial credit via min/max ratio\), and series data references \(40%: Jaccard overlap of reference strings\)\. Axis labels, legend styling, and color choices are not evaluated\.

#### Pivot Tables:

Pivot similarity is scored across five sub\-criteria: sheet placement \(15%\), row fields \(25%: full credit for correct ordered list, 15% for correct fields in wrong order\), column fields \(20%: full credit for correct ordered list, 10% for correct fields in wrong order\), data field names \(25%: Jaccard set overlap\), and cache fields / source schema \(15%: Jaccard overlap of source column names\)\. Aggregation function names are not directly evaluated; the data field name match serves as a proxy\.

#### Conditional Formatting:

Scoring covers three criteria: correct worksheet \(30%\), exact cell range after normalisation \(40%\), and rule type set overlap \(30%, e\.g\.cellIs,colorScale,dataBar\)\. Formatting style properties such as colors, fonts, and borders are not evaluated, nor is rule priority ordering\.

![Refer to caption](https://arxiv.org/html/2608.07873v1/figures/gen.png)\(a\)Generated
![Refer to caption](https://arxiv.org/html/2608.07873v1/figures/gt.png)\(b\)Ground Truth

Figure 14:Evaluation metrics comparison\. For it, Hard match:Fail, Soft match:Pass\.

## Appendix IExamples fromWTM\-Bench

Examples fromWTM\-Benchare presented in the subsequent Figs\.[15](https://arxiv.org/html/2608.07873#A9.F15),[16](https://arxiv.org/html/2608.07873#A9.F16), and[17](https://arxiv.org/html/2608.07873#A9.F17)\.

![Refer to caption](https://arxiv.org/html/2608.07873v1/x16.png)Figure 15:Example 1: Clubbed formula group being added\. Where the colored texts are the utterances corresponding to the task\.![Refer to caption](https://arxiv.org/html/2608.07873v1/x17.png)\(a\)Mixed multi\-step tasks: formulas, charts
![Refer to caption](https://arxiv.org/html/2608.07873v1/x18.png)\(b\)Pivot table on new worksheet

Figure 16:Examples of complex spreadsheet tasks![Refer to caption](https://arxiv.org/html/2608.07873v1/x19.png)Figure 17:Example 4: Multi\-step formula tasks