Forecast Workflow Bench: Evaluating Language-Model Decisions with Budgeted Forecast Tools

# EVALUATING LANGUAGE-MODEL DECISIONSWITH BUDGETED FORECAST TOOLS
Source: [https://arxiv.org/html/2609.27385](https://arxiv.org/html/2609.27385)
## FORECAST WORKFLOW BENCH: EVALUATING LANGUAGE\-MODEL DECISIONS WITH BUDGETED FORECAST TOOLS

###### Abstract

Time\-series foundation models \(TSFMs\) provide forecasts for operational decisions, but accuracy alone does not determine their value\. Evaluating agents that use these models requires measuring decision quality and forecast cost\. FWBench evaluates this capability on 1,251 electricity and cycle\-hire cases using fixed forecast tools and simulated capacity contracts\. Agents select models, histories and horizons, then submit capacities to minimize a stated loss–cost objective\. We evaluated two hosted and eight local configurations, including small language models, and tested local models with and without TSFMs\. GPT\-6 Astra bought inexpensive short\-horizon forecasts selectively, using 2\.5% of the budget, and outperformed fixed policies when the saved decisions were scored with three loss–cost weightings\. FWBench enables reproducible evaluation of how language models select and use time\-series forecasts to make decisions under cost constraints\.

###### Index Terms:

Benchmarks, agents, large language models, time\-series forecasting, decision\-making

††address:Neurogica Inc\.Figure 1:FWBench inputs \(a\), forecasting tools, optimizer and scorer \(b\), and Astra’s submitted capacities \(c\): BANC, July 24; Hop Exchange, May 11\. Capacities use physical units; histories show 3\-hour medians\.## 1Introduction

Effective decision\-making requires anticipating future conditions\. Time\-series foundation models \(TSFMs\) provide accurate forecasts across domains\[[1](https://arxiv.org/html/2609.27385#bib.bib1),[2](https://arxiv.org/html/2609.27385#bib.bib2)\]; large language models \(LLMs\) can plan with tools\[[3](https://arxiv.org/html/2609.27385#bib.bib3)\]\. Existing evaluations examine forecast accuracy and forecaster selection\[[4](https://arxiv.org/html/2609.27385#bib.bib4),[5](https://arxiv.org/html/2609.27385#bib.bib5)\], but do not measure whether a forecast improves capacity decisions enough to justify its cost\.

Recent advances have enabled local forecasting with open\-weight TSFMs smaller than one billion parameters\[[6](https://arxiv.org/html/2609.27385#bib.bib6),[7](https://arxiv.org/html/2609.27385#bib.bib7)\]\. Combined with small language models \(SLMs\), they could support decisions without cloud communication\. However, existing forecasting benchmarks have not established how well these local systems make cost\-constrained decisions or how they compare with proprietary LLMs\. Forecast Workflow Bench \(FWBench; Fig\.[1](https://arxiv.org/html/2609.27385#S0.F1)\) evaluates these capabilities using fixed forecasters, real demand and simulated capacity contracts\.

The code111[https://github\.com/Neurogica/forecast\-workflow\-bench](https://github.com/Neurogica/forecast-workflow-bench)and dataset222[https://huggingface\.co/datasets/Neurogica/forecast\-workflow\-bench](https://huggingface.co/datasets/Neurogica/forecast-workflow-bench)are publicly available for reproducing the evaluation\. Our contributions are \(i\) a 1,251\-case electricity and cycle\-hire benchmark for evaluating forecast selection, its cost and the resulting capacity decisions; \(ii\) comparisons of ten hosted and local configurations with fixed and adaptive policies; and \(iii\) paired ablation studies of TSFM access and thinking, oracle comparisons and sensitivity to objective weights\.

## 2Related Work

Recent work on time\-series foundation models has explored transferring pretrained knowledge across tasks\[[8](https://arxiv.org/html/2609.27385#bib.bib8)\]\. Benchmarks evaluate forecast accuracy\[[4](https://arxiv.org/html/2609.27385#bib.bib4)\], forecaster selection\[[5](https://arxiv.org/html/2609.27385#bib.bib5)\]and broader time\-series tasks\. For agent\-based systems, TimeSeriesGym evaluates time\-series engineering\[[9](https://arxiv.org/html/2609.27385#bib.bib9)\], while TemporalBench assesses contextual prediction and event\-informed reasoning\[[10](https://arxiv.org/html/2609.27385#bib.bib10)\]\.

Recent language\-model agents combine planning, memory and external tools\[[11](https://arxiv.org/html/2609.27385#bib.bib11)\]\. Benchmarks test how they use these tools and whether they complete tasks within resource limits\. Existing benchmarks assess API planning and calling\[[3](https://arxiv.org/html/2609.27385#bib.bib3)\], policy compliance\[[12](https://arxiv.org/html/2609.27385#bib.bib12)\], stateful tool use\[[13](https://arxiv.org/html/2609.27385#bib.bib13)\]and industrial operations\[[14](https://arxiv.org/html/2609.27385#bib.bib14)\]\. CostBench measures planning cost gaps\[[15](https://arxiv.org/html/2609.27385#bib.bib15)\], while EcoAgent\-Bench evaluates budgeted economic decisions\[[16](https://arxiv.org/html/2609.27385#bib.bib16)\]\.

FWBench evaluates how forecast selection affects capacity decisions and cost\. Hosted and local models use the same fixed tools, capacity constraints and forecast budget\.

## 3Benchmark Design

### 3\.1Data construction and observation windows

FWBench uses U\.S\. Energy Information Administration form EIA\-930 electricity demand\[[17](https://arxiv.org/html/2609.27385#bib.bib17),[18](https://arxiv.org/html/2609.27385#bib.bib18)\]and Transport for London \(TfL\) cycle\-hire departures\[[19](https://arxiv.org/html/2609.27385#bib.bib19)\]\. Windows must have unique timestamps and pass provider quality checks\. We exclude forecast, adjusted, imputed and nonpositive electricity values, but retain zero cycle\-hire counts\.

Agents observe hourly histories and submit capacities one or seven days before the target day\. Timestamps mark the start of each UTC hour; agents cannot access future demand \(ground truth, GT\)\. Cycle\-hire capacities represent departures per hour; the task does not model inventory or rebalancing\.

### 3\.2Actions and decision loss

Forecast error alone does not measure the quality of capacity decisions\. We score submitted capacities against observed demand using a decision loss that penalizes unused capacity and unmet demand under the simulated contract\.

ForTTtarget hoursh=0,…,T−1h=0,\\ldots,T\-1,yhy\_\{h\}is observed demand andzh=yh/sz\_\{h\}=y\_\{h\}/sdemand normalized by the visible\-history means\>0s\>0\. Planaacontains capacitiesaba\_\{b\}forT/dT/dconsecutivedd\-hour blocks \(b=0,…,T/d−1b=0,\\ldots,T/d\-1;dddividesTT\)\. Feasible actions satisfyab∈𝒢a\_\{b\}\\in\\mathcal\{G\}and\|ab−ab−1\|≤r\|a\_\{b\}\-a\_\{b\-1\}\|\\leq rforb≥1b\\geq 1, where𝒢\\mathcal\{G\}contains allowed normalized capacities andrrlimits changes between adjacent blocks\. Submissions use physical capacitiess​absa\_\{b\}\. For demand vectorz=\(zh\)z=\(z\_\{h\}\), loss is

ℓ⁡\(a,z\)=1T​∑h=0T−1\[α​\(a¯h−zh\)\+\+wh​\(zh−a¯h\)\+\],\\ell\(a,z\)=\\frac\{1\}\{T\}\\sum\_\{h=0\}^\{T\-1\}\\left\[\\alpha\(\\bar\{a\}\_\{h\}\-z\_\{h\}\)\_\{\+\}\+w\_\{h\}\(z\_\{h\}\-\\bar\{a\}\_\{h\}\)\_\{\+\}\\right\],\(1\)wherea¯h=a⌊h/d⌋\\bar\{a\}\_\{h\}=a\_\{\\lfloor h/d\\rfloor\}is normalized capacity at hourhh,\(⋅\)\+\(\\cdot\)\_\{\+\}clips negative values to zero, andα\\alphaandwhw\_\{h\}are the surplus and hourly shortage weights supplied to the agent\. Submissions require unique block IDs, finite grid values and feasible ramps; invalid answers are penalized above every feasible loss\.

### 3\.3Forecast tools and decision support

Table[1](https://arxiv.org/html/2609.27385#S3.T1)lists the tools, including fixed Chronos\-Bolt Tiny, Chronos\-2 and TimesFM 2\.5 services\[[1](https://arxiv.org/html/2609.27385#bib.bib1),[2](https://arxiv.org/html/2609.27385#bib.bib2),[7](https://arxiv.org/html/2609.27385#bib.bib7),[6](https://arxiv.org/html/2609.27385#bib.bib6)\]\. Unaffordable requests are refused\. A shared optimizer uses dynamic programming to minimize expected contract loss given classical or TSFM forecasts\. Agents can test past forecast accuracy through replay: a model predicts from a historical origin using only earlier observations\. The tool returns pinball loss on observed hours and charges the forecast tariff\.

Table 1:Tools available to all agents\. Forecast charges follow the selected model, history and horizon; other tools are free\.Prompts provide timestamps, block IDs, units, contract terms and the objective below\. Agents submit capacities by block ID; browsing and file access are disabled\.

### 3\.4Cost accounting, objective and references

CPU inference times determine fixed charges by model, history length and forecast horizon\. Credits exclude language\-model inference, loading and optimization; they do not measure evaluation runtime\.

To balance decision quality and forecast cost, caseiiin domainjjwith credit budgetBBreceives score

Si=12​\[ℓi−Fiσj\+ciB\],S\_\{i\}=\\tfrac\{1\}\{2\}\\left\[\\frac\{\\ell\_\{i\}\-F\_\{i\}\}\{\\sigma\_\{j\}\}\+\\frac\{c\_\{i\}\}\{B\}\\right\],\(2\)whereℓi\\ell\_\{i\}is loss from Eq\. \([1](https://arxiv.org/html/2609.27385#S3.E1)\) or the invalid penalty,cic\_\{i\}is the credit charge, andFiF\_\{i\}is the loss obtained by optimizing capacities with the actual target demand known\. The scaleσj\>0\\sigma\_\{j\}\>0is the hour\-of\-day policy’s mean excess loss on pre\-evaluation replay cases\. This policy uses past observations at each target hour\. Prompts omitFiF\_\{i\}because it requires unseen demand; this constant does not affect the optimal action\. Equal loss–cost weights express a preference, not an economic exchange rate \(Section[4\.5](https://arxiv.org/html/2609.27385#S4.SS5)\)\. Normalized excess loss isRi=\(ℓi−Fi\)/σjR\_\{i\}=\(\\ell\_\{i\}\-F\_\{i\}\)/\\sigma\_\{j\}; lowerRiR\_\{i\}andSiS\_\{i\}are better\. We averageRiR\_\{i\},SiS\_\{i\}, loss and credits by series and lead, then average these means equally within each cohort\. Cohort weights \(Section[4\.1](https://arxiv.org/html/2609.27385#S4.SS1)\) give the reportedRR,SS, loss and creditscc\. Invalid submissions retain their loss penalties in these averages\.

Table 2:1,251 cases, full tools\. Valid: feasible plans;cc: credits\.SSincludes invalid penalties;SvS\_\{v\}excludes them and renormalizes weights over differing valid subsets\.†\\dagger:SvS\_\{v\}from only three cases\. T: thinking; bold: agents\.ConfigurationValid↑\\uparrowLoss↓\\downarrowcc↓\\downarrowS↓S\\downarrowSv↓S\_\{v\}\\downarrowSLMsGemma 4 E4B\[[20](https://arxiv.org/html/2609.27385#bib.bib20)\]10190\.7074744\.62941\.187Qwen3 4B\[[21](https://arxiv.org/html/2609.27385#bib.bib21)\]12420\.717504\.54784\.113Qwen3 4BT12370\.519602\.52322\.429Gemma 4 E4BT12510\.364912091\.04651\.046Larger local modelsQwen3\.6 35B\-A3B\[[22](https://arxiv.org/html/2609.27385#bib.bib22)\]3†2\.0080117617\.97730\.544Qwen3\.6 35B\-A3BT1601\.8533146416\.46321\.003Gemma 4 12BT\[[20](https://arxiv.org/html/2609.27385#bib.bib20)\]4901\.28375810410\.63581\.151Gemma 4 12B7660\.9611460917\.32810\.983Hosted LLMsGPT\-5\.4 nano\[[23](https://arxiv.org/html/2609.27385#bib.bib23)\]11390\.516028862\.64110\.893GPT\-6 Astra\[[24](https://arxiv.org/html/2609.27385#bib.bib24)\]12510\.304428020\.39640\.396Policies without language modelsFixed Chronos\-2\[[1](https://arxiv.org/html/2609.27385#bib.bib1)\]12510\.30501087830\.8651–Backtest \(8 windows\)12510\.326500\.6175–Backtest selector12510\.325900\.6109–Fixed Chronos\-Bolt Tiny\[[6](https://arxiv.org/html/2609.27385#bib.bib6)\]12510\.3087391000\.6014–Weekday empirical12510\.322000\.5726–Lead\-matched selector12510\.321100\.5633–Hour\-of\-day empirical12510\.316500\.5147–Hindsight references \(not agents\)Catalog oracle–0\.289910530\.2393–Perfect demand–0\.2676–0\.0000–Baselines without language models include fixed policies that use one method for all cases, and selectors that choose free methods by 24\-hour backtest loss\. Selectors break ties by name and refit on full histories\.

Two oracles measure the score achievable by selecting plans with true future demand: the catalog oracle minimizesSiS\_\{i\}over affordable full\-history plans from eight classical and three TSFM methods, paying only for its choice; the free\-menu oracle excludes TSFMs\.

Figure 2:SRP, July 25, one\-day lead: demand and capacities\. Case fixed before the run; configurations selected post hoc\.

## 4Experiments

### 4\.1Experimental settings

We setT=24T=24,d=3d=3\(eight actions\),α=0\.3\\alpha=0\.3,𝒢=\{0\.40,0\.45,…,2\.00\}\\mathcal\{G\}=\\\{0\.40,0\.45,\\ldots,2\.00\\\}, andr=0\.30r=0\.30\. Shortage weights followed six\-hour segments1,1,2,1\.51,1,2,1\.5, shifted by target weekday\. Each case allowedB=113,726B=113\{,\}726forecast credits and 12 tool calls\. Prompts instructed agents to minimizeSSin Eq\. \([2](https://arxiv.org/html/2609.27385#S3.E2)\)\. We setσj=0\.0542\\sigma\_\{j\}=0\.0542for electricity andσj=0\.0434\\sigma\_\{j\}=0\.0434for cycle hire\. Selectors used four or eight short\-lead replays, or lead\-matched replays with shorter histories\.

We used 1,091 electricity cases from 50 authorities in February, July and August 2026, and 160 cycle\-hire cases from 20 stations in May 2026\. Cohorts comprised three electricity seasons \(weight 1/6 each\) and transport \(1/2\)\. The target day began one or seven days after the decision\. Agents had 336 hourly observations ending three hours before the decision and chose how much history to use\. Forecasts began at the next hourly timestamp; 50 or 194 steps covered the target day\.

The two hosted models used high reasoning effort; the four quantized local models were tested with thinking off and on \(Table[2](https://arxiv.org/html/2609.27385#S3.T2)\)\. SLMs had<10<10B total parameters; larger local models had≥10\\geq 10B\. In total, 22,518 conversations were scored\. Post\-hoc normalization accepted equivalent JSON and quoted numbers without changing numerical values\.

Table 3:Resident SLMs, thinking on, on eight common cases\. Peak GPU/host PSS; mean case time, GPU power and energy\.Table[3](https://arxiv.org/html/2609.27385#S4.T3)profiles one case per cohort–lead pair on an RTX PRO 6000 Blackwell with CPU tools\. All submissions were valid and used no forecast credits\. GPU energy included idle periods; host memory used proportional set size \(PSS\)\. Bolt on one CPU thread took 7\.5/65\.9 ms for 50/194\-step forecasts with 0\.45 GiB peak PSS; CPU energy was not measured\.

### 4\.2Quantitative results

Overall comparison\.Table[2](https://arxiv.org/html/2609.27385#S3.T2)shows decision loss, forecast credits andSSfor agents and reference policies\. Astra achieved similar loss to fixed Chronos\-2 using 2\.6% of its forecast credits\. It bought only short\-horizon Bolt Tiny forecasts \(Fig\.[3](https://arxiv.org/html/2609.27385#S4.F3)\)\. It ranked first inSSoverall and in all eight cohort–lead subsets by point estimate\. Astra’s improvements inSSand loss over the best free policy and three selectors had paired 95% cluster\-bootstrap intervals excluding zero\.

Local decision quality and completion\.All local configurations scored worse than hour\-of\-day empirical\. Gemma 4 E4B with thinking completed every case yet scored more than twice as high, identifying forecast selection and use as improvement targets beyond valid output\. Valid\-onlySvS\_\{v\}excludes failures but compares different subsets\. Qwen3\.6 without thinking exhausted the 12\-call limit in 1,245 cases, so workflow non\-completion dominated its score\.

### 4\.3TSFM access and thinking

Table 4:TSFM removal, 1,251 cases\. Valid/SS: no\-TSFM;Δ​S\\Delta S: full minus no\-TSFM \(negative favors access\)\. 95% CI: paired cluster\-bootstrap confidence interval; bold excludes zero\.ConfigurationValid↑\\uparrowS↓S\\downarrowΔ​S\\Delta S95% CISLMsGemma 4 E4B\[[20](https://arxiv.org/html/2609.27385#bib.bib20)\]10084\.052\+0\.577\+0\.577\[−0\.277,1\.321\]\[\-0\.277,1\.321\]Qwen3 4B\[[21](https://arxiv.org/html/2609.27385#bib.bib21)\]12424\.514\+0\.034\+0\.034\[−0\.042,0\.159\]\[\-0\.042,0\.159\]Qwen3 4BT12432\.464\+0\.059\+0\.059\[−0\.117,0\.230\]\[\-0\.117,0\.230\]Gemma 4 E4BT12501\.139−0\.092\-0\.092\[−0\.197,0\.008\]\[\-0\.197,0\.008\]Larger local modelsQwen3\.6 35B\-A3B\[[22](https://arxiv.org/html/2609.27385#bib.bib22)\]118\.090−0\.113\-0\.113\[−0\.346,0\.119\]\[\-0\.346,0\.119\]Qwen3\.6 35B\-A3BT44812\.436\+4\.027\+4\.027\[3\.338,4\.732\]\[3\.338,4\.732\]Gemma 4 12BT\[[20](https://arxiv.org/html/2609.27385#bib.bib20)\]5849\.662\+0\.974\+0\.974\[−0\.345,2\.184\]\[\-0\.345,2\.184\]Gemma 4 12B11462\.062\+5\.266\+5\.266\[4\.573,5\.944\]\[4\.573,5\.944\]Figure 3:Loss–cost trade\-off on 1,251 cases; lower left is better\. Diagonals: equalSS; filled/hollow triangles: Gemma 4 E4B with thinking, with/without TSFMs; star: catalog oracle\.Table[4](https://arxiv.org/html/2609.27385#S4.T4)shows no significant score benefit from TSFM access for any local configuration\. Access significantly increasedSSfor Gemma 4 12B without thinking and Qwen3\.6 with thinking \(paired 95% intervals excluded zero\)\. On cases valid with and without TSFMs, access raisedSSfor Gemma \(714 cases\) but lowered it for Qwen \(58 cases\)\. Thus, Gemma’s worse score was not explained by failed submissions alone\. Qwen’s lower score on this subset did not establish a benefit across all cases, which included failures\.

With all tools available, thinking loweredSSfor three models but raised it for Gemma 4 12B; all four paired intervals excluded zero\. This comparison measured how agents used TSFMs, rather than TSFM forecast accuracy\.

### 4\.4Effect of the stated objective

We tested whether including forecast cost in the objective changed purchases\. Astra and nano received either loss\-only instructions or instructions to minimize the combined loss–cost scoreSS\(Eq\. \([2](https://arxiv.org/html/2609.27385#S3.E2)\)\) on the same 16 cases, with identical tools and limits\. Relative to loss\-only instructions, minimizingSSreduced Astra’s credits by 98\.5% with 1\.6% higher decision loss\. All Astra plans were valid\. Only Astra showed a significant reduction inSS\. The change in purchases under identical tools and budgets indicates that the stated objective influenced how Astra used forecasts\.

In Fig\.[2](https://arxiv.org/html/2609.27385#S3.F2), Astra bought a Bolt Tiny forecast after three free plans\. Gemma 4 12B without thinking spent 91% of its budget on forecasts ending before the target, but submitted a free plan\.

### 4\.5Weight sensitivity and repeated executions

We rescored the saved decisions to test sensitivity to loss–cost weights\. Astra remained first under both alternatives \(0\.25/0\.75 and 0\.75/0\.25\), with significant improvements over each setting’s best fixed policy\. Giving more weight to loss made Bolt Tiny score better than hour\-of\-day empirical\.

We repeated Astra on 100 cases to assess run\-to\-run stability\. Both runs returned valid plans and ranked first against existing results on those cases\. However, the paired 95% interval against the best fixed policy included zero, so the repeated ranking did not establish statistical superiority\.

## 5Limitations and Future Work

The main limitation is coverage: we evaluated two domains over 16 dates under simulated contracts\. Cycle\-hire counts excluded unmet demand\. Weather could correlate results across sites\. Confidence intervals were conditional on the sampled dates and were not adjusted for multiple testing\.

Fixed tariffs and one calibration week limited economic interpretation\. Hosted comparisons lacked TSFM\-removal tests and full\-set repeated runs\. The common harness did not evaluate differences between complete agent systems\.

In future work, we plan to evaluate task\-specific decision models, extend temporal and domain coverage, compare complete agent systems with their own harnesses, and repeat TSFM\-access comparisons across calibration periods\. We will measure end\-to\-end latency, memory and energy on edge devices while retaining common tools and scoring\. Updated tool catalogs will require new oracle references\.

## 6Conclusion

FWBench evaluates LLM and SLM decisions with fixed forecast tools and a common loss–cost objective\. Astra’s selective purchasing achieved similar loss to fixed Chronos\-2 at lower forecast cost, while fully valid SLM plans still lagged behind free policies\. Better forecast selection could narrow the gap to the hindsight catalog oracle; stronger TSFMs could further improve decision quality\. FWBench supports evaluating improvements in forecast selection and decision quality for both LLMs and SLMs under common tools and cost constraints\.

## References

- \[1\]Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, et al\.,“Chronos\-2: From univariate to universal forecasting,”arXiv preprint arXiv:2510\.15821, 2025\.
- \[2\]Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou,“A decoder\-only foundation model for time\-series forecasting,”inProceedings of ICML, 2024, pp\. 10148–10167\.
- \[3\]Minghao Li, Yingxiu Zhao, Bowen Yu, et al\.,“API\-Bank: A comprehensive benchmark for tool\-augmented LLMs,”inProceedings of EMNLP, 2023, pp\. 3102–3116\.
- \[4\]Taha Aksu, Gerald Woo, Juncheng Liu, et al\.,“GIFT\-Eval: A benchmark for general time series forecasting model evaluation,”arXiv preprint arXiv:2410\.10393, 2024\.
- \[5\]Kanghui Ning, Yushan Jiang, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song,“TimeRouter: Efficient and adaptive routing of time\-series foundation models,”arXiv preprint arXiv:2606\.11625, 2026\.
- \[6\]Amazon Science,“Chronos\-Bolt Tiny model card,”[https://huggingface\.co/amazon/chronos\-bolt\-tiny](https://huggingface.co/amazon/chronos-bolt-tiny),Accessed: 2026\-09\-15\.
- \[7\]Google Research,“TimesFM\-2\.5\-200M model card,”[https://huggingface\.co/google/timesfm\-2\.5\-200m\-pytorch](https://huggingface.co/google/timesfm-2.5-200m-pytorch), 2025,Accessed: 2026\-09\-15\.
- \[8\]Yuxuan Liang, Haomin Wen, Yuqi Nie, et al\.,“Foundation models for time series analysis: A tutorial and survey,”inProceedings of KDD, 2024, pp\. 6555–6565\.
- \[9\]Yifu Cai, Xinyu Li, Mononito Goswami, et al\.,“TimeSeriesGym: A scalable benchmark for \(time series\) machine learning engineering agents,”arXiv preprint arXiv:2505\.13291, 2025\.
- \[10\]Muyan Weng, Defu Cao, Wei Yang, Yashaswi Sharma, and Yan Liu,“TemporalBench: A benchmark for evaluating LLM\-based agents on contextual and event\-informed time series tasks,”inProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2, 2026, pp\. 9997–10008\.
- \[11\]Lei Wang, Chen Ma, Xueyang Feng, et al\.,“A survey on large language model based autonomous agents,”Frontiers of Computer Science, vol\. 18, no\. 6, pp\. 186345, 2024\.
- \[12\]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan,“τ\\tau\-bench: A benchmark for tool\-agent\-user interaction in real\-world domains,”inProceedings of ICLR, 2025\.
- \[13\]Jiarui Lu, Thomas Holleis, Yizhe Zhang, et al\.,“ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities,”inFindings of the Association for Computational Linguistics: NAACL 2025, 2025, pp\. 1160–1183\.
- \[14\]Dhaval Patel et al\.,“AssetOpsBench: Benchmarking AI agents for task automation in industrial asset operations and maintenance,”arXiv preprint arXiv:2506\.03828, 2025\.
- \[15\]Jiayu Liu, Cheng Qian, Zhaochen Su, et al\.,“CostBench: Evaluating multi\-turn cost\-optimal planning and adaptation in dynamic environments for LLM tool\-use agents,”inProceedings of ACL \(Volume 1: Long Papers\), 2026, pp\. 12826–12858\.
- \[16\]Jie Wu, Ming Gong, Feixiang Cheng, and Qinqin Zhao,“EcoAgent\-Bench: Evaluating economic decision\-making in budget\-constrained LLM agents,”arXiv preprint arXiv:2608\.05519, 2026\.
- \[17\]U\.S\. Energy Information Administration,“Form EIA\-930: Hourly and daily balancing authority operations report,”[https://www\.eia\.gov/survey/](https://www.eia.gov/survey/), 2026,Accessed: 2026\-09\-15\.
- \[18\]U\.S\. Energy Information Administration,“Copyrights and reuse,”[https://www\.eia\.gov/about/copyrights\_reuse\.php](https://www.eia.gov/about/copyrights_reuse.php), 2026,Accessed: 2026\-09\-15\.
- \[19\]Transport for London,“Our open data: Cycle hire trip data,”[https://tfl\.gov\.uk/info\-for/open\-data\-users/our\-open\-data](https://tfl.gov.uk/info-for/open-data-users/our-open-data), 2026,Accessed: 2026\-09\-15\.
- \[20\]Gemma Team,“Gemma 4 technical report,”arXiv preprint arXiv:2607\.02770, 2026\.
- \[21\]An Yang, Anfeng Li, Baosong Yang, et al\.,“Qwen3 technical report,”arXiv preprint arXiv:2505\.09388, 2025\.
- \[22\]Qwen Team,“Qwen3\.6\-35B\-A3B model card,”[https://huggingface\.co/Qwen/Qwen3\.6\-35B\-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), 2026,Accessed: 2026\-09\-22\.
- \[23\]OpenAI,“Introducing GPT\-5\.4 mini and nano,”[https://openai\.com/index/introducing\-gpt\-5\-4\-mini\-and\-nano/](https://openai.com/index/introducing-gpt-5-4-mini-and-nano/), 2026,Accessed: 2026\-09\-15\.
- \[24\]OpenAI,“GPT\-6 Astra: A new generation of intelligence,”[https://openai\.com/index/gpt\-6\-astra/](https://openai.com/index/gpt-6-astra/), 2026,Accessed: 2026\-09\-15\.