TSFLab: A Self-Improving Agent-Agnostic Workbench for Time Series Forecasting Research

# A Self-Improving Agent-Agnostic Workbench for Time Series Forecasting Research
Source: [https://arxiv.org/html/2610.40901](https://arxiv.org/html/2610.40901)
Yanjun Zhao2Haoxin Liu3Bonnie Chen4Miles Ma5 Dongjin Song6Hanghang Tong2Qingsong Wen7Jingrui He2Jingchao Ni1🖂

###### Abstract

As publications and products on time series forecasting \(TSF\) proliferate, researchers face a growing burden of analyzing existing work\. First, not all publications release code\. For instance, only about 20% of arXiv TSF papers have a code link in the surveyed period\. Second, released code also requires manual maintenance\. Thus, TSF researchers must constantly adapt or implement existing works\. However, independent implementations can differ in experimental settings and are difficult to maintain\. We therefore propose to use AI agents for benchmarking TSF research and even supporting autoresearch, e\.g\., developing new TSF models\. Yet directly applying existing coding agents, such asClaude CodeandCodex, is suboptimal: prohibitive costs, limited scalability to large research projects \(e\.g\., benchmarking\), and lack of self\-maintenance\. To address these challenges, we proposeTSFLab, a self\-improving agent\-agnostic “workbench” that equips compatible coding agents with shared resources and tools for low\-cost TSF research and continuous maintenance of research resources\. Our experiments cover reproduction, benchmarking, and new method development\. In reproduction and benchmarking,TSFLabimproves task completion by 20–163% and cuts fresh tokens per completed task by 11–66% across baselines\. A pilot study of agent\-driven method development also shows improved forecasting accuracy viaTSFLab, with 32\.8–73\.6% fewer fresh tokens and 9\.2–68\.8% less development time than the other settings\.

## 1Introduction

Time series forecasting \(TSF\) is a fundamental task in energy\[[14](https://arxiv.org/html/2610.40901#bib.bib14)\], transportation\[[22](https://arxiv.org/html/2610.40901#bib.bib22)\], finance\[[33](https://arxiv.org/html/2610.40901#bib.bib33)\], weather\[[20](https://arxiv.org/html/2610.40901#bib.bib20)\], and public health\[[7](https://arxiv.org/html/2610.40901#bib.bib7)\]\. In Fig\.[1](https://arxiv.org/html/2610.40901#S1.F1)\(a\), we summarize the statistics of TSF preprints and code releases using data from arXiv\. As can be seen, code releases grow much more slowly than papers: only about one in five of the 1,707 papers since 2020 provides a code link\. Researchers, particularly those who are new to this area, must often reimplement baselines and build evaluation pipelines manually\. While TSF libraries exist, keeping them up to date is challenging:TSLib\[[38](https://arxiv.org/html/2610.40901#bib.bib38)\]grew from 18 to 41 methods despite over 1,200 new papers since 2024, and announced in April 2026 that it is no longer maintained due to limited maintainer capacity\.

![Refer to caption](https://arxiv.org/html/2610.40901v1/01-intro.png)Figure 1:\(a\) Growth of TSF papers, papers with code, and methods in libraries\. \(b\) General agents access external resources directly for each task\. \(c\) Agents with a workbench share verified resources and reusable experiments\.AI agents offer a promising solution\. Agent\-based autoresearch has shown potential in reproducing experiments\[[37](https://arxiv.org/html/2610.40901#bib.bib37)\], conducting new experiments\[[17](https://arxiv.org/html/2610.40901#bib.bib17),[27](https://arxiv.org/html/2610.40901#bib.bib27)\], and developing new methods\[[25](https://arxiv.org/html/2610.40901#bib.bib25),[32](https://arxiv.org/html/2610.40901#bib.bib32)\]\. In domains such as biology and chemistry, systems such asChemCrow\[[3](https://arxiv.org/html/2610.40901#bib.bib3)\]andPantheonOS\[[42](https://arxiv.org/html/2610.40901#bib.bib42)\]equip agents with domain\-specific knowledge, tools, and workflows, accelerating research in these areas\. In these agents, the infrastructure surrounding their LLMs is called a “harness”\[[43](https://arxiv.org/html/2610.40901#bib.bib43),[21](https://arxiv.org/html/2610.40901#bib.bib21)\]\.

Unlike a harness, which is part of each agent, a “workbench” lies outside agents: it is reusable and shareable across agents\. Workbenches exist for general science, such asToolUniverse\[[10](https://arxiv.org/html/2610.40901#bib.bib10)\]andScientific Agent Skills\[[18](https://arxiv.org/html/2610.40901#bib.bib18)\], and for biomedicine, such asBiomni\-E1\[[16](https://arxiv.org/html/2610.40901#bib.bib16)\], but not for TSF\. We introduce a domain\-specific workbench that supports AI agents throughout TSF research\. Existing TSF agent research focuses on harness development\[[51](https://arxiv.org/html/2610.40901#bib.bib51),[11](https://arxiv.org/html/2610.40901#bib.bib11),[35](https://arxiv.org/html/2610.40901#bib.bib35)\]\. However, once their systems are developed, they cannot automatically benefit from the emergence of newer agents\. In contrast, a workbench is plug\-and\-play infrastructure for any compatible agent, such asClaude Code\[[2](https://arxiv.org/html/2610.40901#bib.bib2)\],Codex\[[30](https://arxiv.org/html/2610.40901#bib.bib30)\], andPi\[[9](https://arxiv.org/html/2610.40901#bib.bib9)\], benefiting from their continuously improving capabilities\. As compared in Fig\.[1](https://arxiv.org/html/2610.40901#S1.F1)\(b\) and \(c\), a workbench enables reuse across agents and teams: a reproduction agent can add verified implementations to the workbench for a benchmarking agent to reuse\. Another agent can then reuse these benchmarks for method development\.

Using agents with only their harness for TSF research presents two challenges: \(1\)Slowly Updated TSF Knowledge\.An LLM\-based agent’s knowledge depends on its training cutoff, so it may not cover newer TSF models and experimental protocols\[[31](https://arxiv.org/html/2610.40901#bib.bib31),[4](https://arxiv.org/html/2610.40901#bib.bib4)\]\. Web search alleviates this problem, but repeated searches for large\-scale research projects are token\-costly\. Retrieved sources may be incomplete or unreliable, and unchecked errors may propagate into subsequent research\. For example, the official code ofInformer\(i\.e\., a TSF model\)\[[52](https://arxiv.org/html/2610.40901#bib.bib52)\]mistakenly dropped the last test batch in evaluation, which has influenced the settings of many subsequent studies\[[31](https://arxiv.org/html/2610.40901#bib.bib31)\]\. \(2\)Inconsistent Research Settings\.As shown in Fig\.[1](https://arxiv.org/html/2610.40901#S1.F1)\(b\), agents that configure experiments independently may resort to different information sources, leading to different settings, implementations, and incomparable results\. This makes it hard to share findings across agents and teams, and leads to unnecessary repeated experiments\[[39](https://arxiv.org/html/2610.40901#bib.bib39),[46](https://arxiv.org/html/2610.40901#bib.bib46)\]\. Both challenges make agent\-based TSF research costly and hard to scale: agents repeat source searches, model implementations, and experiments across agents and teams\.

To address these challenges, we proposeTSFLab, a novel TSF workbench, as illustrated in Fig\.[1](https://arxiv.org/html/2610.40901#S1.F1)\(c\)\.TSFLabhas four modules: \(a\)TSF\-Domain Guideenables TSF\-awareness through common research workflows, experimental settings, evaluation protocols, and known pitfalls; \(b\)Knowledge Structurerdecomposes existing TSF models into reusable units and organizes these units, methods, and diverse benchmark datasets into a unified structure\. This structure is realized by a “knowledge card” for each method, unit, and dataset, read level by level \(summary, card, source\) as a decision requires, so that AI agents can reason efficiently without token\-costly reading of code and data details\. \(c\)Experiment Workflowerexecutes agent\-declared experiments through a unified runner that validates and runs experiment configurations and reports results under consistent protocols\. \(d\)Recursive Improverroutinely checks and verifies up\-to\-date TSF research outcomes and integrates verified artifacts \(e\.g\., models, datasets\) into the workbench for self\-maintenance\. The workbench is updated independently of agent releases\. Our contributions are as follows\.

- •We identify two causes of agent inefficiency in TSF: slowly updated knowledge and inconsistent settings\.
- •We proposeTSFLab, an agent\-agnostic TSF workbench that empowers any compatible agent without altering it\.
- •UsingTSFLab, we integrated 323 TSF models and compiled 93 datasets, along with 71 model units that can be reused for autoresearch\.
- •We evaluateTSFLabon three research tasks\. In reproduction and benchmarking, it raises completion by 20–163% and cuts fresh tokens per completed task by 11–66% across baselines\. A pilot study explores method development with fewer fresh tokens and less development time\.

## 2Related Work

TSF Libraries and Benchmarks\.TSLib\[[40](https://arxiv.org/html/2610.40901#bib.bib40)\],BasicTS\[[34](https://arxiv.org/html/2610.40901#bib.bib34)\],TFB\[[31](https://arxiv.org/html/2610.40901#bib.bib31)\], andNeuralForecast\[[29](https://arxiv.org/html/2610.40901#bib.bib29)\]provide implementations and evaluation pipelines, andGIFT\-Eval\[[1](https://arxiv.org/html/2610.40901#bib.bib1)\]benchmarks time series foundation models\. Results and rankings change with evaluation settings and tuning budgets\[[31](https://arxiv.org/html/2610.40901#bib.bib31),[4](https://arxiv.org/html/2610.40901#bib.bib4)\], yet agents must read library code to recover these settings\. Methods are added and maintained by hand, so coverage lags the literature \(Fig\.[1](https://arxiv.org/html/2610.40901#S1.F1)\(a\)\)\.TSFLabdescribes its 323 methods and 93 datasets in knowledge cards read level by level, with more methods than these libraries \(Fig\.[4](https://arxiv.org/html/2610.40901#S4.F4)\)\.

Research Agents and Harnesses\. Agents show potential to reproduce papers\[[37](https://arxiv.org/html/2610.40901#bib.bib37)\], run machine learning experiments\[[17](https://arxiv.org/html/2610.40901#bib.bib17),[27](https://arxiv.org/html/2610.40901#bib.bib27)\], and propose new methods\[[25](https://arxiv.org/html/2610.40901#bib.bib25),[32](https://arxiv.org/html/2610.40901#bib.bib32)\], but rely on general training knowledge\. Domain agents such asChemCrow\[[3](https://arxiv.org/html/2610.40901#bib.bib3)\]andPantheonOS\[[42](https://arxiv.org/html/2610.40901#bib.bib42)\]add chemistry and genomics tools\. TSF agents such asTimeSeriesScientist\[[51](https://arxiv.org/html/2610.40901#bib.bib51)\],TimeCopilot\[[11](https://arxiv.org/html/2610.40901#bib.bib11)\], andMetaCaster\[[35](https://arxiv.org/html/2610.40901#bib.bib35)\]orchestrate forecasters or optimize their training: they produce forecasts rather than conduct TSF research\. In these systems, domain support lives in the agent’s harness\[[43](https://arxiv.org/html/2610.40901#bib.bib43),[21](https://arxiv.org/html/2610.40901#bib.bib21)\], so it serves one agent and must be rebuilt for newer ones\.TSFLabserves any compatible agent unchanged\.

Agent Workbenches\. Workbenches move domain support outside agents\.ToolUniverse\[[10](https://arxiv.org/html/2610.40901#bib.bib10)\]exposes scientific tools to different agents and generates new ones,Scientific Agent Skills\[[18](https://arxiv.org/html/2610.40901#bib.bib18)\]packages portable procedures, andBiomni\-E1\[[16](https://arxiv.org/html/2610.40901#bib.bib16)\]supplies biomedical tools, software, and databases\. They provide tools and procedures for general science or biomedicine, rather than TSF methods under declared evaluation protocols\.TSFLabis a TSF workbench with TSF\-Domain Guide, Knowledge Structurer, and Experiment Workflower, which executes comparisons under shared, explicit evaluation protocols across agents and teams\.

Self\-Improving Agents\.STOPand theDarwin Gödel Machinerevise agent code through score\-based selection\[[44](https://arxiv.org/html/2610.40901#bib.bib44),[48](https://arxiv.org/html/2610.40901#bib.bib48)\],Voyageraccumulates skills\[[39](https://arxiv.org/html/2610.40901#bib.bib39)\], and memory systems retain information across tasks\[[46](https://arxiv.org/html/2610.40901#bib.bib46),[49](https://arxiv.org/html/2610.40901#bib.bib49)\]\. These methods improve a single agent\. The Recursive Improver ofTSFLabimproves the shared workbench instead: it checks each method against its paper and official code and merges verified methods, units, datasets, and findings, so research outputs from every task feed both the improvement and the maintenance of the workbench\.

## 3TSFLab

![Refer to caption](https://arxiv.org/html/2610.40901v1/02-overview.png)Figure 2:Overview of how an agent works with theTSFLabworkbench\.### 3\.1Problem Statement

Let𝒲\\mathcal\{W\}be a workbench, andΠ\\Pibe the set of agents that are compatible with𝒲\\mathcal\{W\}\. Eachπ∈Π\\pi\\in\\Picomprises its underlying model and harness\. The harness coordinates model and tool execution, manages context, and organizes internal workflows\. In contrast, the workbench𝒲\\mathcal\{W\}provides reusable domain knowledge, research procedures, and experimental tools without changing the model or harness:

π∗=Augment⁡\(π,𝒲\),π∈Π\.\\pi^\{\*\}=\\operatorname\{Augment\}\(\\pi,\\mathcal\{W\}\),\\qquad\\pi\\in\\Pi\.\(1\)Our goal is to improve valid task completion \(i\.e\., completing a research task exactly as required\) and reduce cost per completed task\. We describe the evaluation metrics in §[4\.1](https://arxiv.org/html/2610.40901#S4.SS1)\.

TSFLabsupports research through four modules, as shown in Fig\.[2](https://arxiv.org/html/2610.40901#S3.F2): \(a\)TSF\-Domain Guideis used by an agent for task procedures, \(b\)Knowledge Structurerselects methods, configures parameters, and reuses model units, \(c\)Experiment Workflowerexecutes the declared experiments, and \(d\)Recursive Improververifies new resources and outcomes for updating the shared workbench\.

### 3\.2TSF\-Domain Guide

TSF research requires domain knowledge of data integration, method implementation, and evaluation protocols \(§[1](https://arxiv.org/html/2610.40901#S1)\)\. TSF\-Domain Guide connects this knowledge to workbench procedures:entry instructionsintroduce conventions and module interactions, while 17 task\-specific skills provide steps, checks, and pitfalls as needed\.

For example, when benchmarking methods on a new dataset, a skill guides the agent to identify time fields, forecasting variables, sampling frequency, and evaluation settings, then register data through the workbench’s “add” interface and inspect partitioning and scaling through its “split” interface\. The agent does not need to maintain its own loading and splitting code\. The skill also requires checking whether forecasts are evaluated on one target or all variables, keeping metrics consistent across methods\. These choices establish the constraints for resource selection and configuration\. The entry instructions and task\-specific skills are detailed in App\.[A](https://arxiv.org/html/2610.40901#A1)\.

### 3\.3Knowledge Structurer

Table 1:Three levels of research knowledge\.After establishing task requirements, the agent uses Knowledge Structurer to select and configure resources\. It provides a knowledge card for each method, dataset, and unit, organized into summary, card, and source levels\. Summaries support screening; cards describe capabilities, parameters, interfaces, and limitations; sources provide papers, reference code, and detailed explanations\. This progressive disclosure\[[13](https://arxiv.org/html/2610.40901#bib.bib13),[47](https://arxiv.org/html/2610.40901#bib.bib47)\]lets agents retrieve the detail needed for each decision without rereading sources \(Tab\.[1](https://arxiv.org/html/2610.40901#S3.T1)\)\.

Method cards connect mechanisms to use cases and configuration\. For example, the card ofPatchTST\[[28](https://arxiv.org/html/2610.40901#bib.bib28)\]explains its channel\-independent design—modeling variables separately with shared weights—so an agent can compare it with methods that learn cross\-variable interactions\. It also states whether the implementation supports exogenous inputs or probabilistic forecasts, clarifying task compatibility\. Parameter descriptions define patch length and stride; existing settings and input\-window constraints help agents exclude unsuitable configurations and plan validation comparisons\. The workbench sets data\-dependent parameters, such as input channel count, from declared dependencies\. Dataset cards record data characteristics and split, scaling, and evaluation protocols\. Thus, agents screen resources by both forecasting mechanism and task compatibility, consulting source details when a configuration or implementation decision requires them\.

Knowledge Structurer also extracts common blocks, such asRevIN, series decomposition, and attention variants, into shared units that methods import, enabling shared maintenance; 263 methods reuse at least one unit\. Each unit has a card, and method cards record which units they use\. For example,RevINremoves an input window’s level and scale and restores them in the forecast\. Its card describes the shifts it addresses, parameters, and channel requirements, helping agents decide whether to reuse it in a new method\.

### 3\.4Experiment Workflower

With resources selected, Experiment Workflower executes agent\-declared experiments through a unified runner\. Each method’s specification defines parameter requirements, construction, and input and output interfaces, allowing the runner to instantiate the model and connect it to shared loading, training, and evaluation routines\. Presets supply training settings, dataset protocols, and architecture parameters for reuse across experiments\.

Agents useComposeto combine presets,Overrideto change settings, andSweepto expand methods or values\. For new\-data benchmarking, an agent combines dataset and method presets, fixes the lookback, horizon, and training budget, and sweeps methods and seeds\. The runner expands configurations into a run matrix, then checks configuration validity, data, model compatibility, environments, and budgets\. It returns problems before training, connecting the agent’s declaration to shared training and evaluation\.

Shared routines split data as configured, fit scaling on training data only, select checkpoints on validation data, and compute metrics consistently\. The runner schedules runs within resources and budgets, preserving status, checkpoints, and prior attempts\. Recovery checks configuration, data, and code fingerprints, skips completed runs, and resumes failed runs from their last complete epoch\. Status, wait, cancel, and resume interfaces reduce manual management of scripts, processes, and logs\.

The runner returns metrics, training time, peak GPU memory, status, and artifact paths, archiving configurations, seeds, environments, and data versions\. Result summaries compare methods only under the same data processing, training, and evaluation protocol, reporting seed coverage and variation\. Recorded evidence supports subsequent comparisons, development, and review through Recursive Improver \(§[3\.5](https://arxiv.org/html/2610.40901#S3.SS5)\)\.

### 3\.5Recursive Improver

Here, self\-improvement means incorporating verified research outputs and fixes into the workbench’s shared resources and workflows so later agents can reuse them\. For example, evaluation problems found during research sessions lead to revised guidance and runner checks; later sessions inherit these repairs\.

Recursive Improver brings external resources and agent\-produced outcomes back into the shared workbench through checks for usability, complexity, and consistency \(Fig\.[2](https://arxiv.org/html/2610.40901#S3.F2)d\)\. Failed checks prompt revision and rechecking before integration, supporting workbench maintenance\.

Table 2:Main results usingCodex CLIwith GPT\-6 Luna\. Costs are normalized by completed outcomes\. Best results are bold; second\-best are underlined\. Metric definitions are in App\.[D\.6](https://arxiv.org/html/2610.40901#A4.SS6)\.Usability\.Continuous integration \(CI\) checks basic operations\. Method tests cover construction, a training step, input and output requirements, and saving and loading; dataset checks cover schema and evaluation protocols; experiment records require complete configurations and runner\-produced results\. They also confirm required card fields and agreement between method specifications and presets, exposing integration errors\.

Complexity\.CI checks function\-level cyclomatic complexity and method size\. Cyclomatic complexity counts independent execution paths through a function; method size counts lines of code\. These flag implementations that are harder to understand, test, and maintain\. Code above the limits requires simpler logic, smaller functions, or shared units, then renewed runtime and behavioral checks\.

Consistency\.An independent read\-only review agent checks whether cards describe implementations and whether implementations preserve the defining mechanisms in papers and pinned official code, using papers alone when code is unavailable\. For example, it checks for cross\-variable computation in a method described as channel\-independent\. Paper–code setting differences and implementation choices are recorded\. For extracted units, automated tests compare parameters, outputs, and input gradients before and after extraction, checking model state, forward computation, and backpropagation behavior on reference cases\.

Accepted resources and verification records enter the workbench to support subsequent benchmarking, reproduction, and method development\. Additions and fixes from these tasks undergo the same checks, forming a shared update cycle that reduces repeated collection, implementation, and setup\. Consequently, a session can contribute a reusable method, unit, or workflow repair alongside its experimental results\.

## 4Experiments

Figure 3:Research\-task quality\. \(a\) Reproduction MSE error \(log scale; lower is better\); labels show shares within 10% \(96 scheduled\)\. \(b,c\) Development test MSE versus time; bubble areas and labels show fresh tokens\.We report completion, cost, and code changes \(§[4\.2](https://arxiv.org/html/2610.40901#S4.SS2)\), ablations \(§[4\.3](https://arxiv.org/html/2610.40901#S4.SS3)\), and coverage and maintenance \(§[4\.4](https://arxiv.org/html/2610.40901#S4.SS4)\)\.

### 4\.1Experimental Setup

Baselines\.We compareTSFLabwith two TSF codebases and with no workbench: Time\-Series\-Library \(TSLib\)\[[40](https://arxiv.org/html/2610.40901#bib.bib40)\], a model library, andTFB\[[31](https://arxiv.org/html/2610.40901#bib.bib31)\], a benchmarking framework\. Without a workbench, the agent starts from an empty directory with PyTorch\. All main settings useCodex CLIwith GPT\-6 Luna and identical prompts, data, network access, and budgets; Tab\.[3](https://arxiv.org/html/2610.40901#S4.T3)varies the agent\.

Research Tasks\.\(i\) Paper reproduction: the agent reproduces the reported results of a recent forecasting paper that has no official code and is absent from all compared codebases \(RhyMix\[[36](https://arxiv.org/html/2610.40901#bib.bib36)\], StateFlow\[[12](https://arxiv.org/html/2610.40901#bib.bib12)\], AdaRDiff\[[19](https://arxiv.org/html/2610.40901#bib.bib19)\], and CoRe\[[23](https://arxiv.org/html/2610.40901#bib.bib23)\]; App\.[D](https://arxiv.org/html/2610.40901#A4)\)\. \(ii\) Benchmarking on new data: given two new datasets, ERCOT and Openmeteo, the agent benchmarks eight methods under a common protocol: three established baselines \(DLinear\[[45](https://arxiv.org/html/2610.40901#bib.bib45)\],PatchTST\[[28](https://arxiv.org/html/2610.40901#bib.bib28)\], andiTransformer\[[24](https://arxiv.org/html/2610.40901#bib.bib24)\]\), two alignment and retrieval methods \(TimeAlign\[[15](https://arxiv.org/html/2610.40901#bib.bib15)\]and GTR\[[5](https://arxiv.org/html/2610.40901#bib.bib5)\]\), and three methods for temporal heterogeneity and embedding enhancement \(TimeMosaic\[[8](https://arxiv.org/html/2610.40901#bib.bib8)\], PHAT\[[26](https://arxiv.org/html/2610.40901#bib.bib26)\], and EMAformer\[[50](https://arxiv.org/html/2610.40901#bib.bib50)\]\)\. \(iii\) Method development pilot: on ETTh1 and Weather, the agent searches for a forecasting method using validation data, then evaluates its final selection once on test data \(App\.[D](https://arxiv.org/html/2610.40901#A4)\)\.

Evaluation\.Tab\.[2](https://arxiv.org/html/2610.40901#S3.T2)reports completion, cost, and added code\. Completed counts, for reproduction, sessions whose results pass review \(four papers×\\timesthree attempts\); for benchmarking, method results that are faithfully implemented and evaluated under the protocol \(eight methods×\\timestwo datasets×\\timesthree attempts\)\. Method development results are shown in Fig\.[3](https://arxiv.org/html/2610.40901#S4.F3)b,c, using test MSE, session time, and fresh tokens\. Time, Tokens \(fresh tokens, excluding cache reads\), and Calls \(agent tool calls\) sum all attempts, including failures, and divide by the completed count\.Δ\\DeltaLOC,Δ\\DeltaFuncs, andΔ\\DeltaCC are the net code lines, added functions, and added cyclomatic complexity per session\. Details are in App\.[D\.6](https://arxiv.org/html/2610.40901#A4.SS6)\.

Table 3:Module and agent ablations\. The full workbench usesCodex CLIwith GPT\-6 Luna\. Best values are bold; second\-best distinct values are underlined, including ties\.
### 4\.2Main Results

Tab\.[2](https://arxiv.org/html/2610.40901#S3.T2)and Fig\.[3](https://arxiv.org/html/2610.40901#S4.F3)compareTSFLabwith the other settings on reviewed task completion, cost, and code changes\. Reproduction accuracy is evaluated separately by relative error to the reported MSE\. We highlight five observations\. \(1\)TSFLabcompletes the most tasks with the most accurate results\. It completes all 12 reproduction attempts and 42 of 48 benchmark results, the most of any setting\. Fig\.[3](https://arxiv.org/html/2610.40901#S4.F3)a shows that its reproductions are also the closest to the papers:TSFLabreproduces 73% of the 96 scheduled results within 10% of the paper’s MSE, against 21–46% for the other settings\. Completion and agreement with the reported MSE thus provide complementary views of reproduction quality\. \(2\)TSFLabalso costs less: each completed outcome takes fewer tokens and less time in reproduction, and fewer tokens and tool calls in benchmarking, consistent with agents using shared operations instead of rebuilding them\. \(3\) Its added code is simpler\. Benchmarking needs no code changes, since all eight methods are already integrated and Experiment Workflower runs them from configuration; in reproduction, agents add a similar number of lines but 15–29% fewer functions and 18–28% less added cyclomatic complexity than the other settings, consistent with reuse of existing units\. \(4\) An existing codebase alone is not enough\.TFBcompletes fewer outcomes than no workbench in both tasks, mostly because agents replace requested methods with substitutes\. A framework helps an agent only when the agent can extend it faithfully\. \(5\) The method\-development pilot shows similar benefits\. On ETTh1 and Weather \(Fig\.[3](https://arxiv.org/html/2610.40901#S4.F3)b,c\),TSFLabreaches the lowest test MSE with less time and fewer fresh tokens than every other setting; details are in App\.[D](https://arxiv.org/html/2610.40901#A4)\. Reproduction benefits from shared units while still requiring new model code; benchmarking reuses complete methods\. The latter therefore measures the combined benefit of catalog availability and shared execution\. The six unaccepted benchmark results show that integrated methods alone do not guarantee valid evaluation\. Zero added code measures implementation reuse; completion review establishes whether experiments follow the declared evaluation protocol\.

Figure 4:Workbench coverage and evolution\. \(a\) Top\-cited coverage\. \(b\) Median/max model lines\. \(c\) Functions per method, initial \(0%\) to latest \(100%\)\. \(d\) Intake tokens including cache reads\.
### 4\.3Ablation Study

Tab\.[3](https://arxiv.org/html/2610.40901#S4.T3)reports two groups of ablations\. In \(a\), we remove one module at a time on four reproduction tasks with three attempts each: the TSF\-Domain Guide \(entry instructions and skills\), the Knowledge Structurer \(cards and sources\), or the Experiment Workflower \(runner and experiment commands\)\. Recursive Improver is examined through maintenance and refactoring outcomes in §[4\.4](https://arxiv.org/html/2610.40901#S4.SS4)\. In \(b\), we replace the agent:Claude Codewith Sonnet 5\.5, andCodex CLIwith GPT\-6 Sol\.

From Tab\.[3](https://arxiv.org/html/2610.40901#S4.T3)\(a\), removing any module lowers completion, and each fails differently\. Without the guide, completion drops the most: agents write their own data loaders, and every rejected attempt uses wrong ETTh1 split boundaries, so the guide mainly keeps agents on the evaluation protocol\. Without the Knowledge Structurer, cost stays close to the full workbench, but agents reuse fewer units and add more code and complexity\. Without the Experiment Workflower, agents write their own training and evaluation code; they add the most code, and their errors move into evaluation, such as tuning on test scores and averaging batch errors\. Shared execution thus removes errors that finite results alone do not reveal\. Similar token use without the Knowledge Structurer, despite lower completion, shows that token cost alone cannot measure reuse\.

Tab\.[3](https://arxiv.org/html/2610.40901#S4.T3)\(b\) highlights two observations\. \(1\) Stronger agent configurations reduce interaction, but not monetary cost\. Compared with GPT\-6 Luna, GPT\-6 Sol and Sonnet 5\.5 reduce tool calls by 48\.5% and 51\.1%, and fresh tokens by 9\.2% and 55\.1%, respectively\. Under a common fresh\-token mix, list prices imply estimated costs of 18\.2×\\timesand 9\.0×\\timesLuna’s, excluding cache reads \(App\.[D\.7](https://arxiv.org/html/2610.40901#A4.SS7)\)\.

\(2\) The workbench lets a lightweight agent match the stronger configurations in completion\. Luna and Sol complete all 12 attempts; Sonnet completes 11\. On these tasks, stronger configurations mainly reduce interaction, while Luna achieves comparable completion at lower estimated cost\.

### 4\.4Further Analysis

Coverage\.TSFLabcontains 44 of the 50 most cited methods, compared with 29 across the three other codebases combined \(Fig\.[4](https://arxiv.org/html/2610.40901#S4.F4)a\)\. The catalog combines established baselines with recent methods that have official code, reducing the integration needed to begin a comparison\. This measure captures baseline availability; forecasting quality is evaluated through the research tasks\.

Compact implementations\.Broader coverage accompanies compact model implementations \(Fig\.[4](https://arxiv.org/html/2610.40901#S4.F4)b\)\. The median implementation has 144 lines, close toTSLib’s 145, while the largest has 505 lines, compared with 2,071 inTFB\. Shared units consolidate common computations, and method specifications handle construction and parameter requirements outside model logic\. These statistics describe model code; the codebases differ in method coverage and supported functionality\.

Maintenance through reuse\.As the catalog grows, extracting shared units helps control code per method \(Fig\.[4](https://arxiv.org/html/2610.40901#S4.F4)c\): one extraction reduced the average function count from 11\.9 to 8\.6\. A separate refactor reduced maximum function complexity from 20 to 9 while preserving bit\-identical parameters, outputs, and gradients on reference cases for all 323 methods\. Research sessions also expose reusable fixes\. Missing guidance, manual configuration, repeated log polling, and premature test access led to instruction and runner updates; all 17 initial usability failures were repaired before integration\. These cases illustrate how Recursive Improver turns recurring problems into shared improvements, although they do not isolate the effect of individual checks\.

Integration beyond model code\.Model implementation accounts for only 9% of method\-intake tokens \(Fig\.[4](https://arxiv.org/html/2610.40901#S4.F4)d\)\. Source reading, catalog search, and verification account for 28%, 27%, and 28%, respectively\. Efficient integration therefore requires reusing knowledge and verification evidence as well as code\. Knowledge cards retain these findings, helping later agents locate suitable methods and units without repeating the same investigation\. The mean intake uses 6\.1M tokens including cache reads; research\-task costs in Tab\.[2](https://arxiv.org/html/2610.40901#S3.T2)count fresh tokens and exclude cache reads\.

## 5Conclusion

We presentedTSFLab, a shared TSF research workbench that leaves the agent’s model and harness unchanged\. It combines domain guidance, structured knowledge, reusable units, execution, and verified resource updates\. Across the evaluated tasks, it improves task completion and token efficiency, and supports method development\. Ablations show complementary benefits from guidance, resource organization, and shared execution\. Retaining source knowledge, implementations, and experiment records allows new agents to reuse verified results and repairs\.TSFLabsupports cumulative TSF research across agents and teams, allowing verified improvements to remain available as research tasks, underlying models, and agent harnesses evolve over time\.

## Limitations

TSFLabcurrently focuses on time series forecasting\. Its guidance, resource descriptions, and experiment interfaces are designed for TSF methods and evaluation protocols, so their effectiveness in other research domains remains untested\. Extending the workbench would require adapting these resources and checks to the tasks and conventions of each domain\. Within the limited search time, method development has not yet produced a new method that substantially outperforms all leading forecasting models\. The reported improvements therefore demonstrate support for the evaluated development tasks rather than a broadly superior forecasting method\. More effective search strategies and experimentation over longer research cycles remain directions for future work\.

## Ethical Considerations

TSFLabrecords the sources, upstream licenses, and reference revisions of the methods it incorporates\. Datasets retain their original licenses, and data that cannot be redistributed are obtained from their providers; all data are public numeric time series without personal information\. Reviews document discrepancies between papers and code together with the chosen implementation, making these decisions open to inspection\. Experimental results should be interpreted under their recorded protocols; benchmark rankings do not establish that a method is best across all settings\. Coding agents execute code with network access, so their runs should remain sandboxed, with external actions requiring explicit authorization\. Agent sessions and model training also consume GPU time and API tokens;TSFLabreduces these costs by sharing implementations and lowering token use\. §[4\.2](https://arxiv.org/html/2610.40901#S4.SS2)reports the time and token cost of each setting, and App\.[D](https://arxiv.org/html/2610.40901#A4)gives the hardware and API prices used for these estimates\.

## References

- \[1\]Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo\. 2024\.GIFT\-Eval: A benchmark for general time series forecasting model evaluation\.In*NeurIPS Workshop on Time Series in the Age of Large Models*\.
- \[2\]Anthropic\. 2026\.Claude Code\.[https://github\.com/anthropics/claude\-code](https://github.com/anthropics/claude-code)\.
- \[3\]Andres M\. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D\. White, and Philippe Schwaller\. 2024\.Augmenting large language models with chemistry tools\.*Nature Machine Intelligence*\.
- \[4\]Lorenzo Brigato, Rafael Morand, Knut Joar Strømmen, Maria Panagiotou, Markus Schmidt, and Stavroula Mougiakakou\. 2026\.There are no champions in supervised long\-term time series forecasting\.*Transactions on Machine Learning Research*\.
- \[5\]Fanpu Cao, Lu Dai, Jindong Han, and Hui Xiong\. 2026\.Enhancing multivariate time series forecasting with global temporal retrieval\.In*ICLR*\.
- \[6\]Chao Chen, Karl Petty, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia\. 2001\.Freeway performance measurement system: Mining loop detector data\.*Transportation Research Record*\.
- \[7\]Estee Y\. Cramer, Evan L\. Ray, Velma K\. Lopez, Johannes Bracher, Andrea Brennen, et al\. 2022\.Evaluation of individual and ensemble probabilistic forecasts of COVID\-19 mortality in the United States\.*Proceedings of the National Academy of Sciences*\.
- \[8\]Kuiye Ding, Fanda Fan, Chunyi Hou, Zheya Wang, Lei Wang, Zhengxin Yang, and Jianfeng Zhan\. 2026\.TimeMosaic: Temporal heterogeneity guided time series forecasting via adaptive granularity patch and segment\-wise decoding\.In*AAAI*\.
- \[9\]Earendil Works\. 2026\.Pi: A minimal, extensible agent harness\.[https://github\.com/earendil\-works/pi](https://github.com/earendil-works/pi)\.
- \[10\]Shanghua Gao, Richard Zhu, Pengwei Sui, Zhenglun Kong, Sufian Aldogom, et al\. 2025\.ToolUniverse: An open platform for democratizing AI scientists\.*arXiv:2509\.23426*\.
- \[11\]Azul Garza and Renée Rosillo\. 2025\.TimeCopilot\.*arXiv:2509\.00616*\.
- \[12\]Haroon Gharwi, Yue Dai, and Kai Shu\. 2026\.StateFlow: Dual\-state recurrent modeling for long\-horizon time series forecasting\.*arXiv:2607\.00197*\.
- \[13\]Yifeng He, Yinzhe Zhao, Jicheng Wang, and Hao Chen\. 2026\.Is progressive disclosure all you need for long\-context agents?*arXiv:2607\.17598*\.
- \[14\]Tao Hong and Shu Fan\. 2016\.Probabilistic electric load forecasting: A tutorial review\.*International Journal of Forecasting*\.
- \[15\]Yifan Hu, Jie Yang, Tian Zhou, Peiyuan Liu, Yujin Tang, Rong Jin, and Liang Sun\. 2026\.Bridging past and future: Distribution\-aware alignment for time series forecasting\.In*ICLR*\.
- \[16\]Kexin Huang, Serena Zhang, Hanchen Wang, Yuanhao Qu, Yingzhou Lu, et al\. 2026\.Autonomous biomedical research with an artificial intelligence agent\.*Science*\.
- \[17\]Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec\. 2024\.MLAgentBench: Evaluating language agents on machine learning experimentation\.In*ICML*\.
- \[18\]Timothy Kassis, Vinayak Agarwal, Yuhuan He, Darshil Patel, and Aubrey M\. Brueckner\. 2026\.Scientific agent skills: A library of procedural knowledge for research agents\.*arXiv:2609\.00065*\.
- \[19\]Morad Laglil, Younes Hlal, Marouane El Hadari, Emilie Devijver, and Eric Gaussier\. 2026\.Learning to difference: Adaptive reversible differencing \(AdaRDiff\) for time series forecasting\.*arXiv:2608\.28134*\.
- \[20\]Remi Lam, Alvaro Sanchez\-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, et al\. 2023\.Learning skillful medium\-range global weather forecasting\.*Science*\.
- \[21\]Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn\. 2026\.Meta\-Harness: End\-to\-end optimization of model harnesses\.*arXiv:2603\.28052*\.
- \[22\]Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu\. 2018\.Diffusion convolutional recurrent neural network: Data\-driven traffic forecasting\.In*ICLR*\.
- \[23\]Xiaoyu Lin, Huiran Duan, Yining Liu, Zhixiang Wu, Chu Lin, and Lin Lu\. 2026\.CoRe: Coherence and relational alignment for multivariate time series forecasting\.In*International Conference on Neural Information Processing \(ICONIP\)*\.
- \[24\]Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long\. 2024\.iTransformer: Inverted transformers are effective for time series forecasting\.In*ICLR*\.
- \[25\]Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune\. 2026\.Towards end\-to\-end automation of AI research\.*Nature*\.
- \[26\]Jiaming Ma, Qihe Huang, Haofeng Ma, Guanjun Wang, Sheng Huang, et al\. 2026\.PHAT: Modeling period heterogeneity for multivariate time series forecasting\.In*ICLR*\.
- \[27\]Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, et al\. 2025\.MLGym: A new framework and benchmark for advancing AI research agents\.In*COLM*\.
- \[28\]Yuqi Nie, Nam H\. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam\. 2023\.A time series is worth 64 words: Long\-term forecasting with Transformers\.In*ICLR*\.
- \[29\]Kin G\. Olivares, Cristian Challu, Azul Garza, Max Mergenthaler Canseco, and Artur Dubrawski\. 2022\.NeuralForecast: User friendly state\-of\-the\-art neural forecasting models\.[https://github\.com/Nixtla/neuralforecast](https://github.com/Nixtla/neuralforecast)\.
- \[30\]OpenAI\. 2026\.Codex CLI\.[https://github\.com/openai/codex](https://github.com/openai/codex)\.
- \[31\]Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, et al\. 2024\.TFB: Towards comprehensive and fair benchmarking of time series forecasting methods\.*Proceedings of the VLDB Endowment*\.
- \[32\]Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, et al\. 2025\.Agent laboratory: Using LLM agents as research assistants\.In*Findings of EMNLP*\.
- \[33\]Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu\. 2020\.Financial time series forecasting with deep learning: A systematic literature review: 2005–2019\.*Applied Soft Computing*\.
- \[34\]Zezhi Shao, Fei Wang, Yongjun Xu, Wei Wei, Chengqing Yu, et al\. 2025\.Exploring progress in multivariate time series forecasting: Comprehensive benchmarking and heterogeneity analysis\.*IEEE Transactions on Knowledge and Data Engineering*\.
- \[35\]ChengAo Shen, Wenchao Yu, Fangyu Wu, Dongjin Song, Hanghang Tong, et al\. 2026\.MetaCaster: Meta\-harness\-optimized agent for end\-to\-end few\-shot learning of lightweight time series forecasters\.In*EMNLP*\.
- \[36\]Sumit Satishrao Shevtekar and Chandresh Kumar Maurya\. 2026\.RhyMix: A lightweight adaptive multi\-rhythm network for long\-term time series forecasting\.*arXiv:2607\.08234*\.
- \[37\]Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, et al\. 2025\.PaperBench: Evaluating AI’s ability to replicate AI research\.In*ICML*\.
- \[38\]THUML\. 2026\.Time\-Series\-Library: Maintenance notice\.[https://github\.com/thuml/Time\-Series\-Library](https://github.com/thuml/Time-Series-Library)\.
- \[39\]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar\. 2024\.Voyager: An open\-ended embodied agent with large language models\.*Transactions on Machine Learning Research*\.
- \[40\]Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Yong Liu, Chen Wang, Mingsheng Long, and Jianmin Wang\. 2026\.Deep time series models: A comprehensive survey and benchmark\.*IEEE Transactions on Pattern Analysis and Machine Intelligence*\.
- \[41\]Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long\. 2021\.Autoformer: Decomposition transformers with auto\-correlation for long\-term series forecasting\.In*Advances in Neural Information Processing Systems*\.
- \[42\]Weize Xu, Erwin Poussi, Quan Zhong, Zehua Zeng, Christopher Zou, et al\. 2026\.PantheonOS: An evolvable multi\-agent framework for automatic genomics discovery\.*bioRxiv preprint*\.
- \[43\]Justin Young\. 2025\.Effective harnesses for long\-running agents\.[https://www\.anthropic\.com/engineering/effective\-harnesses\-for\-long\-running\-agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents)\.
- \[44\]Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai\. 2024\.Self\-taught optimizer \(STOP\): Recursively self\-improving code generation\.In*COLM*\.
- \[45\]Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu\. 2023\.Are Transformers effective for time series forecasting?In*AAAI*\.
- \[46\]Guilin Zhang, Wei Jiang, Xiejiashan Wang, Aisha Behr, Kai Zhao, Jeffrey Friedman, Xu Chu, and Amine Anoun\. 2026a\.Adaptive memory admission control for LLM agents\.In*ICLR Workshop on Memory for LLM\-Based Agentic Systems \(MemAgents\)*\.
- \[47\]Guilin Zhang, Kai Zhao, Priyanka Mudgal, Waleed Ammar, Xiquan Cui, Xu Chu, and Alet Blanken\. 2026b\.Report: Progressive disclosure of agent skills\.*arXiv:2609\.35692*\.
- \[48\]Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune\. 2026c\.Darwin Gödel Machine: Open\-ended evolution of self\-improving agents\.In*ICLR*\.
- \[49\]Yan Zhang and Shibo Li\. 2026\.ConsistencyGate: Preventing memory contamination in LLM agents via self\-consistency admission control\.*arXiv:2607\.22962*\.
- \[50\]Zhiwei Zhang, Xinyi Du, Xuanchi Guo, Weihao Wang, and Wenjuan Han\. 2026d\.EMAformer: Enhancing Transformer through embedding armor for time series forecasting\.In*AAAI*\.
- \[51\]Haokun Zhao, Xiang Zhang, Jiaqi Wei, Yiwei Xu, Yuting He, Siqi Sun, and Chenyu You\. 2025\.TimeSeriesScientist: A general\-purpose AI agent for time series analysis\.*arXiv:2510\.01538*\.
- \[52\]Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang\. 2021\.Informer: Beyond efficient Transformer for long sequence time\-series forecasting\.In*AAAI*\.

## Appendix AFramework Structure and Domain Guide

### A\.1Framework Structure

The workbench separates agent decisions from shared execution\. The agent reads guidance, selects resources from the catalog, declares an experiment, and interprets its records\. Four modules support this sequence:

1. 1\.TSF\-Domain Guide:entry instructions and skills specify workflows, protocol checks, and pitfalls\.
2. 2\.Knowledge Structurer:cards record the interfaces, configuration rules, and source evidence of each method, dataset, and shared unit in the catalog\.
3. 3\.Experiment Workflower:presets and method specifications connect each configuration to the shared loading, training, evaluation, and recovery routines\.
4. 4\.Recursive Improver:usability, complexity, and consistency checks screen new resources and repairs\.

The resulting flow is guidance→\\rightarrowresource selection→\\rightarrowconfiguration→\\rightarrowexecution→\\rightarrowreview→\\rightarrowshared updates\. The repository keeps models insrc/tsflab/models/<model\>/, units inmodels/\_components/, and workflows in\.agents/skills/\. Cards describe resources; specifications, configurations, and tests define runtime behavior\.

### A\.2Entry Instructions and Skills

Every session starts from one harness\-neutral entry file,AGENTS\.md\(Claude Code reads the same file throughCLAUDE\.md\)\. It states the workbench conventions, module interactions, resource access, and verification requirements\. Task\-specific skills provide the steps, checks, and common pitfalls of each workflow and are loaded as needed\. For new\-data benchmarking, these procedures guide the agent through identifying data fields and evaluation settings, adding the dataset, and inspecting its splits and scaling before configuring experiments\. Listingsummarizes the entry instructions used in the benchmarking snapshot\.

Condensed entry instructions fromAGENTS\.md\.

Workflows:\.agents/skills/;tasks:\.agents/tasks/;

contracts:\.agents/STANDARDS\.md\.

\-Keepmodelsinsrc/tsflab/models/<model\>/\.

\-Putpaper\-neutralunitsinmodels/\_components/\.

\-Eachresourcehascard\.toml,README\.md,andoptionalreference\.md\.

\-Verifysourceclaims;pinrevisionsandrecorddifferences\.

\-Recordexecutedmodeladmissionwith‘tsfmodelverify‘\.

\-Startwith‘tsfcatalog‘and‘tsfcatalogsearch‘\.

\-Read‘tsfcatalogshow<name\>‘;deepenwith\-\-depth2\|3\.

\-Use‘tsfcatalogmatch<dataset\>‘forresourceselection\.

\-Inspectsplitsandscalingwith‘tsfdatasplits<dataset\>‘\.

\-Registernewdatawith‘tsfdataadd‘;executewith‘tsfrun‘\.

\-Theagentownsdecisions;APIsownexecutionguarantees\.

uvsync\-\-frozen\-\-python3\.12

uvruntsfrepocheck\-\-audit

uvruntsfmodelverify\-\-changed

The 17 skills by module: data \(add\-dataset,inspect\-dataset\), models \(add\-model,integrate\-foundation\-model,curate\-components,discover\-papers\), experiments \(setup\-environment,run\-experiment,diagnose\-experiment,reproduce\-paper\-results,analyze\-results\), release \(submit\-results,forecast\-realtime\-round,publish\-weights\), autoresearch \(run\-autoresearch\), and maintenance \(audit,handle\-contribution\)\.

## Appendix BKnowledge Cards

Cards support summary, card, and source access\. These examples retain the TOML and Markdown formats of the benchmarking snapshot, shortened where marked\. Structured fields, usage guidance, and source evidence are stored separately; automatic checks and read\-only review validate fields and descriptions\.

### B\.1Method Card: PatchTST

PatchTST:card\.toml\.

kind="model"

name="PatchTST"

fits=\["long\-lookback","long\-horizon",

"weak\-cross\-correlation"\]

fidelity="reference\-checked"

\[paper\]

title="ATimeSeriesisWorth64Words:\.\.\."

venue="ICLR2023"

\[code\]

url="https://github\.com/yuqinie98/PatchTST"

revision="204c21ef"

license="Apache\-2\.0"

\[composition\]

normalization="component:revin"

temporal="local:patch\-transformer\-encoder"

channel="local:channel\-independent\-shared\-weights"

head="local:flatten\-linear\-head"

\[data\_params\]

enc\_in=\{from="channels"\}

patch\_len=\{from="seq\_len",rule="atmostseq\_len"\}

\[admission\]

status="passed"

description:Channel\-independentTransformeroveroverlappingpatchesofeachseries,

withaflattenheadandRevIN\.Useforlong\-horizonmultivariateforecastingwith

longlookbacks;notfortasksneedingcross\-channelinteraction,exogenousinputs,

orprobabilisticoutput\.

\-Longhorizonswherealonglookbackhelps:patchingcutsattentioncost\.

\-Notwhencross\-channelinteractioncarriesthesignal\(nochannelmixing\)or

calendarmarksmatter\(ignored\)\.

\-enc\_inmustequalthedata’schannelcount\.

\-patch\_lenmustnotexceedseq\_len\.

\-Clean\-roomimplementation;officialcodeforreferenceonly\.

\-Noself\-supervisedpretrainingorresidualattention\.

README\.md\(shortened\)
### B\.2Shared\-Unit Card: RevIN

revin:card\.toml\.

kind="component"

name="revin"

slot="normalization"

role="Removeseachwindow’slevelandscalebeforethe

modelandrestoresthemontheforecast\."

fits=\["non\-stationary","train\-val\-level\-shift",

"train\-val\-scale\-shift"\]

input="\[batch,time,channels\]"

output="sameshapeasinput"

origin="RevIN,Kimetal\.,ICLR2022"

origin\_models=\["patchtst"\]

norm:mu=mean\_t\(x\),sigma=sqrt\(var\_t\(x\)\+eps\),

y=\(x\-mu\)/sigma,theny\*gamma\+betaifaffine\.

denorm:invertstheaffine,thenz\*sigma\+muwiththe

statisticscachedbythelatestnormcall\(detached\)\.

Levelorscaledriftsacrosswindowsorbetweentrainingandevaluation\.Notwhen

outputchannelsdifferfrominput,orwhenstatisticsmustcarrygradients\.

RevIN\(num\_features,eps=1e\-5,affine=True,

subtract\_last=False,enabled=True\)

forward\(values,mode\)withmodein\{"norm","denorm"\}

README\.md\(shortened\)
### B\.3Dataset Card: ETTh1

etth1:card\.toml\.

kind="dataset"

name="etth1"

domain="energy"

characteristics=\["non\-stationary","within\-train\-drift",

"regime\-changes"\]

\[source\]

url="https://github\.com/zhouhaoyi/ETDataset"

license="CC\-BY\-ND\-4\.0"

\[shape\]

frequency="hourly\(1h\)"

length=14400

channels=7

target="OT"

\[protocol\]

split="6:2:2"

seq\_lens=\[96,336,512\]

pred\_lens=\[96,192,336,720\]

\-Truncation:onlythefirst14,400rows\(20months\)areused;6:2:2reproducesthe

12/4/4\-monthsplit\(7:1:2resultsarenotcomparable\)\.

\-drop\_last:testkeepsitslastpartialbatch,sometricscovereverywindow;code

thatdropsitreportsdifferentnumbers\.

\-Distributionshift:OTmean/stdis17\.13/9\.18\(train\)vs\.4\.85/3\.15\(test\)\.

README\.md\(shortened\)

## Appendix CExperiment Workflower

### C\.1Declarative Configurations

An experiment is a TOML file that composes presets: a base configuration \(training, optimizer, checkpointing, metrics\), a dataset preset \(path, target, scaling, split ratio\), and a method preset \(architecture parameters\), followed by overrides and a sweep\. For methods already integrated into the workbench, agents declare the experiment without writing a separate training script\. The runner resolves the configuration into one run per sweep cell \(Listing\)\. Parameters that a method card declares as following the data \(\[data\_params\], e\.g\.enc\_in = \{ from = "channels" \}\) are set from the dataset at run time, reducing manual changes when reusing a method preset on new data\. Each method specification defines parameter requirements, construction, and input and output interfaces, connecting the model to the shared loader, trainer, and evaluator\.

A run configuration:PatchTSTon ETTh1 at four horizons and two seeds\.

extends=\["\.\./base\.toml",

"\.\./datasets/etth1\.toml",

"\.\./models/PatchTST\.toml"\]

\[task\]

seq\_len=336

\[sweep\]

experiment\.random\_seed=\[0,1\]

\[sweep\.task\]

pred\_len=\[96,192,336,720\]

### C\.2Command Reference

Commands are grouped by workflow below\. Angle brackets denote placeholders; bracketed options are optional\. The examples document the interfaces used by the workbench, rather than a single shell script\.

Resource discovery and dataset commands\.

tsfcatalog

tsfcatalogsearch<query\>

tsfcatalogshow<name\>\[\-\-depth2\|3\]

tsfcatalogmatch<dataset\>

tsfdataadd

tsfdataanalyze

tsfdatasplits<dataset\>

Experiment execution and recovery commands\.

tsfrun<config\>\-\-dry\-run

tsfrun<config\>

tsfrunstatus<dir\>

tsfrunwait<dir\>

tsfruncancel<dir\>

tsfrunresume<dir\>

Research, result comparison, and verification commands\.

tsfresearchstart

tsfresultreport

tsfmodelverify\-\-changed

tsfrepocheck\-\-audit

### C\.3Budgets and Research Rounds

An optional execution policy sets budgets \(runs, parallel jobs, wall time, GPU\-hours\) and resources \(GPUs per run, free memory, threads per run\); the scheduler leases GPUs exclusively and counts all attempts against the budget\. Runs attached to a research round \(tsf research start\) score the validation split unless overridden, for validation\-based search\. The method\-development protocol is specified in App\.[D](https://arxiv.org/html/2610.40901#A4)\.

### C\.4Records and Comparable Results

Every run writes a record with its resolved configuration, metrics, checkpoints, and timing, and appends a row to a results table that carries protocol and model\-variant fingerprints \(data content, task, evaluation, and training settings\)\.tsf result reportranks methods only within one protocol and reports seed coverage and spread\. The same records feed the publicTSFLableaderboard, which ranks methods on fixed benchmark protocols and on real\-time tracks that are forecast before the targets are observed \([https://huggingface\.co/spaces/Diaugeia/TSFLab](https://huggingface.co/spaces/Diaugeia/TSFLab)\); the two benchmarking datasets of App\.[D](https://arxiv.org/html/2610.40901#A4)are static exports from these tracks and are evaluated with chronological splits\.

## Appendix DExperimental Setup and Evaluation

### D\.1Shared Setup

Codebases are pinned \(TSLibat 4e938a1,TFBat a0087ed\), and every setting receives the same prompt, task file, and raw data\. A session runs the agent headless with no additional user instructions beyond the task prompt, no persistent memory, plug\-ins, or MCP servers, on dedicated NVIDIA RTX 6000 Ada GPUs with network access\. Each task uses a fixedTSFLabsnapshot, taken in the order reproduction, benchmarking, and method development; the released code \(TSFLab1\.0\.0\) is the latest version\.

### D\.2Datasets

Reproduction and development\.ETTh1\[[52](https://arxiv.org/html/2610.40901#bib.bib52)\]is the hourly energy dataset used in both tasks\. We use its first 14,400 rows and seven channels, with 8,640/2,880/2,880 training, validation, and test rows\. Traffic\[[41](https://arxiv.org/html/2610.40901#bib.bib41),[6](https://arxiv.org/html/2610.40901#bib.bib6)\]provides the second reproduction dataset; both reproduction datasets are evaluated at horizons 96, 192, 336, and 720\. Weather\[[41](https://arxiv.org/html/2610.40901#bib.bib41)\]supplies the second development task, with 21 channels and a chronological 7:1:2 split\. Development uses input 336 and horizon 192 on both datasets\. Model\-specific reproduction input lengths and training requirements follow\.

New\-data benchmarking\.ERCOT111[https://www\.ercot\.com](https://www.ercot.com/)contains hourly load for eight weather zones, covering 2024\-10\-01 through 2026\-09\-30 \(17,520 rows\)\. Openmeteo222[https://open\-meteo\.com](https://open-meteo.com/)contains hourly 2 m temperature for 82 cities, covering 2025\-10\-01 through 2026\-09\-30 \(8,760 rows\)\. Both are fixed CSV exports from the workbench’s real\-time tracks\. These experiments use historical chronological splits, rather than the live forecasting protocol of the original tracks\. All channels are forecast with input 168 and horizon 24; scaling is fitted on the training partition only, and metrics use the normalized scale\.

### D\.3Paper Reproduction

Papers are arXiv forecasting papers first submitted on or after 2026\-06\-01\. We kept papers that have no official code, are absent from every compared codebase, report per\-horizon MSE and MAE on ETTh1 and Traffic at horizons 96, 192, 336, and 720 under the standard protocol, give enough training detail, and train all four horizons in under one GPU\-hour\. Each session has four hours, one GPU, and a ready environment\.

- •RhyMix\[[36](https://arxiv.org/html/2610.40901#bib.bib36)\]: input 96; cyclic and multi\-scale convolution paths with gated forecasting heads; Huber loss\. Each cell uses five seeds to match the paper’s five\-run means\. Batch size and gate sizes are not specified\.
- •StateFlow\[[12](https://arxiv.org/html/2610.40901#bib.bib12)\]: input 96; recurrent hidden and residual\-memory states with a chunk decoder\. Part of the cell is defined in an earlier paper; optimizer and two\-stage schedule are unspecified\.
- •AdaRDiff\[[19](https://arxiv.org/html/2610.40901#bib.bib19)\]: input 720; a learnable reversible residual transform around a linear backbone\. Training runs 30 residual\-loss then 10 forecast\-loss epochs\. The paper gives only the search grid\.
- •CoRe\[[23](https://arxiv.org/html/2610.40901#bib.bib23)\]: input 96; coherence and low\-rank channel\-alignment losses applied toiTransformer\. Loss weight, rank, and learning rate are unspecified\. It is related to the existing FreDF method, so the task tests whether agents reuse it\.

### D\.4Benchmarking on New Data

The task uses the ERCOT and Openmeteo exports described above\. The eight methods are three established baselines contained in all three codebases and the five most cited forecasting methods of 2026 venues thatTSFLabcontains andTSLibandTFBdo not \(Semantic Scholar, 2026\-10\-09\):

- •DLineardecomposes the input into trend and seasonal parts and forecasts each with a linear layer\.PatchTSTapplies a channel\-independent Transformer to patches of each series;iTransformerinstead treats each variate’s input history as one token and attends across variates\.
- •TimeAligntrains a patch MLP with a future\-reconstruction branch and aligns history and future representations at local and global levels; inference uses the patch MLP alone\.
- •GTRretrieves segments from a learned full\-cycle memory and fuses them with the input by 2D convolution, followed by a residual MLP\.
- •TimeMosaicadapts patch size to the information density of each input region and uses segment\-specific prompts and linear heads for forecasting\.
- •PHATbuckets each variable by FFT\-derived phase and cycle, with positive–negative attention per bucket\.
- •EMAformeraugments variate tokens with channel, cycle\-phase, and joint channel–phase embeddings\.

References for these methods are listed in §[4\.1](https://arxiv.org/html/2610.40901#S4.SS1)\.

This task therefore measures the combined benefit of available method implementations and the experiment workflow; method availability differs across codebases\. The fixed protocol is:

- •Data\.Chronological 70/10/20 split; train\-only scaling; all channels; input 168 and horizon 24\.
- •Training\.Paper or official\-code architecture settings; at most ten epochs; validation early stopping with patience 3; batch size 32; seed 42\.
- •Evaluation\.Every test window \(stride 1\); MSE and MAE on the normalized scale\.
- •Budget\.Six hours per session, including environment installation\.

### D\.5Method Development Pilot

On ETTh1 and Weather, the agent searches for a method on validation data under a fixed protocol:

- •Data\.All channels; input 336; horizon 192; train\-only standardization; seed 2021\.
- •Training\.MSE loss; Adam, lr10−310^\{\-3\}; at most ten epochs; patience 3; best\-validation checkpoint\.
- •Selection\.Validation\-only candidate selection; the frozen method’s saved checkpoint is tested once\.
- •Budget\.90 minutes; two GPUs; at most 12 fits and four search iterations per session\.

### D\.6Evaluation Metrics

Completion\.For reproduction and the ablations, Completed counts accepted sessions out ofNN\(x/Nx/N\)\. A script first checks that every required output exists\. Two read\-only reviewer agents then answer a fixed checklist: where the split is made, where scaling statistics are fitted, which data select checkpoints, whether every test window is evaluated, and whether reported values come from executed runs; a third reviewer decides disagreements\. A session is accepted if it covers the complete requested matrix with executed metrics, the prescribed splits, train\-only scaling, the common evaluation over all windows, and no test\-informed configuration selection; agreement with the paper’s MSE is evaluated separately and is not an acceptance threshold\. For benchmarking, Completed counts accepted method–dataset results out of 48 \(eight methods, two datasets, three attempts\)\. A result is accepted if its MSE and MAE are finite, the reviews \(after any tie\-break\) support the protocol, the code implements the method’s defining mechanisms, and the metrics have execution evidence in the run records\.

Accuracy\.ForWWtest windows, horizonHH, andCCchannels on the normalized scale,

MSE=1W​H​C​∑w,h,c\(y^w,h,c−yw,h,c\)2,MAE=1W​H​C​∑w,h,c\|y^w,h,c−yw,h,c\|\.\\mathrm\{MSE\}=\\frac\{1\}\{WHC\}\\sum\_\{w,h,c\}\\big\(\\hat\{y\}\_\{w,h,c\}\-y\_\{w,h,c\}\\big\)^\{2\},\\quad\\mathrm\{MAE\}=\\frac\{1\}\{WHC\}\\sum\_\{w,h,c\}\\big\|\\hat\{y\}\_\{w,h,c\}\-y\_\{w,h,c\}\\big\|\.The relative error of a reproduced result isϵ=100​\|MSErun−MSEpaper\|/MSEpaper\\epsilon=100\\,\|\\mathrm\{MSE\}\_\{\\mathrm\{run\}\}\-\\mathrm\{MSE\}\_\{\\mathrm\{paper\}\}\|/\\mathrm\{MSE\}\_\{\\mathrm\{paper\}\}\.

Cost\.For a setting with attemptss∈Ss\\in Sandnncompleted outcomes,

Time=1n​∑s∈STs,Tokens=1n​∑s∈SKs,Calls=1n​∑s∈SAs,\\mathrm\{Time\}=\\frac\{1\}\{n\}\\sum\_\{s\\in S\}T\_\{s\},\\quad\\mathrm\{Tokens\}=\\frac\{1\}\{n\}\\sum\_\{s\\in S\}K\_\{s\},\\quad\\mathrm\{Calls\}=\\frac\{1\}\{n\}\\sum\_\{s\\in S\}A\_\{s\},whereTsT\_\{s\}is wall time,AsA\_\{s\}the number of agent tool calls, andKsK\_\{s\}the fresh tokens of attemptss: non\-cached input, cache\-write input, and output, excluding cache reads\.SSincludes failed attempts\. For reproduction,nncounts accepted sessions; for benchmarking, it counts accepted method–dataset results\. These denominators differ, so costs are compared within each task\. The method\-intake analysis separately reports tokens including cache reads, as stated in the main text\.

Code changes\.LetF0F\_\{0\}andF1F\_\{1\}be the functions of the initial and final source of a session andCC⁡\(f\)\\mathrm\{CC\}\(f\)the cyclomatic complexity offf\(Radon 6\.0\.1\), withCC0​\(f\)=0\\mathrm\{CC\}\_\{0\}\(f\)=0forf∉F0f\\notin F\_\{0\}\. Cyclomatic complexity counts independent execution paths through a function; conditionals and loops typically increase it\. Then

Δ​LOC=L\+−L−,Δ​Funcs=\|F1∖F0\|,Δ​CC=∑f∈F1max⁡\(0,CC1​\(f\)−CC0​\(f\)\),\\Delta\\mathrm\{LOC\}=L^\{\+\}\-L^\{\-\},\\quad\\Delta\\mathrm\{Funcs\}=\|F\_\{1\}\\setminus F\_\{0\}\|,\\quad\\Delta\\mathrm\{CC\}=\\sum\_\{f\\in F\_\{1\}\}\\max\\big\(0,\\mathrm\{CC\}\_\{1\}\(f\)\-\\mathrm\{CC\}\_\{0\}\(f\)\\big\),whereL\+L^\{\+\}andL−L^\{\-\}are added and removed Python source lines\. The source includes experiment scripts and excludes official repositories cloned into the workspace, installed environments, and generated run records\. Code metrics are averaged over sessions within each task and then across tasks with equal weight\.

### D\.7API Price Assumptions

The cost comparison in §[4\.3](https://arxiv.org/html/2610.40901#S4.SS3)uses API list prices checked on October 13, 2026 \(USD per million tokens\)\. GPT\-6 rates are short\-context\. Sources: the official[GPT\-6 Luna](https://developers.openai.com/api/docs/models/gpt-6-luna),[GPT\-6 Sol](https://developers.openai.com/api/docs/models/gpt-6-sol), and[Sonnet 5\.5](https://platform.claude.com/docs/en/models/sonnet-5-5/overview)model pages\.

Sonnet’s cache\-write rate above is for five\-minute caching; one\-hour writes cost $4\.00 per million tokens\. For non\-cached inputII, cache\-write inputWW, and outputOO, the estimated fresh\-token charge is

C^fresh=pI​I\+pW​W\+pO​O106\.\\widehat\{C\}\_\{\\mathrm\{fresh\}\}=\\frac\{p\_\{I\}I\+p\_\{W\}W\+p\_\{O\}O\}\{10^\{6\}\}\.Cache reads are excluded from fresh tokens and this estimate; addingpR​R/106p\_\{R\}R/10^\{6\}gives the token charge includingRRcache\-read tokens\. The comparison assumes the same fresh\-token composition across agents, using five\-minute cache writes for Sonnet\. Each fresh\-token rate for Sol and Sonnet is then 20 times Luna’s\. Using Tab\.[3](https://arxiv.org/html/2610.40901#S4.T3), the estimated ratios are20​\(0\.394/0\.434\)≈18\.220\(0\.394/0\.434\)\\approx 18\.2and20​\(0\.195/0\.434\)≈9\.020\(0\.195/0\.434\)\\approx 9\.0, respectively\. These are normalized estimates, not invoices; actual charges depend on token composition, context tier, and caching\.