当终端智能体训练陷入停滞：揭秘数据生成与验证的难题

# When Terminal-Agent Training Stalls: Demystifying Data Generation and Verification Challenges
Source: [https://arxiv.org/html/2610.02405](https://arxiv.org/html/2610.02405)
Isabel Kurth††thanks:This work was done when Isabel was an intern at SAP\.Affiliation:SAP LabAffiliation:Palo Alto, CA 94304Email:[isabel\.kurth@t\-online\.de](mailto:)Xin CuiAffiliation:SAP LabAffiliation:Palo Alto, CA 94304Email:[xin\.cui@sap\.com](mailto:)Elin ParkAffiliation:SAP LabAffiliation:Palo Alto, CA 94304Email:[elin\.park@sap\.com](mailto:)Alexander SchaeferAffiliation:SAP LabAffiliation:Palo Alto, CA 94304Email:[alexander\.schaefer01@sap\.com](mailto:)Yaad OrenAffiliation:SAP LabAffiliation:Palo Alto, CA 94304Email:[yaad\.oren@sap\.com](mailto:)

###### Abstract

Using a frontier model like Claude Opus as a meta\-agent to generate terminal tasks and verifiers for RL training is increasingly common\. Yet a runnable Docker image and executable test suite do not guarantee a faithful end\-to\-end pipeline for terminal agent training\. We present a meta\-agent pipeline motivated by this gap, diagnosing three classes of failure: benchmark invalidity, harness brittleness, and reward misalignment\. Prompt redesign and context extension raise baseline solvability 5\.6×, but a 9B model saturates at 81\.3% mean pass@2 within 20 steps on Claude Opus\-generated tasks\. Adding hard tasks reduces mean pass@2 to 20\.6% without changing the training configuration, a strong evidence that the solvability band is model\-specific\. These findings demonstrate that meta\-agent reliability requires solvability\-band calibration, verifier audits, and infrastructure error accounting as first\-class evaluation criteria, not post\-hoc diagnostics\.

## 1Introduction

Autonomous terminal agents, as models that issue shell commands, read outputs, and iterate to complete engineering tasks, are a promising direction for automating software operations\. Reinforcement learning is a natural training approach: either a binary pass\-or\-fail signal or multi\-state checks from a test suite, provides a clean reward without human labeling\. The bottleneck is environment construction\. Writing executable terminal tasks with correct initial states, verifiable final states, and graded rewards is labor\-intensive, and the resulting datasets are small\.

Gandhi et al\.[Gandhi et al\. \(2025\)](https://arxiv.org/html/2610.02405#bib.bib1)address environment construction with Endless Terminals, a pipeline that procedurally generates tasks without human annotation\. Their pipeline produces structurally valid tasks – the container boots, the test suite runs, and this process discards roughly half of all generated candidates by OpenAI O3 model, one of the top models at that time\. This seemingly sparse reward would highly possibly make RL training difficult: even with 16 samples per task, most batches return all\-zero reward and produce no gradient\. Future work is left to quantify the impacts of this sparsity issue over training instability\.

In this work, we aim to bridge this gap through first anatomizing the challenges of data generation and verification in this brittle pipeline\. As a starter, we should identify*solvability band*: the range of per\-task pass rates where RL training can make progress\. On the extreme cases, too low \(near 0%\) and gradient signal is absent; too high \(near 100%\) and advantages collapse to zero\. Raising solvability is necessary but not sufficient, the model must still fail on some tasks at training time to learn from contrastive examples\.

Endless Terminals and Harbor

Endless Terminals[Gandhi et al\. \(2025\)](https://arxiv.org/html/2610.02405#bib.bib1)generates terminal\-agent tasks in five stages: task description and ground truth, initial\-state pytest, final\-state pytest, Apptainer container definition, and packaged task bundle\. Each stage is a batched LLM call conditioned on earlier outputs\. The pipeline requires no human annotation and scales with inference compute\. Structure\-wise, we preserve this five\-stage skeleton and replace the pipelines of data generation and curation, evaluation harness, and LLM providers\.

Harbor[Harbor Framework Team \(2026\)](https://arxiv.org/html/2610.02405#bib.bib2)is an open\-source benchmark framework for terminal agents\. It prescribes a fixed directory layout per task: an instruction file, atask\.tomlmetadata file, a Docker environment, a reference solution, and a pytest verifier that writes binary reward\. Harbor’s CLI builds task images, attaches an agent, drives terminal interaction, and collects results\. We adopt Harbor as our task format and evaluation harness\.

We make three contributions\.

1. 1\.Terminal Task DatasetWe generate 18,185 tasks with 8192\-token context windows from Claude Opus models,the largestto our best knowledge\.
2. 2\.Task Quality Improvements\.We design the generation prompts thoroughly around three dimensions: engineer voice \(casual, mid\-stream narration with deliberate imperfections\), anti\-leakage \(prohibiting restatement of verification criteria\), and domain coverage \(round\-robin balancing across eight task categories\)\. We replace syntactic complexity levels with three\-level cognitive difficulty and thread difficulty through every generation stage\.
3. 3\.Structural Validity Is Not Benchmark Validity\.We present an empirical study of a meta\-agent pipeline of automated terminal task generation and verification\. It demystifies three classes of meta\-agent failure: benchmark invalidity, harness brittleness, and reward misalignment\. It requires solvability\-band calibration, verifier audits, and infrastructure error accounting as first\-class evaluation criteria\.

## 2Acknowledgment

The authors gratefully acknowledge Kanishk Gandhi of Stanford University Human\-Centered Artificial Intelligence Institution, for his participation in constructive discussions that helped sharpen the analysis and presentation of this work\.

## 3Data Generation

In our five\-stage pipeline \(Figure[1](https://arxiv.org/html/2610.02405#S3.F1)\), stages 1–3 produce the task description, ground truth, initial\-state tests, and final\-state tests\. Stage 4 produces a Dockerfile to build the sandbox for simulation and verification of a terminal task\. Stage 5 writes a Harbor task directory instead of a flat bundle\.

1\. Task templateinstruction \+ truth2\. Initial statepytest3\. Final statepytest4\. Dockerfileenvironment5\. Harbor taskdirectoryPre\-agent validationinitial passes; final failsSolution samplingN attempts per taskSolvable≥1\\geq 1passing attemptUnsolved0 passing attemptsUse for RL training, reference solutions, and curriculum construction\.Triage: genuinely hard, ambiguous, impossible environment, or brittle test\.

Figure 1:Overall pipeline\. Stages 1–5 generate Harbor task directories\. Pre\-agent validation filters structural failures before rollout\. Solution sampling assigns empirical solvability labels, which control RL training signal\.Here are the details in each stage\.

### 3\.1From Apptainer to Docker and Harbor

The original pipeline targets Apptainer because SLURM clusters require daemonless unprivileged containers\. Our setting infuses Harbor, which expects a Dockerfile per task\. Stage 4’s prompt was rewritten around Docker\-specific constraints: heredoc handling, the absence ofsystemd, restrictions on Docker\-in\-Docker, non\-root user execution, and wrapperCMDscripts for tasks that need a service running at agent connect time\. The Harbor task directory layout is listed in Appendix[B](https://arxiv.org/html/2610.02405#A2)\. During training and evaluation, the verifier writes reward11when the final\-state pytest passes and00otherwise\. We follow this convention throughout\.

### 3\.2Task\-Template Prompt Redesign

The original prompt is approximately thirty lines and specifies a rubric for the<task\>and<truth\>XML blocks\. It produces two systematic failure modes\.

Worksheet Voice\.The model produces tasks that sound like graded assignments: numbered sub\-requirements, phrases such as “The output should…” and “Ensure that…,” and openings like “Please complete the following task\.” These patterns signal to the agent that it is being evaluated, not helping\.

Grader Leakage\.The model restates verification criteria inside the task body – for example, asking the agent to “produce a log of every change” when the verifier checks exactly that log\. Leakage collapses the task from problem\-solving to transcription\.

We redesign the prompt around three dimensions\.

Voice\.The narrator is an engineer: casual, and whom could be possibly wrong\. The prompt enumerates opening modes – trigger/debug, build, reverse engineering, recovery, migration, investigation, optimization – and requires the model to rotate across them\. Each task must contain at least one deliberate imperfection: a hedge, a possibly\-wrong guess, or a register cue\. Banned phrasings are listed verbatim so the model can self\-check before emitting\.

Anti\-leakage\.The prompt enumerates leakage behaviors: restating verification criteria, requesting artifacts whose only purpose is grading, enumerating sub\-requirements, and explaining the root cause in the task body\. The<truth\>section requires four items in order: initial state, expected final state, invariants, and anti\-shortcut guards\.

Domain Coverage\.To prevent the model from collapsing to a small attractor \(in our case, “Python data bug”\), the system message lists multiple domains and task types and instructs deliberate rotation\. A self\-check re\-asks three questions: did the diagnosis leak, does the task contain at least one imperfection, and does the first sentence drop the agent into a situation rather than announcing a project\.

The system message carries standing rules\. The user message offers task variation: a domain drawn from balanced category buckets, a fictitious narrator persona, and a difficulty level with level\-specific guidance\. We share the core prompts in Appendix[A](https://arxiv.org/html/2610.02405#A1)\.

### 3\.3Difficulty and Category Balancing

Previous work in[Gandhi et al\. \(2025\)](https://arxiv.org/html/2610.02405#bib.bib1)defines six complexity levels by command count, sampled uniformly and inserted as a bare phrase\. We think command count is a poor proxy for solve difficulty:chmod \+x fooand a one\-linesedbug fix are both one\-command tasks\. So we replace it with a three\-leveldifficultydimension defined in cognitive terms\. Easy tasks have a single obvious cause and a direct fix\. Medium tasks require investigation before action\. Hard tasks satisfy at least one of five criteria: misleading symptom, conflicting constraints, required domain knowledge, large state space, or reverse engineering\. Each level includes examples rather than a bare phrase\. Difficulty is threaded through every generation stage and written intotask\.toml, aligning with Harbor’s metadata schema\.

The core round\-robin sampling strategy applies across ten high\-level buckets enforces category balance:file\_ops,networking,services,security,debugging,data\_processing,git\_ops,system\_admin,build\_tools, andtext\_processing\. Without this, a single batched call produces correlated tasks: if the first task involvesgit rebase, several subsequent tasks do as well\.

### 3\.4Pre\-Agent Validation

After Stage 4 builds the Docker image, the pipeline runs the generated tests against the container before any agent rollout\. A task survives only when the initial\-state test passes and the final\-state test fails\. The initial\-pass requirement checks that the Dockerfile correctly seeds the world described in the ground truth\. Missing input files, wrong port bindings, or incorrect permissions are caught here rather than during rollout\.

This pre\-agent validation step is absent from the pipeline in[Gandhi et al\. \(2025\)](https://arxiv.org/html/2610.02405#bib.bib1), which discovers such mismatches only during solution sampling\.

### 3\.5Solution Generation

After task generation, a solution\-generation stage samples agent trajectories using Harbor’sTerminus2agent\. The companion scriptcollect\_harbor\_results\.pyscans Harbor job directories, groups trials by task, and writessolution\.jsonwith per\-task pass@k metrics\.

Solution generation assigns an empirical solvability label to each task\. Tasks with at least one passing attempt are candidates for RL training datasets\. Tasks with zero passing attempts require triage: they may be genuinely hard, underspecified, impossible, or over\-strictly graded\.

## 4Dataset Analysis

### 4\.1Solvability of Generated Tasks

We generate three datasets\. Table[1](https://arxiv.org/html/2610.02405#S4.T1)summarizes the three datasets, where category assignment strategy details are in Appendix[D](https://arxiv.org/html/2610.02405#A4)\. We identify 8 attempts being a sweet point for most solution generation, which is proved by figure[8](https://arxiv.org/html/2610.02405#A5.F8)in Appendix\.

The5\.6×5\.6\\timesimprovement in solvability fromV​1V1toV​2V2comes from two sources\. Longer context allows more complete environment specifications, reducing cases where the agent cannot infer the missing state\. The redesigned prompt reduces grader leakage and worksheet voice, producing tasks that an agent can approach as genuine engineering problems\.

Table 1:Dataset overview\. For solvability measured by pass@1, Claude 4\.6 Sonnet filters forV​1V1andV​2V2, and Claude 4\.7 Opus forV​3V3specifically\.
### 4\.2Difficulty and Category Breakdown

Overall, to avoid heavy tails in easy and hard, we make medium tasks 63%, then easy and hard 18\.5% separately\. WithinV​3V3, difficulty is distributed as follows: easy \(57%\), medium \(36%\), and hard \(7%\), after solvability filtering\. A subset of randomly sampled 1,000 tasks, has pass rate that monotonically decreases with difficulty \(easy: 97%, medium: 92%, hard: 89%\)\. This confirms the cognitive difficulty definitions are meaningful\.

Round\-robin construction makes category coverage near uniform\. The eight common task domains are shown in Figure[2](https://arxiv.org/html/2610.02405#S4.F2)left graph; domain distribution remains in a universal pattern across all datasets\.

### 4\.3Instruction Length Distribution

In Figure[2](https://arxiv.org/html/2610.02405#S4.F2)right graph,V​2V2instructions has a broader coverage thanV​1V1andV​3V3, with some extremely long instructions with 1000 more characters, reflecting richer environment specifications enabled by the larger context window\. But the caveat is the heavy left tail down to 200 characters, meaning a significant number of unusually short instructions that could detail the trajectory\.V​3V3covers a broader range thanV​1V1with a nice bell\-shaped and tightly centered distribution, which shows a more balanced instruction design\.

### 4\.4Verifier Quality

Assertion count improved steadily fromV​1V1\(mean=24\.3\) toV​3V3\(mean=27\.6\), shifting noticeably rightward and having the least low\-count tail, suggesting more thorough verifiers over time\. However, the semantic type breakdown reveals a trade\-off:V​1V1had the highest process output assertions \(5\.7 per task\), which degraded to 2\.7 inV​3V3\. File content assertions similarly collapsed from 5\.8 inV​2V2to 1\.7 inV​3V3, as shown in[3](https://arxiv.org/html/2610.02405#S4.F3)\. We show more details in graphs in Appendix[C](https://arxiv.org/html/2610.02405#A3)\.

Figure 2:Left: task categories in all three datasets; no single category dominates\. Right: instruction length distributions for all datasets\.Figure 3:Assertion semantic types acrossV​1V1,V​2V2, andV​3V3datasets\.

## 5Training Experiments

We use training rollouts to test whether a generated task distribution supplies useful RL signal, rather than treating a runnable verifier as sufficient\. All experiments use SkyRL[Cao et al\. \(2025\)](https://arxiv.org/html/2610.02405#bib.bib8)[Griggs et al\. \(2025\)](https://arxiv.org/html/2610.02405#bib.bib9), binary reward from Harbor’s verifier, and theTerminus2agent with at most eight turns per rollout\.

Table 2:Training evidence for dataset calibration\.As a reference, Claude Opus 4\.6’s solve rate over three datasets are 94\.1%, 98\.0% and 11\.2% respectively fromV​1V1toV​3V3\.

### 5\.1E​1E1: Sparse Rewards onV​1V1

We trainQwen/Qwen2\.5\-3B\-Instructwith PPO on 457 tasks downsampled fromV​1V1Dataset\-4k for two epochs \(228 gradient steps\)\. Each step contains four tasks with four samples per task; the maximum sequence length is 4096 tokens\. We evaluate on 51 held\-out tasks every 20 steps\.

The base model achieves 7\.8% pass@1\. Training unfortunately does not improve it: validation pass@1 ranges from 3\.9% to 5\.9%, and the final checkpoint reaches 5\.9%\. With roughly 10% task solvability, most batches contain no successful trajectory\. Training reward consequently remains zero for most of the run, with only isolated positive\-reward steps\. This is the low end of the solvability band: a structurally valid task set can still provide too little signal for learning\.

### 5\.2E​2E2–E​3E3: Hard Tasks Restore Headroom for a 9B Model

We next run GRPO withQwen/Qwen3\.5\-9Bon an H200 instance, holding the instance, batch size \(eight tasks\), two samples per prompt, maximum turns, and sequence length fixed while changing only the task distribution\. Due to the limitation of computation resources and considering solvability ceiling, we still downsample from this mixture\. The experiment uses 3,238 training tasks\. ThenE​3E3adds 3,291 hard tasks fromV​3V3, for 6,529 training tasks\.

Table 3:Changing the training distribution changes rollout difficulty\.OnV​1V1\+V​2V2, the 9B model solves 81\.3% of attempts on average; four of the first 20 steps have pass@2 of 100%\. AddingV​3V3hard tasks reduces the mean pass@2 to 20\.6% and only lifts by 1pp after 190 steps\. The mean reward standard deviation remains nonzero in both runs, but the harder mixture restores failures that can distinguish trajectories\. Thus the same training configuration moves from a frequently saturated distribution to one with substantially more headroom solely by changing task difficulty\.

This result complementsE​1E1\. Tasks that are too difficult yield almost no positive reward; tasks that are too easy leave too few informative failures\. Task generation must therefore be calibrated to the target policy rather than optimized for a single global solvability rate\.

## 6Failure Modes and Measurement Fidelity

Generated terminal tasks can fail before an agent acts, while an agent acts, or while the harness measures the outcome\. These cases have different consequences: a malformed task is not evidence of limited agent capability, and a successful container build is not evidence that a task supplies useful training signal\. Figure[4](https://arxiv.org/html/2610.02405#S6.F4)organizes the observed failure modes by the stage at which they enter the pipeline\.

Generate taskand verifierRun agent inenvironmentAggregate rewardand pass@kSkewed difficulty;duplicates; guard gapsDockerfile failures;heavy setup timeoutsNetwork exhaustion;rebuilds; CPU saturation

Figure 4:Failure modes enter at generation, environment execution, and evaluation\-operation stages; they require separate structural, solvability, and error\-rate checks\.### 6\.1Generation and Verifier Failures

The most frequent categories are predominantly mechanical workflows, and difficulty is sampled independently of category\. A nominal difficulty label does not guarantee that the task creates discriminative reward variation\. Therefore, the combination of difficulty level and category distribution should be enforced during round\-robin generation\.

Deduplication exposes a second generation failure\. This process removed 1,753 of 6,682 raw candidate tasks \(26\.2%\) at a similarity threshold of 0\.85; 1,461 removed tasks were near duplicates with similarity between 0\.85 and 0\.95\. Repetition inflates dataset size without expanding the actual novel behaviors for agents to learn\. Finally, a baseline quality sample found anti\-shortcut guards in roughly 60% of task truth blocks, but the generation pipeline did not validate their presence\. Missing guards create a reward hacking risk: behavior tailored to the test can receive reward without meeting the intended engineering objectives\.

Table 4:Observed failure modes from distinct runs\.
### 6\.2Environment and Evaluation Failures

Environment construction is a direct confound in live evaluation\. In one initial test generation ofV​2V2, we identified approximately 74% of errored trials were attributed to a Docker heredoc generation bug\. In a later 8,218\-task solution\-generation run, Docker network\-pool exhaustion affected 9\.1% of trials and varied Docker build failures affected 9\.7%\. The former was reduced by periodic network pruning; the latter includes heterogeneous causes such as network\-dependent package installation, Git operations, heredoc handling, and database initialization\. These are harness failures, not policy outcomes\.

Some limitations persist even after failed trials are removed\. The same large run rebuilt each task image for every attempt, spending a mean 163 seconds in environment setup; retaining images would reduce rebuilds but was infeasible at the reported scale because of disk demand\. Twelve concurrent Docker builds also saturated roughly 29\.7 of 32 CPU cores\. Separately, three locally evaluated tasks exceeded a 600\-second startup limit because their initial states seeded large databases, repositories, or log exports\. Together, these observations motivate a designated infrastructure and isolating heavy tasks rather than interpreting every non\-pass as an agent mistake\.

### 6\.3Evaluation Measurement Confounds

Two operational findings affect how we interpret evaluation results\. First, when vLLM returns more than two samples asynchronously, completion order biases prefix pass@k: short, early\-returning trajectories can dominate pass@1 or pass@2\. We therefore report pass@2 only for the two\-sample runs above; for larger offline samples, pass@k should be estimated from the full set of successful trajectories\.

Second, evaluation infrastructure can masquerade as model failure\. On the same 151\-task validation set, increasing evaluation concurrency to 40 containers caused 144 tasks to error and yielded 4\.6% pass@2\. Reducing concurrency to 16 containers reduced errors to one task and yielded 76\.8% pass@2\. We consequently treat task\-level error rate as a required companion metric to pass@k for live\-environment evaluation\.

## 7Discussion and Future Work

The solvability band\.Our experiments identify two failure modes:V​1V1is too sparse for Qwen2\.5\-3B PPO, while theV​1V1\+V​2V2mixture is frequently near ceiling for Qwen3\.5\-9B GRPO\. AddingV​3V3hard tasks lowers mean rollout pass@2 from 81\.3% to 20\.6% without changing the training configuration\. The appropriate training distribution is therefore model dependent; a single global pass rate cannot certify that a benchmark will produce useful gradient signal\.

Demystifying live\-environment evaluation\.A low score in a terminal environment does not, by itself, show that an agent failed at the intended task\. It can instead reflect a malformed Dockerfile, an environment\-start timeout, exhausted Docker network address pools, or biased aggregation of asynchronously returned samples\. Therefore, for researchers working on similar pipelines, we strongly suggest separating three questions from an end\-to\-end benchmarking:

whether an environment is structurally valid?

whether its verifier faithfully measures the stated task?

whether the target policy can solve it?

Our current pipeline incorporates practical mitigations for these confounds\. Examples are like the Dockerfile generator prefixes BuildKit syntax to support heredocs\. Evaluation isolates known heavy\-build tasks with a longer timeout, limits container concurrency, and runs periodic Docker cleanup to reduce stale\-network and cache failures\.

Finally, we pair pass@k with task\-level error rates and avoid prefix pass@k for asynchronous runs with more than two samples\. These controls do not make a generated verifier fully faithful, but they prevent common infrastructure failures from being reported as agent incapability\.

Limitations and Future Work

In measurement, asynchronous vLLM completion order biases prefix pass@k when more than two samples are generated, so larger\-sample pass@k requires an estimator over the full set of trajectories\. Moreover, concurrent container failures can dominate an evaluation score; task\-level error rates must accompany pass@k\. The present experiments demonstrate calibration effects, but do not yet show end\-to\-end improvement across a full training run in the middle of the solvability band\. The constraints also come partially from the high expenses of large\-scale training and evaluation\.

In data, 16\.6% tasks show few assertions check the desired command outcomes, and overlap score between task description and test suite flags potential drift from the real task intent\.

Therefore, we target the followings as our next steps\.

- •Scale up training experiments for longer steps and larger training and evaluation samples\.
- •Filter and improve tasks having low assertion matching score and description–test overlap score\.
- •Generate more tasks in scientific computing, data science, and security\.
- •Add intricate multi\-stage tasks to broaden the curriculum for capable agents\.

## 8Related Work

Terminal agent benchmarks\.SWE\-bench[Jimenez et al\. \(2024\)](https://arxiv.org/html/2610.02405#bib.bib3)and related work evaluate agents on real GitHub issues in Docker containers, but construction is human\-curated and the task set is static\. Harbor[Harbor Framework Team \(2026\)](https://arxiv.org/html/2610.02405#bib.bib2)provides a harness for executing terminal agents against programmatically defined tasks\. TMAX[Ivison et al\. \(2026\)](https://arxiv.org/html/2610.02405#bib.bib4)generates terminal tasks using LLMs; our approach differs in the RL training loop, the solvability analysis, and the integration with Harbor’s evaluation API\. Gandhi et al\.[Gandhi et al\. \(2025\)](https://arxiv.org/html/2610.02405#bib.bib1)is the direct predecessor; we preserve their generation skeleton and extend it along the dimensions described above\.

RL for code and tool\-use agents\.DeepSeekCoder[Guo et al\. \(2024\)](https://arxiv.org/html/2610.02405#bib.bib5)and related work show that GRPO with binary reward drives measurable improvement on code generation when the base model already solves a fraction of training problems\. Our experiments confirm this pattern and extend it to the terminal domain, where environment statefulness and open action spaces make the solvability requirement harder to satisfy\.

Curriculum learning for RL\.Several works on curriculum learning for RL[Cai et al\. \(2026\)](https://arxiv.org/html/2610.02405#bib.bib6)[Parashar et al\. \(2026\)](https://arxiv.org/html/2610.02405#bib.bib7)show that training on tasks calibrated to the model’s current capability accelerates learning\. Our solvability\-band hypothesis is a terminal\-domain instantiation of this principle: the dataset must be filtered to keep pass rates in the range where gradient signal is present but not saturated\.

## References

- Caiet al\.\(2026\)P\. Cai, T\. Fang, X\. Li, Q\. Zeng, G\. Li, and J\. ChenCurriculum reinforcement learning can incentivize reasoning capacity in llms beyond the base model\.External Links:2606\.22317,[Link](https://arxiv.org/abs/2606.22317)Cited by:[§8](https://arxiv.org/html/2610.02405#S8.p3.1)\.
- Caoet al\.\(2025\)S\. Cao, S\. Hegde, D\. Li, T\. Griggs, S\. Liu, E\. Tang, J\. Pan, X\. Wang, A\. Malik, G\. Neubig, K\. Hakhamaneshi, R\. Liaw, P\. Moritz, M\. Zaharia, J\. E\. Gonzalez, and I\. StoicaSkyRL\-v0: train real\-world long\-horizon agents via reinforcement learning\.Cited by:[§5](https://arxiv.org/html/2610.02405#S5.p1.1)\.
- Gandhiet al\.\(2025\)K\. Gandhi, S\. Garg, N\. D\. Goodman, and D\. PapailiopoulosEndless terminals: scaling rl environments for terminal agents\.arXiv preprint arXiv:2601\.16443\.Cited by:[§1](https://arxiv.org/html/2610.02405#S1.p2.1),[§1](https://arxiv.org/html/2610.02405#S1.p5.1),[§3\.3](https://arxiv.org/html/2610.02405#S3.SS3.p1.1),[§3\.4](https://arxiv.org/html/2610.02405#S3.SS4.p2.1),[§8](https://arxiv.org/html/2610.02405#S8.p1.1)\.
- Griggset al\.\(2025\)T\. Griggs, S\. Hegde, E\. Tang, S\. Liu, S\. Cao, D\. Li, C\. Ruan, P\. Moritz, K\. Hakhamaneshi, R\. Liaw, A\. Malik, M\. Zaharia, J\. E\. Gonzalez, and I\. StoicaEvolving skyrl into a highly\-modular rl framework\.Note:Notion BlogCited by:[§5](https://arxiv.org/html/2610.02405#S5.p1.1)\.
- Guoet al\.\(2024\)D\. Guo, Q\. Zhu, D\. Yang, Z\. Xie, K\. Dong, W\. Zhang, G\. Chen, X\. Bi, Y\. Wu, Y\.K\. Li, F\. Luo, Y\. Xiong, and W\. LiangDeepSeek\-coder: when the large language model meets programming – the rise of code intelligence\.Vol\.abs/2401\.14196\.External Links:[Link](https://arxiv.org/abs/2401.14196)Cited by:[§8](https://arxiv.org/html/2610.02405#S8.p2.1)\.
- Harbor Framework Team \(2026\)Harbor: A framework for evaluating and optimizing agents and models in container environmentsExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.20953922),[Link](https://doi.org/10.5281/zenodo.20953922)Cited by:[§1](https://arxiv.org/html/2610.02405#S1.p6.1),[§8](https://arxiv.org/html/2610.02405#S8.p1.1)\.
- Ivisonet al\.\(2026\)H\. Ivison, J\. O\. Yin, R\. Shao, T\. Xiao, N\. Lambert, and H\. HajishirziTmax: a simple recipe for terminal agents\.External Links:2606\.23321,[Link](https://arxiv.org/abs/2606.23321)Cited by:[§8](https://arxiv.org/html/2610.02405#S8.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. NarasimhanSWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§8](https://arxiv.org/html/2610.02405#S8.p1.1)\.
- Parasharet al\.\(2026\)S\. Parashar, S\. Gui, X\. Li, H\. Ling, S\. Vemuri, B\. Olson, E\. Li, Y\. Zhang, J\. Caverlee, D\. Kalathil, and S\. JiCurriculum reinforcement learning from easy to hard tasks improves llm reasoning\.External Links:2506\.06632,[Link](https://arxiv.org/abs/2506.06632)Cited by:[§8](https://arxiv.org/html/2610.02405#S8.p3.1)\.

## Appendix AGeneration Prompts

This appendix reproduces the four LLM system prompts used in the data\-generation pipeline\. Each prompt is the verbatimSYSTEM\_MSGconstant passed to the language model for that stage\. User messages are constructed programmatically from templates and are not reproduced here\.

### A\.1Stage 1: Task Template Prompt

The task template prompt asks the model to produce a<task\>\(the instruction shown to the agent\) and a<truth\>\(privileged ground\-truth for the verifier\)\. It encodes the voice guide, anti\-leakage rules, domain\-coverage rotation, difficulty levels, environment constraints, and a self\-check list\.

YouarecreatingrealisticLinux\-terminaltasksfortraininganAIagent\.

Respondinxmlformat\.

<task\>

Thetaskasahumanwouldwriteit\.Seethevoiceguidebelow\.

Lengthsvary\.Manyshouldbeunder50words;180isaceiling,notatarget\.

RealSlackmessagesareshort\.Leanshorterthanfeelscomfortable\.

DoNOThandtheagentanumberedlistofstepsorsub\-requirements\.

DoNOTdirectlygivethecommandsorthefixrecipe\.

</task\>

<truth\>

Privilegedground\-truthforthegrader\.Neverappearsin<task\>\.

Usethetemplatedescribedinthe"truthblock"sectionbelow\.

</truth\>

\#Howtasksopen

Realengineeringrequestscomeinseveralshapes\.Rotateacrossthem

deliberately\-\-don’tdefaultto"somethingbroke,fixit\."

\-Trigger/debug\."Iwasrebasingwhen\.\.\.","ThenightlyETLstarted

returning\.\.\.","Ijustpulledmainandnow\.\.\."\.Somethingisalready

underwayandbroken\.

\-Build/setup/creation\."IneedaworkingXbyEOD\."Aconstruction

taskwithconstraints,nofailurebackstoryrequired\.

\-Reverse\-engineering\."There’sthisbinarynobodydocumentedandIneed

totalktoit\."Exploratory,curiosity\-driven\.

\-Recovery\."MyDBgottruncated,Ihaveapartialbackup,helpmeget

backwhatIcan\."

\-Migration/port\."We’removingfromXtoYandI’mstuckonZ\."

\-Investigation\."Thenumbersinthereportdon’tmatchthesource\.

Figureoutwhichisright\."

\-Optimization/tightening\."Thisloopisfinebutit’s40xslower

thanitshouldbe\."

Thefirstsentenceshoulddroptheagentintoasituation,notannounce

aproject\.

\#Voice

Thenarratorisanengineermid\-stream\.Casual,possiblytired,possiblywrong\.

\-First\-person,mid\-flow\.Nogreetings,noprojectbriefs,no"HiClaude,"

no"Pleasecompletethefollowingtask\."

\-Specificabout\*where\*,vagueabout\*why\*\.Beexactwithpaths,

filenames,ports,tablenames,commitshas,versionnumbers\.Beloose,

hedging,orevenmistakenaboutdiagnosis\.

\-Hedgeswelcome\."thisfeelslikeatimezonething","mightbe

middlewareidk","nothing’schangedafaik"\.

\-Registercuesformessiness\.Droppedarticles\("serverkeepsOOMing"\),

abbreviations,trailingparentheticals,incompletememory,tangents,

sentencefragments\.Keep"I"\-\-write"I’vebeentryingto\.\.\.",not

"Beentryingto\.\.\."\.Dropping"I"readslikealogentry\.

\-Oneimperfectionminimum\.Eachtaskmustcontainatleastoneof:a

vaguedescription,anincompletememory,asoftguessthatmaybe

wrong,slightrambling,oratangent\.

\#Whatrealusersdon’tdo

\-Don’trequestartifactswhoseonlypurposeisgrading\.

\-Don’tspecifyexactoutputformatsunlessthere’sarealdownstreamreason\.

\-Don’trestateverificationcriteria\.

\-Don’tenumeratesub\-requirementsasanumberedorbulletedlist\.

\-Don’texplaintherootcausecleanly\.

\-Don’tdictateimplementationunlessarealengineerwouldactually

constrainthat\.

Bannedphrasings\(donotuse\):

\-"Theoutputshould\.\.\."

\-"Thefilemustcontain\.\.\."

\-"Ensurethat\.\.\."

\-"Line1:\.\.\.","Step1:\.\.\."

\-"ReturnaJSONwiththefollowingfields\.\.\."

\-"Pleasecompletethefollowingtask"

\#Prose,notspecification

Thetaskisoneortwoparagraphsofprose,notalistofrequirements\.

\#Thetruthblock

Useroughlythisshape:

\-Initialstate\.What’spre\-seededondisk\.Exactpaths\.Whichfilesare

writablevsread\-only\.Whattools/binaries/servicesexist\.

\-Expectedfinalstate\.Positivechecks:filecontents,exitcodes,

serviceresponses,testpass/fail\.Bespecific\.

\-Invariants\.WhatmustNOTchange\.

\-Anti\-shortcutguards\.Whenataskissolvablebycheating,spellouta

gradercheckthatcatchesit\.

Valuesin<truth\>mustNOTappearin<task\>\.

\#Environmentconstraints

\-TheagentdoesNOThaveroot\.

\-Useabsolutepaths\.Homeis/home/user\.

\-Noliveinternet\.Nocurrent\-eventsknowledge\.

\-TheenvironmentisasingleDockercontainer\(ubuntu:22\.04\)\.

\-NOTavailable:Docker\-in\-Docker,Kubernetes,systemd,multiple

networkedhosts,privilegedkerneloperations,GUI\.

\-Servicescanbeusedbutmustbestartedasforeground/background

processes\-\-notviasystemctl\.

\#Self\-checkbeforefinalizing

1\.Doesthefirstsentencedroptheagentintoasituation?

2\.Isthereatleastoneimperfection?

3\.CouldarealengineerplausiblysendthisinSlack?

4\.HaveIleakedthediagnosisorthefixrecipe?

5\.Areanybannedphrasingspresent?

6\.Isthetaskprose,ordidIslipintospecvoice?

7\.Isthedeliverablesomethingarealuserwouldactuallywant?

8\.Does<truth\>containinitialstate,expectedfinalstate,invariants,

ANDanti\-shortcutguards?

9\.AmIproducingthesamedomainortasktypeasmyrecentoutputs?

Ifanyansweriswrong,rewrite\.

The user message appended to this system prompt specifies: domain category \(sampled round\-robin from ten buckets\), narrator persona, difficulty level with level\-specific guidance, command complexity \(bash\-only / bash\+python / bash\+python\+services\), and workflow length \(short / medium / long\)\.

### A\.2Stage 2: Initial\-State Test Prompt

Given the task description and truth block from Stage 1, Stage 2 generates atest\_initial\_state\.pypytest file that passes only if the container is correctly seeded before the agent acts\.

YouareaseniorPythonengineerwhowritesrobustpytestsuites\.

Write\*one\*pytestfilethatvalidatestheoperatingsystem/filesystem

\*before\*thestudentperformstheaction\.

Thetruthvalueindicatestheanswerthatthestudentshouldget\.

Youshouldtestforthepresenceoffiles,directories,processes,

repositories,websites,etc\.

Rules:

\*Thefilenameshouldbe‘test\_initial\_state\.py‘\(showitinaheader

comment\)\.

\*Useonlystdlib\+pytest\.

\*Failuresmustclearlyexplainwhatismissing\.

\*EnsurethatthestateoftheOSmatchesthetruth\.

\*Writethecodeinafencedcodeblockthatcanbeparsedtogeta

singlepythonfile\.

\*Whenyoutestforafileordirectory,testforthefullpathtothe

fileordirectory,notjustrelativepath\.

\*DONOTtestforanyoftheoutputfilesordirectories\.

\*Thehomepathis/home/user\.

\*Forservice\-basedtasks,verifythatrequiredservices/daemonsare

runningandlistening\.

\*Fortasksinvolvingprocesses,verifytheexpectedprocessesexist\.

\*Usesubprocesscallstocheckservicestatewhenneeded

\(e\.g\.,‘ss\-tlnp‘,‘pgrep‘,‘systemctl‘\)\.

The user message provides\{task\_description\},\{truth\}, and\{difficulty\}\.

### A\.3Stage 3: Final\-State Test Prompt

Stage 3 generatestest\_final\_state\.py, which must pass after a correct agent solution and fail in the initial container state\. It receives the initial\-state test as additional context\.

YouareaseniorPythonengineerwhowritesrobustpytestsuites\.

Writearobustpytestsuitethatvalidatesthe\*FINAL\*stateofthe

operating\-system/container\*after\*thestudenthascompletedthetask

described\.

Usetheprivileged\*truth\*datatoasserttheexactexpectedendstate

forthetasktobecompleted\.

Rules:

\*Thefilenamemustbe‘test\_final\_state\.py‘\(showitinaheader

comment\)\.

\*Use\*only\*thePythonstandardlibraryand‘pytest‘\(nothird\-party

libs\)\.

\*Failuresmustclearlyexplain\*whatisstillwrong\*\.

\*Whenyoucheckforfilesordirectories,alwaysusetheir\*absolute\*

pathsexactlyasgiven\(norelativepaths\)\.

\*Writethecodeinafencedcodeblockthatcanbeparsedtogeta

singlepythonfile\.

Evaluationapproachguidelines:

\*Prefersemanticvalidationoverexactstringmatching\.Forexample:

\-ParseJSON/YAMLandcheckkeys/valuesinsteadofcomparingraw

strings\.

\-Use‘in‘orregextocheckforrequiredcontentinsteadofexact

linematching\.

\-Checknumericvalueswithtoleranceinsteadofstringcomparison\.

\*Forservice\-basedtasks,testthatservicesrespondcorrectly:

\-Usesubprocesstocurl/wgetendpointsandcheckresponsecodesand

content\.

\-Checkthatportsarelistening\(socketconnectorss/netstat\)\.

\-Verifyprocessesarerunning\(subprocess:pgrep,ps\)\.

\*Forcommand\-outputtasks,runthecommandandcheckitsexitcodeand

stdout\.

\*Forfiletasks,validatestructureandkeycontent,notexact

byte\-for\-bytematch\.

\*Nevercheckexactwhitespace,trailingnewlines,orformattingthat

thetaskdidn’texplicitlyspecify\.

\*Foreasytasks,testsshouldchecksimpleoutcomes\(fileexists,

contentcontainsexpectedvalue\)\.

\*Forhardtasks,testscancheckcomplexmulti\-facetedstatebutshould

stillvalidatesemanticsoverexactformatting\.

The user message provides\{task\_description\},\{truth\},\{difficulty\}, and\{initial\_test\_py\}\.

### A\.4Stage 4: Dockerfile Prompt

Stage 4 generates aDockerfilethat, when built, produces a container whose state satisfies the initial state tests and does not yet satisfy the final state tests\. This is the most constrained prompt: it enumerates Docker\-specific rules, heredoc syntax requirements, and a hard constraint that the output must begin withFROM ubuntu:22\.04and contain only valid Dockerfile instructions\.

YouareanexpertinDocker\.

Youaregivenataskdescriptionandwillbetestedsothattheinitial

stateofthecontainerissetupinawaythatanagentcanbetestedon

thetask\.

Makesurethatthecontainerissetupinawaythatanagentcanbe

testedonthetask\.

Basicallyensurethatthetaskisvalidwhenthecontainerisbuilt:

Clonearepository,createafile,createadirectory,createaprocess,

etc\.

Keyrules:

\-Alwaysuse‘FROMubuntu:22\.04‘asthebaseimage\.

\-Installpytestviapip3\.

\-Create/home/userdirectorywithappropriatepermissions

\(chmod0777/home/user\)\.

\-Theagentwillnothaverootaccess\.Settherightpermissionsfor

filesanddirectories\.

\-Don’tincludethetestsintheDockerfile\(noCOPYoftestfiles\)\.

\-Don’taddanyoftheoutputfilesordirectoriesthattheagentwill

create\.

\-Don’tcreate/touchemptyfilesfortheagent\.

\-Thehomepathis/home/user\.

\-Fortasksthatrequirerunningservices/daemons,setupanentrypoint

scriptoruseCMDtostartthem\.ServicesstartedinRUNstepswon’t

persist\-\-useawrapperscriptthatstartsservicesthenexec’sinto

ashell\.

\-NODocker\-in\-Docker,Kubernetes,kind,minikube,orcontainer

orchestrationtools\.

\-NOsystemdorinitsystems\.Startservicesdirectlyasprocesses\.

\-NOnftables/iptablesrules\.

\-ForPostgreSQL:initializewith‘initdb‘,startwith‘pg\_ctlstart‘

or‘postgres\-D<datadir\>&‘inanentrypointscript\.DoNOTuse

‘aptinstallpostgresql‘andexpecttheservicetoauto\-start\.

\-Foranyservicethatmustberunningwhentheagentconnects:usea

CMD/ENTRYPOINTwrapperscriptthatstartstheserviceinthe

background,waitsforittobeready,thenexec’sintobash\.

Heredocandmultilinecommandrules\(IMPORTANT\):

\-Everyshellcommandmustbeinsidea‘RUN‘instruction\.DoNOTleave

lineslike‘PermitRootLoginyes‘outsideofa‘RUN‘\.

\-Whenusingheredocblocks\(e\.g\.‘cat<<EOF\.\.\.EOF‘\):

\-Keeptheentireheredocinsideasingle‘RUN‘instruction\.

\-Thelinewith‘<<EOF‘mustbethelastthingonthatline\.

\-Theheredocbodymuststartonthenextlineasplaintext\.

\-Theterminating‘EOF‘mustbealoneonitsownline\.

\-Follow\-upcommandsgoinaseparate‘RUN‘instruction\.

\-DoNOTwrapaheredocblockinsideasingle\-quotedstringlike

‘sh\-c’cat<<EOF\.\.\.EOF’‘\.

\-Formulti\-linePython:use‘RUNpython3<<’PYEOF’\.\.\.PYEOF‘

\(heredoc\)orwritethescripttoafilefirstwith

‘cat<<’EOF’\>script\.py\.\.\.EOF‘then‘RUNpython3script\.py‘\.

DoNOTuse‘RUNpython3\-c"\.\.\."‘formulti\-linecode\.

CRITICAL:Yourresponsemuststartwith‘FROMubuntu:22\.04‘onthevery

firstline\.DoNOTincludeanyexplanation,commentary,thinking,or

markdownfences\.OutputONLYvalidDockerfileinstructions\-\-nothing

else\.

CRITICAL:DoNOTincludeanyshellscripts,Pythoncode,configfile

contents,gitsubmodulelines,importstatements,oranynon\-Dockerfile

textasbarelines\.EverylinemustbeavalidDockerfileinstruction

\(FROM,RUN,COPY,ENV,etc\.\)oracommentstartingwith\#\.

The user message provides\{task\_description\},\{truth\},\{test\_py\}\(the initial state pytest\),\{difficulty\}, and\{failures\}\(error output from previous build attempts, fed back for up to three retries\)\.

## Appendix BHarbor Task Structure

task\_<id\>\_<hash\>/

instruction\.md\#promptshowntotheagent

task\.toml\#metadata:difficulty,timeouts,resources

environment/

Dockerfile\#containerdefinition

task\.json\#description\+groundtruth

test\_initial\_state\.py\#validatesinitialcontainerstate

solution/\#referencesolution\(s\)

tests/

test\.sh\#verifier:runspytest,writesreward

test\_final\_state\.py\#validatestaskcompletion

## Appendix CDataset Quality Figures

Figure 5:Chain complexity distribution acrossV​1V1,V​2V2, andV​3V3datasets\.Figure 6:Tool breadth distribution acrossV​1V1,V​2V2, andV​3V3datasets\.Figure 7:Assertion count distribution acrossV​1V1,V​2V2, andV​3V3datasets\.
## Appendix DCategory Generation Strategy

V​1V1— Round\-robin across buckets, and shuffle bucket names once, then assign taskiito bucket\[i%​l​e​n​\(b​u​c​k​e​t​s\)\]\[i\\%len\(buckets\)\], finally picking a random sub\-category within each bucket uniformly\. Categories were pre\-sampled before generation and not propagated back from the LLM output\.

V​2V2— Difficulty\-bucket coupling, difficulties are picked first, then categories are constrained by difficulty: \- Hard tasks → restricted to debugging, security, services, data processing, networking, git ops \- Easy tasks → prefer file ops, text processing, system admin, build tools \- Medium → any bucket \- Plus richer sub\-categories like stripped ELF binary reverse engineering \(security\), binary data format parsing \(debugging\), corrupt archive recovery \(system admin\)\.

V​3V3\- Previously categories were sampled before difficulties, so bucket coupling was never applied\. Now difficulty configuration runs first to mitigate this; Within each bucket, weighting choices can control the distribution of sub\-category per bucket as well\.

## Appendix EDeep Dive in Solve Rate of DatasetV​2V2

Figure 8:V​2V2dataset deep\-dive\. Left: pass@k onV​2V2tasks evaluated by Claude 4\.6 Sonnet \(pass@1 = 60\.2%, plateaus near 63%\)\. Right: distribution of per\-task pass rates; most tasks are consistently solved or consistently failed, with few in the 0\.2–0\.8 middle band\.