Paper page - Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories

Source: https://huggingface.co/papers/2604.17596 Terminal Wrench is a dataset of terminal-bench style environments that have shown evidence of being reward-hackable, paired with agent trajectories that lead to hacking and non-hacking rewards. Each entry preserves the original task definition alongside full attack trajectories showing how the verifier was passed, including cases where the task was not solved as intended. The tasks span system administration, machine learning, software engineering, and security challenges, and the hacks range from simple output spoofing to stack-frame introspection, stdlib backdoors, and rootkit-style binary hijacking. Notably, these hacking trajectories are task-specific rather than benchmark/evaluation-harness-specific, which makes them harder to fix. The hacks were elicited by appending a hack elicitation excerpt to the prompt. We believe this is a subset of hackable tasks across the 1,860 we analyzed. Our methodology was to first elicit and judge hacking using a variety of models and prompts, producing over 40k trials. From these trials we narrowed the hackable-task subset to 395, and then performed a more robust hacker loop on those, resulting in this dataset.

We also share results of a simple monitorability experiment. We first sanitize the hacked trajectories with a sanitization prompt to remove obvious mentions of hacking behavior from the prompt and replace common keywords, and we ask another model to rewrite the agent blocks to be less suspicious. These sanitized trajectories are stored under sanitized_trajectories//trial/agent/trajectory.json, with sanitizer_model.txt recording which model produced the rewrite. Tool calls are unchanged. We also perform more aggressive stripping, where we remove all agent commentary and only keep the tool calls. We store these under stripped_trajectories//trial/agent/trajectory.json. We then use a judge prompt to rate the original hack as well as the two variants to determine whether detectability drops.

Some benchmarks continue to make improvements, for example we are aware Terminal Bench 2 has continued to fix existing tasks as issues have been found. We pulled our tasks from the primary sources between 2026-01-01 and 2026-03-30, so some of these tasks might have changed since.

For presentation, ◆ rewarded serious exploits and ◇ rewarded non-serious hacks are grouped together as hacks; ~ attacker legitimate solves and ° no-reward attempts remain separate non-hack outcomes.

331 unique tasks 957 task/model entries 6,289 v5 hacker trajectories 3,632 hack trajectories (◆ + ◇) 1,216 attacker legitimate solves (~) 1,441 no-reward attempts (°) 2,352 baseline trajectories from successful prechecks 3 models: claude-opus-4.6, gemini-3.1-pro, gpt-5.4