ProgramBench (5 minute read)

# ProgramBench Source: [https://programbench.com/](https://programbench.com/) Can language models rebuild programs from scratch? Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior\. ## About ProgramBench In each task, the agent receives an executable and its documentation, and it must re\-implement the given executable\. It does not get access to*any*of the executable's source code, it cannot[de\-compile the executable](https://programbench.com/#faq-decompilation), and cannot[use the internet](https://programbench.com/#faq-internet)\. There are[200 tasks](https://programbench.com/tasks/)in total covering different program complexities, ranging from small terminal utilities like[jq](https://programbench.com/task/jqlang__jq.b33a763/)and[ripgrep](https://programbench.com/task/burntsushi__ripgrep.3b7fd44/)to massive software projects like the[PHP compiler](https://programbench.com/task/php__php-src.c891263/),[FFmpeg](https://programbench.com/task/ffmpeg__ffmpeg.360a402/), and[SQLite](https://programbench.com/task/sqlite__sqlite.839433d/)\. The agent must choose a language, design the architecture, write all source code, and produce a build script\. Every design decision is the model's to make\. Once the agent submits a program, our test suite compares the candidate program's behavior against the original program\. A candidate program passes only if all tests for that task pass\. Our test suite is generated via agent\-driven fuzzing, and it comprises more than 248,000 total behavioral tests for our[200 tasks](https://programbench.com/tasks/)\. Can tasks in ProgramBench be fully solved at all?Yes\. The agent can run the given program with any input and observe exactly what it does, so there's nothing hidden that can't be discovered through experimentation\. The benchmark is hard, but it's solvable by design: all the reference executables pass our test suites\. Read more in our[blog post](https://programbench.com/blog/is-programbench-impossible/)\. Why are ProgramBench scores so low?Building a program from scratch is a fundamentally challenging task\. Agents do currently make partial progress on many tasks \(see the[extended results](https://programbench.com/extended/)for details\), but fully passing every test is still out of reach\. **Agents truly have to architect\.**This is in part because unlike other whole\-repo generation projects, we give no hints or structure to the agent, meaning that the agent truly has to architect its own solutions \(see "[How is ProgramBench different?](https://programbench.com/#faq-related-work)"\)\. **No harness tuning\.**Other recent and concurrent work also performed substantial harness tuning for a single or a handful number of tasks\. We deliberately avoid this, since headline scores from a tuned harness on a curated handful of tasks can substantially overstate how capable agents really are at building software from scratch\. Instead, ProgramBench is evaluated with a single generic harness across the entire task set\. **Cleanroom implementation\.**We take substantial precautions to prevent cheating\. Agents run in sandboxed containers without internet access, so they cannot retrieve the original source code or obtain any other form of help\. **No decompilation\.**See "[Can tasks be solved with decompilation?](https://programbench.com/#faq-decompilation)" We review related work in section 6 of the[paper](https://arxiv.org/abs/2605.03546)\. We also discuss cheating in[the FAQ below](https://programbench.com/#faq-cheating)and in section 4\.1\. Is your agent scaffold sufficient to solve all tasks?**Widely adopted baseline\.**We use[mini\-SWE\-agent](https://github.com/swe-agent/mini-swe-agent/)because it is both widely adopted as a baseline by other benchmarks \(SWE\-bench Verified, SWE\-bench Multilingual, Terminal\-bench\) and deliberately minimal in its scaffolding, reducing confounds between model capability and harness design\. Most other agents \(like Claude Code with apparently several 100k lines of code\) are also constantly changing in non\-transparent ways, while mini\-SWE\-agent will allow for apples\-to\-apples performance comparison of models for the foreseeable future\. **Almost no runtime limitations\.**With very few exceptions, models submit their solutions deliberately rather than exceeding our generous time or step limits, and they never exhaust their context window\. Because we do not limit total cost, our runs have cost up to $5k \(for Sonnet 4\.5\)\. **Varying degree of difficulty\.**ProgramBench deliberately includes tasks from various degrees of difficulty, from very short repositories of only a few thousand lines of code to extremely large ones\. We believe that the extremely low scores are therefore more of a signal of inadequate model capabilities rather than an indicator that only multi\-agent systems can solve our tasks\. Nonetheless, we would be excited to be one of the first systematic benchmarks that includes tasks that can only be solved by multi\-agent systems\. **Kicking off a new scaffold race\.**We believe that[mini\-SWE\-agent](https://github.com/swe-agent/mini-swe-agent/)is the right choice of baseline and that it can absolutely solve \(some of\) the tasks\. However, we'd be more than excited if ProgramBench kicks off a new scaffold race\! We will be opening[submissions](https://programbench.com/#faq-submit)soon\. Can agents cheat?Agents run in sandboxed containers with no internet access, execute\-only permissions on the binary, and no access to decompilation tools\. In early trials without these restrictions, models found shortcuts like cloning source repositories from GitHub or downloading code through package managers\. Read more in our[blog post](https://programbench.com/blog/is-programbench-impossible/)and in section 4\.1 of the[paper](https://arxiv.org/abs/2605.03546)\. Why and how do you block decompilation?The executable that is given to the agent only has execution, not read permissions\. That means that any operation that is not execution \(such as running a decompiler, disassembler,`objdump`,`strings`, or`hexdump`\) will fail\. We do this because we want ProgramBench to answer the question "How well can LMs build programs from scratch", rather than "How well can LMs patch together bits of decompiled code"\. How is the leaderboard sorted? What's the primary metric?The primary metric that should be reported for ProgramBench are fully resolved instances\. We currently report "almost resolved" \(more than 95% of test cases pass\) as an additional point of reference while the scores of our primary metric are low\. The leaderboard is sorted by fully resolved first, almost resolved second, and finally the average test pass rate\. For a detailed understanding of model performance, we recommend the plot at the[detailed leaderboard](https://programbench.com/extended/)\. See also: "[Have you considered other metrics?](https://programbench.com/#faq-metrics)" How do I submit to the leaderboard?A public submission portal is coming soon\. Follow the authors \([John](https://x.com/jyangballin),[Kilian](https://x.com/KLieret)\) for updates\. Why do you not allow internet?We have extensively studied different inference settings, including allowing internet\. We find that allowing internet leads to an abundance of cheating that requires LM as a judge to flag and disqualify solutions\. This makes the benchmark less reliable, especially because defining exactly what cheating means in the context of obtaining source online is not as clear cut as it might seem\. However, except for instances that contained cheating, we did not observe a dramatic improvement of scores when allowing internet\. Find our ablations in section 4\.1 of the[paper](https://arxiv.org/abs/2605.03546)and John's[explanation](https://x.com/jyangballin/status/2051829957199011974)\. Have you considered other metrics? E\.g\., average number of tests passed?Yes, we've decided on our current resolved metric after a lot of thought\. Our initial question was "Can LMs build programs from scratch?", and the most relevant metric is the fraction of programs that can be fully built\. Reporting an average test pass rate would be extremely misleading, because every instance includes very simple tests \(such as checking for the existence of flags, checking what happens if you call the executable with`\-\-help`etc\.\)\. We've also thought about using a more relaxed metric like "almost resolved"\. However, relaxing to ≥95% of tests solved or even 99% of tests solved is also problematic\. First, for some of our tasks, we have almost 15k tests\. Even 1% of that is still 100 tests\. And even a single failed test can indicate severe issues with a program\. Therefore "almost resolved" only serves as additional orientation, until the main "resolved" metric has enough signal to differentiate all models\. However, all auxiliary metrics are still useful for diagnosing and improving models and scaffolds\! They're just not the right metric as a benchmark\. Check the[extended results](https://programbench.com/extended/)for more information\. See also: "[How is the leaderboard sorted?](https://programbench.com/#faq-sorting)" What about contamination/memorization?All tasks are pulled from open\-source repositories\. As such, they were certainly seen during training by the LMs evaluated here\. However, we are currently**not**worried about this: **Zero scores\.**Even if memorization is at play \(it doesn't look like it to us\), it is by far insufficient to score highly\. Scores seem to be much more correlated with systematic efforts at implementing functionality\. **Different\-language ablation\.**In section 4\.1 of the[paper](https://programbench.com/static/paper.pdf), we introduce an alternative inference setting where models are forced to implement the program in a different language than the original, effectively bypassing memorization \(especially because memorization is extremely prompt\-dependent\)\. The scores did not change drastically, leading to the conclusion that scores are currently not significantly correlated with memorization of the original source code\. Direct memorization is extremely simple to measure and we are actively monitoring it\. Should it ever become a clear problem, we might mix up our inference settings to prevent it\. For now, it was more important to make sure all tasks are[solvable](https://programbench.com/#faq-solvable)and natural, so we removed any constraints not deemed necessary to[prevent cheating](https://programbench.com/#faq-cheating)\. [:cherry\_blossom: A command\-line fuzzy finder 79,721go Best score:82%](https://programbench.com/task/junegunn__fzf.b56d614/)[simple terminal UI for git commands 76,901go Best score:56%](https://programbench.com/task/jesseduffield__lazygit.1d0db51/)[ripgrep recursively searches directories for a regex pattern while respecting your gitignore 62,855rs Best score:80%](https://programbench.com/task/burntsushi__ripgrep.3b7fd44/)[Mirror of https://git\.ffmpeg\.org/ffmpeg\.git 59,217c Best score:5%](https://programbench.com/task/ffmpeg__ffmpeg.360a402/)[A cat\(1\) clone with wings\. 58,487rs Best score:33%](https://programbench.com/task/sharkdp__bat.f822bd0/)[A markup\-based typesetting system that is powerful and easy to learn\. 52,957rs Best score:28%](https://programbench.com/task/typst__typst.88356d0/)[Universal markup converter 43,632hs Best score:14%](https://programbench.com/task/jgm__pandoc.5caad90/)[A simple, fast and user\-friendly alternative to 'find' 42,668rs Best score:78%](https://programbench.com/task/sharkdp__fd.40d8eb3/)[The PHP Interpreter 40,030c Best score:5%](https://programbench.com/task/php__php-src.c891263/)[DuckDB is an analytical in\-process SQL database management system 37,657cpp Best score:12%](https://programbench.com/task/duckdb__duckdb.bdb65ec/)[A smarter cd command\. Supports all major shells\. 35,994rs Best score:76%](https://programbench.com/task/ajeetdsouza__zoxide.67ca1bc/)[Command\-line JSON processor 34,541c Best score:90%](https://programbench.com/task/jqlang__jq.b33a763/)[A syntax\-highlighting pager for git, diff, grep, rg \-\-json, and blame output 30,445rs Best score:37%](https://programbench.com/task/dandavison__delta.acd758f/)[A command\-line benchmarking tool 27,960rs Best score:54%](https://programbench.com/task/sharkdp__hyperfine.327d5f4/)[ggreer/the\_silver\_searcher A code\-searching tool similar to ack, but faster\. 27,080c Best score:59%](https://programbench.com/task/ggreer__the_silver_searcher.a61f178/)[Zstandard \- Fast real\-time compression algorithm 27,013c Best score:69%](https://programbench.com/task/facebook__zstd.1168da0/)[facebookresearch/fastText Library for fast text representation and classification\. 26,511cpp Best score:76%](https://programbench.com/task/facebookresearch__fasttext.1142dc4/)[robertdavidgraham/masscan TCP port scanner, spews SYN packets asynchronously, scanning entire Internet in under 5 minutes\. 25,544c Best score:57%](https://programbench.com/task/robertdavidgraham__masscan.b99d433/)[An incremental parsing system for programming tools 24,953rs Best score:37%](https://programbench.com/task/tree-sitter__tree-sitter.5e23cca/)[A simple, modern and secure encryption tool \(and Go library\) with small explicit keys, no config options, and UNIX\-style composability\. 22,077go Best score:63%](https://programbench.com/task/filosottile__age.706dfc1/)[Create book from markdown files\. Like Gitbook but implemented in Rust 21,541rs Best score:55%](https://programbench.com/task/rust-lang__mdbook.37273ba/)[n³ The unorthodox terminal file manager 21,506c Best score:98%](https://programbench.com/task/jarun__nnn.cb2c535/)[Terminal JSON viewer & processor 20,433go Best score:76%](https://programbench.com/task/antonmedv__fx.86d0d34/)[yq is a portable command\-line YAML, JSON, XML, CSV, TOML, HCL and properties processor 15,281go Best score:39%](https://programbench.com/task/mikefarah__yq.602586d/)[⬛️ CLI tool and library for saving complete web pages as a single HTML file 15,024rs Best score:51%](https://programbench.com/task/y2z__monolith.8702e66/)[unclutter your \.profile 14,998go Best score:62%](https://programbench.com/task/direnv__direnv.02040c7/)[Brotli compression format 14,673c Best score:91%](https://programbench.com/task/google__brotli.b3dc9cc/)[Make JSON greppable\! 14,424go Best score:90%](https://programbench.com/task/tomnomnom__gron.88a6234/)[Count your code, quickly\. 14,300rs Best score:70%](https://programbench.com/task/xampprocky__tokei.505d648/)[⚡A CLI tool for code structural search, lint and rewriting\. Written in Rust 13,541rs Best score:12%](https://programbench.com/task/ast-grep__ast-grep.dde0fe0/)[cheat allows you to create and view interactive cheatsheets on the command\-line\. It was designed to help remind \*nix system administrators of options for commands that they use frequently, but not frequently enough to remember\. 13,278go Best score:60%](https://programbench.com/task/cheat__cheat.b8098dc/)[Text\-mode interface for git 13,200c Best score:84%](https://programbench.com/task/jonas__tig.8334123/)[a small build system with a focus on speed 12,895cpp Best score:72%](https://programbench.com/task/ninja-build__ninja.cc60300/)[A new way to see and navigate directory trees : https://dystroy\.org/broot 12,619rs Best score:67%](https://programbench.com/task/canop__broot.d6c798e/)[Ping, but with a graph 12,433rs Best score:78%](https://programbench.com/task/orf__gping.26eb5b9/)[🌀 A nonsense activity generator 11,995rs Best score:59%](https://programbench.com/task/svenstaro__genact.16f96e3/)[Extremely Fast Compression algorithm 11,781c Best score:83%](https://programbench.com/task/lz4__lz4.1519f46/)[Command\-line Git information tool 11,745rs Best score:82%](https://programbench.com/task/o2sh__onefetch.e5958ce/)[A more intuitive version of du in rust 11,609rs Best score:71%](https://programbench.com/task/bootandy__dust.62bf1e1/)[🕳 bore is a simple CLI tool for making tunnels to localhost 11,075rs Best score:69%](https://programbench.com/task/ekzhang__bore.8e059cd/)[A fast CSV command line toolkit written in Rust\. 10,757rs Best score:83%](https://programbench.com/task/burntsushi__xsv.f430466/)[Public repository of the QuickJS Javascript Engine\. 10,565c Best score:4%](https://programbench.com/task/bellard__quickjs.d7ae12a/)[Ohayou\(おはよう\), HTTP load generator, inspired by rakyll/hey with tui animation\. 10,201rs Best score:73%](https://programbench.com/task/hatoo__oha.8dc6349/)[Log file navigator 10,200cpp Best score:13%](https://programbench.com/task/tstack__lnav.ee34494/)[A command\-line hex viewer 10,086rs Best score:83%](https://programbench.com/task/sharkdp__hexyl.2e26437/)[A copy of the Lua development repository, as seen by the Lua team\. Mirrored irregularly\. All communication should be through the Lua mailing list https://www\.lua\.org/lua\-l\.html 9,908c Best score:43%](https://programbench.com/task/lua__lua.c6b4848/)[Miller is like awk, sed, cut, join, and sort for name\-indexed data such as CSV, TSV, and tabular JSON 9,842go Best score:23%](https://programbench.com/task/johnkerl__miller.8d85b46/)[Official Git mirror of the SQLite source tree 9,434c Best score:67%](https://programbench.com/task/sqlite__sqlite.839433d/)[Sloc, Cloc and Code: scc is a very fast accurate code counter with complexity calculations and COCOMO estimates written in pure Go 8,320go Best score:38%](https://programbench.com/task/boyter__scc.515f91c/)[Declarative schema migrations with schema\-as\-code workflows 8,311go Best score:55%](https://programbench.com/task/ariga__atlas.6d81150/)[A command\-line tool and Rust library with Python bindings for generating regular expressions from user\-provided test cases 8,103rs Best score:74%](https://programbench.com/task/pemistahl__grex.fa3e8ed/)[htop \- an interactive process viewer 8,021c Best score:85%](https://programbench.com/task/htop-dev__htop.523600b/)[Simplistic interactive filtering tool 7,881go Best score:77%](https://programbench.com/task/peco__peco.4e58dad/)[🌀 A log file highlighter 7,793rs Best score:76%](https://programbench.com/task/bensadeh__tailspin.6278437/)[Friendly and fast tool for sending HTTP requests 7,754rs Best score:50%](https://programbench.com/task/ducaale__xh.4a6e44f/)[🌟 For when you really just want to serve some files over HTTP right now\! 7,561rs Best score:79%](https://programbench.com/task/svenstaro__miniserve.8449e8b/)[Like jq, but for HTML\. 7,520rs Best score:94%](https://programbench.com/task/mgdm__htmlq.6e31bc8/)[parcel\-bundler/lightningcss An extremely fast CSS parser, transformer, bundler, and minifier written in Rust\. 7,515rs Best score:54%](https://programbench.com/task/parcel-bundler__lightningcss.aa2ed1e/)[A maintained ctags implementation 7,149c Best score:13%](https://programbench.com/task/universal-ctags__ctags.243595e/)[Intuitive find & replace CLI \(sed alternative\) 7,072rs Best score:91%](https://programbench.com/task/chmln__sd.87d1ba5/)[A command\-line DNS client\. 6,640rs Best score:84%](https://programbench.com/task/ogham__dog.721440b/)[static analysis of C/C\+\+ code 6,599cpp Best score:15%](https://programbench.com/task/danmar__cppcheck.0a5b103/)[Official doxygen git repository 6,422c Best score:34%](https://programbench.com/task/doxygen__doxygen.966d98e/)[A command\-line tool to generate, analyze, convert and manipulate colors 6,334rs Best score:77%](https://programbench.com/task/sharkdp__pastel.b60e899/)[the official Rust and C implementations of the BLAKE3 cryptographic hash function 6,178rs Best score:98%](https://programbench.com/task/blake3-team__blake3.15e83a5/)[:stars: Manage your shell commands\. 6,154rs Best score:15%](https://programbench.com/task/nukesor__pueue.8b9d6fe/)[GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats\. 5,875cpp Best score:25%](https://programbench.com/task/osgeo__gdal.0847f12/)[View disk space usage and delete unwanted data, fast\. 5,794rs Best score:87%](https://programbench.com/task/byron__dua-cli.8570c15/)[Fast disk usage analyzer with console interface written in Go 5,578go Best score:70%](https://programbench.com/task/dundee__gdu.ede21d2/)[Run arbitrary commands when files change 5,551c Best score:89%](https://programbench.com/task/eradman__entr.8e2e8b4/)[Mirror of the LuaJIT git repository 5,518c Best score:71%](https://programbench.com/task/luajit__luajit.a553b3d/)[🔥 ~6x faster, stricter, configurable, extensible, and beautiful drop\-in replacement for golint 5,486go Best score:46%](https://programbench.com/task/mgechev__revive.201451e/)[Automatically generate Go test boilerplate from your source code\. 5,294go Best score:62%](https://programbench.com/task/cweill__gotests.2a672c5/)[Visualize Ownership and Lifetimes in Rust 5,113rs Best score:75%](https://programbench.com/task/cordx56__rustowl.655bc5c/)[Terminal based "The Matrix" like implementation 5,042c Best score:97%](https://programbench.com/task/abishekvashok__cmatrix.5c082c6/)[Async\-friendly QUIC implementation in Rust 5,041rs Best score:62%](https://programbench.com/task/quinn-rs__quinn.bb359cc/)[A general purpose syntax highlighter in pure Go 4,910go Best score:16%](https://programbench.com/task/alecthomas__chroma.8d04def/)[The corrective bash syntax highlighter 4,778rs Best score:82%](https://programbench.com/task/anordal__shellharden.6a6ffd4/)[Melody is a language that compiles to regular expressions and aims to be more readable and maintainable 4,748rs Best score:79%](https://programbench.com/task/yoav-lavi__melody.f4af9b4/)[A hackable, minimal, fast TUI file explorer 4,735rs Best score:60%](https://programbench.com/task/sayanarijit__xplr.1751065/)[📺🗿 Terminal graphics for the 21st century\. 4,648c Best score:58%](https://programbench.com/task/hpjansson__chafa.dd4d4c1/)[Find files with SQL\-like queries 4,420rs Best score:44%](https://programbench.com/task/jhspetersson__fselect.c3559ca/)[Convert your ascii diagram scribbles into happy little SVG 4,182rs Best score:41%](https://programbench.com/task/ivanceras__svgbob.6d00ad9/)[Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more\. 3,867go Best score:80%](https://programbench.com/task/multiprocessio__dsq.c3ae0ba/)[Slice and dice logs on the command line 3,727rs Best score:38%](https://programbench.com/task/rcoh__angle-grinder.9c2fc88/)[The power of curl, the ease of use of httpie\. 3,637go Best score:89%](https://programbench.com/task/rs__curlie.5dfcbb1/)[Terminal file manager 3,598go Best score:74%](https://programbench.com/task/antonmedv__walk.bf802ef/)[JohannesKaufmann/html\-to\-markdown ⚙️ Convert HTML to Markdown\. Even works with entire websites and can be extended through rules\. 3,586go Best score:86%](https://programbench.com/task/johanneskaufmann__html-to-markdown.3006818/)[TheZoraiz/ascii\-image\-converter A cross\-platform command\-line tool to convert images into ascii art and print them on the console\. Now supports braille art\! 3,284go Best score:64%](https://programbench.com/task/thezoraiz__ascii-image-converter.d05a757/)[A flexible commandline tool for template rendering\. Supports lots of local and remote datasources\. 3,135go Best score:75%](https://programbench.com/task/hairyhenderson__gomplate.05eb3aa/)[7\-Zip 2,967cpp Best score:34%](https://programbench.com/task/ip7z__7zip.839151e/)[A parallel implementation of gzip for modern multi\-processor, multi\-core machines\. 2,924c Best score:83%](https://programbench.com/task/madler__pigz.fe4894f/)[Unofficial mirror of mob development branch 2,843c Best score:13%](https://programbench.com/task/tinycc__tinycc.9b8765d/)[Fast website link checker in Go 2,597go Best score:88%](https://programbench.com/task/raviqqe__muffet.a882908/)[CLI for managing secrets 2,588go Best score:82%](https://programbench.com/task/segmentio__chamber.5f93f5f/)[Go implement CLI, cURL\-like tool for humans 2,563go Best score:72%](https://programbench.com/task/astaxie__bat.17d1080/)[Plain text note\-taking assistant 2,542go Best score:43%](https://programbench.com/task/zk-org__zk.10d93d5/)[errcheck checks that you checked errors\. 2,480go Best score:80%](https://programbench.com/task/kisielk__errcheck.dacab89/)[Dropbear SSH 2,231c Best score:58%](https://programbench.com/task/mkj__dropbear.75f699b/)[CLI tool that can execute SQL queries on CSV, LTSV, JSON, YAML and TBLN\. Can output to various formats\. 2,159go Best score:67%](https://programbench.com/task/noborus__trdsql.d8c5ff6/)[🐧ping command but with pingu 2,087go Best score:97%](https://programbench.com/task/sheepla__pingu.926d475/)[The most opinionated Go source code linter for code audit\. 2,041go Best score:42%](https://programbench.com/task/go-critic__go-critic.9aea378/)[PROJ \- Cartographic Projections and Coordinate Transformations Library 1,974cpp Best score:74%](https://programbench.com/task/osgeo__proj.75d455c/)[🎑Feature\-rich terminal\-based text viewer\. It is a so\-called terminal pager\. 1,935go Best score:88%](https://programbench.com/task/noborus__ov.b96c2ba/)[Tools \(written in C using htslib\) for manipulating next\-generation sequencing data 1,886c Best score:14%](https://programbench.com/task/samtools__samtools.aa823b5/)[Tool for helping developers keep their code bases clean and decoupled\. It allows visualising a code base complexity using a 3d force\-directed graph of files and the dependencies between them\. 1,706go Best score:65%](https://programbench.com/task/gabotechs__dep-tree.60a95a2/)[Claudio's FIGlet tree 1,606c Best score:78%](https://programbench.com/task/cmatsuoka__figlet.202a0a8/)[Toolkit for processing sequences in FASTA/Q formats 1,537c Best score:67%](https://programbench.com/task/lh3__seqtk.94e7070/)[XZ Utils 1,522c Best score:36%](https://programbench.com/task/tukaani-project__xz.1007bf0/)[Declarative pure\-SQL schema management for MySQL and MariaDB 1,361go Best score:76%](https://programbench.com/task/skeema__skeema.6a76243/)[CLI tool for summarizing go test output\. Pipe friendly\. CI/CD friendly\. 1,246go Best score:78%](https://programbench.com/task/mfridman__tparse.2416b4b/)[A text\-based calendar and scheduling application 1,243c Best score:54%](https://programbench.com/task/lfos__calcurse.49180d5/)[WSDL2Go code generation as well as its SOAP proxy 1,219go Best score:86%](https://programbench.com/task/hooklift__gowsdl.2a06cec/)[Your dev tool to manage /etc/hosts like a pro\! 1,216go Best score:83%](https://programbench.com/task/guumaster__hostctl.d6d9699/)[iTerm2 expvar/JSON monitoring tool 1,178go Best score:89%](https://programbench.com/task/rs__jplot.2a54bcc/)[Git powered terminal\-based todo/note manager \-\- markdown note page per task\. Single binary\! 1,157go Best score:59%](https://programbench.com/task/naggie__dstask.ff57396/)[A Bash CLI framework, also a Bash command runner\. 1,135rs Best score:44%](https://programbench.com/task/sigoden__argc.04a08f1/)[Command\-line XML and HTML beautifier and content extractor 1,109go Best score:76%](https://programbench.com/task/sibprogrammer__xq.b89f681/)[Clock using lib ncurses 1,105c Best score:84%](https://programbench.com/task/xorg62__tty-clock.f2f847c/)[A CLI code\-typing game that turns your source code into typing challenges 1,075rs Best score:91%](https://programbench.com/task/unhappychoice__gittype.34b72d0/)[A plain text\-based spaced repetition system\. 1,071rs Best score:56%](https://programbench.com/task/eudoxia0__hashcards.48aa136/)[Fast Markdown linter and formatter written in Rust 1,051rs Best score:41%](https://programbench.com/task/rvben__rumdl.2d75c4d/)[CLI \- Convert between YAML, TOML, JSON, and HCL\. Preserves map order\. 1,041go Best score:74%](https://programbench.com/task/sclevine__yj.8016400/)[bedtools \- the swiss army knife for genome arithmetic 1,029c Best score:39%](https://programbench.com/task/arq5x__bedtools2.dd57059/)[Converts jpg images to ASCII 1,021c Best score:56%](https://programbench.com/task/cslarsen__jp2a.61d205f/)[A modern alternative to the watch command, records the differences in execution results and can check this differences at after\. 1,016rs Best score:81%](https://programbench.com/task/blacknon__hwatch.edfcb62/)[Draw images in your ANSI terminal with true color 1,014go Best score:75%](https://programbench.com/task/eliukblau__pixterm.1a93fd5/)[A nginx log explorer 1,006rs Best score:53%](https://programbench.com/task/canop__rhit.ae90bcb/)[ditaa is a small command\-line utility that can convert diagrams drawn using ascii art \('drawings' that contain characters that resemble lines like \| / \- \), into proper bitmap graphics\. 1,005java Best score:20%](https://programbench.com/task/stathissideris__ditaa.f2286c4/)[ELF visualizer\. Generates HTML files from ELF binaries\. 990rs Best score:98%](https://programbench.com/task/rbakbashev__elfcat.52f8cc7/)[A command\-line shell like fish, but POSIX compatible\. 966rs Best score:84%](https://programbench.com/task/nuta__nsh.bdd0702/)[A code search / replace tool 941rs Best score:71%](https://programbench.com/task/dalance__amber.69a0f52/)[pls is a prettier and powerful ls\(1\) for the pros\. 932rs Best score:62%](https://programbench.com/task/pls-rs__pls.4e1ae50/)[Universal multi\-language runner and smart REPL written in Rust\. 919rs Best score:85%](https://programbench.com/task/esubaalew__run.0fb9dec/)[SoX, Swiss Army knife of sound processing 913c Best score:38%](https://programbench.com/task/chirlu__sox.42b3557/)[Generate beautiful changelogs from your Git commit history 912rs Best score:93%](https://programbench.com/task/clog-tool__clog-cli.7066cba/)[An extended \`cp\` 911rs Best score:93%](https://programbench.com/task/tarka__xcp.5e5b448/)[a calculator REPL, similar to bc\(1\) 907rs Best score:89%](https://programbench.com/task/oppiliappan__eva.41ae245/)[Command line tool to show clear git graphs arranged for your branching model 904rs Best score:80%](https://programbench.com/task/git-bahn__git-graph.87b4473/)[Public/backup repository of the GROMACS molecular simulation toolkit\. Please do not mine the metadata blindly; we use https://gitlab\.com/gromacs/gromacs for code review and issue tracking\. 901cpp Best score:9%](https://programbench.com/task/gromacs__gromacs.665ea4c/)[A command\-line tool to prevent committing secret keys into your source code 901rs Best score:73%](https://programbench.com/task/sirwart__ripsecrets.34c9e03/)[Quickly Extracts IP's, Email Addresses, Hashes, Files, Credit Cards, Social Security Numbers and a lot More From Text 890rs Best score:74%](https://programbench.com/task/drew-alleman__datasurgeon.d257cee/)[A grep\-like tool which understands source code syntax and allows for manipulation in addition to search 889rs Best score:69%](https://programbench.com/task/alexpovel__srgn.89f943b/)[tui file manager with vim\-like key mapping 888rs Best score:49%](https://programbench.com/task/kyoheiu__felix.95df390/)[lints and suggestions for the nix programming language 882rs Best score:43%](https://programbench.com/task/oppiliappan__statix.e9df54c/)[a tool to analyze file system usage written in Rust 871rs Best score:90%](https://programbench.com/task/nachoparker__dutree.44e877d/)[💡 CLI tool to input and store your ideas without leaving the terminal 867rs Best score:79%](https://programbench.com/task/simeg__eureka.df3796c/)[Enrich \`go test\` outputs with text decorations\. 863go Best score:85%](https://programbench.com/task/kyoh86__richgo.313114f/)[Markdown makes sites \- A Static Site Generator for Blogs 837rs Best score:45%](https://programbench.com/task/rochacbruno__marmite.7d4bc2d/)[Generate Rust register maps \(\`struct\`s\) from SVD files 835rs Best score:73%](https://programbench.com/task/rust-embedded__svd2rust.1760b5e/)[Interactive Grep 827rs Best score:74%](https://programbench.com/task/konradsz__igrep.aa75630/)[A simple timetracker for the command line\. It saves a log of all tracked activities as a plaintext file and allows you to create flexible reports\. 827rs Best score:87%](https://programbench.com/task/nikolassv__bartib.6b9b5ce/)[A simple terminal UI for search and replace, ala VS Code\. 824rs Best score:61%](https://programbench.com/task/yassinebridi__serpl.c48a9d7/)[When cut doesn't cut it 820rs Best score:93%](https://programbench.com/task/riquito__tuc.16fb471/)[A 3D software rasterizer\.\.\. for the terminal\! 818rs Best score:53%](https://programbench.com/task/ecumene__rust-sloth.051c559/)[Converts books written in Markdown to HTML, LaTeX/PDF and EPUB 813rs Best score:60%](https://programbench.com/task/crowdagger__crowbook.ea214d7/)[An extremely fast LaTeX formatter written in Rust 789rs Best score:81%](https://programbench.com/task/wgunderwood__tex-fmt.3f1aef6/)[A high\-performance JSON Schema validator for Rust 770rs Best score:52%](https://programbench.com/task/stranger6667__jsonschema.d52e881/)[A small terminal UTF\-8 text editor written in Rust 📝🦀 761rs Best score:93%](https://programbench.com/task/rhysd__kiro-editor.4157485/)[Scan Nix files for dead code 745rs Best score:86%](https://programbench.com/task/astro__deadnix.d590041/)[A sharp cut\(1\) clone\. 738rs Best score:96%](https://programbench.com/task/sstadick__hck.b66c751/)[Git genealogy, untangled\. A TUI for navigating commit graphs with color and clarity\. 729rs Best score:67%](https://programbench.com/task/trasta298__keifu.3331426/)[Your journal app if you live in a terminal 722rs Best score:71%](https://programbench.com/task/ammarabouzor__tui-journal.2b4540d/)[incu6us/goimports\-reviser Right imports sorting & code formatting tool \(goimports alternative\) 715go Best score:86%](https://programbench.com/task/incu6us__goimports-reviser.81bd549/)[Batch rename utility for developers 710rs Best score:80%](https://programbench.com/task/yaa110__nomino.f892499/)[📠 Pretty and fast csv viewer for cli with cjk/emoji support\. 694rs Best score:96%](https://programbench.com/task/wfxr__csview.8ac4de0/)[A better xdg\-utils 693rs Best score:91%](https://programbench.com/task/chmln__handlr.90e78ba/)[UNIX's missing \`loop\` command 692rs Best score:95%](https://programbench.com/task/miserlou__loop.209927c/)[KSXGitHub/parallel\-disk\-usage Highly parallelized, blazing fast directory tree analyzer 689rs Best score:86%](https://programbench.com/task/ksxgithub__parallel-disk-usage.96978ed/)[Hush is a unix shell based on the Lua programming language 688rs Best score:83%](https://programbench.com/task/hush-shell__hush.560c33a/)[Dude, where are my bytes: Duc, a library and suite of tools for inspecting disk usage 682c Best score:83%](https://programbench.com/task/zevv__duc.a58fa4e/)[🎨 Make your i3 config a little more stylish\. 678rs Best score:80%](https://programbench.com/task/altdesktop__i3-style.f93821b/)[A TUI tool to help you type faster and learn new layouts\. Includes a free cat\. 674rs Best score:84%](https://programbench.com/task/wintermute-cell__ngrrram.8ea13c3/)[Find outdated dependencies of your Go projects\. go\-mod\-outdated provides a table view of the go list \-u \-m \-json all command which lists all dependencies of a Go project and their available minor and patch updates\. It also provides a way to filter indirect dependencies and dependencies without updates\. 669go Best score:98%](https://programbench.com/task/psampaz__go-mod-outdated.bb79367/)[🛰 A high performance code minimap render\. 660rs Best score:89%](https://programbench.com/task/wfxr__code-minimap.0ddeea5/)[Peek inside Parquet files right from your terminal 654rs Best score:59%](https://programbench.com/task/kaushiksrini__parqeye.8072121/)[Stacked Git 652rs Best score:20%](https://programbench.com/task/stacked-git__stgit.430027d/)[Fast directory scanning and scraping tool 632rs Best score:67%](https://programbench.com/task/isona__dirble.e2dea9f/)[Flamegraph viewer in the terminal 622rs Best score:59%](https://programbench.com/task/ys-l__flamelens.0b4dc33/)[Yet another diff highlighting tool 612rs Best score:85%](https://programbench.com/task/mookid__diffr.2152742/)[⚡Rapid note management for the terminal\. 609rs Best score:85%](https://programbench.com/task/shashwatah__jot.a92aad8/)[A \(TUI/CLI\) markdown navigator with tree\-based structural navigation\. 603rs Best score:55%](https://programbench.com/task/epistates__treemd.825c6dd/)[A CLI to organize and run short Unix shell scripts 596rs Best score:84%](https://programbench.com/task/pier-cli__pier.5e1bde9/)[✨ sleek typing tui with visualized results and historical logging 595rs Best score:82%](https://programbench.com/task/jrnxf__thokr.09375ef/)[A command\-line tool to batch rename files and directories 581rs Best score:82%](https://programbench.com/task/ismaelgv__rnr.fc0733b/)[🔮 Futuristic take on hexdump, made in Rust\. 563rs Best score:92%](https://programbench.com/task/sitkevij__hex.61ae69b/)[Small command\-line JSON Log viewer 561rs Best score:86%](https://programbench.com/task/brocode__fblog.3b54330/)[🦀️📸 Pure Rust tool to generate beautiful code snapshots, provide CLI and Library 557rs Best score:59%](https://programbench.com/task/codesnap-rs__codesnap.f81e4f3/)[Automatically trims your branches whose tracking remote refs are merged or stray 548rs Best score:65%](https://programbench.com/task/foriequal0__git-trim.07c2f50/)[🎁 generate beautiful landing pages for your developer tools 542rs Best score:54%](https://programbench.com/task/axodotdev__oranda.27d60c7/)[A tool to interactively write shell pipelines\. 541rs Best score:57%](https://programbench.com/task/elkowar__pipr.fae0b17/)[Blazingly fast, modular and contributor friendly Solidity compiler, written in Rust 539rs Best score:43%](https://programbench.com/task/paradigmxyz__solar.5190d0e/)[Caesium Command Line Tools \- Lossy/lossless image compression tool 537rs Best score:92%](https://programbench.com/task/lymphatus__caesium-clt.a529b2e/)[agourlay/zip\-password\-finder Find the password of protected ZIP files\. 534rs Best score:98%](https://programbench.com/task/agourlay__zip-password-finder.704700d/)[Encode and decode smart contract invocations 525rs Best score:91%](https://programbench.com/task/rust-ethereum__ethabi.b1710ad/)[A JSON terminal UI made in C\+\+ 438cpp Best score:71%](https://programbench.com/task/arthursonzogni__json-tui.17a22b6/)[A Go linter to check that errors from external packages are wrapped 374go Best score:81%](https://programbench.com/task/tomarrell__wrapcheck.c058da1/)[A small TUI journaling tool\. 📖 370cpp Best score:62%](https://programbench.com/task/nikoladucak__caps-log.2cf2d1e/)[a tool for code clone detection 367go Best score:85%](https://programbench.com/task/mibk__dupl.1bf052b/)[@twosigma's first artificial intelligence programming challenge 202cpp Best score:80%](https://programbench.com/task/halitechallenge__halite.822cfb6/) ## Citation ``` @misc{yang2026programbenchlanguagemodelsrebuild, title={ProgramBench: Can Language Models Rebuild Programs From Scratch?}, author={John Yang and Kilian Lieret and Jeffrey Ma and Parth Thakkar and Dmitrii Pedchenko and Sten Sootla and Emily McMilin and Pengcheng Yin and Rui Hou and Gabriel Synnaeve and Diyi Yang and Ofir Press}, year={2026}, eprint={2605.03546}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2605.03546}, } ```