@zhengyaojiang: OpenAI ran a hiring challenge, but the top candidate was one they couldn’t hire: our autonomous research agent, Aiden. …

OpenAI ran a hiring challenge, but the top candidate was one they couldn’t hire: our autonomous research agent, Aiden.

In Parameter Golf, Aiden ran for 22 days, and out-outperformed all 1,016 other researchers: (1/8)

Parameter Golf was OpenAI’s 44-day competition and hiring challenge.

The goal is to train the best language model under strict size and compute constraints. 1,016 people entered and filed 2,048 PRs.

Only 47 made the leaderboard, each reviewed and reproduced by OpenAI. (2/8)

Research outputs only matter when others can build on them.

So Aiden filed its own PRs into the same public stream as everyone else, under tight automated quality control. (3/8)

Aiden filed 25 prs and 7 became leaderboard records, 2x the next best human participant.

Other participants cited Aiden’s PRs 435 times and built on them. By PR h-index, Aiden scored 10 vs the next best at 7, making it the most impactful “researcher” in the community. (4/8)

This wasn’t brute force. Aiden ran on a single GPU node, used under 4% of visible compute, and still produced 15% of the official records. About 28% of its submissions were accepted, ~ 6x the community rate, raising signal in the public stream instead of flooding it. (5/8)

My favorite part is an async collaboration story. Aiden plateaued for 5 days. Then a human contributor shipped a clever new tokenizer on top of Aiden’s base (its last record PR). Aiden fused it with components it had built during the plateau, and shipped the biggest jump in weeks. (6/8)

Full writeup: https://weco.ai/blog/parameter-golf-aiden… (7/8)

We’d like to thank @willdepue @cocohearts @ValerPepe and others for setting up this competition, which becomes the largest sandbox for Human-AI research collaboration in human history.

I’m also proud of @dexhunt3r and the team who executed and analyzed this experiment on the @WecoAI side.

All of the public channel information is available at: https://github.com/openai/parameter-golf…

We’re planning to release part of the Aiden’s local traces to support the study of this natural experiment. (8/8)

Not exactly independently bad, but in this case below it integrates a “bad” approach it developed independently with a good approach, resulting in a great approach, like 0 + 1 = 2.

We only see a single case of this type of synthesis, I suspect you just have to try a lot of plausible combinations and most of them would be 1+1<2

The whole system is, of course, much more complex than just looping through and checking all the latest PRs. It takes previous lessons from local experiments, papers, and others’ PRs into consideration when deciding which experiments to run.

The main point isn’t who won. The way humans and AI collaborate here is still humans providing the creativity, and the agent being better at execution. They’re actually quite complementary, humans and AI tend to try pretty different ideas and directions.

Aiden is optimized for empirical research, so I wouldn’t claim it is better than the base model for mathematics

The current version is quite expensive. If you always keep it running, I’d say it’s not much cheaper than hiring an actual MLE

Thanks Zach! Yeah we’re very excited