The author argues for reviving 'Needle in a haystack' benchmarks to evaluate AI capabilities, sharing private test results that show many models performing poorly in remembering instructions, questioning their trustworthiness for real-world tasks.
We really need to bring back Needle in a haystack comparisons as the primary way to gauge AI abilities because Jesus Christ every AI is progressively getting worse at it ever since it left the public scene. Everyone used to talk about this benchmark all the time and it honestly did far more for AI capabilities than people remember it doing. Like seriously I have no idea how anybody can trust an AI with coding literally anything at all if it ignores or forgets over 20% of your instructions. I started running my own private benchmark for this some time in late 2024 and so far here are the results of my most recent run. Gemini 2.5 pro: 72% Gemini 3.1 pro: 66% Gemini 3 flash: 54% Gemini 3.5 flash: 62% Gemini 3.6 flash: 67% Gemini 3.7 flash: 63% Claude 3 Opus: 69% Claude 3.5 Sonnet: 72% Claude 3.5 Sonnet v2: 75% Claude 3.7 Sonnet: 80% Claude 4 Sonnet: 76% Claude 4 Opus: 74% Claude 4.1 Opus: 79% Claude 4.5 Heiku: 73% Claude 4.5 Sonnet: 70% Claude 4.5 Opus: 72% Claude 4.6 Sonnet: 69% Claude 4.6 Opus: 73% Claude 4.7 Opus: 71% Claude 4.8 Opus: 70% Claude 5 Fable: 65% Claude 5 Sonnet: 67% Claude 5 Opus: 71% GPT 4: 74% GPT 4o mini: 73% o1 preview: 77% o1 mini: 78% o1: 75% o3 mini: 71% GPT 4.1: 76% GPT 4.1 mini: 69% GPT 4.1 nano: 66% GPT 4.5: 59% o3: 60% o4 mini: 61% codex 1: 68% GPT 5: 63% GPT 5 Codex: 62% GPT 5.1: 66$ GPT 5.1 Codex Max: 71% GPT 5.2: 67% GPT 5.3 Codex: 72% GPT 5.4: 72% GPT 5.4 mini: 69% GPT 5.4 nano: 58% GPT 5.5: 65% GPT 5.6 Luna: 66% GPT 5.6 Terra: 67% GPT 5.6 Sol: 67% Other AI models aren't any better.... To put these numbers into perspective, a normal human on this test should be able to score 100% without even putting a tiny bit of effort into it. If a human actually put real effort into doing this benchmark, they would get greater than 100% because the scale wasn't designed for going above and beyond the minimum requirements. The entire purpose of my private tests was to find out which models you could trust to actually do the work you give them and its honestly horrifying to me that anybody is out here trusting AI to do literally ANY amount of work at all. A score of less than 60% means the AI was only able to remember or include 70 out of 100 rules / details given to it. Yep, you read that right, the AI was only able to remember 70 out of 100 rules / details. Look at the scores. Realize how terrible they are and wonder what the fuck are we even doing here. The way the benchmark is formatted is to give the AI a simple task. Write the first 4 chapters of a web novel. The format of the request varies from conversational to sudo-code all the way to actual code block formatting. And I'll be honest here people, the formatting wasn't the issue. It didn't matter if the provided request was formatted with hyper precision, with every thing segmented into easy to understand categories vs a run on sentence gigantic block of text. Every AI failed the test. To pass the test would require a score of at least 90%. And honestly I personally would never trust an AI to code anything if it couldn't score at least 100%, being on par with a high school freshman. I will not be posting the benchmark and I wont go deep into the details because I don't want it to find its way into future AI training data. You may think that there isn't any overlap between asking an AI to write stories vs Coding, but you would be dead wrong. The only thing that mattered in this test was for the AI to follow the instructions given to it and not leave out or skip over anything and how that effects its future work. I wasn't looking for Amazing story telling. I was looking at its ability to remember key details, rules and how its memory of those things effect its later writing. For example, If in chapter 1, it forgot to say "Charly forgave Sarah." Does that effect how the AI sets the tone in chapter 4 where I hand off to the AI a choice on how Charly will treat Sarah. And the answer is, yes, it does effect how the AI interprets how Charly treats Sarah when it wrongly leaves out the earlier forgiveness scene. This means that yes, your AI will likely overlook or not include rules you gave it early on in its workflow, and its later workflow will suffer because of that, even though the rule exists in the prompt / context window. If you tell it to write you a draft for an App and include rules like "don't put passwords in plain text.". If the AI does not include that detail in its draft, it will in fact become an issue later on even though the prompt is still in the context window. Anyways, just wanted to drop this somewhere and rant a bit about it. I'm not nearly as tech savvy as the rest of yall so forgive me for not spouting any buzzwords or using technical jargon. It's honestly dead simple for anybody and their dog to create a benchmark thanks to youtube videos and guides online and I really hope that some of yall out there will take the time to create your own because seriously the states of some of these AI are absolutely abysmal.
The article discusses the growing disconnect between high AI benchmark scores and actual real-world performance, highlighting issues like consistency, latency, and context handling.
The article argues that AI benchmarks and flashy demos are overemphasized; the real test for AI trustworthiness is how models handle boring real-world responsibilities like following instructions, admitting uncertainty, handling edge cases, and being auditable.
The tweet critiques traditional AI benchmarks and introduces TRACES, a new benchmark that evaluates AI's discovery process by focusing on how models reach answers, including tool usage, error correction, and evidence tracing.
This paper from July 2026 defines principles for designing effective benchmark tasks for AI agents, drawing on experience from the Terminal Bench project. Good tasks are correct, solvable, verifiable, well-specified, and hard for interesting reasons, with an emphasis on real-world relevance and outcome-based verification.