On the Value of Human Ideas: What data poisoning research reveals about "autonomous" AI breakthroughs

I was reading up on the recent controversy around Tristan Buckmaster, Levent Alpöge, OpenAI, and the Navier–Stokes result, and it got me thinking about something broader than this particular dispute. Buckmaster says that he and Alpöge had been putting drafts from their project into Codex while working on it. OpenAI says that neither its researchers nor its agents accessed their specific user data while solving Navier–Stokes, but also says that it cannot rule out that de-identified data derived from their use of OpenAI products helped improve its models. Whatever ultimately happened in this particular case, that last possibility raises a question I haven't really seen discussed enough: How much could the accumulated half-finished ideas of millions of human users actually contribute to what we later call "AI discoveries"? There is a relevant result from AI security research by researchers at the UK AI Security Institute, Anthropic, the Alan Turing Institute, Oxford and others. They studied data-poisoning attacks and found that the number of poisoned documents needed to implant a particular backdoor behavior remained surprisingly close to constant as they scaled both the model and the amount of clean training data. In their largest pretraining experiment, a 13B-parameter model was trained on 260 billion tokens. Just 250 poisoned documents (about 420,000 tokens, or 0.00016% of the training tokens) were enough to reliably implant the tested backdoor. This same attack worked across models from 600M to 13B parameters despite the largest model seeing more than twenty times as much clean data. In their fine-tuning experiments they found similar dynamics; in one GPT-3.5 experiment, roughly 50–90 poisoned examples could produce greater than 80% attack success even as the amount of clean fine-tuning data varied by two orders of magnitude. Obviously, teaching a model to respond to a backdoor trigger is not the same thing as teaching it a new piece of mathematics. I don't want to make the leap that 250 clever research notes are enough to make a model solve Navier–Stokes. But I do think it undermines a very intuitive argument people make about training data: "A few conversations are nothing compared with hundreds of billions or trillions of tokens. They would just be diluted away." Apparently, at least for some kinds of learning, that's not how it works. A tiny absolute amount of highly consistent, targeted data can have an effect wildly disproportionate to its percentage of the dataset. Now think about how researchers actually use LLMs. Someone asks ChatGPT whether an unusual substitution makes sense. Someone else uploads a half-written proof to Claude to find a weak point. A PhD student tries an obscure lemma, discovers it would require months of technical estimates, and abandons it. A professor talks through an approach that seems promising but not enough to pursue. Someone notices a strange analogy between two fields, discusses it with an AI for twenty minutes, then forgets the conversation. Most of these things never become papers. They are fragments: intuitions, failed approaches, potentially useful transformations, conjectures, objections, shortcuts, and little pieces of tacit knowledge about where a problem might yield. Individually, almost all of them are probably worthless. But imagine the aggregate. A frontier AI company potentially sits at the intersection of an enormous amount of human intellectual activity. Thousands of people might independently poke at the same famous open problem without knowing what others tried. But the provider of the tool is in a fundamentally different position: depending on its data policies and training pipeline, information derived from all of those interactions could eventually influence later models. Maybe researcher A contributes a useful ansatz but abandons it. Researcher B independently notices the obstruction. Researcher C knows an obscure theorem that gets around part of the obstruction. Researcher D tries a numerical experiment that suggests which parameter regime matters. Researcher E has almost the whole idea but decides the remaining proof would be too tedious. No one person solved the problem. There is nothing to plagiarize in the traditional sense. But collectively, humans may have supplied a remarkable amount of the search landscape. Then a later model, combined with enormous inference-time search, formal verification, or agents, connects the pieces and finishes the job. What exactly should we call that? It might still be an extraordinary achievement in machine reasoning. Synthesizing ideas that no human had connected, filling in technical gaps and verifying the result could itself be genuinely novel. But it would be a very different kind of achievement from the image suggested by the phrase "the AI independently solved an open problem." It would be something more like distributed human-machine discovery: humans collectively generating a huge cloud of partial ideas and the model becoming extremely good at remembering, recombining, extending and searching through that cloud. This is where the poisoning result is conceptually interesting. While it does not establish that this is happening with mathematical ideas, it gives us reason to be careful about assuming that an idea must appear millions of times before it can meaningfully affect a model. I increasingly think AI may be less an independent inventor than an extremely powerful tool for organizing and recombining information that was previously too sparse or disconnected for any one person to put together. As that ability improves, we may see more "discoveries" that are genuinely new combinations, but whose raw ingredients came from many different humans. In a "perfect" world where everyone freely shared every half-formed idea and unfinished proof without worrying about credit, science would move much faster. AI may be creating something close to that shared intellectual space, but without preserving who contributed which pieces. If so, the question is: how can we design a system where even our weakest ideas can be contributed and synthesized into groundbreaking discovery, with proper credit? Is that even possible? And what would it look like?