Andrej Karpathy: From Vibe Coding to Agentic Engineering

**TL;DR:** Andrej Karpathy discusses the December 2024 shift where LLMs reached a new level of reliability, coining “vibe coding” for the floor-raising and “agentic engineering” for the ceiling-raising, and argues that verifiability is key to understanding AI’s jagged capabilities while human taste, judgment, and understanding remain irreplaceable. ## The December Shift: From Vibe Coding to Agentic Engineering Andrej Karpathy opens by recounting the moment last December when his relationship with AI coding tools fundamentally changed. He had been using agentic tools like AlphaCode for about a year – good for chunks of code, but often requiring corrections. Then in December, while on a break, he noticed that the latest models “just came out fine. And then I kept asking for more, and it just came out fine. I can't remember the last time I corrected it.” That trust led him straight into what he later called “vibe coding.” He stresses that this wasn’t a gradual improvement – it was “a very stark transition.” Many people still think of AI as a ChatGPT-like assistant from a year ago, but as of December, “things have changed fundamentally, especially on this agentic coherent workflow that really started to actually work.” The result: his side projects folder became “extremely full” and he has been “coding all the time.” ## Software 3.0: A New Computing Paradigm Karpathy revisits his framework: Software 1.0 is writing explicit rules, Software 2.0 is programming by creating datasets and training neural networks. Now Software 3.0 is different – “programming turns to prompting, and what's in the context window is your lever over the interpreter that is the LLM.” The LLM is a kind of programmable computer that interprets context and performs computation in digital information space. ### The OpenCL Installation Example When you want to install OpenCL, traditionally you’d write a shell script – but those scripts balloon as they try to target many platforms. Karpathy points out that the correct way now is to give your agent a block of text: a copy-paste prompt that instructs the agent to install OpenCL. The agent uses its own intelligence to look at your environment, debug in the loop, and make things work. “What is the piece of text to copy-paste to your agent? That's the programming paradigm now.” ### MenuGen: A Striking Demonstration Karpathy built **MenuGen**, an app that lets you photograph a restaurant menu and get pictures of each dish. He coded it traditionally: upload photo, OCR, image generator, rendering on Vercel. Then he saw the Software 3.0 version: “literally just take your photo, give it to Gemini, and say 'use Nano Banana to overlay the things onto the menu.'” Nano Banana returned an image with the items rendered directly into the pixels. “This blew my mind because actually all of my MenuGen is spurious – it's working in the old paradigm. That app shouldn’t exist.” The neural network does all the work; no app layer is needed. He warns that people still think in terms of speeding up existing workflows. “It’s actually new things that are available now – new opportunities of things that couldn't be possible before. And I almost think that's more exciting.” ## Verifiability and Jagged Intelligence Karpathy introduced the concept of **verifiability**: traditional computers automate what you can specify in code; LLMs easily automate what you can verify. Frontier labs train models in giant reinforcement learning environments, giving verification rewards. The models then peak in capability in verifiable domains like math and code, and stagnate elsewhere – creating “jagged” intelligence. ### Why Are Models Jagged? He gives two reasons: (1) the labs focus on what is economically valuable (code, math) so they create RL environments in those areas; (2) whatever happens to be in the data distribution matters. Example: chess improved a lot from GPT-3.5 to GPT-4, not just because of scaling but because someone at OpenAI added a huge amount of chess data to the pre-training set. The classic “how many letters in strawberry?” question is being patched, but the new test is: “I want to go to a car wash 50 m away – should I drive or walk?” State-of-the-art models will say walk. “How is it possible that a model can refactor a 100k line codebase or find zero-day vulnerabilities and yet tells me to walk to the car wash? This is insane.” That jaggedness means you must treat these tools with awareness: “You need to actually be in the loop a little bit and stay in touch with what they're doing.” ### What Becomes Automatable? He notes that almost everything can be made verifiable to some extent – even writing can use a council of LLM judges. But some domains are easier than others. For founders: if you’re in a verifiable setting where you can create RL environments, you can do your own fine-tuning and pull the lever. He hints there are valuable RL environments not yet part of the labs’ focus, but declines to name them directly. ## Agentic Engineering vs. Vibe Coding Karpathy draws a clear distinction: - **Vibe coding** is about raising the floor – enabling anyone to build software with AI. - **Agentic engineering** is about preserving the quality bar of professional software. “You're not allowed to introduce vulnerabilities due to vibe coding. You're still responsible.” Doing it properly is “an engineering discipline” – coordinating fallible, stochastic, yet powerful agents to go faster without sacrificing quality. He believes the ceiling is very high: “People used to talk about the 10x engineer. I think that this is magnified a lot more. 10x is not the speed-up you gain. People who are very good at this peak a lot more than 10x.” When asked what makes someone an AI-native coder versus mediocre, Karpathy points to investing in your setup and utilizing all available features – just like you’d invest in Vim or VS Code before. Hiring for agentic engineering should change too: instead of puzzles, give a big project like “write a Twitter clone for agents, secure it, and have other agents try to break it. See if they can break it.” ## What Human Skills Matter? Right now, agents are “catalog internal entities” – they lack taste, judgment, and oversight. Karpathy gives a concrete example from MenuGen: when purchasing credits, the agent used email addresses from Stripe and Google to associate funds, but both emails can be arbitrary. “Why would you use email addresses to cross-correlate funds? This is such a weird thing to do. People have to be in charge of the spec, the plan.” He even says he doesn’t like “plan mode” – instead, you design a detailed spec (docs) and let agents write under the hood. He admits he already forgets API details like `keepdim` vs `keepdims` or `axis` vs `dim` – “the intern handles that.” But you still need to understand fundamentals (e.g., views vs. storage in tensors) to avoid inefficiencies. “You’re in charge of the taste, the engineering, the design, and that it makes sense.” Could that change? He hopes so – he once tried to get an LLM to simplify his `microGPT` project further and “it just can’t. You feel like you’re outside the RL circuits. It’s like pulling teeth.” There’s nothing fundamental preventing it; the labs just haven’t done it yet. ## The Future: Agent-Native Infrastructure Karpathy envisions a world where everything is built for agents first. He laments that most docs are written for humans – “Why are people still telling me what to do? I don't want to do anything. What is the thing I should copy-paste to my agent?” He dreams of an agent native environment: decompose workloads into sensors over the world and actuators, describe everything to agents first, and have data structures that are legible to LLMs. He points to his own pain deploying MenuGen – not the code, but configuring DNS, Vercel, etc. “I would hope that MenuGen could be deployed by just giving a prompt to an LLM, and I didn’t have to touch anything. That would be a good test of whether infrastructure is becoming more agent native.” Ultimately, we’re heading toward a world where people and organizations have agent representations. “I’ll have my agent talk to your agent to figure out details of our meetings.” ## Education and Understanding Asked what remains worth learning deeply when intelligence becomes cheap, Karpathy recalls a tweet that blew his mind: “You can outsource your thinking, but you can’t outsource your understanding.” He still feels like the bottleneck – information has to make it into his brain to direct agents, know why something is worth doing, and have good taste. “The LLMs certainly don’t excel at understanding. You still are uniquely in charge of that.” He is excited about tools that enhance understanding – like building wikis from articles, asking questions, using synthetic data generation over fixed data. “These are tools to enhance understanding. And this is still a bit of a bottleneck because you can’t be a good director if you still… the LLMs don’t excel at understanding.” He closes with a wry hope: “I’m excited to be back here in a couple years to see if we’ve been fully automated out of the loop and they actually take care of understanding as well.” *Source: [Andrej Karpathy: From Vibe Coding to Agentic Engineering – YouTube](https://www.youtube.com/watch?v=96jN2OCOfLs)*