[Mike Pound] Why AI Tokens are so Expensive - Computerphile

### TL;DR The token cost of AI models comes mainly from the autoregressive generation process: the model must predict output token by token, and each prediction requires re-processing the entire context (including previous input and generated output), causing computation to grow linearly with context length. Coding agents autonomously read and write files and engage in multi-turn interactions, quickly accumulating extremely long contexts and causing token costs to skyrocket — this is the root cause of recent billing increases. ## What is a Token? A token is the basic unit of text processed by large language models. It can be a word, part of a word, punctuation, or a special character. For example: - In English, "the cat sat on the mat" — each word plus space is usually one token. - Periods and braces are also tokens. - Complex characters like Chinese characters may correspond to 1 or 2 tokens. The exact segmentation depends on the **tokenizer** used by the model. Tokenizers are based on frequency statistics: extremely common words (e.g., "the") become a single token. Modern language models typically have access to about 100,000 tokens, covering standard English vocabulary, other languages, code symbols, and special Unicode characters. In practice, the active tokens are concentrated in the few thousand most frequent ones, but the rest can be called upon when needed. ## How Does the Model Use Tokens? When tokens enter the model, they are first converted into numerical vectors (embeddings). These vectors are learned by the model during training and represent the meaning of the token. Tokens with similar meanings (e.g., "cat" and "dog") are closer together in vector space. The tokenizer is the fixed front-end and back-end of the LLM, while the embeddings are trained specifically for that model. ## Autoregressive Generation: Why It's So Expensive Large language models work in an **autoregressive** manner: - The model receives the complete input (e.g., "the cat sat on the"). - It outputs the next most likely token (e.g., "mat"). - Then the new token "mat" is appended to the input, and the whole input is sent through the model again to predict the next token. - This process repeats until the full output is generated. **Key issue**: Each time it generates the next token, the model must re-feed the **entire history context** (the original input + all generated tokens) into the GPU for computation. This means: - For a simple Q&A, say input 500 tokens, reasoning process 10,000 tokens, response 500 tokens, the actual number of computations = 10,000 + 500 = 10,500 forward passes. - Each pass sees an increasing context length, so computation rises accordingly. This explains why running these models requires so much power — not generating the whole reply at once, but generating token by token with repeated computation. ## The Role of KV Cache (But Still Limited) To mitigate repeated computation, models use **KV cache (key-value cache)**: - During generation, the model caches the intermediate representations (mainly Key and Value in the attention mechanism) for each layer. - When adding the next token, only the interaction between the new token and the existing cache needs to be computed, not all old token relationships. - This is very effective for very long contexts (e.g., 1 million tokens). However, the cache has a lifespan: if the user pauses too long, the cache on the GPU may be discarded (because other users need service), and next time the user asks a question, it must be recomputed (called prefill). So the cache's effectiveness depends on user query frequency and the system's caching strategy. ## Coding Agents: A Textbook Case of Rapidly Soaring Costs Typical chatbot interactions are relatively short, with limited token costs. But coding agents (e.g., GitHub Copilot) are completely different: they can autonomously read files, search code, execute tool calls, and engage in continuous multi-turn conversations. **Cost growth example**: 1. System prompt (fixed instructions): about 4,000 tokens. 2. User initial query (e.g., "Help me find and fix the bug in the authentication code"): about 200 tokens. 3. Agent starts thinking: it outputs a long reasoning process (maybe 10,000+ tokens), then outputs the actual code changes (maybe 500 tokens). 4. Total cost for a single round: input 4,200 tokens, output 10,500 tokens → 10,500 forward passes. **Explosion with multi-turn interaction**: - User asks a follow-up → context now includes original query, reasoning, reply (total ~11,100 tokens) plus new 200 tokens → input becomes 11,300 tokens. - Another question → input grows to 20,000 tokens. - Agent reads a 50,000-token file → from then on, every interaction includes that file, input skyrockets to 70,000+ tokens. Modern provider pricing is roughly: $2–3 per million input tokens, $15 per million output tokens. When context reaches hundreds of thousands of tokens, a single conversation can cost several dollars or more. This also explains why products like GitHub Copilot recently switched from flat monthly fees to token-based billing. ## Summary - A token is the basic processing unit of a language model; frequency determines segmentation. - Autoregressive generation forces the model to recompute the entire context for each output, which is the root cause of high computational cost. - KV cache can partially optimize, but is limited by cache invalidation and context growth. - Coding agents, due to autonomous file access and multi-turn conversations, push context length up extremely fast, leading to a sharp increase in token costs. As AI model context windows continue to expand (e.g., million-token level), token costs will remain a key factor affecting user experience and product pricing. --- **Source**: Why AI Tokens are so Expensive - Computerphile (https://www.youtube.com/watch?v=-0HRzXk8vlk)