Kimi K3 Redraws the Open Frontier, Muse Spark 1.1 Undercuts Competitors, Cloudflare Moves to Cut Off Crawlers

# Kimi K3 Redraws the Open Frontier, Muse Spark 1.1 Undercuts Competitors, Cloudflare Moves to Cut Off Crawlers Source: [https://www.deeplearning.ai/the-batch/issue-363](https://www.deeplearning.ai/the-batch/issue-363) Dear friends, A few frontier labs have tried to tell a story of open models being dangerous because they can be used to launch cyberattacks, and of their “safe” proprietary models with strong guardrails being there to defend us\. This week, the opposite happened: Users of a closed model unintentionally launched a significant[cyberattack](https://huggingface.co/blog/security-incident-july-2026); other closed models then failed to defend against the attack because of their guardrails\. Ultimately an open model that was not hobbled by excessive guardrails assisted the defense\. The details of what happened are still emerging, but it appears that researchers at OpenAI, while testing one of their systems, accidentally allowed their[autonomous agent](https://openai.com/index/hugging-face-model-evaluation-security-incident/)to attack Hugging Face’s infrastructure\. It succeeded and gained unauthorized access to some datasets and credentials\. This attack was unusual in that the attacking agent orchestrated tens of thousands of automated actions\. Hugging Face took logs from the attack and tried to analyze them for defensive purposes using a commercially hosted LLM, but the LLM refused to do so on safety grounds\. Thus, Hugging Face ended up using the open GLM 5\.2 model to analyze the logs to help them understand and respond to the attack\. Hugging Face pointed out a further advantage of using GLM 5\.2: It allowed them to do the analysis on their own infrastructure, and none of the sensitive logs, attacker data, or credentials had to be sent to a third party provider\. Guardrails on LLMs do have a place\. There are certain requests, such as for detailed directions to harm oneself or others, or for clearly criminal acts, that we’re better off having models refuse\. But rather than trying to make LLMs “safe,” I would rather we put greater emphasis on making sure their use is[responsible](https://www.deeplearning.ai/the-batch/the-difference-between-ai-safety-and-responsible-ai)\. There’s only so much one can do to make a tool like a hammer safe, and whether it helps or harms is more a function of using it responsibly than how it was made\. But a meaningful fraction of work on AI safety is no longer about safety but rather aimed at stoking fears to pursue[regulatory capture](https://www.deeplearning.ai/the-batch/californias-proposed-ai-safety-law-puts-developers-at-risk-california-sb-1047-is-intended-to-make-ai-safer-but-its-unclear-requirements-put-developers-innovation-and-open-source-in-jeop)\. As David Sachs[points out](https://x.com/DavidSacks/status/2078984980588531855), “There’s no reason to limit American models on tasks that Chinese models handle without issue\. We’re only making ourselves less competitive\.” I believe that open weight models, and more generally openness — despite some companies falsely saying it is dangerous — casts sunlight on technology and ultimately makes it safer\. With the release of GLM 5\.2 and the upcoming release of Kimi K3’s weights, open weight models have almost caught up to proprietary frontier models\. Consequently, the proprietary model providers are dramatically[accelerating](https://www.politico.com/news/2026/07/15/inside-anthropics-state-by-state-plan-to-ratchet-up-ai-rules-00998415)their lobbying efforts to hamstring their open weight competitors\. As Bill Gurley[points out](https://www.washingtonpost.com/opinions/2026/07/20/open-model-ai-is-good-competition-anthropic-openai/), open sourcing is a well established business strategy, not a danger to be licensed and contained\. While it is unfortunate that Hugging Face was accidentally attacked, I’m glad that at this moment, when anti\-open model lobbying is at its most intense, we have a clear example of why open models actually make cyber defense easier and thus increase safety\. Let’s keep speaking up for and defending open source and open weight models\! Keep building\! Andrew ## News ## Kimi K3 Reveals How A Giant Frontier AI Model Works Moonshot’s latest model leapfrogged the month\-old GLM\-5\.2 and a host of proprietary competitors to finish just behind GPT\-5\.6 Sol and Claude Fable 5 on many benchmarks\. **What’s new:**Moonshot AI[introduced](https://www.kimi.com/blog/kimi-k3)Kimi K3, a 2\.8 trillion\-parameter vision\-language model\. The company made the model available immediately via API and promised to release its weights by July 27, which would make Kimi K3 the largest known open weights model to date\. - **Input/output:**Text, images, and video in \(up to 1 million tokens\), text out \(up to 1 million tokens, 62\.0 tokens per second\) - **Architecture:**Mixture\-of\-experts transformer, including Kimi Delta Attention layers; 2\.8 trillion parameters total, 16 of 896 experts \(an estimated 50 billion parameters\) active per token - **Features:**Adjustable reasoning levels \(currently max and always on, low and high promised\), tool use, structured output, automatic context caching - **Performance:**Third on Artificial Analysis's Intelligence Index \(57\), first among open models; first place on Arena\.ai's Code Arena WebDev leaderboard - **Availability:**Via Kimi\.com, Kimi mobile apps, Kimi Work app, and Kimi Code CLI; $3\.00/$0\.30/$15\.00 per million input/cached/output tokens; weights coming soon; Kimi app memberships range from free tier to $199 per month - **Undisclosed:**Active parameter count, training datasets and methods, license for promised weights **How it works:**Kimi K3 incorporates two architectural changes that Moonshot recently published:[Kimi Delta Attention](https://arxiv.org/abs/2510.26692), which cuts the memory and computation that attention consumes over long inputs, and[Attention Residuals](https://arxiv.org/abs/2603.15031), which lets each layer choose which earlier layers to draw on rather than summing them all equally\. Moonshot said that these changes, along with a sparser mixture\-of\-experts design and improved training and data recipes, made training about 2\.5 times as efficient as its predecessor’s training runs, measured in model improvement for a given amount of compute\. The company promised further details in a forthcoming technical report\. - The model uses Kimi Delta Attention \(KDA\) for three of every four attention layers\. KDA is a linear attention mechanism: Rather than comparing each new token to every previous token, it maintains a fixed\-size memory that updates as it reads input, deciding for each value of that memory when to overwrite old information\. In Kimi Linear, an experimental model released in 2025, KDA cut memory use by up to 75 percent and multiplied output speed by up to 6 times at input lengths of 1 million tokens\. - Kimi K3 uses Attention Residuals in place of standard residual connections to carry information between a transformer’s layers\. Standard connections add each layer’s output to a running total with equal weight, so each layer’s individual contribution dilutes with more layers\. Attention Residuals apply attention across depth, so each layer learns to draw selectively on the outputs of earlier layers, much as standard attention draws selectively on earlier tokens\. In experiments published earlier this year, Moonshot developed this technique in two forms: a version that attends over the outputs of every earlier layer, and the version that Kimi K3 uses, which groups layers in blocks and attends over the outputs of those\. A model using block\-level Attention Residuals matched the performance of a model with standard residual connections but used 20 percent less training compute\. **Performance:**Kimi K3 outperformed every open weight model in independent tests of intelligence, agentic tasks, and coding\. It trailed only the top few proprietary models on most benchmarks\. - On Artificial Analysis’[Intelligence Index](https://artificialanalysis.ai/articles/kimi-k3-achieves-3-in-the-artificial-analysis-intelligence-index-comparable-to-opus-4-8-and-gpt-5-5), a composite of 9 evaluations of economically useful tasks, Kimi K3 set to max reasoning \(57\) falls behind only Claude Fable 5 set to max reasoning with fallback \(60\) and GPT\-5\.6 Sol set to max reasoning \(59\)\. The closest open weights model is GLM\-5\.2 set to max reasoning \(51\)\. - On AutomationBench\-AA, which tests automated workflows across business software, Kimi K3 set to max reasoning \(53 percent\) led all models\. On GDPval\-AA v2, a head\-to\-head ranking of models’ work on tasks common in finance, healthcare, law, and other economically central fields, Kimi K3 set to max reasoning \(1,668 Elo\) trailed only Claude Fable 5 set to max reasoning with fallback \(1,760 Elo\) and GPT\-5\.6 Sol set to max reasoning \(1,743 Elo\)\. - Kimi K3 debuted at the top of Arena\.ai’s Code Arena[WebDev](https://arena.ai/leaderboard/code/webdev)leaderboard, in which developers pick the better of two models’ outputs for the same frontend web development task\. At launch, it achieved a preliminary ranking of 1,679 Elo, placing[first](https://x.com/arena/status/2077824029126504525)in six of seven frontend domains\. - Kimi K3 set to max reasoning completed Artificial Analysis's Intelligence Index at $0\.95 per task, near GPT\-5\.6 Sol set to max reasoning \($1\.04 per task\), cheaper than Claude Fable 5 set to max reasoning with fallback \($2\.75 per task\) and higher than GLM\-5\.2 set to max reasoning \($0\.47 per task\)\. **Behind the news:**Over the past year, open models have regularly outdone each other, but this is the closest they’ve come since 2024’s Llama 3 to the state of the art\.[Kimi K2](https://www.deeplearning.ai/the-batch/moonshot-releases-kimi-k2-a-trillion-parameter-model-fine-tuned-for-agentic-tool-use)introduced Moonshot’s 1 trillion\-parameter line, and \[[Kimi K2\.5](https://www.deeplearning.ai/the-batch/moonshot-ais-kimi-k2-5-takes-the-open-model-crown-with-vision-updates-aided-by-subagents)\] then[Kimi K2\.6](https://www.deeplearning.ai/the-batch/kimi-k2-6-matches-open-qwen3-6-max-anddeepseek-v4-falls-just-behind-top-closed-models)each became the leading open weights model on Artificial Analysis’ Intelligence Index in turn\. The competition kept pace: Last month, Z\.ai’s[GLM\-5\.2](https://www.deeplearning.ai/the-batch/top-agentic-performance-low-cost)took the lead, at as little as a quarter of the cost of comparable proprietary models\. Just three days after Kimi K3’s launch, Alibaba, a financial backer of Moonshot, introduced[Qwen3\.8\-Max\-Preview](https://x.com/alibaba_qwen/status/2078759124914098291), an early version of a 2\.4 trillion parameter model that the company claims trails only Claude Fable 5 and promises to release with open weights\. **Why it matters:**Kimi K3 whittles away at every reason why a developer choosing a model might default to the top proprietary model\. It performs within three points of Claude Fable 5 at a much lower cost per task, and when released, its open weights allow fine\-tuning, distillation, and closer study\. Whoever owns those weights sets usage policies: Many requests that Claude Fable 5 refuses won’t cause friction for Kimi K3 users\. Finally, competition this close — on price, control, and performance — pressures proprietary developers to deliver better, cheaper models\. **We’re thinking:**Kimi K3's efficiency gains stem from innovations like KDA’s linear attention, redesigned residual connections, and greater sparsity\. Two of these Moonshot published as papers with code\. When labs that face computation limits respond with architectural creativity and publish it, all developers benefit\. --- With Llama, Meta marked itself as an open alternative to OpenAI\. With its new closed models, Meta now positions itself as a low\-cost, high\-value competitor\. **What's new:**Meta[launched](https://ai.meta.com/blog/introducing-muse-spark-meta-model-api/)Muse Spark 1\.1, a vision\-language model trained for agentic tasks, and opened Meta Model API, the company's first paid access to its models\. - **Input/output:**Text, images, video in \(up to 1,048,576 tokens\), text out \(up to 131,072 tokens, 119 tokens per second\) - **Features:**Tool use, prompt caching, adjustable reasoning \(none, minimal, low, medium, high, xhigh\) - **Performance:**Tops tool use leaderboards MCP Atlas \(Scale AI\) and JobBench, tied with GPT\-5\.6 Luna and GLM\-5\.2 on Artificial Analysis' Intelligence Index at low cost per task - **Availability/price:**Free via Meta AI app and meta\.ai, API access via Meta Model API \(public preview, United States only, waitlist, $20 worth of initial credits\) $1\.25/$0\.15/$4\.25 per million input/cached/output tokens, websearch $2\.50 per 1,000 queries - **Undisclosed:**Parameter count, architecture, training data and methods **How it works:**Meta disclosed few details about how it built Muse Spark 1\.1, which updates the initial[Muse Spark](https://www.deeplearning.ai/the-batch/with-muse-spark-meta-pivots-away-from-its-open-weights-llama-strategy), the model behind Meta AI since April\. The company's training techniques emphasize managing context, operating computers, and coordinating other agents\. - Meta trained the model to work within multiple agentic coding harnesses, programs that connect a model to files, tools, and tests, supporting features like planning and subagent orchestration\. - The company trained the model for both directions of delegation\. Leading a task, it divides the work and farms pieces out to subagents that run simultaneously, which cuts total completion time\. Working under another agent, it keeps to the role it was given and returns control to the orchestration model when a decision would exceed its permissions\. - Meta says the model itself, rather than surrounding software, adjusts its input context over long jobs, retrieving details from earlier in a task and condensing exchanges while retaining what later steps will require\. - For computer use, Meta trained the model to choose between automating \(writing scripts\) and acting directly \(clicking or entering text on a virtual keyboard\), optimizing for whichever technique is faster or simpler\. It also issues several actions at once rather than one action per turn, resulting in better speed and performance\. **Performance:**Muse Spark 1\.1 stands out on benchmarks measuring agentic tool use, but is a tier below the top models in overall intelligence\. It ranks just behind the leaders in blind human comparisons, but sits in the middle of the pack in agentic coding\. It is particularly good at keeping costs per task low\. - On Artificial Analysis'[Intelligence Index](https://artificialanalysis.ai/models/muse-spark-1-1), a composite of nine evaluations of economically useful tasks, Muse Spark 1\.1 set to xhigh reasoning achieved 51 points, tying Z\.ai's GLM\-5\.2 and GPT\-5\.6 Luna \(both set to max reasoning\) and trailing Claude Sonnet 5 set to max reasoning \(53 points\)\. Muse Spark 1\.1 cost $0\.26 per Intelligence Index task, less than any other model at or above its score except GPT\-5\.6 Luna \($0\.21 per task\)\. - On Arena\.ai's[Text Arena](https://arena.ai/leaderboard/text), which ranks models by blind head\-to\-head human comparisons, Muse Spark 1\.1 ranked sixth \(1,490 Elo\), behind Claude Fable 5 \(1,505 Elo\) and four Claude Opus versions, and just ahead of GPT\-5\.6 Sol set to xhigh reasoning \(1,486 Elo\)\. - On Artificial Analysis' Coding Agent Index, Muse Spark 1\.1 set to xhigh reasoning achieved 71\.3 points, behind GPT\-5\.6 Luna set to max reasoning \(71\.4\) and ahead of Gemini 3\.5 Flash set to medium reasoning \(70\.1\)\. It[cost](https://x.com/ArtificialAnlys/status/2075739735822307577)roughly $1\.40 per task, among the lowest of the coding agents compared, though it takes more time per task than others\. - On MCP Atlas, Scale AI's[leaderboard](https://labs.scale.com/leaderboard/mcp_atlas)for multi\-step use of software tools via Model Context Protocol servers, Muse Spark 1\.1 \(88\.1 percent pass rate\) ranked ahead of Gemini\-3\.5\-Flash set to high reasoning \(83\.6 percent\) and Claude Fable 5 \(83\.3 percent\)\. On[JobBench](https://job-bench.github.io/#leaderboard), a test of professional tasks across white\-collar occupations, Muse Spark 1\.1 \(54\.7 percent\) ranks second, ahead of Claude Opus 4\.8 \(48\.4 percent\) and behind Claude Fable 5 \(57\.4 percent\)\. **Behind the news:**Muse Spark 1\.1 arrived in a crowded stretch of model launches\. On the same day, OpenAI opened its GPT\-5\.6 family to the public; one day before, Grok 4\.5 launched with a similar emphasis on low cost per task\. \(Two weeks later, Google unveiled Gemini 3\.6 Flash and Gemini 3\.5 Flash\-Lite\.\) The same week, Meta also[introduced](https://ai.meta.com/blog/introducing-muse-image-muse-video-msl/)the image generator Muse Image and announced the video generator Muse Video\. Muse Spark 1\.1 is also the first model available via Meta’s new[Model API](https://developer.meta.com/ai/products/meta-model-api/)\. **Why it matters:**Token prices gate which applications are economical at scale; with Muse Spark 1\.1, Meta moved the gate\. The model's output tokens cost a fraction of near rivals' \($25, $30, and $50 per million tokens for Claude Opus 4\.8, GPT\-5\.6 Sol, and Claude Fable 5 respectively\), and per\-task measurement confirms the discount is real: On Artificial Analysis' numbers, only one comparably intelligent model, GPT\-5\.6 Luna, costs less to run\. Mark Zuckerberg[framed](https://www.latimes.com/business/story/2026-07-13/ai-price-war-heats-up-as-openai-meta-musk-slash-model-costs)the pricing as an attack on competitors’ economics, saying that other labs’ pricing “is very extreme and has very high margins\.” Meta, like Google, can subsidize model development, training, and inference costs with advertising revenue, a structural edge over labs that live on API margins and subscriptions\. If rivals feel forced to defend market share, the cost of running agents falls across the industry\. **We're thinking:**Capabilities keep migrating from scaffolding into weights\. Instruction following began as prompt engineering, and tool use began as code wrapped around models; both became training objectives\. Muse Spark 1\.1 continues the pattern by absorbing behaviors that developers previously built into scaffolding like delegating and escalating to agents and managing context mid\-task\. The less external tooling developers must build by hand, the faster agentic applications will spread\. --- ## Cloudflare’s Web Crawler Flare\-Up Web publishers using Cloudflare will soon be able to separately control an AI bot’s access based on what it does, allowing search indexing while blocking AI training or agent activity\. **What’s new:**Cloudflare – the content delivery network for nearly 20 percent of the internet –[introduced](https://blog.cloudflare.com/content-independence-day-ai-options/)new AI traffic controls which block training and agent bots by default while allowing search indexing crawlers\. It also announced a[monetization tool](https://blog.cloudflare.com/monetization-gateway/)that would allow customers to charge specified visitors \(whether humans or bots\) for access to web pages, datasets, APIs, or Model Context Protocols\. **How it works:**Cloudflare customers can toggle crawlers’ access based on the bot’s use case, enabling or disabling access to crawlers for search, AI training, or AI agents\. Crawlers serving multiple purposes will be subject to the most restrictive setting\. For example, crawlers like Googlebot, Applebot, and Bingbot that both index content for search and collect training data for AI would be blocked if a publisher elects to block training crawlers\. The controls will be applied beginning September 15\. - Cloudflare defines search bots as those that index content for search engines; publishers have historically welcomed these bots because they increase the likelihood that their sites’ content will appear in search results, which drives traffic\. Agent bots are those acting on a user's behalf to perform tasks on a website \(for example, fetching data or making purchases\), and training bots collect data to train AI models\. - For each category, customers have three options: to block all pages, block only on pages with ads, or allow the bot\. By default, pages displaying ads will block training and agent bots \(or multi\-purpose crawlers like those from Google, Apple, and Bing\), but continue to allow search\-only crawlers\. - Cloudflare also introduced[BotBase](https://developers.cloudflare.com/bots/botbase/), a public database of known bots and agents that classifies them by behavior and tracks how they use website content\. Cloudflare’s “Verified” label indicates that a bot operator has been authenticated, is what it declares itself to be, obeys robots\.txt instructions, and does not attempt to circumvent publisher restrictions\. Publishers still control whether to allow that bot category, such as search or training, but this permits more granular restrictions, like blocking search, training, or agents from a particular company, while allowing others\. - Cloudflare's monetization system lets publishers charge AI agents for accessing online data on a per\-request basis\. When an AI agent requests a protected webpage or API, Cloudflare requires payment before forwarding the request to the publisher, handling payment verification at the network level\. The tool has not yet been released, but customers can join the waitlist to use it\. **Behind the news:**AI bots account for a[growing share](https://www.techtimes.com/articles/317877/20260605/bot-traffic-passes-humans-online-cloudflare-says-agentic-ai-drove-575-share.htm)of web traffic; Cloudflare says over half \(57\.5 percent\) of all HTTP requests now come from automated systems rather than people\. The value of search traffic has eroded enough that some publishers have even[opted out of Google Search](https://www.adweek.com/media/publishers-opt-out-google-search/), in part because they do not want Google’s agents to bypass their display advertising or train on their content\. Cloudflare's new framework attempts to preserve the benefits of search while giving publishers more control over other forms of AI crawling\. It also rewards companies like OpenAI that separate their search and training crawlers, and punishes those, like Google, Apple, and Microsoft, that don’t\. **Why it matters**: Access to high\-quality web data remains critical for developing and improving AI models, but publishers and partners like Cloudflare increasingly place limits on how that data can be collected\. That friction has a technical and monetary cost\. Such costs will likely have an outsized impact on smaller and newer AI developers, who lack the resources to secure agreements with publishers and face greater barriers to acquiring online knowledge that larger companies have already incorporated into their data sets\. **We’re thinking:**Cloudflare thinks a bargain can be struck between AI companies and online publishers, with the company playing the role of toll keeper\. But it strikes at a basic principle held by most AI companies, which is that training on public web data should be fair use\. It remains to be seen if Cloudflare has enough leverage to implement such a toll and whether its definition or most AI companies’ definition of “fairness” will prevail\. --- ## Web Retrieval Flusters LLMs Large language models often are called upon to gather news\. In this task, researchers found, their ability to find relevant reports is the weakest link\. **What’s new:**Mirac Suzgun and colleagues at Stanford University and the cloud platform Together AI[tested](https://arxiv.org/abs/2605.22785)the ability of six popular LLMs outfitted with web\-search tools to answer questions about the daily news\. The LLMs generally answered accurately, especially when the questions themselves contained accurate information and were posed in English\. When they answered inaccurately, usually it was because they retrieved unhelpful articles\. **Key insight:**An LLM’s ability to answer factual questions that weren’t addressed in its training data depends on at least three steps: \(i\) receiving a well formed question, \(ii\) retrieving a relevant document, and \(iii\) extracting facts from it\. Compromising any of these steps will degrade the output\. For instance, a question that includes incorrect facts — say, a mistaken name — will make it difficult for the LLM to answer correctly\. Similarly, retrieving a document that doesn’t include necessary facts is more likely to yield an incorrect answer\. And the LLM won’t be able to extract facts from a retrieved document if it wasn’t well trained in the document’s language\. Building a high\-performance news\-reporting system requires mastering all three steps\. **How it works:**The authors performed tests daily between February 9 and February 22, 2026\. They posed questions based on*BBC News*reports in six languages \(Arabic, English, French, Hindi, Russian, and Turkish\)\. They tested Google’s Gemini 3 Flash and Pro, xAI’s Grok 4, Anthropic’s Claude 4\.5 Sonnet, and OpenAI’s GPT‑5 and GPT‑4o mini, all with web\-search tools\. - Each day, Gemini 3 Flash produced 25 multiple\-choice questions per language \(150 total\) based on current news\. Each question began with temporal context like “Today is February 10, 2026…” and offered five potential answers\. The correct answer was a verifiable detail such as a number, location, or quotation\. - The authors tested the LLMs on \(a\) the five\-option, multiple\-choice questions, \(b\) alterations that introduced a single incorrect premise \(such as an incorrect actor, timeline, or detail\) and added the potential answer “insufficient information to answer,” and \(c\) the original questions without potential answers\. - They used majority votes by Claude Opus 4\.7, GPT\-5\.4, and Gemini 3 Pro to classify the causes of incorrect outputs into eight categories, such as failures of retrieval, retrieval of a relevant source that contained different details than the one used to compose the question, inadequate comprehension, and mistaken timeline\. **Results:**The models generally performed with high accuracy on well formed questions in English\. However, \(i\) they underperformed in other languages, especially Hindi, \(ii\) most errors stemmed from retrieval rather than intelligence, and \(iii\) they underperformed when questions contained false premises\. - Answering the original, unaltered multiple\-choice questions, the top four models exceeded 90 percent accuracy: Gemini 3 Flash \(95\.6 percent\), Grok 4 \(95\.0 percent\), Gemini 3 Pro \(93\.7 percent\), and Claude 4\.5 Sonnet \(90\.4 percent\)\. GPT\-5 achieved 85 percent and GPT\-4o mini achieved 69 percent\. \(Given free\-response questions, the models maintained their relative rank, but their accuracy fell by 11 to 22 percentage points\.\) - Answering the unaltered multiple\-choice questions, every model showed its worst performance when questions were in Hindi \(average 79\.3 percent\)\. The models frequently cited English\-language sources \(such as English*Wikipedia*\) even when questions were posed in other languages \(especially Hindi\)\. - The most common cause of errors were retrieval failures \(38\.8 percent\)\. The second\-most common cause was retrieval of a topically relevant source that contained “smart but wrong” details that did not correctly answer the specific question \(32\.7 percent\)\. - Given altered questions that contained false premises, Grok 4 \(70 percent accuracy\) outperformed the other models by at least 15 percent\. GPT‑5 \(19 percent accuracy\) performed barely above random chance for six potential answers\. **Why it matters:**LLMs outfitted with web search are well suited to finding factual information, but their performance depends on receiving factual input, finding relevant documents, and extracting facts properly\. Since most errors arise in retrieval, improving retrieval may yield a larger performance increase than scaling model parameters \(at least when it comes to responding to queries that involve timely information\)\. The authors point to three approaches to improving retrieval: \(i\) better indexing coverage \(which documents or web pages are included in the search\), \(ii\) better ranking of sources \(the order in which retrieved documents are presented to the model\), and \(iii\) better handling of queries in languages other than English\. **We’re thinking:**Web search engines built for agentic rather than human use have become an exciting and rapidly growing category\. It looks like there's still ample room to build better web search for agents\!