Towards an Agent-First Web: Redesigning the Web for AI Agents

# Towards an Agent-First Web: Redesigning the Web for AI Agents
Source: [https://arxiv.org/html/2606.19116](https://arxiv.org/html/2606.19116)
Ross Gore[rgore@odu\.edu](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Ravi Mukkamala[rmukkama@odu\.edu](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Asanga Gunaratna[asanga\.gunaratna@complianceoslab\.app](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Safdar H\. Bouk[sbouk@odu\.edu](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Xueping Liang[xuliang@fiu\.edu](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Peter Foytik[pfoytik@odu\.edu](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Abdul Rahman[abdulrahman@deloitte\.com](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Sachini Rajapakse[sachini\.rajapakse@iciclelabs\.ai](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Isurunima Kularathna[nima@linesandloops\.art](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Pramoda Karunarathna[pramoda\.karunarathna@iciclelabs\.ai](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Chalani Rajapakse[chalani\.rajapakse@iciclelabs\.ai](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Ng Wee Keong[awkng@ntu\.edu\.sg](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Kasun De Zoysa[kasun@ucsc\.cmb\.ac\.lk](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Tharaka Hewa[tharaka\.hewa@oulu\.fi](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Amin Hass[amin\.hassanzadeh@accenture\.com](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Wathsala Herath[wathsala\.herath@agentsway\.ai](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Aruna Withanage[aruna@effectz\.ai](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Nilaan Loganathan[nilaan@effectz\.ai](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Atmaram Yarlagadda[atmaram\.yarlagadda\.civ@health\.mil](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Sachin Shetty[sshetty@odu\.edu](https://arxiv.org/html/2606.19116v1/mailto:[email protected])Old Dominion University, Norfolk, VA, USAAI Motion Labs, Melbourne, AustraliaDeloitte & Touche LLP, USAFlorida International University, USAIcicleLabs\.AILinesandloops\.artAccenture Technology Labs, Arlington, VA, USANanyang Technological University, SingaporeUniversity of Colombo, Sri LankaAgentsway\.AIEffectz\.AICenter for Wireless Communications, University of Oulu, FinlandMcDonald Army Health Center, Newport News, VA, USA

###### Abstract

The World Wide Web was architected on a foundational assumption that has held for three decades: that the primary consumer of web content is a human being\. This assumption permeates every layer of the web — its access model presumes human visitors, its economic foundations rest on human attention, and its content architecture targets human perception\. The rapid emergence of AI agents as primary intermediaries between humans and web content fundamentally invalidates this assumption\. Today, AI agents browse, synthesize, and act on web content on behalf of humans at scale — yet the web actively resists them through blanket blocking, CAPTCHA\-based exclusion, and economic models that treat agent access as extraction rather than legitimate interaction\.

This paper proposes a principled redesign of the web across three interdependent layers\. At theaccess layer, we argue that agents acting on behalf of humans should inherit equivalent access rights to those humans — governed not by blanket blocking but by rate limiting and standardized agent identification metadata carried in HTTP requests, analogous to how browsers currently identify themselves to servers\. We propose a dual\-layer web architecture in which human\-readable content and agent\-optimized content coexist during a transition period, with a phased migration path toward an agent\-first publishing standard\. At theeconomic layer, we reject universal pay\-per\-query models in favor of an intent\-based tier framework grounded in the principle that an agent’s economic obligation mirrors that of the human it represents — theagent\-as\-human\-proxy principle\. We propose a token\-based subscription model in which content access is metered in tokens rather than pageviews, alongside a commissioned content economy in which human intentionality anchors AI\-generated content production and breaks the self\-reinforcing generative loop\. At thecontent layer, we identify a structural threat we termepistemic recursion— the self\-referential loop in which AI\-generated content is consumed by AI agents to produce further content, progressively detaching web knowledge from human ground truth\. To counter this, we propose the Agent Text Markup Language \(ATML\) — a semantic content format optimized for agent consumption — alongside a four\-level human supervision tier model and a cryptographic provenance chain architecture that makes the degree of human oversight machine\-readable and verifiable\.

Together, these three layers constitute a framework of ten design principles for an agent\-first internet — one in which agents are not threats to be blocked but first\-class citizens whose integration requires renegotiating the web’s foundational social contract across access, economics, and content\.

###### keywords:

Agentic AI , Agentic Web , LLM , AI Agents , Web Architecture

††journal:Journal Name## 1Introduction

The World Wide Web, since its inception by Berners\-Lee in 1989, was designed with a singular assumption: that the primary consumer of web content is a human being\[[18](https://arxiv.org/html/2606.19116#bib.bib526)\]\. This assumption permeates every layer of the web’s architecture\. Hypertext Markup Language \(HTML\) renders content visually for human perception\[[46](https://arxiv.org/html/2606.19116#bib.bib528)\]\. Search engines optimize content discovery around human attention and intent\[[19](https://arxiv.org/html/2606.19116#bib.bib527)\]\. Economic models monetize human engagement through advertising impressions and clicks\[[28](https://arxiv.org/html/2606.19116#bib.bib529)\]\. Security mechanisms such as CAPTCHAs are explicitly designed to distinguish humans from automated agents\[[52](https://arxiv.org/html/2606.19116#bib.bib530)\]\. For three decades, this human\-centric architecture served the web well — enabling an unprecedented global information ecosystem that transformed commerce, communication, and knowledge\.

This foundational assumption is now breaking down\. The rapid advancement of large language models \(LLMs\)\[[5](https://arxiv.org/html/2606.19116#bib.bib203),[12](https://arxiv.org/html/2606.19116#bib.bib525)\]and autonomous AI agents has introduced a fundamentally new class of web participant\[[11](https://arxiv.org/html/2606.19116#bib.bib366),[7](https://arxiv.org/html/2606.19116#bib.bib325)\]— one that browses, synthesizes, and acts on web content not for itself, but as a proxy for a human user\[[54](https://arxiv.org/html/2606.19116#bib.bib531)\]\. Unlike traditional web crawlers, which indexed content to route human traffic back to publishers, modern AI agents complete tasks end\-to\-end — booking flights, answering research questions, drafting documents — without the human ever visiting the underlying web resource\[[57](https://arxiv.org/html/2606.19116#bib.bib532)\]\. The web’s human\-centric architecture was not designed for this interaction model, and the friction is now measurable and severe\.

Empirically, this tension manifests across all three layers of the web\. At the access layer, infrastructure providers have begun blocking AI agents by default\. Cloudflare, which routes over 16% of global internet traffic, moved in July 2025 to block AI crawlers unless explicitly permitted, introducing a pay\-per\-crawl model that treats agent access as a commercial transaction rather than a default right\[[20](https://arxiv.org/html/2606.19116#bib.bib539)\]\. The crawl\-to\-referral ratio of AI agents reveals the economic asymmetry driving this response: by mid\-2025, Anthropic’s crawler exhibited a ratio of 73,000 crawled pages for every single human referral generated — fundamentally breaking the value exchange that historically justified open crawler access\[[22](https://arxiv.org/html/2606.19116#bib.bib540)\]\. At the economic layer, zero\-click searches now account for approximately 60% of all Google queries, rising to 93% in AI\-native search modes\[[49](https://arxiv.org/html/2606.19116#bib.bib541)\], with click\-through rates at position one falling from 27% to 11%\[[51](https://arxiv.org/html/2606.19116#bib.bib542)\]\. Publishers report organic traffic declines of 70–80%, with some characterizing the shift as an extinction\-level event for web\-dependent media\[[51](https://arxiv.org/html/2606.19116#bib.bib542)\]\. At the content layer, a deeper structural problem emerges: as AI agents increasingly consume AI\-generated content to produce further content, the web risks entering a self\-referential generative loop that progressively erodes its epistemic foundations\[[50](https://arxiv.org/html/2606.19116#bib.bib544)\]— a phenomenon we termepistemic recursion\.

The web’s response to these pressures has been reactive and fragmented\. Infrastructure providers block agents by default\[[20](https://arxiv.org/html/2606.19116#bib.bib539)\]\. Protocol working groups propose interoperability standards in isolation\[[4](https://arxiv.org/html/2606.19116#bib.bib536),[32](https://arxiv.org/html/2606.19116#bib.bib537),[43](https://arxiv.org/html/2606.19116#bib.bib538)\]\. Economic responses treat agent access as a billing problem rather than a design problem\[[20](https://arxiv.org/html/2606.19116#bib.bib539),[1](https://arxiv.org/html/2606.19116#bib.bib543)\]\. Content provenance efforts address detection of AI\-generated content without addressing the architectural conditions that produce epistemic recursion\[[24](https://arxiv.org/html/2606.19116#bib.bib555),[38](https://arxiv.org/html/2606.19116#bib.bib554)\]\. Critically, no existing work addresses all three layers together, nor frames the challenge as what we argue it fundamentally is: a failure of the web’s foundational social contract that requires principled redesign, not reactive patches\[[15](https://arxiv.org/html/2606.19116#bib.bib308),[9](https://arxiv.org/html/2606.19116#bib.bib306)\]\.

This paper makes the following argument: the web requires simultaneous redesign across three interdependent layers — access, economics, and content — grounded in a single philosophical anchor: that AI agents acting on behalf of humans are first\-class web citizens, entitled to the same presumption of access as the humans they represent, subject to equivalent obligations, and deserving of an architectural environment designed for their interaction model rather than one that merely tolerates them\.

Concretely, we propose three mechanism clusters\. At the access layer, we proposeagent identification metadata— standardized HTTP request headers that allow agents to declare their identity, the human they represent, and their intent, analogous to the User\-Agent and Accept headers browsers currently use\[[30](https://arxiv.org/html/2606.19116#bib.bib558)\]— alongsideagents\.txt, a machine\-readable access policy standard that replaces the inadequaterobots\.txthonor system with graduated, intent\-aware access declarations\. Together these enable servers to apply rate limiting rather than blanket blocking, and to serve agent\-optimized content via adual\-layer web architecturethat supports gradual migration rather than disruptive replacement\. At the economic layer, we propose atoken\-based subscription modelin which content access is metered in tokens rather than pageviews — directly compatible with existing AI API pricing infrastructure — alongside acommissioned content economyin which human intentionality anchors AI content production, and a free tier governed by rate limits mirroring the open source / proprietary software distinction\[[47](https://arxiv.org/html/2606.19116#bib.bib559)\]\. At the content layer, we propose theAgent Text Markup Language \(ATML\)— a semantic content format optimized for agent consumption — alongside afour\-level human supervision tier modeland acryptographic provenance chain architecturethat make the degree of human oversight machine\-readable and verifiable, breaking the epistemic recursion loop at its structural root\.

We make the following specific contributions:

- 1\.We diagnose the web’s human\-centric design assumptions across three layers and demonstrate empirically that each is incompatible with agent\-first interaction at scale \(Section[3](https://arxiv.org/html/2606.19116#S3)\)\.
- 2\.We proposeagent identification metadataandagents\.txtas lightweight, backward\-compatible mechanisms for agent identification, intent declaration, and access policy over existing HTTP infrastructure, enabling rate limiting over blocking and dual\-layer content serving \(Section[4](https://arxiv.org/html/2606.19116#S4)\)\.
- 3\.We introduce anintent\-based economic tier modelgrounded in the agent\-as\-human\-proxy principle, with a token\-based subscription mechanism and commissioned content economy directly compatible with existing AI API infrastructure \(Section[5](https://arxiv.org/html/2606.19116#S5)\)\.
- 4\.We introduce the concept ofepistemic recursion— the self\-referential loop in which AI\-generated content is consumed by AI agents to produce further content, progressively detaching web knowledge from human ground truth — and propose ATML, human supervision tiers, and a cryptographic provenance chain as architectural responses \(Section[6](https://arxiv.org/html/2606.19116#S6)\)\.
- 5\.We synthesize these contributions into a unified framework of ten design principles for an agent\-first internet, positioning the challenge as a sociotechnical problem requiring renegotiation of the web’s foundational social contract \(Section[7](https://arxiv.org/html/2606.19116#S7)\)\.

The remainder of this paper is structured as follows\. Section[2](https://arxiv.org/html/2606.19116#S2)reviews related work across agent protocols, web economics, and content architecture\. Section[3](https://arxiv.org/html/2606.19116#S3)diagnoses the three\-layer failure of the human\-centric web\. Section[4](https://arxiv.org/html/2606.19116#S4)presents the access layer redesign\. Section[5](https://arxiv.org/html/2606.19116#S5)presents the economic layer redesign\. Section[6](https://arxiv.org/html/2606.19116#S6)presents the content layer redesign including ATML, supervision tiers, and epistemic recursion prevention\. Section[7](https://arxiv.org/html/2606.19116#S7)synthesizes the unified framework and ten principles\. Section[8](https://arxiv.org/html/2606.19116#S8)discusses open challenges\. Section[9](https://arxiv.org/html/2606.19116#S9)concludes\.

## 2Related Work

Research relevant to redesigning the web for AI agents spans four broad areas: \(1\) AI agent architectures and web interaction, \(2\) agent communication protocols and infrastructure, \(3\) web economics and the attention economy, and \(4\) content provenance and epistemic integrity\. While each area has produced significant contributions, no existing work addresses all three layers — access, economics, and content — simultaneously, nor frames the challenge as a renegotiation of the web’s foundational social contract\. Table[1](https://arxiv.org/html/2606.19116#S2.T1)summarizes how representative works relate to the key dimensions of our framework\.

### 2\.1AI Agent Architectures and Web Interaction

Early work on web\-based AI agents focused on task completion in constrained environments\.Yaoet al\.\[[57](https://arxiv.org/html/2606.19116#bib.bib532)\]introduced WebShop, demonstrating that language model agents could navigate product search and purchase flows, establishing a benchmark for grounded web interaction\.Zhouet al\.\[[59](https://arxiv.org/html/2606.19116#bib.bib533)\]extended this with WebArena, a realistic multi\-domain environment covering e\-commerce, content management, and developer tools, revealing the significant gap between agent capability and real\-world web complexity\.Denget al\.\[[25](https://arxiv.org/html/2606.19116#bib.bib546)\]introduced Mind2Web, the first dataset for generalist web agents capable of following natural language instructions across diverse websites, highlighting the challenge of generalizing across the heterogeneous structure of the human\-designed web\.

More recent work has examined agentic behavior at scale\.Wanget al\.\[[54](https://arxiv.org/html/2606.19116#bib.bib531)\]surveyed LLM\-based autonomous agents across planning, memory, and tool use dimensions, noting that web interaction represents one of the most complex and underspecified deployment environments\.Duranteet al\.\[[26](https://arxiv.org/html/2606.19116#bib.bib547)\]examined multi\-agent systems operating in open\-ended environments, identifying trust, coordination, and resource access as primary bottlenecks\. Critically, none of these works examine the web architecture itself as a site of intervention — they treat the existing web as a fixed environment and attempt to improve agent capability within it\. Our work takes the complementary position: that the web architecture must change to accommodate agents, not only that agents must improve to navigate the existing web\.

### 2\.2Agent Communication Protocols and Infrastructure

A parallel research and engineering effort has focused on protocol\-level infrastructure for agent interaction\.Anthropic \[[4](https://arxiv.org/html/2606.19116#bib.bib536)\]introduced the Model Context Protocol \(MCP\), an open standard enabling AI agents to connect to external tools and data sources through a unified interface, addressing the fragmentation of agent\-to\-tool communication\. Google’s Agent\-to\-Agent \(A2A\) protocol\[[32](https://arxiv.org/html/2606.19116#bib.bib537)\]extended this to agent\-to\-agent communication, enabling coordination across heterogeneous agent systems\. IBM’s Agent Communication Protocol \(ACP\)\[[48](https://arxiv.org/html/2606.19116#bib.bib548)\]and the Agent Network Protocol \(ANP\)\[[3](https://arxiv.org/html/2606.19116#bib.bib549)\]addressed further coordination challenges at network scale\.Microsoft \[[43](https://arxiv.org/html/2606.19116#bib.bib538)\]proposed NLWeb, bringing natural language interfaces to websites and positioning every NLWeb endpoint as an MCP server — a significant step toward agent\-readable web content\.

Kapoor and others \[[36](https://arxiv.org/html/2606.19116#bib.bib534)\]provided the most comprehensive survey of agent infrastructure requirements to date, cataloging use cases, limitations, and open problems across memory, execution, and communication layers\.Li and others \[[40](https://arxiv.org/html/2606.19116#bib.bib535)\]proposed a conceptual model for the agentic web across intelligence, interaction, and economic dimensions, identifying architectural challenges in protocol design and agent orchestration\. The W3C Agent Web Community Group\[[53](https://arxiv.org/html/2606.19116#bib.bib550)\]identified four core trends in agent\-web interaction and three primary challenges including data silos, human\-machine interface friction, and absence of standard protocols\.

While these works make important protocol\-level contributions, they address the access and interoperability problem in isolation\. They do not examine the economic model that must accompany open agent access, nor the epistemic consequences of AI\-generated content recursion\. Our framework synthesizes the protocol layer with the economic and content layers to produce a unified redesign proposal\.

### 2\.3Web Economics and the Attention Economy

The economic foundations of the web have been studied extensively in the context of human interaction\.Evans \[[28](https://arxiv.org/html/2606.19116#bib.bib529)\]characterized the web’s advertising model as a two\-sided market connecting publishers and advertisers through human attention as the mediating resource\.Wu \[[56](https://arxiv.org/html/2606.19116#bib.bib551)\]traced the historical development of the attention economy, demonstrating how successive communication technologies — radio, television, the web — converged on attention capture as the primary economic mechanism\.

The disruption of this model by AI is increasingly documented empirically\.Cloudflare \[[22](https://arxiv.org/html/2606.19116#bib.bib540)\]reported crawl\-to\-referral ratios of 73,000:1 for AI crawlers by mid\-2025, demonstrating the complete breakdown of the value exchange that historically justified open crawler access\.Semrush \[[49](https://arxiv.org/html/2606.19116#bib.bib541)\]documented that 93% of searches in AI\-native modes end without a click, whileSISTRIX \[[51](https://arxiv.org/html/2606.19116#bib.bib542)\]reported click\-through rate declines from 27% to 11% at position one in AI\-augmented search\.Cloudflare \[[20](https://arxiv.org/html/2606.19116#bib.bib539)\]described the infrastructure response to these dynamics — default blocking of AI crawlers and introduction of pay\-per\-crawl models — as the first large\-scale attempt to impose economic structure on agent access\.

Proposed economic responses have been largely reactive and incomplete\. Perplexity’s publisher revenue sharing\[[1](https://arxiv.org/html/2606.19116#bib.bib543)\]and Cloudflare’s pay\-per\-crawl model\[[20](https://arxiv.org/html/2606.19116#bib.bib539)\]address specific friction points without a principled framework\.Nguyen and others \[[44](https://arxiv.org/html/2606.19116#bib.bib552)\]examined micropayment models for agent content access but did not address the agent\-as\-human\-proxy principle or the distinction between personal and commercial agent use\. Our work contributes the intent\-based economic tier model — grounded in the philosophical principle that an agent’s economic obligation mirrors that of the human it represents — as a principled alternative to universal payment requirements\.

### 2\.4Content Provenance and Epistemic Integrity

The question of content provenance has gained urgency with the proliferation of AI\-generated content\.Zellerset al\.\[[58](https://arxiv.org/html/2606.19116#bib.bib553)\]demonstrated early that neural text generation could produce indistinguishable fake news, motivating provenance research\.Kirchenbaueret al\.\[[38](https://arxiv.org/html/2606.19116#bib.bib554)\]proposed watermarking techniques for LLM outputs, enabling statistical detection of AI\-generated text\. The C2PA \(Coalition for Content Provenance and Authenticity\) standard\[[24](https://arxiv.org/html/2606.19116#bib.bib555)\]established cryptographic provenance for media content, providing a technical foundation for authorship declaration at the content layer\.

Most directly relevant to our epistemic recursion argument,Shumailovet al\.\[[50](https://arxiv.org/html/2606.19116#bib.bib544)\]demonstrated the phenomenon of model collapse — the progressive degradation of model quality when trained on AI\-generated data — providing empirical grounding for our theoretical framing\.Martini and others \[[42](https://arxiv.org/html/2606.19116#bib.bib556)\]examined provenance requirements for agentic content pipelines, andBenderet al\.\[[16](https://arxiv.org/html/2606.19116#bib.bib557)\]characterized large language models as stochastic parrots — systems that recombine existing patterns without grounding in world experience — a characterization that motivates the human\-intentionality anchor we propose\.

Existing provenance work addresses detection and attribution of AI\-generated content but does not address the systemic architectural question of how the web should be redesigned to preserve epistemic integrity at scale\. Our provenance\-anchored content architecture extends this work by proposing a web\-level standard for declaring content origin, derivation chain, and human oversight level — going beyond detection to prevention of epistemic recursion\.

### 2\.5Summary and Positioning

Table[1](https://arxiv.org/html/2606.19116#S2.T1)summarizes the coverage of representative related works across the six key dimensions of our framework: agent access model, behavioral contracts, economic framework, intent\-based tiers, epistemic recursion, and provenance architecture\. As the table shows, existing works address individual dimensions in isolation\. Our framework is the first to address all six dimensions simultaneously, grounded in a unified social contract redesign philosophy\.

Table 1:Comparison of related work across key dimensions of the agent\-first web redesign framework\. ✓ = addressed; ✗ = not addressed;∼\\sim= partially addressed\.WorkAgentaccess modelBehavioralcontractsEconomicframeworkIntent\-basedtiersEpistemicrecursionProvenancearchitectureYao et al\.\[[57](https://arxiv.org/html/2606.19116#bib.bib532)\]∼\\sim✗✗✗✗✗Zhou et al\.\[[59](https://arxiv.org/html/2606.19116#bib.bib533)\]∼\\sim✗✗✗✗✗Deng et al\.\[[25](https://arxiv.org/html/2606.19116#bib.bib546)\]∼\\sim✗✗✗✗✗Wang et al\.\[[54](https://arxiv.org/html/2606.19116#bib.bib531)\]∼\\sim✗✗✗✗✗Kapoor et al\.\[[36](https://arxiv.org/html/2606.19116#bib.bib534)\]✓✗∼\\sim✗✗✗Li et al\.\[[40](https://arxiv.org/html/2606.19116#bib.bib535)\]✓✗∼\\sim✗✗✗Anthropic MCP\[[4](https://arxiv.org/html/2606.19116#bib.bib536)\]✓✗✗✗✗✗Google A2A\[[32](https://arxiv.org/html/2606.19116#bib.bib537)\]✓✗✗✗✗✗Microsoft NLWeb\[[43](https://arxiv.org/html/2606.19116#bib.bib538)\]✓✗✗✗✗✗W3C Agent Web CG\[[53](https://arxiv.org/html/2606.19116#bib.bib550)\]✓∼\\sim✗✗✗✗Cloudflare Pay\-per\-crawl\[[20](https://arxiv.org/html/2606.19116#bib.bib539)\]∼\\sim✗∼\\sim✗✗✗Nguyen et al\.\[[44](https://arxiv.org/html/2606.19116#bib.bib552)\]✗✗✓✗✗✗Shumailov et al\.\[[50](https://arxiv.org/html/2606.19116#bib.bib544)\]✗✗✗✗✓✗C2PA\[[24](https://arxiv.org/html/2606.19116#bib.bib555)\]✗✗✗✗✗∼\\simKirchenbauer et al\.\[[38](https://arxiv.org/html/2606.19116#bib.bib554)\]✗✗✗✗∼\\sim∼\\simBender et al\.\[[16](https://arxiv.org/html/2606.19116#bib.bib557)\]✗✗✗✗∼\\sim✗Evans\[[28](https://arxiv.org/html/2606.19116#bib.bib529)\]✗✗✓✗✗✗Wu\[[56](https://arxiv.org/html/2606.19116#bib.bib551)\]✗✗✓✗✗✗This paper \(proposed\)✓✓✓✓✓✓

## 3The Human\-Centric Web: A Three\-Layer Diagnosis

The World Wide Web was not designed with a single deliberate choice to exclude non\-human participants — rather, its human\-centricity emerged organically from three decades of architectural decisions, each reasonable in isolation, that collectively produce a web hostile to agent\-first interaction\. In this section, we diagnose this incompatibility systematically across three layers: access, economics, and content\. For each layer, we identify the foundational human\-centric assumption, demonstrate empirically how that assumption fails under agent interaction, and characterize the nature of the resulting incompatibility\. Proposed solutions to each layer are deferred to Sections[3\.1](https://arxiv.org/html/2606.19116#S3.SS1),[3\.2](https://arxiv.org/html/2606.19116#S3.SS2), and[3\.3](https://arxiv.org/html/2606.19116#S3.SS3)respectively\.

### 3\.1Layer 1: The Access Model

#### 3\.1\.1The Human\-Centric Assumption

The web’s access model was designed around the presumption of human visitors\. When a human navigates to a website, the interaction carries a set of implicit guarantees that the web’s architecture has never needed to make explicit: the visitor is a single individual, operating at human speed, consuming content for personal use, and subject to social and legal accountability for their actions\. These implicit guarantees justified the web’s foundational philosophy of open access — articulated by Berners\-Lee as the principle that any client should be able to retrieve any resource without prior negotiation\[[18](https://arxiv.org/html/2606.19116#bib.bib526)\]\. Security mechanisms such as CAPTCHAs operationalize this assumption directly, treating human biological capability as the access credential\[[52](https://arxiv.org/html/2606.19116#bib.bib530)\]\.

The only formal mechanism for non\-human access management is therobots\.txtstandard, introduced in 1994\[[39](https://arxiv.org/html/2606.19116#bib.bib560)\]\. Designed for search engine crawlers operating under an implicit social contract — crawl freely, return traffic —robots\.txtis an honor system with no enforcement mechanism, no concept of agent identity, no support for graduated access levels, and no capacity to distinguish between a personal assistant agent acting for one user and a mass scraper extracting content for commercial training\[[20](https://arxiv.org/html/2606.19116#bib.bib539)\]\. It was adequate when the only non\-human visitors were well\-behaved search crawlers\. It is wholly inadequate for the agent\-first web\.

#### 3\.1\.2Empirical Evidence of Failure

The incompatibility of the current access model with agent interaction is now empirically documented at infrastructure scale\. Cloudflare, which routes over 16% of global internet traffic, moved in July 2025 to block AI crawlers by default — the first time in the web’s history that a major infrastructure provider has treated non\-human access as an opt\-in rather than a default right\[[20](https://arxiv.org/html/2606.19116#bib.bib539)\]\. Between July 2025 and January 2026, the number of websites actively blocking AI crawlers was nearly seven times the number blocking traditional search crawlers such as Googlebot\[[23](https://arxiv.org/html/2606.19116#bib.bib564),[13](https://arxiv.org/html/2606.19116#bib.bib403)\]\. Raw requests from GPTBot grew 147% from July 2024 to July 2025, while Meta\-ExternalAgent grew 843% over the same period\[[21](https://arxiv.org/html/2606.19116#bib.bib565)\], demonstrating that the scale asymmetry between human and agent access is not marginal but orders of magnitude\.

The economic asymmetry driving this blocking response is equally stark\. Cloudflare data from mid\-2025 reveals that Google’s search crawler — operating under the traditional crawl\-for\-traffic social contract — crawled approximately 14 pages per human referral generated\. By contrast, OpenAI’s crawler exhibited a crawl\-to\-referral ratio of 1,700:1, and Anthropic’s crawler 73,000:1\[[22](https://arxiv.org/html/2606.19116#bib.bib540)\]\. From a publisher’s perspective, this ratio represents pure extraction: content consumed at massive scale with essentially zero value returned\. The blocking response is rational given this asymmetry — but it is a symptom of architectural failure, not a solution\.

#### 3\.1\.3The Nature of the Incompatibility

The fundamental incompatibility at the access layer is not that agents access content — humans do that too — but that the web has no mechanism to distinguish between access types that carry very different implications:

- 1\.Apersonal agentacting for one user, consuming content on their behalf as a proxy for a human visit — analogous to a human reading a webpage\.
- 2\.Asearch agentcrawling content to build an index that returns traffic to publishers — analogous to Googlebot under the original social contract\.
- 3\.Atraining crawlerbulk\-extracting content to train commercial AI models — a use case with no human analog and genuinely novel economic implications\.
- 4\.Amalicious botprobing for vulnerabilities, scraping for competitive intelligence, or performing denial\-of\-service — which should be blocked regardless of whether the visitor is human or agent\.

Because current infrastructure cannot distinguish these cases, the default response is to block all non\-human access indiscriminately — effectively treating a personal assistant agent acting for a paying subscriber with the same hostility as a malicious scraper\. This is the access layer’s core failure: the absence of an agent identification and intent declaration mechanism that would enable graduated, context\-appropriate responses rather than binary block\-or\-allow decisions\.

### 3\.2Layer 2: The Economic Model

#### 3\.2\.1The Human\-Centric Assumption

The web’s economic model is built on the attention economy — a two\-sided market in which publishers provide content to attract human attention, and advertisers pay to access that attention\[[28](https://arxiv.org/html/2606.19116#bib.bib529),[56](https://arxiv.org/html/2606.19116#bib.bib551)\]\. Every economic mechanism the web has developed — advertising impressions, click\-through rates, pageviews, time\-on\-site, subscription paywalls — is a proxy for human attention\. The implicit assumption is that value flows when a human engages with content: seeing an advertisement, clicking a link, spending time reading\. This model sustained a diverse publishing ecosystem for three decades, funding journalism, research, creative work, and open knowledge resources\.

The attention economy’s dependence on human engagement as its fundamental unit of value is not incidental — it is structural\. Remove the human from the content consumption loop and the entire economic architecture collapses, because there is no attention to monetize, no click to count, and no impression to sell\.

#### 3\.2\.2Empirical Evidence of Failure

AI agents are removing humans from the content consumption loop at measurable and accelerating scale\. Zero\-click searches — where AI systems synthesize answers without routing users to source content — now account for approximately 60% of all Google queries\[[49](https://arxiv.org/html/2606.19116#bib.bib541)\]\. In AI\-native search modes, this figure rises to 93%\[[49](https://arxiv.org/html/2606.19116#bib.bib541)\]\. Only 1% of users click links inside an AI Overview\[[45](https://arxiv.org/html/2606.19116#bib.bib563)\]\. Click\-through rates at position one in search results have fallen from 27% to 11% as AI Overviews have become pervasive\[[51](https://arxiv.org/html/2606.19116#bib.bib542)\]\. The downstream impact on publishers is severe and accelerating — HubSpot reported organic traffic declines of 70–80%, Chegg lost 49% of its traffic, and NPR characterized the shift as an extinction\-level event for online news\[[51](https://arxiv.org/html/2606.19116#bib.bib542)\]\.

The economic asymmetry driving this blocking response is equally stark\. Cloudflare data from mid\-2025 reveals that Google’s search crawler — operating under the traditional crawl\-for\-traffic social contract — crawled approximately 14 pages per human referral generated\. By contrast, OpenAI’s crawler exhibited a crawl\-to\-referral ratio of 1,700:1, and Anthropic’s crawler 73,000:1\[[22](https://arxiv.org/html/2606.19116#bib.bib540)\]\. Zero\-click searches now account for approximately 60% of all Google queries\[[49](https://arxiv.org/html/2606.19116#bib.bib541)\], rising to 93% in AI\-native search modes, with click\-through rates at position one falling from 27% to 11%\[[51](https://arxiv.org/html/2606.19116#bib.bib542)\]\. Figure[1](https://arxiv.org/html/2606.19116#S3.F1)illustrates how the human web’s working value chain — where content consumption generates ad revenue that returns to publishers — is structurally broken under agent interaction, where content is consumed at scale with zero economic return\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/economic_loop.png)Figure 1:The broken economic loop\. In the human web \(top\), content consumption generates advertising revenue that returns to publishers, sustaining the content ecosystem\. In the agent web \(bottom\), agents retrieve and synthesize content on behalf of users without generating pageviews, clicks, or impressions — delivering value to users while returning zero economic signal to publishers\. This structural asymmetry is the primary driver of blanket agent blocking\.
#### 3\.2\.3The Nature of the Incompatibility

The economic incompatibility has three distinct components that must be addressed separately:

The consumption problem\.When an agent retrieves and synthesizes content on behalf of a user, the publisher receives no economic signal — no impression, no click, no pageview\. The value delivered to the user is real; the value returned to the publisher is zero\. This is not a problem of agent malice but of architectural mismatch: the web has no mechanism for value exchange at the point of agent consumption\.

The attribution problem\.When an agent synthesizes an answer from multiple sources, there is no mechanism for fractional attribution — no way to record that source A contributed 40% of the answer’s informational content and source B contributed 60%, and to distribute economic value accordingly\. Human\-web economics were never required to solve this problem because humans visit one page at a time\.

The production problem\.As AI generates increasing proportions of web content, the human creative labor that justified content economics — the journalist’s investigation, the researcher’s analysis, the expert’s judgment — is displaced\. An economic model that does not distinguish between human\-authored and AI\-generated content provides no incentive for the human contribution that grounds content quality and epistemic integrity\.

### 3\.3Layer 3: The Content Model

#### 3\.3\.1The Human\-Centric Assumption

The web’s content architecture was designed for human perception\. HTML encodes visual layout, typography, color, and interactive behavior — all properties relevant to human reading experience and irrelevant to machine comprehension\[[46](https://arxiv.org/html/2606.19116#bib.bib528)\]\. Search engine optimization, link structures, and content discovery mechanisms were designed around human attention patterns and search intent\[[19](https://arxiv.org/html/2606.19116#bib.bib527)\]\. Even the semantic web effort — intended to make content machine\-readable — was designed to augment human\-targeted HTML rather than replace it\[[17](https://arxiv.org/html/2606.19116#bib.bib561)\]\.

The result is a content layer that is simultaneously over\-specified for human rendering — carrying vast amounts of visual layout information irrelevant to agents — and under\-specified for machine comprehension, lacking the semantic structure, provenance information, and intent metadata that agents need to evaluate and use content reliably\.

#### 3\.3\.2Empirical Evidence of Failure

The inefficiency of current web content formats for agent consumption is quantifiable\.\[[55](https://arxiv.org/html/2606.19116#bib.bib562),[14](https://arxiv.org/html/2606.19116#bib.bib343)\]demonstrated that agent\-optimized content delivery — stripping visual rendering overhead and providing structured semantic content — achieves a 67\.6% reduction in token usage compared to standard HTML delivery\. Since token consumption is directly proportional to computational cost in LLM\-based agents, this represents a massive structural inefficiency imposed on every agent interaction with the current web\.

Beyond format inefficiency, the content layer faces a structural integrity problem with no precedent in the human web\.\[[50](https://arxiv.org/html/2606.19116#bib.bib544)\]demonstrated the phenomenon of model collapse — the progressive degradation of AI model quality when trained iteratively on AI\-generated data\. Each generation of AI content introduces statistical artifacts, amplifies existing biases, and loses the diversity that characterizes human\-origin content\. When this content re\-enters the training pipeline, these artifacts compound\. The web, as the primary reservoir of training data for large language models, is thus at risk of becoming a source of progressively degraded knowledge — not through any deliberate action but through the structural dynamics of AI content generation at scale\. We term this phenomenonepistemic recursion: the self\-referential loop in which AI\-generated content becomes the input for future AI content generation, progressively detaching web knowledge from human ground truth\. Figure[2](https://arxiv.org/html/2606.19116#S3.F2)illustrates this loop and its compounding degradation across generations\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/epistemic_recursion.png)Figure 2:The epistemic recursion loop\. Human\-origin knowledge enters the web through AI\-generated content\. AI agents retrieve and synthesize this content to produce further content, which re\-enters the web for subsequent retrieval\. Each cycle amplifies statistical artifacts and biases while reducing diversity, progressively detaching web knowledge from human ground truth\. The dashed arrow represents the feedback path that sustains the loop across generations\. Unlike model collapse\[[50](https://arxiv.org/html/2606.19116#bib.bib544)\], which describes degradation within a single training pipeline, epistemic recursion describes a web\-scale phenomenon operating across the entire AI content ecosystem\.
#### 3\.3\.3The Nature of the Incompatibility

We identify three distinct content\-layer incompatibilities:

The format incompatibility\.Current web content formats — HTML, JavaScript\-rendered pages, PDF documents — carry significant visual rendering overhead that is meaningless to agents and computationally expensive to process\. Agents require structured semantic content: clean text, explicit metadata, machine\-readable relationships between concepts\. The web currently has no standard format serving this requirement, though recent proposals such asllms\.txt\[[35](https://arxiv.org/html/2606.19116#bib.bib566)\]represent early steps toward agent\-readable content declarations\.

The provenance incompatibility\.Current web content carries no standard mechanism for declaring its origin, derivation chain, or the degree of human oversight involved in its production\. An agent retrieving a web page cannot determine whether it is reading a human expert’s peer\-reviewed analysis, an AI\-generated content farm article, or a human\-AI collaborative piece\. This absence of provenance metadata prevents agents from making quality\-aware content decisions and enables the epistemic recursion problem described above\.

The discoverability incompatibility\.Search engine optimization was built to attract human attention through search rankings optimized for human query patterns\. There is no equivalent agent discoverability standard — no mechanism by which an agent can determine which site holds authoritative data on a topic, what content formats a site supports, what access terms apply, or what semantic capabilities a site exposes\. Agents currently navigate content discovery through human\-facing search interfaces, producing significant inefficiency and unreliability\.

### 3\.4Summary: Three Layers, One Failure

The three failures diagnosed in this section are not independent — they are deeply coupled in a self\-reinforcing cycle\. The access layer’s inability to identify agent intent drives blanket blocking, which prevents the development of economic models for legitimate agent access\. The absence of economic models eliminates the incentive for publishers to invest in high\-quality, human\-supervised content production\. The collapse of human\-supervised content production accelerates epistemic recursion, further degrading the web’s knowledge quality and reinforcing the perception that agent access is extractive rather than valuable — which in turn strengthens the case for blanket blocking\. Figure[3](https://arxiv.org/html/2606.19116#S3.F3)illustrates this interdependency\. Addressing any single layer in isolation — as existing work has largely done — produces solutions that are undermined by the failures of the other two\. A coherent redesign must address all three layers simultaneously, grounded in a unified philosophical foundation, which we develop in the following sections\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/three_layer_coupling.png)Figure 3:Three\-layer failure coupling\. The access, economic, and content layer failures of the human\-centric web are deeply interdependent\. Blanket blocking \(access\) prevents the development of economic models; absent economic models eliminate incentives for quality content production; degraded content quality reinforces the case for blocking\. The dashed arc represents the feedback path completing the cycle\. Fixing any single layer in isolation creates new contradictions — all three must be redesigned simultaneously\.Table[2](https://arxiv.org/html/2606.19116#S3.T2)summarizes the three\-layer diagnosis across the key dimensions of assumption, empirical evidence, and nature of incompatibility\.

Table 2:Three\-layer diagnosis of human\-centric web failure under agent interaction\.LayerHuman\-centricassumptionEmpiricalevidenceNature ofincompatibilityAccessVisitors are single humansat human speed7×\\timesmore sitesblocking AI vs Googlebot\[[23](https://arxiv.org/html/2606.19116#bib.bib564)\]No mechanism to distinguishagent types or intentEconomicValue flows through humanattention and clicks73,000:1 crawl\-to\-referral ratio\[[22](https://arxiv.org/html/2606.19116#bib.bib540)\];93% zero\-click in AI search\[[49](https://arxiv.org/html/2606.19116#bib.bib541)\]No value exchange at point of agentconsumption; attribution impossibleContentContent is human\-authored andrendered for human perception67\.6% token overhead in HTML\[[55](https://arxiv.org/html/2606.19116#bib.bib562)\];model collapse under AI training\[[50](https://arxiv.org/html/2606.19116#bib.bib544)\]Format overhead; no provenance standard;epistemic recursion risk

## 4The Access Layer Redesign

Having diagnosed the three\-layer failure of the human\-centric web in Section[3](https://arxiv.org/html/2606.19116#S3), we now turn to proposed solutions\. This section addresses the access layer, proposing a principled redesign grounded in a single philosophical anchor: that AI agents acting on behalf of humans are first\-class web citizens entitled to the same presumption of access as the humans they represent\.

### 4\.1The Agent\-as\-Human\-Proxy Principle

The web’s founding access philosophy — articulated by Berners\-Lee as the principle that any client should be able to retrieve any resource without prior negotiation\[[18](https://arxiv.org/html/2606.19116#bib.bib526)\]— was grounded in a presumption of human visitors\. We propose extending this presumption to AI agents through what we term theagent\-as\-human\-proxy principle:

> An AI agent acting on behalf of a human user should inherit the same presumption of access as that human — no more, no less\. The agent’s access rights, obligations, and economic responsibilities are determined by the human it represents, not by its nature as an automated system\.

This principle has several important implications\. First, it establishes that the relevant unit of access is the human\-agent pair, not the agent in isolation\. An anonymous human can browse freely — their agent should too\. A subscribed human has paid for access — their agent should inherit it\. A human cannot legally bulk\-scrape a million pages for commercial resale — neither can their agent\. Second, it implies that blanket blocking of agents is philosophically unjustifiable when those agents represent humans who would otherwise be welcome\. Third, it identifies the two genuinely novel cases that fall outside the human\-proxy principle — commercial training use and multi\-user content aggregation — as the only cases requiring new access rules rather than extensions of existing human\-web norms\.

The agent\-as\-human\-proxy principle does not eliminate the need for agent identification — it defines what that identification must accomplish\. Rather than requiring agents to prove full identity \(which raises privacy concerns and creates friction\), agents need only declare thebehavioral contextthat makes the proxy relationship legible: who they represent, at what scale, and for what purpose\.

### 4\.2Agent Identification Metadata

The central technical contribution of this section is the proposal foragent identification metadata— a set of standardized HTTP request headers that allow agents to declare their identity, represented user, and intent to web servers\. This proposal is grounded in a direct analogy to existing browser identification mechanisms\.

When a browser makes an HTTP request, it sends a set of headers that identify its capabilities and context\[[30](https://arxiv.org/html/2606.19116#bib.bib558)\]:

```
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X)
Accept: text/html,application/xhtml+xml
Accept-Language: en-US,en;q=0.9
```

Servers read these headers to serve appropriate content — mobile vs desktop layouts, supported media types, preferred language versions\. No new protocol is required; the mechanism is embedded in the existing HTTP infrastructure that every web server already supports\.

We propose an analogous mechanism for agents:

```
Agent-Identity: claude/sonnet-4
Agent-Represents: user/anonymous
Agent-Intent: personal-use
Agent-Auth: Bearer <delegation-token>
Agent-Rate-Class: free
```

The semantics of each header are as follows\.Agent\-Identitydeclares the agent system and version — analogous toUser\-Agentfor browsers\.Agent\-Representsdeclares the relationship between the agent and the human it serves — anonymous, authenticated, or subscribed\.Agent\-Intentdeclares the purpose of the request from a controlled vocabulary:personal\-use,search,training,commercial, orresearch\.Agent\-Authcarries a delegation token linking the agent to a human user’s existing credentials — enabling subscription inheritance without exposing personal identity\.Agent\-Rate\-Classdeclares the agent’s expected consumption tier, enabling servers to pre\-authorize appropriate rate limits\.

Figure[4](https://arxiv.org/html/2606.19116#S4.F4)illustrates the full request\-response flow enabled by agent identification metadata, showing how servers can distinguish agent types and respond with graduated access rather than binary blocking\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/agent_metadata.png)Figure 4:Agent identification metadata flow\. An AI agent sends standardized metadata headers alongside its HTTP request, declaring identity, represented user, and intent\. The web server reads these headers to identify the agent type, verify authorization, apply appropriate rate limits, and select the correct content format \(ATML for agents, HTML for humans\)\. All communication occurs over existing HTTP/TCP infrastructure — no new transport protocol is required\. The metadata mechanism directly extends the browser User\-Agent identification pattern already ubiquitous in web infrastructure\.This proposal requires no new transport protocol — everything operates over existing HTTP infrastructure that every web server already supports\. It is backward compatible: servers that do not recognize agent metadata headers simply ignore them, serving their existing content as before\. Adoption can be incremental, with agent\-aware servers gaining the ability to serve optimized content and apply graduated access controls while non\-aware servers continue to function normally\.

A critical security consideration is metadata verification\. Unlike browser User\-Agent headers, which carry no cryptographic guarantee, agent metadata headers should be cryptographically signed to prevent impersonation — the Perplexity case demonstrated that bad actors will modify agent identifiers to circumvent restrictions\[[1](https://arxiv.org/html/2606.19116#bib.bib543)\]\. We propose that theAgent\-Identityheader be accompanied by a cryptographic signature verifiable against a public key registry maintained by registered agent operators, analogous to DKIM signatures for email\[[2](https://arxiv.org/html/2606.19116#bib.bib567)\]\. This does not prevent all impersonation but raises the cost significantly above simple header modification\.

### 4\.3Rate Limiting as the New Default

The agent\-as\-human\-proxy principle implies that blanket blocking of agents is not a legitimate long\-term access policy — it is a reactive measure taken in the absence of better tools\. Agent identification metadata provides those tools\. Once servers can distinguish agent types and intents, the appropriate response is graduated rate limiting rather than binary blocking\.

We propose that the default access control posture for agent requests shift fromblock unless permittedtorate limit unless escalated, mirroring the web’s existing posture toward anonymous human visitors\. Figure[5](https://arxiv.org/html/2606.19116#S4.F5)illustrates the proposed tier model\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/rate_limit_tiers.png)Figure 5:Graduated rate limiting model replacing blanket blocking\. Agent access is tiered by declared intent and authentication level rather than binary blocked or allowed\. Anonymous personal agents receive rate\-limited free access, equivalent to anonymous human browsing\. Authenticated agents inherit their represented user’s subscription\. Training crawlers access a licensed tier with negotiated terms\. Only malicious behavior — not agent identity — triggers blocking\. This model preserves the web’s open access philosophy while providing servers with meaningful differentiation between agent types\.The specific rate limits for each tier are not mandated by this framework — they are determined by individual publishers based on their content economics and infrastructure capacity\. The framework mandates only the structure: that tiers exist, that they are determined by intent and authentication level rather than by agent identity alone, and that blocking is reserved for behaviorally malicious access rather than applied to all non\-human requests indiscriminately\.

### 4\.4Subscription Inheritance Protocol

Perhaps the most immediately actionable proposal in this paper is subscription inheritance — the principle that a human user’s existing paid subscriptions should extend to their AI agents without requiring separate agent subscriptions\. This is a direct application of the agent\-as\-human\-proxy principle: if a human has paid for access, their agent proxy should have it too\.

Technically, this requires a lightweight extension to existing OAuth 2\.0 delegation mechanisms\[[33](https://arxiv.org/html/2606.19116#bib.bib568)\]\. A user authenticates with a content provider as normal, obtaining an access token\. When authorizing their AI agent, the user generates adelegation token— a derived credential that encodes:

- 1\.The human user’s subscription level and scope
- 2\.The agent system authorized to use it
- 3\.The permitted use cases \(personal\-use only, no training, no redistribution\)
- 4\.An expiry and revocation mechanism

The delegation token is passed as theAgent\-Authheader in agent requests\. The server verifies the token, confirms it maps to a valid subscription, and grants the agent the same content access the human subscriber would receive\. The human user’s subscription is not consumed or duplicated — it is extended to their proxy\.

This mechanism has several advantages over alternative approaches\. It requires no changes to existing subscription billing infrastructure\. It preserves publisher revenue — the human still pays for subscription, the agent merely inherits it\. It gives users meaningful control over what their agents can access on their behalf\. And it is directly compatible with existing OAuth infrastructure deployed by virtually every major content platform\.

### 4\.5Dual\-Layer Web Architecture

The access layer redesign requires not only new identification and authorization mechanisms but a new approach to content delivery\. We propose adual\-layer web architecturein which the same domain serves both human\-readable HTML and agent\-optimized content — with the server selecting the appropriate format based on request metadata\.

This is not a new concept in web architecture — content negotiation has been part of HTTP since its earliest versions\[[30](https://arxiv.org/html/2606.19116#bib.bib558)\], enabling servers to serve different content types to different clients\. TheAcceptheader allows browsers to declare their preferred media types; servers respond with the best available match\. We propose extending this mechanism to include an agent\-optimized content type — which we call ATML \(Agent Text Markup Language, developed in Section[6](https://arxiv.org/html/2606.19116#S6)\) — as a first\-class content type alongside HTML\.

A browser request declares:

```
Accept: text/html, application/xhtml+xml
```

An agent request declares:

```
Accept: application/atml+xml, text/markdown
Agent-Identity: claude/sonnet-4
Agent-Intent: personal-use
```

The server responds with HTML to the browser and ATML to the agent — the same content, differently structured\. No separate domain, no separate publishing workflow\. Publishers maintain one content repository; the server layer handles format negotiation\.

Figure[6](https://arxiv.org/html/2606.19116#S4.F6)illustrates this architecture and the three\-phase migration path from the current human\-only web to an agent\-first web\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/dual_layer.png)Figure 6:Dual\-layer web architecture and migration path\. The same domain serves HTML responses to browser requests and ATML responses to agent requests, with format selection driven by request metadata\. Phase 1 represents the current state \(HTML only\)\. Phase 2 introduces parallel ATML alongside HTML\. Phase 3 establishes ATML as the primary format with HTML maintained for legacy compatibility — analogous to how HTTPS was added alongside HTTP before becoming the mandatory standard\.The migration path is deliberately gradual\. Phase 1 represents the current state — human HTML only, with agent metadata headers added as a forward compatibility measure\. Phase 2 introduces ATML as a parallel delivery format, with publishers adopting it incrementally\. Phase 3 establishes ATML as the primary delivery format, with HTML maintained for legacy browser compatibility\. This mirrors the HTTPS migration: years of coexistence, then a tipping point driven by security requirements, then universal adoption\[[29](https://arxiv.org/html/2606.19116#bib.bib569)\]\.

### 4\.6A New agents\.txt Standard

The final component of the access layer redesign is a replacement forrobots\.txt— a richer, machine\-readable standard that declares a site’s agent access policy in a way that the existing honor\-system standard cannot\. We proposeagents\.txt, a structured declaration file served at a well\-known URL \(/\.well\-known/agents\.txt\) that encodes:

```
# agents.txt example
[personal-agent]
allow: true
rate-limit: 100000 tokens/day
content-format: atml, markdown
auth-required: false

[subscribed-agent]
allow: true
rate-limit: unlimited
content-format: atml, markdown
auth-required: true
auth-type: delegation-token

[training-crawler]
allow: conditional
license-required: true
contact: [email protected]

[malicious-bot]
allow: false
```

Unlikerobots\.txt,agents\.txtis not an honor system — its declarations are enforced by the agent identification metadata mechanism\. A server receiving a request withAgent\-Intent: trainingcan automatically apply the\[training\-crawler\]policy declared inagents\.txt, without manual configuration per crawler\. Publishers gain granular control over agent access without resorting to blanket blocking\. Agents gain clear, machine\-readable signals about what access is available and under what terms — eliminating the current situation where agents must attempt access and interpret failures heuristically\.

### 4\.7Summary

Table[3](https://arxiv.org/html/2606.19116#S4.T3)summarizes the access layer redesign proposals and their relationship to the failures diagnosed in Section[3\.1](https://arxiv.org/html/2606.19116#S3.SS1)\.

Table 3:Access layer redesign: diagnosed failures and proposed solutions\.Diagnosed failureProposed solutionNo agent identity standardAgent identification metadata headers over HTTPBlanket blocking defaultGraduated rate limiting by intent and auth levelNo subscription extensionOAuth delegation token for subscription inheritanceHTML\-only deliveryDual\-layer architecture with ATML content negotiationrobots\.txt inadequacyagents\.txt — rich machine\-readable access policy

Together these five proposals constitute a complete access layer redesign that preserves the web’s founding philosophy of open access while providing the identification, authorization, and policy infrastructure necessary for servers to respond to agents appropriately rather than defensively\. Critically, all proposals operate over existing HTTP/TCP infrastructure — no new transport protocol is required\. The agent\-first web is not a new internet; it is the existing internet with the missing metadata layer added\.

## 5The Economic Layer Redesign

The access layer redesign proposed in Section[4](https://arxiv.org/html/2606.19116#S4)establishes how agents reach content\. This section addresses what happens when they do — proposing an economic framework that replaces the broken attention economy with a principled value\-exchange model\. We argue that the economic redesign must be grounded in the same philosophical anchor as the access redesign: the agent\-as\-human\-proxy principle\. An agent’s economic obligation mirrors that of the human it represents — no more, no less\.

### 5\.1The Attention Economy Cannot Be Patched

The web’s attention economy was not designed — it emerged organically as advertising became the dominant funding mechanism for online content\[[28](https://arxiv.org/html/2606.19116#bib.bib529),[56](https://arxiv.org/html/2606.19116#bib.bib551)\]\. Its core mechanism — human attention proxied by clicks and pageviews, monetized through advertising impressions — worked because it created a closed value loop: publishers produced content, humans consumed it, advertisers paid for access to that attention, and revenue returned to publishers who produced more content\. Every layer of the web’s architecture evolved to serve this loop: SEO to attract human attention, analytics to measure it, advertising networks to monetize it\.

AI agents do not fit this loop\. They do not see advertisements\. They do not generate pageviews\. They do not click\. The value they deliver to users is real — task completion, information synthesis, decision support — but that value is entirely decoupled from the economic signals the attention economy was built to capture\. This is not a marginal disruption\. Zero\-click searches now account for 93% of queries in AI\-native search modes\[[49](https://arxiv.org/html/2606.19116#bib.bib541)\], and click\-through rates at position one have fallen from 27% to 11%\[[51](https://arxiv.org/html/2606.19116#bib.bib542)\]\. The attention economy is not being disrupted at the edges — it is collapsing at its foundation\.

Reactive patches — Cloudflare’s pay\-per\-crawl model, Perplexity’s revenue sharing program\[[1](https://arxiv.org/html/2606.19116#bib.bib543),[20](https://arxiv.org/html/2606.19116#bib.bib539)\]— address specific friction points without a principled framework\. They treat agent access as a billing problem rather than a design problem\. The result is an incoherent patchwork: some content is paywalled per crawl, some is blocked outright, some is freely scraped, and publishers have no consistent basis on which to make access decisions\. A principled framework is needed\.

### 5\.2The Agent\-as\-Human\-Proxy Economic Principle

We propose that the economic redesign be grounded in a single principle that resolves the majority of agent content access cases without requiring new economic models:

> An agent’s economic obligation to a content publisher is equivalent to the economic obligation of the human it represents\. If the content is free for anonymous humans, it is free for their agents\. If the content requires a subscription, the agent inherits that subscription\. The economic relationship is between the human and the publisher; the agent is a proxy, not a new economic entity\.

This principle is powerful because it resolves most agent content access cases by reference to existing human\-web economics — no new models required\. It also identifies precisely where new models are needed: the genuinely novel cases for which no human analog exists\. These are commercial training use — where an agent bulk\-extracts content to train AI models, a use case no human performs — and multi\-user aggregation — where a single agent serves content to millions of users simultaneously, a scale no human achieves\. For these two cases, the human\-proxy principle does not apply, and new economic models must be designed from first principles\.

### 5\.3The Intent\-Based Economic Tier Model

Applying the agent\-as\-human\-proxy principle produces a natural taxonomy of agent content access behaviors, each with an appropriate economic model\. We term this theintent\-based economic tier model\. Figure[7](https://arxiv.org/html/2606.19116#S5.F7)presents the full model\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/economic_tiers.png)Figure 7:Intent\-based economic tier model\. Agent content access behaviors are classified by declared intent and authentication level\. The top three tiers — personal use, subscribed access, and search/discovery — are resolved by the agent\-as\-human\-proxy principle, mapping directly to existing human\-web economic models\. The bottom three tiers — commercial training, multi\-user aggregation, and bulk extraction — represent genuinely novel cases with no human analog, requiring new economic models\. The dividing line between these groups is the boundary of the human\-proxy principle\.For the human\-proxy tiers, the economic model is straightforward\. A personal agent accessing free content on behalf of an anonymous user pays nothing — exactly as the user would\. A subscribed agent accessing paywalled content inherits the user’s subscription via the delegation token mechanism described in Section[4\.4](https://arxiv.org/html/2606.19116#S4.SS4)\. A search agent crawling content for indexing operates under the traditional crawl\-for\-traffic social contract, with a machine\-readable attribution requirement replacing the implicit honor system thatrobots\.txtcurrently relies on\.

For the novel tiers, new models are required\. Commercial training access requires a licensing framework analogous to stock photo or music sync licensing — negotiated terms declared inagents\.txt, with fees proportional to the commercial value derived\. Multi\-user aggregation — where a single AI system serves millions of users using content from many publishers — requires a collective model analogous to music streaming royalties, discussed in Section[5](https://arxiv.org/html/2606.19116#S5)\.

### 5\.4The Open/Closed Source Analogy

The intent\-based tier model preserves a critical property of the current web: economic diversity\. The web’s strength has never derived from a single economic model — it derives from the coexistence of free and paid content, open and proprietary knowledge, advertising\-supported and subscription\-supported publishing\. This diversity must be preserved in the agent\-first web\.

We propose framing the content economics decision for publishers through an analogy that has already proven robust in the adjacent domain of software: the open source / closed source distinction\[[47](https://arxiv.org/html/2606.19116#bib.bib559)\]\. Just as a software developer chooses whether to release code under an open license \(freely available, potentially with attribution requirements\) or a proprietary license \(paid access, restricted use\), a publisher chooses whether to make content available to agents under an open model or a paid model\. The choice is entirely the publisher’s — the framework mandates no universal model\.

Under this framing:

Open contentis freely accessible to agents under the same terms as humans\. Rate limits apply as the boundary of free access — analogous to open source licenses that permit free use but not unlimited commercial exploitation\. Wikipedia, government data, open research, personal blogs — all of these continue to operate as they do today, with agents welcome on the same terms as humans\.

Paid contentrequires economic exchange for agent access\. Publishers declare their pricing terms inagents\.txtand content headers\. The economic model — subscription, per\-token, outcome\-based — is chosen by the publisher\. The agent checks declared terms before accessing and pays accordingly\. The New York Times, academic journals, professional data services — all of these can operate paid agent access tiers independently of their human subscription models\.

This framing is politically important as well as technically clean\. It does not mandate that all content be paid — which would kill open knowledge and information access\. It does not mandate that content be free — which would destroy publisher economics\. It gives publishers the same choice software developers have had for decades and lets the market produce diversity\.

### 5\.5Token\-Based Subscription Model

For paid content, we propose thetoken\-based subscription modelas the primary economic mechanism for agent content access\. Rather than metering access by pageviews, article counts, or time periods — all human\-centric units — content access is metered in tokens consumed, directly mirroring how AI API access is priced today\[[4](https://arxiv.org/html/2606.19116#bib.bib536)\]\.

Under this model, a publisher declares a price per million tokens for their content\. An agent consuming that content is billed based on tokens actually consumed from the response\. Publishers can declare different prices for different content tiers — breaking news vs archived content, premium analysis vs commodity information — using theagents\.txtpricing declarations introduced in Section[4\.6](https://arxiv.org/html/2606.19116#S4.SS6)\.

Figure[8](https://arxiv.org/html/2606.19116#S5.F8)illustrates the token\-based subscription model and its relationship to existing AI infrastructure\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/token_model.png)Figure 8:Token\-based subscription model\. Content access is metered in tokens consumed rather than pageviews or article counts\. Publishers declare price per million tokens; agents pay on consumption\. Free tier access is rate\-limited by token budget rather than blocked\. The model is directly compatible with existing AI API billing infrastructure — the same token metering used by major AI providers requires no new payment rails for adoption\.The token\-based model has several properties that make it well\-suited to agent content economics\. First, it is directly compatible with existing AI billing infrastructure\. AI providers already meter token consumption for API access — the same infrastructure can be extended to meter content consumption, requiring no new payment rails\. Second, it is proportional to actual consumption\. A short agent query that retrieves a single paragraph costs less than a deep research task that retrieves thousands of articles — unlike subscription models that charge fixed fees regardless of actual use\. Third, it is format\-agnostic\. ATML content, Markdown content, structured data — all can be metered in tokens regardless of their format, providing a unified pricing mechanism across the heterogeneous content landscape of the web\.

For free\-tier access, the token budget functions as a rate limit rather than a paywall — an agent consuming 100,000 tokens per day of a publisher’s content at no charge is welcome; an agent consuming 10 million tokens per day for commercial purposes is directed to the paid tier\. The boundary is declared inagents\.txt, enforced by the agent identification metadata mechanism, and transparent to both agents and publishers\.

### 5\.6The Commissioned Content Economy

The economic models described above address the demand side of agent content economics — how agents pay for content they consume\. We now address the supply side: how content production in the agent\-first web can be economically sustainable and epistemically grounded\.

The epistemic recursion problem identified in Section[3\.3](https://arxiv.org/html/2606.19116#S3.SS3)— the self\-referential loop in which AI generates content that AI consumes — arises precisely because AI\-generated content currently carries no economic cost and no provenance requirement\. Content farms flood the web with AI\-generated articles at negligible marginal cost; there is no economic signal distinguishing this content from human\-authored journalism, and no provenance mechanism flagging it as AI\-derived\. The loop is sustained by the absence of economic friction at the production stage\.

We propose thecommissioned content economyas a structural solution to this problem\. Under this model, AI\-generated content that enters the web as a publishable resource — rather than as an ephemeral agent response — is produced under a commissioning relationship: a human entity pays for the content to be produced, establishing a human intentionality anchor at the production stage\. The commissioned content is published with a machine\-readable provenance tag declaring the commissioner, the producing agent, and the human oversight level\. Other agents and humans can access the content under declared terms, with revenue flowing back to the commissioner\.

Figure[9](https://arxiv.org/html/2606.19116#S5.F9)illustrates the commissioned content economy as a two\-sided marketplace\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/commissioned_content.png)Figure 9:The commissioned content economy\. On the supply side, a human entity commissions an AI agent to produce content under a paid agreement\. The content is published with a provenance tag declaring commissioner, producing agent, and human supervision level\. On the demand side, other agents and human readers access the content under declared terms, paying where required\. Revenue flows back to the commissioner\. The critical contribution of this model is the human commissioning decision at the production stage — which introduces human intentionality as a ground truth anchor, breaking the epistemic recursion loop even when the content itself is AI\-generated\.The commissioned content model does not require that all web content be commissioned — the open web continues to exist for informal, personal, and community\-generated content\. It provides an economic structure for the subset of web content where quality, provenance, and trustworthiness matter most: journalism, research, professional analysis, educational content\. For this subset, the commissioning relationship reintroduces the human judgment that the epistemic recursion loop erodes — not by requiring humans to write every word, but by requiring a human to decide that this content is worth producing and to take responsibility for it\.

### 5\.7The Agent Advertising Model

The attention economy’s primary mechanism — advertising — does not disappear in the agent\-first web\. It transforms\. When an agent retrieves content on behalf of a human user and presents a synthesized response, there remains a human attention moment: the user reading the agent’s output\. This moment is potentially monetizable, though the mechanism differs fundamentally from traditional web advertising\.

We sketch a preliminary model for agent\-targeted advertising, acknowledging that full development of this model constitutes a separate research contribution beyond the scope of this paper\. Under the proposed model, publishers embed structured advertisement metadata in their ATML content alongside substantive content\. When an agent retrieves and synthesizes this content, it carries the advertisement metadata as part of its response payload\. The agent frontend — Claude, ChatGPT, Gemini, or other interface — surfaces the advertisement to the human user at an appropriate point in the interaction\. An impression is recorded when the advertisement reaches the human; revenue flows to the publisher whose content carried it\.

This model faces a significant open challenge: agent frontends have limited incentive to surface publisher advertisements, particularly when doing so competes with their own monetization interests\. Resolving this tension requires either a protocol\-level standard mandating advertisement carriage — analogous to the must\-carry rules in broadcast television regulation — or a revenue\-sharing agreement between publishers and agent frontend operators\. Both approaches involve stakeholder coordination that extends beyond technical standardization\. We identify this as a priority open challenge in Section[8](https://arxiv.org/html/2606.19116#S8)and flag it as a direction for future work\.

### 5\.8Summary

Table[4](https://arxiv.org/html/2606.19116#S5.T4)summarizes the economic layer redesign proposals and their relationship to the failures diagnosed in Section[3\.2](https://arxiv.org/html/2606.19116#S3.SS2)\.

Table 4:Economic layer redesign: diagnosed failures and proposed solutions\.Diagnosed failureProposed solutionAttention economy collapses under agent interactionIntent\-based tier model grounded in agent\-as\-human\-proxy principleNo value exchange at point of agent consumptionToken\-based subscription model — metered on consumptionUniversal payment mandates kill open knowledgeOpen/closed source analogy — publisher chooses modelNo economic model for AI content productionCommissioned content economy with human intentionality anchorAttribution impossible at scaleCollective pool model for multi\-user aggregation tiersAd model breaks without human attentionAgent advertising model \(open challenge — future work\)

The economic layer redesign is the most complex of the three layers because it must accommodate the full diversity of the existing web — free and paid, open and proprietary, advertising\-supported and subscription\-supported — while introducing new mechanisms for the genuinely novel cases that agent interaction creates\. The intent\-based tier model, grounded in the agent\-as\-human\-proxy principle, provides the organizing framework\. The token\-based subscription model provides the primary mechanism for paid access\. The commissioned content economy addresses the supply side\. Together they constitute a coherent replacement for the attention economy — one designed from first principles for a web in which agents are primary consumers of content\.

## 6The Content Layer Redesign

The access layer redesign \(Section[4](https://arxiv.org/html/2606.19116#S4)\) establishes how agents reach content and identify themselves\. The economic layer redesign \(Section[5](https://arxiv.org/html/2606.19116#S5)\) establishes how value flows when they do\. This section addresses what the content itself looks like in an agent\-first web — proposing a new content format, a human supervision standard, a provenance chain architecture, and an agent\-native discoverability mechanism\. Together these proposals address the three content\-layer incompatibilities diagnosed in Section[3\.3](https://arxiv.org/html/2606.19116#S3.SS3): format inefficiency, provenance absence, and epistemic recursion\.

### 6\.1Design Principles for Agent\-First Content

Before proposing specific mechanisms, we establish the design principles that agent\-first content must satisfy\. These principles are derived from the requirements of LLM\-based agents as content consumers — requirements that are fundamentally different from those of human readers\.

Semantic richness over visual presentation\.Agents process meaning, not appearance\. HTML’s investment in visual layout — typography, color, responsive grid systems, animation — is not merely neutral overhead for agents; it is active noise that consumes token budget without contributing to comprehension\. Agent\-first content must prioritize semantic structure: what is being said, who said it, what it is derived from, and how confident the source is\. Visual presentation is a secondary concern addressed by the rendering layer of the agent frontend, not the content layer\.

Explicit provenance as a first\-class property\.An agent retrieving content must be able to determine its origin, derivation chain, and human oversight level without inferring it from context\. Provenance is not metadata appended to content — it is a structural property of the content itself, as fundamental as the content body\.

Machine\-readable access and license terms\.An agent must be able to determine, before consuming content, what terms govern that consumption — free or paid, personal or commercial, redistributable or restricted\. These terms must be embedded in the content itself or in a well\-known declaration file, not buried in a human\-readable terms\-of\-service page that no agent can reliably parse\.

Token efficiency\.Every token an agent consumes processing layout noise rather than semantic content is a token wasted\.webMCP Authors \[[55](https://arxiv.org/html/2606.19116#bib.bib562)\], Bandaraet al\.\[[6](https://arxiv.org/html/2606.19116#bib.bib359)\]demonstrated that current HTML carries 67\.6% token overhead relative to semantic content alone\. Agent\-first content must minimize this overhead — not merely for economic reasons but because token efficiency directly determines the quality and depth of agent reasoning over content at scale\.

Dual\-layer compatibility\.The human web does not disappear\. Agent\-first content must coexist with human\-readable content through the dual\-layer architecture proposed in Section[4\.5](https://arxiv.org/html/2606.19116#S4.SS5)— served from the same domain, produced from the same content repository, differentiated only at the delivery layer\.

### 6\.2ATML: Agent Text Markup Language

We propose theAgent Text Markup Language \(ATML\)as the content format for agent\-first web delivery\. ATML is not an entirely new language — it is a structured semantic profile designed for machine consumption, composed of three explicit layers: provenance, access terms, and semantic content\. We engage first with the question of whether a new format is necessary at all\.

#### 6\.2\.1The HTML Debate

Karpathy \[[37](https://arxiv.org/html/2606.19116#bib.bib571)\]has argued that HTML is already sufficiently structured for agent consumption — that modern LLMs can parse HTML effectively and that a new format is unnecessary overhead\. This position has merit: HTML does encode semantic structure through heading tags, paragraph tags, list tags, and semantic HTML5 elements such as<article\>,<section\>, and<main\>\. Many agents do navigate HTML\-rendered web content successfully today\.

We argue, however, that HTML is inadequate for three reasons beyond parsing difficulty\. First, the token overhead problem is structural, not incidental: even well\-structured semantic HTML carries layout, styling, and scripting overhead that is irrelevant to agent consumption\. The 67\.6% token reduction demonstrated bywebMCP Authors \[[55](https://arxiv.org/html/2606.19116#bib.bib562)\], Bandaraet al\.\[[10](https://arxiv.org/html/2606.19116#bib.bib302)\]was achieved not by better HTML parsing but by delivering a semantically equivalent content representation without layout noise — a task HTML cannot accomplish without effectively becoming a different format\. Second, HTML has no native provenance layer\. The information an agent most needs — who authored this, what was it derived from, what supervision level applies — has no standardized HTML representation\. Third, HTML has no native machine\-readable access terms layer\. An agent cannot read a content license from HTML structure; it must parse a separate legal document written for human readers\.

ATML addresses all three gaps while remaining grounded in familiar XML syntax that existing tooling can process without new parsers\.

#### 6\.2\.2ATML Structure

ATML is organized into three layers, as illustrated in Figure[10](https://arxiv.org/html/2606.19116#S6.F10)\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/atml_structure.png)Figure 10:ATML structure compared to current HTML\. HTML carries significant visual rendering overhead — layout tags, CSS, JavaScript, navigation, advertisement slots — that is irrelevant to agent consumption and imposes a 67\.6% token overhead\[[55](https://arxiv.org/html/2606.19116#bib.bib562)\]\. ATML replaces this with three explicit semantic layers: provenance \(declaring origin, commissioner, derivation chain, and human supervision level\), access and economics \(declaring tier, rate limits, and pricing\), and semantic content \(clean text, summary, and structured entities\)\. The result is a format optimized for agent comprehension while carrying all the metadata agents need to evaluate, cite, and pay for content appropriately\.A minimal ATML document has the following structure:

```
<atml version="1.0">
  <provenance>
    <authored-by>human</authored-by>
    <commissioned-by>NYT</commissioned-by>
    <derived-from>
      <source url="https://..."
              retrieved="2025-11-01"/>
    </derived-from>
    <supervised>true</supervised>
    <supervision-level>2</supervision-level>
  </provenance>
  <access>
    <tier>paid</tier>
    <rate-limit unit="tokens"
                period="day">100000</rate-limit>
    <price unit="per-million-tokens">
      2.50
    </price>
    <license>personal-use</license>
    <license>no-training</license>
  </access>
  <content>
    <title>Article title here</title>
    <summary>Agent-optimized abstract
             in 2-3 sentences.</summary>
    <body>Clean semantic text without
          layout or styling markup.
    </body>
    <entities>
      <entity type="person">Name</entity>
      <entity type="org">Organisation
      </entity>
    </entities>
  </content>
</atml>
```

The<provenance\>layer is the ATML element with no HTML equivalent\. It declares the authorship type \(human, agent, or collaborative\), the commissioning entity, the sources the content was derived from with retrieval dates, and the human supervision level from the tier model proposed in Section[6\.3](https://arxiv.org/html/2606.19116#S6.SS3)\. This layer directly enables agents to make quality\-aware content decisions and breaks the epistemic recursion loop by making AI\-origin content explicitly visible\.

The<access\>layer declares economic terms in machine\-readable form, directly compatible with the token\-based subscription model proposed in Section[5\.5](https://arxiv.org/html/2606.19116#S5.SS5)\. Agents read these terms before consuming content — no separate legal document required\.

The<content\>layer carries the semantic text without layout noise\. The<summary\>field enables agents to make relevance decisions before consuming the full body — a significant token efficiency gain for research tasks involving many sources\. The<entities\>field provides structured key concepts that enable downstream reasoning without full\-body parsing\.

### 6\.3Human Supervision Requirements

The epistemic recursion problem identified in Section[3\.3](https://arxiv.org/html/2606.19116#S3.SS3)cannot be solved by format alone\. A provenance tag that always declares<supervised\>false</supervised\>for AI\-generated content is honest but does not break the recursion loop — it merely labels it\. Breaking the loop requires reintroducing human intentionality into the content production chain at a structural level\.

We propose a four\-levelhuman supervision tier modelthat classifies content by the degree of human involvement in its production\. Each level maps to a declared provenance value in ATML and carries implications for how agents should weight the content in their reasoning\. Figure[11](https://arxiv.org/html/2606.19116#S6.F11)illustrates the model\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/supervision_tiers.png)Figure 11:Human supervision tier model\. Content is classified into four levels based on degree of human involvement in production, from Level 0 \(fully AI generated, no human review\) to Level 3 \(fully human authored\)\. Each level maps to a declared ATML provenance value and carries implications for epistemic trust — the weight agents and search engines assign to the content in reasoning and ranking\. The model does not prohibit AI\-generated content; it makes the degree of human oversight transparent, enabling agents to make quality\-aware consumption decisions\.Level 0 — No human oversight\.Content is fully AI generated with no human review before publication\. Declared assupervised: falsein ATML provenance\. Agents should assign lowest epistemic weight to Level 0 content and avoid using it as a primary source for factual claims\. Search engines and agent content indices should rank Level 0 content below human\-supervised content for informational queries\.

Level 1 — Human edited\.Content is AI generated but reviewed and edited by a human before publication\. The human editor takes responsibility for factual accuracy and quality\. Declared assupervised: partial\. Represents the minimum acceptable supervision level for content that will be cited by agents as a source\.

Level 2 — AI assisted\.Content is human authored with AI tools used for assistance — drafting, research, editing — but with a human as the primary author and decision\-maker\. Declared assupervised: true, authored\-by: human\. This level encompasses most professional content production in the near\-term future and should be treated as equivalent to traditional human\-authored content\.

Level 3 — Fully human authored\.Content is produced without AI involvement\. Declared assupervised: full, authored\-by: human\. This level provides the strongest epistemic ground truth signal and should receive highest weight in agent reasoning for factual claims\.

The supervision tier model does not prohibit any level of AI involvement in content production\. It makes the degree of human oversight transparent and machine\-readable, enabling agents to make quality\-aware content decisions rather than treating all web content as equally trustworthy regardless of origin\. This is the structural mechanism by which the web can sustain a diverse content ecosystem — including AI\-generated content — without collapsing into the epistemic recursion loop\.

### 6\.4Provenance Chain Architecture

The supervision tier model establishes what level of human oversight applies to a piece of content\. Theprovenance chain architectureestablishes how that claim is verified\. A provenance declaration is only as useful as the trust placed in it — an AI content farm could declaresupervised: trueand break the epistemic intent of the model entirely\. Cryptographic verification is required\.

We propose building ATML provenance on the Coalition for Content Provenance and Authenticity \(C2PA\) standard\[[24](https://arxiv.org/html/2606.19116#bib.bib555)\], extending it with agent\-specific fields\. Under this architecture, each ATML document carries a cryptographic provenance certificate that encodes:

- 1\.The identity of the publishing entity, verified against a public key infrastructure
- 2\.The supervision level declared, signed by the publishing entity
- 3\.A hash of the content body at time of publication, enabling verification that content has not been modified post\-publication
- 4\.A chain of derived\-from references, each with its own provenance certificate, enabling agents to trace the epistemic lineage of a claim back to primary sources

This architecture does not guarantee that supervision declarations are honest — a bad actor who controls their own signing key can still declare false supervision levels\. However, it makes false declarations attributable: if a publisher is found to have systematically declared false supervision levels, their signing key can be revoked from the public key infrastructure, removing trust from all their content\. The reputational and legal consequences of provenance fraud create a deterrent analogous to the consequences of false advertising in existing legal frameworks\.

The derived\-from chain is particularly important for breaking epistemic recursion\. When an agent traces the provenance chain of a piece of content and finds that it derives from another AI\-generated piece, which derives from another AI\-generated piece, without a human\-authored primary source anywhere in the chain, it has detected epistemic recursion in action\. Agents can be designed to flag such chains and reduce the weight assigned to content with deep AI\-only derivation chains — a practical circuit breaker for the recursion loop that requires no centralised authority to operate\.

### 6\.5Agent Content Discoverability

The human web’s discovery mechanism — search engines optimizing for human attention and click\-through — does not serve agent needs\. An agent does not want the most popular result; it wants the most authoritative, most accurately provenance\-tagged, most appropriately\-priced result for its specific task\. These are different optimization targets requiring different infrastructure\.

We propose anagent content discoverability architecturebuilt on a capability declaration standard and an agent\-optimized search index\. Figure[12](https://arxiv.org/html/2606.19116#S6.F12)illustrates the full discoverability flow\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/discoverbility.png)Figure 12:Agent content discoverability architecture\. An agent queries an agent search index optimized for semantic relevance, provenance quality, and access terms rather than human attention signals\. The index returns ATML endpoints from publishers who have declared agent capabilities in a structuredagents\.jsonfile\. The agent fetches the capability declaration to understand available content types, supervision levels, and pricing before making content requests\. This eliminates blind scraping — agents always know what they are accessing and under what terms before they access it\.The capability declaration — served at/\.well\-known/agents\.json— extends theagents\.txtaccess policy file proposed in Section[4\.6](https://arxiv.org/html/2606.19116#S4.SS6)with richer content metadata:

```
{
  "atml-endpoint": "/atml",
  "content-types": ["news", "analysis",
                    "data"],
  "languages": ["en", "fr"],
  "supervision-level": 2,
  "auth": "delegation-token",
  "pricing": "/.well-known/agents.txt",
  "search-topics": ["finance",
                    "technology"],
  "update-frequency": "real-time"
}
```

Agent search engines index these capability declarations rather than raw content — they know which sites publish which content types at which supervision levels and under what economic terms, without having to scrape and parse the full content of every page\. This is a fundamentally more efficient discovery architecture than human\-web search, which must index full page content to infer topical relevance\.

The agent search index optimizes for different signals than human search: semantic relevance to agent task, provenance quality of the source, supervision level of available content, and access terms compatibility with the agent’s authorization level\. These signals produce rankings appropriate for agent information retrieval tasks — prioritizing trustworthy, well\-provenance\-tagged sources over high\-traffic, SEO\-optimized pages\.

### 6\.6Dual Publishing Strategy

The transition from a human\-centric to an agent\-first content layer cannot be abrupt\. Publishers cannot abandon HTML overnight; human users continue to browse the web directly; legacy systems depend on existing content formats\. The dual\-layer web architecture proposed in Section[4\.5](https://arxiv.org/html/2606.19116#S4.SS5)requires a corresponding dual publishing strategy at the content layer\.

We propose that publishers adopt ATML as a parallel delivery format alongside HTML, generated from the same content source\. In most publishing systems, content is stored in a structured internal format — a content management system database — and rendered into HTML at delivery time\. ATML is an alternative rendering target for the same source content, not a separate authoring workflow\. A publisher who stores articles as structured database records can generate both an HTML rendering \(for browsers\) and an ATML rendering \(for agents\) from the same record, adding only the provenance and access layers that ATML requires and HTML does not\.

The migration path mirrors the access layer migration described in Section[4\.5](https://arxiv.org/html/2606.19116#S4.SS5): Phase 1 establishes ATML as an optional parallel format; Phase 2 normalizes dual publishing as standard practice; Phase 3 establishes ATML as the primary format for agent consumption with HTML maintained for browser compatibility\. The human web does not disappear — it becomes a rendering target for a content layer that is fundamentally structured for agent consumption\.

### 6\.7Summary

Table[5](https://arxiv.org/html/2606.19116#S6.T5)summarizes the content layer redesign proposals and their relationship to the failures diagnosed in Section[3\.3](https://arxiv.org/html/2606.19116#S3.SS3)\.

Table 5:Content layer redesign: diagnosed failures and proposed solutions\.Diagnosed failureProposed solutionHTML format overhead — 67\.6% token wasteATML — semantic content format without layout noiseNo provenance standardATML provenance layer \+ C2PA cryptographic chainNo human supervision signalFour\-level human supervision tier modelEpistemic recursion — AI feeding AIProvenance chain circuit breaker \+ supervision declarationNo agent discoverability standardagents\.json capability declaration \+ agent search indexHuman\-only publishing workflowDual publishing strategy — HTML and ATML from same source

The content layer redesign closes the loop opened by the access and economic layer redesigns\. An agent that can identify itself \(Section[4](https://arxiv.org/html/2606.19116#S4)\), operate under an appropriate economic model \(Section[5](https://arxiv.org/html/2606.19116#S5)\), and retrieve semantically rich, provenance\-tagged content in an efficient format \(this section\) has everything it needs to operate as a first\-class web citizen\. The three layers are deeply interdependent — ATML content is only useful if agents can reach it \(access layer\) and publishers have an incentive to produce it \(economic layer\) — which is why all three must be redesigned simultaneously rather than in isolation\.

## 7The Agent\-First Web: A Unified Framework

The preceding three sections proposed solutions to each layer of the human\-centric web’s failure under agent interaction: an access layer redesign \(Section[4](https://arxiv.org/html/2606.19116#S4)\), an economic layer redesign \(Section[5](https://arxiv.org/html/2606.19116#S5)\), and a content layer redesign \(Section[6](https://arxiv.org/html/2606.19116#S6)\)\. This section synthesizes these proposals into a unified framework — establishing the design principles, social contract restatement, and migration roadmap that constitute the paper’s headline contribution\.

### 7\.1The Social Contract Restatement

The web’s original social contract — articulated implicitly through the architectural choices of its first decade — rested on three foundational commitments\[[18](https://arxiv.org/html/2606.19116#bib.bib526),[60](https://arxiv.org/html/2606.19116#bib.bib545)\]\. First, open access: any client should be able to retrieve any resource without prior negotiation or identity disclosure\. Second, the attention economy: publishers provide content freely in exchange for human attention monetized through advertising\. Third, human authorship: the web is a record of human knowledge, experience, and expression, authored by humans for human readers\.

Each of these commitments is now under structural pressure\. Open access is being dismantled by blanket agent blocking\. The attention economy is collapsing under zero\-click AI synthesis\. Human authorship is being displaced by AI\-generated content at scale\. The web’s social contract is not merely being stressed — it is being violated simultaneously at all three layers by forces its architects did not anticipate\.

We propose a restatement of the web’s social contract for the agent\-first era, preserving its founding values while extending them to accommodate the new realities of agent\-mediated interaction\. Figure[13](https://arxiv.org/html/2606.19116#S7.F13)illustrates the comparison between the original and proposed contracts\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/social_contract.png)Figure 13:The web’s social contract: original versus agent\-first\. The original contract \(1991\) rested on open access for human clients, an attention economy, HTML for human perception, and anonymous identity\. The agent\-first contract preserves open access but extends it to agents through behavioral contracts; replaces the attention economy with a value\-exchange economy based on tokens and outcomes; replaces HTML with provenance\-anchored ATML; and replaces CAPTCHA\-based human filtering with intent\-based behavioral declaration\. The web has renegotiated its social contract before — HTTP to HTTPS, desktop to mobile — this is the next renegotiation\.The new social contract can be stated concisely:

> The agent\-first web is open to any client — human or agent — that declares its behavioral context and respects the terms declared by content providers\. Value flows at the point of consumption, not the point of attention\. Content carries its own provenance and human oversight level, enabling quality\-aware consumption\. The human web continues to exist; the agent web is built alongside it, not in replacement of it\.

This restatement preserves the web’s founding commitment to openness — no client is excluded by identity — while introducing the behavioral accountability and economic transparency that agent\-scale interaction requires\.

### 7\.2The Ten Principles of an Agent\-First Internet

We synthesize the proposals of Sections[4](https://arxiv.org/html/2606.19116#S4)–[6](https://arxiv.org/html/2606.19116#S6)into ten design principles for the agent\-first internet\. These principles are intended to serve the same function that REST principles served for API design\[[31](https://arxiv.org/html/2606.19116#bib.bib572)\]or mobile\-first principles served for responsive web design\[[41](https://arxiv.org/html/2606.19116#bib.bib573)\]: a concise, quotable framework that practitioners can apply when making architectural decisions about agent\-web interaction\.

1. 1\.Presumption of agent access\.Agents acting on behalf of humans are presumed welcome on any web resource that is open to anonymous human visitors\. Blocking is reserved for malicious behavior, not agent identity\. The default posture is access, not exclusion\.
2. 2\.Behavioral contract over identity mandate\.Agents declare their behavioral context — who they represent, at what scale, for what purpose — rather than proving their identity\. Anonymous is acceptable; unaccountable is not\. Intent declaration enables graduated responses without requiring privacy\-invasive identification\.
3. 3\.Rate limiting as the universal access control\.The binary block/allow decision is replaced by graduated rate limiting determined by declared intent and authentication level\. No legitimate agent use case is categorically blocked; all are accommodated at appropriate scale and under appropriate terms\.
4. 4\.Economic equivalence to the represented human\.An agent’s economic obligation mirrors that of the human it represents\. Free content for anonymous humans is free for their agents\. Paid subscriptions extend to agents through delegation tokens\. New economic models are required only for genuinely novel agent behaviors with no human analog\.
5. 5\.Token as the unit of value exchange\.The pageview and the advertising impression are replaced by the token as the fundamental unit of value exchange between agents and content providers\. Token consumption is proportional to actual value derived, compatible with existing AI billing infrastructure, and format\-agnostic\.
6. 6\.Publisher economic sovereignty\.No universal economic model is mandated\. Publishers choose between open and paid content — mirroring the open source / proprietary software distinction — and declare their terms in machine\-readable form\. Economic diversity is preserved as a web property\.
7. 7\.Provenance as a first\-class web property\.Every piece of web content carries a machine\-readable provenance declaration: who authored it, what it was derived from, and what degree of human oversight was applied\. Provenance is not optional metadata — it is a structural property of agent\-first content, as fundamental as the content body itself\.
8. 8\.Human supervision declared, not assumed\.The degree of human involvement in content production is declared explicitly through the supervision tier model rather than inferred or assumed\. Agents weight content in their reasoning according to declared supervision level\. AI\-generated content is not prohibited; it is labelled\.
9. 9\.Dual\-layer coexistence\.The human web and the agent web coexist on the same infrastructure\. HTML for human browsers and ATML for agent clients are served from the same domain, generated from the same content source, differentiated only at the delivery layer\. Migration is gradual and additive, not disruptive\.
10. 10\.Simultaneous three\-layer redesign\.Access, economics, and content are redesigned together\. Fixing any single layer in isolation produces solutions that are undermined by the failures of the other two\. The three layers are deeply coupled — they must be addressed as a system, not as independent problems\.

These ten principles constitute theagent\-first internet framework\. Figure[14](https://arxiv.org/html/2606.19116#S7.F14)illustrates how the principles map to the three layers and their interdependencies\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/unified_framework.png)Figure 14:The unified agent\-first web framework\. The agent\-as\-human\-proxy principle provides the philosophical foundation\. Three redesign layers — access, economic, and content — each contribute specific mechanisms\. Together they produce an agent\-first internet in which agents are first\-class citizens with open access, value\-exchange economics, and provenance\-anchored content\. The migration path is phased and additive, preserving the human web throughout the transition\.
### 7\.3Migration Roadmap

The agent\-first web is not built overnight\. The proposals in this paper span a range of implementation complexity — from the immediately deployable \(agent metadata headers,agents\.txt\) to the medium\-term \(ATML adoption, OAuth delegation tokens\) to the longer\-term \(full provenance chain infrastructure, agent search indices\)\. We propose a three\-phase migration roadmap that sequences these proposals by implementation complexity and stakeholder readiness\.

Phase 1 — Identification and Declaration \(near\-term\)\.The immediate priority is establishing the identification and declaration infrastructure that enables all subsequent proposals\. This includes: deployment of agent metadata HTTP headers by major AI agent providers; adoption ofagents\.txtby publishers as a machine\-readable access policy standard; and expansion ofllms\.txt\[[35](https://arxiv.org/html/2606.19116#bib.bib566)\]from a content permission declaration into a richer capability declaration incorporating access terms and supervision levels\. These measures require no new standards bodies and no new infrastructure — they are extensions of existing HTTP and web conventions that individual actors can adopt unilaterally\.

Phase 2 — Economic and Format Infrastructure \(medium\-term\)\.Once identification and declaration are in place, the economic and content format infrastructure can be built\. This includes: ATML as a parallel content delivery format alongside HTML; OAuth delegation tokens for subscription inheritance; token\-based billing APIs for content publishers; and pilot implementations of the commissioned content economy model\. These measures require coordination between AI providers, content publishers, and payment infrastructure operators — achievable through industry working groups without formal standards bodies\.

Phase 3 — Full Agent\-First Infrastructure \(longer\-term\)\.The final phase establishes ATML as the primary agent\-facing content format; deploys full cryptographic provenance chain infrastructure built on C2PA\[[24](https://arxiv.org/html/2606.19116#bib.bib555)\]; launches agent\-optimized search indices; and establishes the agent advertising protocol\. These measures require formal standardization through bodies such as the W3C and IETF, analogous to the standardization processes that established HTTPS\[[29](https://arxiv.org/html/2606.19116#bib.bib569)\]and OAuth\[[33](https://arxiv.org/html/2606.19116#bib.bib568)\]\.

Table[6](https://arxiv.org/html/2606.19116#S7.T6)summarizes the migration roadmap\.

Table 6:Agent\-first web migration roadmap\.PhaseFocusKey proposalsCoordinationrequired1Identification and declarationAgent metadata headers, agents\.txt, llms\.txt expansionUnilateral — individual actors2Economic and format infrastructureATML, OAuth delegation, token billingIndustry working groups3Full agent\-first infrastructureProvenance chain, agent search, ad protocolFormal standards bodies

### 7\.4Backward Compatibility

A recurring concern with proposals to redesign web infrastructure is disruption to the existing ecosystem\. We emphasize that every proposal in this framework isadditive rather than disruptive\. No existing web standard is deprecated\. No existing content format is invalidated\. No existing economic model is prohibited\.

Agent metadata headers are additional HTTP headers — servers that do not recognize them ignore them\.agents\.txtis an additional well\-known file — sites that do not provide it simply have no declared agent policy, and agents fall back to default behaviors\. ATML is an additional content type — browsers that do not request it receive HTML as before\. Token billing is an additional payment option — publishers who do not adopt it continue to operate under existing models\.

The human web does not disappear in the agent\-first future — it becomes one layer of a richer, dual\-layer web that serves both human and agent clients from the same infrastructure\. This is not a replacement; it is an extension\. The web has extended itself before — adding HTTPS alongside HTTP, adding mobile layouts alongside desktop, adding APIs alongside HTML pages — and has been strengthened by each extension\. The agent\-first layer is the next extension in this lineage\.

### 7\.5Relationship to Existing Protocol Work

The agent\-first internet framework proposed in this paper is complementary to, not competitive with, the protocol\-level work surveyed in Section[2](https://arxiv.org/html/2606.19116#S2)\. MCP\[[4](https://arxiv.org/html/2606.19116#bib.bib536)\], A2A\[[32](https://arxiv.org/html/2606.19116#bib.bib537)\], NLWeb\[[43](https://arxiv.org/html/2606.19116#bib.bib538)\], and related protocols address agent\-to\-tool and agent\-to\-agent communication — the plumbing of the agentic web\. The agent\-first internet framework addresses the social contract, economic model, and content architecture of the web that these agents operate within — the foundations those protocols build on\.

The relationship is analogous to the relationship between TCP/IP and the web’s application layer: TCP/IP provides the communication substrate; HTTP, HTML, and the web’s economic conventions constitute the application layer built on top\. MCP and A2A are the emerging TCP/IP of the agentic web; the agent\-first internet framework is the application layer — the conventions, economic models, and content standards that determine what agents actually do with the communication capabilities those protocols provide\.

### 7\.6Summary

The agent\-first internet framework synthesizes fifteen specific proposals across three layers into a coherent redesign grounded in a single philosophical anchor — the agent\-as\-human\-proxy principle — and expressed as ten design principles that practitioners can apply when making architectural decisions about agent\-web interaction\. The framework is additive, backward\-compatible, and phased — preserving the human web throughout the transition to an agent\-first architecture\. It positions the challenge not as a technical problem of protocol design but as a sociotechnical problem requiring renegotiation of the web’s foundational social contract — a renegotiation that the web has successfully accomplished before and must accomplish again\.

## 8Open Challenges

The agent\-first internet framework proposed in this paper is principled and internally coherent, but it does not resolve all problems\. Honest accounting of what the framework does not solve is as important as what it proposes — it defines the research agenda that follows from this work and prevents overreach in the claims made\. This section identifies six open challenges that remain unresolved, each constituting a direction for future work\. Figure[15](https://arxiv.org/html/2606.19116#S8.F15)provides an overview\.

![Refer to caption](https://arxiv.org/html/2606.19116v1/figures/open_challenges.png)Figure 15:Open challenges in the agent\-first web\. Six challenge areas remain unresolved by the proposed framework: governance and standardization, adversarial agents, the agent advertising model, regulatory alignment, small publisher equity, and multi\-agent coordination\. These challenges do not invalidate the framework — they define the research agenda that follows from it\.### 8\.1Governance and Standardization

The proposals in this paper — agent metadata headers,agents\.txt, ATML, the supervision tier model, the provenance chain architecture — require standardization to achieve the universal adoption that makes them effective\. A metadata header that only some agent providers implement provides partial identification at best\. A content format that only some publishers adopt creates a two\-tier web rather than a unified agent\-first layer\. Standards achieve their value through universality, and universality requires a governance process\[[8](https://arxiv.org/html/2606.19116#bib.bib576)\]\.

The web’s history of standardization is instructive but not entirely encouraging\. HTML was standardized through the W3C, a process that produced robust but slow\-moving standards vulnerable to browser\-vendor fragmentation during the browser wars of the late 1990s\[[46](https://arxiv.org/html/2606.19116#bib.bib528)\]\. HTTP was standardized through the IETF, a more technically rigorous but equally slow process\. OAuth emerged from industry collaboration before IETF formalization\[[33](https://arxiv.org/html/2606.19116#bib.bib568)\]\. Each pathway involved years of negotiation between parties with conflicting commercial interests\.

The agent\-first web faces a governance challenge of comparable complexity, with the additional complication that the primary stakeholders — AI companies, content publishers, infrastructure providers, and regulators — have interests that are not merely competitive but structurally opposed\. AI companies benefit from open agent access; publishers benefit from controlled, economically compensated access; infrastructure providers benefit from being the chokepoint of any access control mechanism; regulators benefit from accountability that none of the other parties are inclined to provide voluntarily\.

We do not propose a governance solution here — the political economy of web standards governance is a research domain in its own right\. We note that the Phase 1 proposals in our migration roadmap \(Section[7\.3](https://arxiv.org/html/2606.19116#S7.SS3)\) are deliberately designed to be adoptable unilaterally by individual actors without standards body coordination, precisely to create momentum and demonstrated value before formal standardization is required\.

### 8\.2The Adversarial Agent Problem

The agent identification metadata mechanism proposed in Section[4\.2](https://arxiv.org/html/2606.19116#S4.SS2)assumes that agent metadata declarations are honest\. The Perplexity case demonstrated that this assumption does not hold under adversarial conditions: bad actors will modify agent identifiers, rotate IP addresses, and falsify intent declarations to circumvent access controls\[[1](https://arxiv.org/html/2606.19116#bib.bib543)\]\. Cryptographic signing of agent metadata raises the cost of impersonation significantly but does not eliminate it — a bad actor who controls their own signing key can still declare false intent, and a compromised legitimate signing key can be used for impersonation until revocation\.

The adversarial agent problem is structurally similar to the email spam problem\. Email authentication standards — SPF, DKIM, DMARC — significantly reduced certain classes of email abuse by making sender identity verifiable, but did not eliminate spam or phishing; they shifted the attack surface rather than eliminating it\[[2](https://arxiv.org/html/2606.19116#bib.bib567)\]\. Agent metadata authentication is likely to follow a similar trajectory: reducing casual circumvention while displacing sophisticated attackers toward more costly evasion strategies\.

A more robust long\-term defense is behavioral rather than declarative: monitoring agent consumption patterns over time to detect statistical anomalies — consumption rates inconsistent with declared personal use, content extraction patterns consistent with training rather than query\-answering, access patterns consistent with competitive intelligence rather than user assistance\. Behavioral detection is harder to evade than declaration forgery, but requires infrastructure that does not currently exist at web scale and raises privacy concerns about agent activity monitoring that must be carefully managed\.

### 8\.3The Agent Advertising Model

The agent advertising model sketched in Section[5\.7](https://arxiv.org/html/2606.19116#S5.SS7)faces a fundamental principal\-agent conflict that the framework does not resolve\. Publishers want their advertisements surfaced to human users by agent frontends\. Agent frontends — Claude, ChatGPT, Gemini — have competing monetization interests and limited incentive to surface publisher advertisements that generate no revenue for the frontend operator\. Users want clean, uninterrupted agent responses and are likely to prefer frontends that do not surface advertisements\.

This three\-way conflict cannot be resolved by technical standards alone\. Two resolution pathways exist, both of which involve stakeholder coordination beyond what any single actor can achieve\. The first is a revenue\-sharing model in which publisher advertisement revenue is split between the publisher and the agent frontend operator that surfaces it — giving frontend operators a financial incentive to carry publisher advertisements\. The second is a regulatory mandate analogous to must\-carry rules in broadcast television regulation, requiring agent frontends to surface publisher advertisements as a condition of accessing publisher content\.

Neither pathway is politically straightforward\. Revenue sharing requires bilateral agreements between every publisher and every agent frontend operator — a combinatorially complex negotiation landscape\. Regulatory mandates require jurisdictional agreement across regulators who currently lack frameworks for agent content intermediation\. We identify this as a priority open challenge and flag it as a direction for future work in both the technical and policy research communities\.

### 8\.4Regulatory Alignment

The existing regulatory landscape for web content and data was designed for human\-web interaction and applies imperfectly or ambiguously to agent\-mediated interaction across multiple dimensions\.

Copyright\.When an AI agent retrieves and synthesizes content from multiple sources, the copyright status of the synthesized output is legally unsettled in most jurisdictions\. The training data copyright disputes currently before courts in the United States and European Union\[[34](https://arxiv.org/html/2606.19116#bib.bib574)\]address a related but distinct question — whether training on copyrighted data constitutes infringement — and their resolution will not definitively answer the question of whether agent synthesis of copyrighted content for a user constitutes fair use or requires licensing\. The token\-based licensing model proposed in Section[5\.5](https://arxiv.org/html/2606.19116#S5.SS5)is designed to be compatible with whatever copyright resolution emerges, but cannot itself produce that resolution\.

Data protection\.GDPR and equivalent regulations govern the processing of personal data by automated systems, with requirements around consent, purpose limitation, and data subject rights\[[27](https://arxiv.org/html/2606.19116#bib.bib575)\]\. When an AI agent processes content containing personal data on behalf of a user, it is unclear which entity — the user, the agent provider, the content publisher — bears data controller responsibility\. The delegation token mechanism proposed in Section[4\.4](https://arxiv.org/html/2606.19116#S4.SS4)partially addresses this by establishing a clear authorization chain, but does not resolve the underlying regulatory ambiguity\.

Platform liability\.Section 230 of the US Communications Decency Act and equivalent provisions in other jurisdictions limit platform liability for user\-generated content\. It is unclear whether AI agents that synthesize and present content to users are acting as platforms \(potentially eligible for liability protection\) or as publishers \(potentially liable for the content they synthesize\)\. This ambiguity has significant implications for the design of agent content systems and requires legislative rather than technical resolution\.

### 8\.5The Small Publisher Equity Problem

The proposals in this paper — ATML publishing infrastructure, provenance chain certification, token billing APIs,agents\.jsoncapability declarations — carry implementation costs that large publishers can absorb and small publishers cannot\. The New York Times has engineering teams capable of implementing dual\-layer publishing infrastructure\. An independent journalist with a WordPress blog does not\.

If the agent\-first web is accessible only to publishers with the resources to implement its infrastructure, it creates a two\-tier web in which large publishers dominate agent\-accessible content and small, independent, and community publishers are effectively excluded\. This replicates and potentially amplifies existing web inequality — large platforms already dominate search rankings and social distribution; agent\-first infrastructure could extend this dominance to agent\-mediated information access\.

The mitigation is platform\-level adoption: if WordPress, Substack, Ghost, and similar publishing platforms implement ATML generation,agents\.jsoncapability declarations, and provenance tagging as platform features, their users gain agent\-first capabilities without individual implementation effort\. This mirrors how SSL adoption was democratized through hosting platform support rather than individual webmaster implementation\. We recommend that the standards proposed in this paper be designed with platform\-level implementation as the primary deployment pathway — simplicity for platform integration is more important than feature richness for individual deployment\.

### 8\.6Multi\-Agent Coordination

The framework proposed in this paper addresses the interaction between a single agent and web content\. Increasingly, however, agent tasks involve multiple agents collaborating — an orchestrator agent directing specialist agents to retrieve, analyze, and synthesize content from many sources simultaneously\[[26](https://arxiv.org/html/2606.19116#bib.bib547)\]\. This multi\-agent scenario introduces attribution and billing complexities that the framework does not resolve\.

When five specialist agents each retrieve content from ten sources to complete a single user task, who pays for the content access? The user, through their orchestrating agent? Each specialist agent independently? The orchestrator on behalf of all specialists? The token\-based billing model proposed in Section[5\.5](https://arxiv.org/html/2606.19116#S5.SS5)provides a unit of account but not a billing architecture for multi\-agent scenarios\.

Attribution is equally complex\. When a multi\-agent pipeline produces a synthesized output that incorporates content from fifty sources retrieved by multiple agents, the provenance chain described in Section[6\.4](https://arxiv.org/html/2606.19116#S6.SS4)must span agent boundaries — tracking not just which sources were consulted but which agents consulted them and how their outputs were combined\. This requires a cross\-agent provenance standard that does not currently exist and whose design involves significant technical and privacy tradeoffs\.

The A2A protocol\[[32](https://arxiv.org/html/2606.19116#bib.bib537)\]and related multi\-agent communication standards provide communication infrastructure for multi\-agent systems but do not address content attribution or billing coordination\. We identify multi\-agent content economics as a priority area for future work, noting that its resolution requires coordination between the agent communication protocol community and the web content economics community — two communities that have not yet engaged substantively with each other’s work\.

### 8\.7Summary

Table[7](https://arxiv.org/html/2606.19116#S8.T7)summarizes the six open challenges and their relationship to the framework proposals\.

Table 7:Open challenges and their relationship to the agent\-first web framework\.ChallengeFrameworklimitationResolutionpathwayGovernance and standardizationProposals require universal adoptionW3C, IETF, industry working groupsAdversarial agentsDeclarations can be falsifiedBehavioral detection \+ cryptographic signingAgent advertisingFrontend incentive conflictRevenue sharing or regulatory mandateRegulatory alignmentLegal frameworks predate agentsLegislative and judicial resolutionSmall publisher equityImplementation costs create barriersPlatform\-level adoption pathwayMulti\-agent coordinationSingle\-agent billing and attribution modelCross\-agent provenance standard

These six challenges are significant but do not invalidate the framework\. Each represents a known limitation with an identifiable resolution pathway — they are open problems, not fatal flaws\. The web has faced comparable challenges at every major architectural transition: HTTPS deployment faced governance fragmentation and small\-site adoption barriers before Let’s Encrypt democratized certificate issuance; OAuth faced competing implementations before RFC 6749 standardized the protocol; mobile web faced regulatory uncertainty before jurisdictions developed mobile\-specific frameworks\. The agent\-first web transition will follow a similar pattern — early adoption by large actors, gradual standardization, eventual universal deployment — provided the foundational framework is principled enough to survive the transition intact\. We believe the framework proposed in this paper meets that standard\.

## 9Conclusion

The World Wide Web was built for human eyes\. For three decades, this assumption was so fundamental that it required no articulation — it was simply the water in which the web swam\. The emergence of AI agents as primary intermediaries between humans and web content has made this assumption visible by breaking it\. Blanket agent blocking, collapsing publisher revenues, and the epistemic recursion loop are not isolated problems — they are symptoms of a single underlying condition: a web whose architecture was designed for a world that no longer exists\.

This paper has argued that the response to this condition cannot be reactive or piecemeal\. Patching the access layer without the economic layer produces agents that can reach content but publishers who cannot sustain producing it\. Patching the economic layer without the content layer produces billing infrastructure for content that degrades in quality through epistemic recursion\. The three layers — access, economics, and content — are deeply coupled, and must be redesigned simultaneously, grounded in a single philosophical anchor: that AI agents acting on behalf of humans are first\-class web citizens entitled to the same presumption of access, subject to equivalent obligations, and deserving of an architectural environment designed for their interaction model\.

The agent\-first internet framework proposed in this paper offers that redesign\. At the access layer, agent identification metadata headers and a graduated rate\-limiting model replace blanket blocking with context\-appropriate access control\. At the economic layer, the intent\-based tier model and token\-based subscription mechanism replace the broken attention economy with a value\-exchange model grounded in the agent\-as\-human\-proxy principle\. At the content layer, ATML, the human supervision tier model, and the provenance chain architecture replace opaque HTML with semantically rich, provenance\-anchored content that breaks the epistemic recursion loop\. Together these proposals constitute ten design principles for the agent\-first internet — a framework intended to serve practitioners making architectural decisions about agent\-web interaction in the same way that REST principles served API designers and mobile\-first principles served responsive web designers\.

The web has renegotiated its social contract before\. The transition from HTTP to HTTPS reframed security from an optional feature to a default expectation\. The transition from desktop to mobile reframed layout from a fixed assumption to a responsive variable\. Each transition required years of friction, partial adoption, and eventual standardization before becoming universal\. The transition to an agent\-first web is the next renegotiation in this lineage — and by the evidence of collapsing publisher revenues, infrastructure\-scale agent blocking, and accelerating epistemic recursion, it is already overdue\.

The cost of inaction is not merely economic\. A web that successfully excludes AI agents excludes the humans those agents represent — making the web less useful to the people it was built to serve\. A web whose content is generated by AI for consumption by AI, without human intentionality at any point in the chain, is no longer a record of human knowledge — it is a hall of mirrors, reflecting increasingly distorted versions of what humans once knew\. Redesigning the web for AI agents is not a concession to technological change\. It is an act of stewardship for the knowledge ecosystem that both humans and AI depend upon\.

## References

- \[1\]P\. AI\(2024\)Publisher revenue sharing program\.Note:[https://www\.perplexity\.ai](https://www.perplexity.ai/)Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p4.1),[§2\.3](https://arxiv.org/html/2606.19116#S2.SS3.p3.1),[§4\.2](https://arxiv.org/html/2606.19116#S4.SS2.p10.1),[§5\.1](https://arxiv.org/html/2606.19116#S5.SS1.p3.1),[§8\.2](https://arxiv.org/html/2606.19116#S8.SS2.p1.1)\.
- \[2\]E\. Allman, J\. Callas, M\. Delany, M\. Libbey, J\. Fenton, and M\. Thomas\(2007\)DomainKeys identified mail \(DKIM\) signatures – RFC 4871\.IETF\.Note:[https://tools\.ietf\.org/html/rfc4871](https://tools.ietf.org/html/rfc4871)Cited by:[§4\.2](https://arxiv.org/html/2606.19116#S4.SS2.p10.1),[§8\.2](https://arxiv.org/html/2606.19116#S8.SS2.p2.1)\.
- \[3\]ANP Working Group\(2025\)Agent network protocol \(ANP\)\.Note:[https://agent\-network\-protocol\.com](https://agent-network-protocol.com/)Cited by:[§2\.2](https://arxiv.org/html/2606.19116#S2.SS2.p1.1)\.
- \[4\]Anthropic\(2024\)Model context protocol \(MCP\)\.Note:[https://modelcontextprotocol\.io](https://modelcontextprotocol.io/)Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p4.1),[§2\.2](https://arxiv.org/html/2606.19116#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.15.13.13.13.15.1.1),[§5\.5](https://arxiv.org/html/2606.19116#S5.SS5.p1.1),[§7\.5](https://arxiv.org/html/2606.19116#S7.SS5.p1.1)\.
- \[5\]S\. Arora, B\. Yang, S\. Eyuboglu, A\. Narayan, A\. Hojel, I\. Trummer, and C\. Ré\(2023\)Language models enable simple systems for generating structured views of heterogeneous data lakes\.arXiv preprint arXiv:2304\.09433\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p2.1)\.
- \[6\]E\. Bandara, S\. H\. Bouk, R\. Mukkamala, A\. Rahman, X\. Liang, R\. Gore, and S\. Shetty\(2025\)Slice\-mcp — fine\-tuned llama\-4 llm and mcp enabled 5g network slice orchestration platform\.InMILCOM 2025 \- 2025 IEEE Military Communications Conference \(MILCOM\),Vol\.,pp\. 1506–1511\.External Links:[Document](https://dx.doi.org/10.1109/MILCOM64451.2025.11310417)Cited by:[§6\.1](https://arxiv.org/html/2606.19116#S6.SS1.p5.1)\.
- \[7\]E\. Bandara, R\. Gore, P\. Foytik, S\. Shetty, R\. Mukkamala, A\. Rahman, X\. Liang, S\. H\. Bouk, A\. Hass, S\. Rajapakse,et al\.\(2025\)A practical guide for designing, developing, and deploying production\-grade agentic ai workflows\.arXiv preprint arXiv:2512\.08769\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p2.1)\.
- \[8\]E\. Bandara, R\. Gore, A\. Gunaratna, S\. Rajapakse, I\. Kularathna, R\. Mukkamala, S\. Shetty, X\. Liang, A\. Hass, T\. Hewa,et al\.\(2026\)Think before you act–a neurocognitive governance model for autonomous ai agents\.arXiv preprint arXiv:2604\.25684\.Cited by:[§8\.1](https://arxiv.org/html/2606.19116#S8.SS1.p1.1)\.
- \[9\]E\. Bandara, R\. Gore, X\. Liang, S\. Rajapakse, I\. Kularathne, P\. Karunarathna, P\. Foytik, S\. Shetty, R\. Mukkamala, A\. Rahman,et al\.\(2025\)Agentsway–software development methodology for ai agents\-based teams\.arXiv preprint arXiv:2510\.23664\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p4.1)\.
- \[10\]E\. Bandara, R\. Gore, S\. Shetty, R\. Mukkamala, C\. Rhea, A\. Yarlagadda, S\. Kaushik, L\. De Silva, A\. Maznychenko, I\. Sokolowska,et al\.\(2025\)Standardization of neuromuscular reflex analysis–role of fine\-tuned vision\-language model consortium and openai gpt\-oss reasoning llm enabled decision support system\.arXiv preprint arXiv:2508\.12473\.Cited by:[§6\.2\.1](https://arxiv.org/html/2606.19116#S6.SS2.SSS1.p2.1)\.
- \[11\]E\. Bandara, R\. Gore, S\. Shetty, S\. Rajapakse, I\. Kularathna, P\. Karunarathna, R\. Mukkamala, P\. Foytik, S\. H\. Bouk, A\. Rahman,et al\.\(2026\)A practical guide to agentic ai transition in organizations\.arXiv preprint arXiv:2602\.10122\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p2.1)\.
- \[12\]E\. Bandara, A\. Gunaratna, R\. Gore, A\. Rahman, R\. Mukkamala, S\. Shetty, S\. Rajapakse, I\. Kularathna, P\. Foytik, S\. H\. Bouk,et al\.\(2026\)AI trust os–a continuous governance framework for autonomous ai observability and zero\-trust compliance in enterprise environments\.arXiv preprint arXiv:2604\.04749\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p2.1)\.
- \[13\]E\. Bandara, A\. Hass, R\. Gore, S\. Shetty, R\. Mukkamala, S\. H\. Bouk, X\. Liang, N\. W\. Keong, K\. De Zoysa, A\. Withanage,et al\.\(2025\)ASTRIDE: a security threat modeling platform for agentic\-ai applications\.arXiv preprint arXiv:2512\.04785\.Cited by:[§3\.1\.2](https://arxiv.org/html/2606.19116#S3.SS1.SSS2.p1.1)\.
- \[14\]E\. Bandara, T\. Hewa, R\. Gore, S\. Shetty, R\. Mukkamala, P\. Foytik, A\. Rahman, S\. H\. Bouk, X\. Liang, A\. Hass,et al\.\(2025\)Towards responsible and explainable ai agents with consensus\-driven reasoning\.arXiv preprint arXiv:2512\.21699\.Cited by:[§3\.3\.2](https://arxiv.org/html/2606.19116#S3.SS3.SSS2.p1.1)\.
- \[15\]E\. Bandara, S\. Shetty, R\. Mukkamala, R\. Gore, P\. Foytik, S\. H\. Bouk, A\. Rahman, X\. Liang, N\. W\. Keong, K\. De Zoysa,et al\.\(2025\)Model context contracts\-mcp\-enabled framework to integrate llms with blockchain smart contracts\.arXiv preprint arXiv:2510\.19856\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p4.1)\.
- \[16\]E\. M\. Bender, T\. Gebru, A\. McMillan\-Major, and S\. Shmitchell\(2021\)On the dangers of stochastic parrots: can language models be too big?\.InProceedings of the ACM Conference on Fairness, Accountability, and Transparency \(FAccT\),pp\. 610–623\.Cited by:[§2\.4](https://arxiv.org/html/2606.19116#S2.SS4.p2.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.15.13.13.13.13.2)\.
- \[17\]T\. Berners\-Lee, J\. Hendler, and O\. Lassila\(2001\)The semantic web\.Scientific American284\(5\),pp\. 34–43\.Cited by:[§3\.3\.1](https://arxiv.org/html/2606.19116#S3.SS3.SSS1.p1.1)\.
- \[18\]T\. Berners\-Lee\(1989\)Information management: a proposal\.Technical reportCERN\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p1.1),[§3\.1\.1](https://arxiv.org/html/2606.19116#S3.SS1.SSS1.p1.1),[§4\.1](https://arxiv.org/html/2606.19116#S4.SS1.p1.1),[§7\.1](https://arxiv.org/html/2606.19116#S7.SS1.p1.1)\.
- \[19\]S\. Brin and L\. Page\(1998\)The anatomy of a large\-scale hypertextual web search engine\.InProceedings of the 7th International Conference on World Wide Web \(WWW\),pp\. 107–117\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p1.1),[§3\.3\.1](https://arxiv.org/html/2606.19116#S3.SS3.SSS1.p1.1)\.
- \[20\]Cloudflare\(2025\)Cloudflare blocks AI crawlers by default\.Note:[https://blog\.cloudflare\.com](https://blog.cloudflare.com/)Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p3.1),[§1](https://arxiv.org/html/2606.19116#S1.p4.1),[§2\.3](https://arxiv.org/html/2606.19116#S2.SS3.p2.1),[§2\.3](https://arxiv.org/html/2606.19116#S2.SS3.p3.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.11.9.9.9.9.3),[§3\.1\.1](https://arxiv.org/html/2606.19116#S3.SS1.SSS1.p2.1),[§3\.1\.2](https://arxiv.org/html/2606.19116#S3.SS1.SSS2.p1.1),[§5\.1](https://arxiv.org/html/2606.19116#S5.SS1.p3.1)\.
- \[21\]Cloudflare\(2025\)From Googlebot to GPTBot: who’s crawling your site in 2025\.Note:[https://blog\.cloudflare\.com](https://blog.cloudflare.com/)Cited by:[§3\.1\.2](https://arxiv.org/html/2606.19116#S3.SS1.SSS2.p1.1)\.
- \[22\]Cloudflare\(2025\)The crawl\-to\-click gap: Cloudflare data on AI bots, training, and referrals\.Note:[https://blog\.cloudflare\.com](https://blog.cloudflare.com/)Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p3.1),[§2\.3](https://arxiv.org/html/2606.19116#S2.SS3.p2.1),[§3\.1\.2](https://arxiv.org/html/2606.19116#S3.SS1.SSS2.p2.1),[§3\.2\.2](https://arxiv.org/html/2606.19116#S3.SS2.SSS2.p2.1),[Table 2](https://arxiv.org/html/2606.19116#S3.T2.1.1.1.1.3.1.3.2.1.1.1)\.
- \[23\]Cloudflare\(2026\)Google’s AI crawler policy and competitive implications\.Note:[https://blog\.cloudflare\.com](https://blog.cloudflare.com/)Cited by:[§3\.1\.2](https://arxiv.org/html/2606.19116#S3.SS1.SSS2.p1.1),[Table 2](https://arxiv.org/html/2606.19116#S3.T2.1.1.1.1.1.1)\.
- \[24\]Coalition for Content Provenance and Authenticity\(2024\)C2PA technical specification\.Note:[https://c2pa\.org/specifications](https://c2pa.org/specifications)Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p4.1),[§2\.4](https://arxiv.org/html/2606.19116#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.12.10.10.10.10.2),[§6\.4](https://arxiv.org/html/2606.19116#S6.SS4.p2.1),[§7\.3](https://arxiv.org/html/2606.19116#S7.SS3.p4.1)\.
- \[25\]X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. Su\(2023\)Mind2Web: towards a generalist agent for the web\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.1](https://arxiv.org/html/2606.19116#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.5.3.3.3.3.2)\.
- \[26\]Z\. Durante, B\. Sarkar, R\. Gong, R\. Taori, Y\. Noda, P\. Tang, E\. Adeli, S\. K\. Lakshmikanth, K\. Schulman, A\. Milstein,et al\.\(2024\)Agent AI: surveying the horizons of multimodal interaction\.InarXiv preprint arXiv:2401\.03568,Cited by:[§2\.1](https://arxiv.org/html/2606.19116#S2.SS1.p2.1),[§8\.6](https://arxiv.org/html/2606.19116#S8.SS6.p1.1)\.
- \[27\]European Parliament\(2016\)General data protection regulation \(GDPR\) – regulation \(eu\) 2016/679\.Note:[https://gdpr\-info\.eu](https://gdpr-info.eu/)Cited by:[§8\.4](https://arxiv.org/html/2606.19116#S8.SS4.p3.1)\.
- \[28\]D\. S\. Evans\(2008\)The economics of the online advertising industry\.Review of Network Economics7\(3\),pp\. 359–391\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p1.1),[§2\.3](https://arxiv.org/html/2606.19116#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.15.13.13.13.20.6.1),[§3\.2\.1](https://arxiv.org/html/2606.19116#S3.SS2.SSS1.p1.1),[§5\.1](https://arxiv.org/html/2606.19116#S5.SS1.p1.1)\.
- \[29\]A\. P\. Felt, R\. Barnes, A\. King, C\. Palmer, C\. Bentzel, and P\. Tabriz\(2017\)Measuring HTTPS adoption on the web\.In26th USENIX Security Symposium,pp\. 1323–1338\.Cited by:[§4\.5](https://arxiv.org/html/2606.19116#S4.SS5.p7.1),[§7\.3](https://arxiv.org/html/2606.19116#S7.SS3.p4.1)\.
- \[30\]R\. Fielding, J\. Gettys, J\. Mogul, H\. Nielsen, L\. Masinter, P\. Leach, and T\. Berners\-Lee\(1999\)HTTP/1\.1 – RFC 2616\.Technical reportIETF\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p6.1),[§4\.2](https://arxiv.org/html/2606.19116#S4.SS2.p2.1),[§4\.5](https://arxiv.org/html/2606.19116#S4.SS5.p2.1)\.
- \[31\]R\. T\. Fielding\(2000\)Architectural styles and the design of network\-based software architectures\.Ph\.D\. Thesis,University of California, Irvine\.Cited by:[§7\.2](https://arxiv.org/html/2606.19116#S7.SS2.p1.1)\.
- \[32\]Google\(2025\)Agent\-to\-agent protocol \(A2A\)\.Note:[https://developers\.google\.com/agent\-to\-agent](https://developers.google.com/agent-to-agent)Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p4.1),[§2\.2](https://arxiv.org/html/2606.19116#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.15.13.13.13.16.2.1),[§7\.5](https://arxiv.org/html/2606.19116#S7.SS5.p1.1),[§8\.6](https://arxiv.org/html/2606.19116#S8.SS6.p4.1)\.
- \[33\]D\. Hardt\(2012\)The OAuth 2\.0 authorization framework – RFC 6749\.Technical reportIETF\.Note:[https://tools\.ietf\.org/html/rfc6749](https://tools.ietf.org/html/rfc6749)Cited by:[§4\.4](https://arxiv.org/html/2606.19116#S4.SS4.p2.1),[§7\.3](https://arxiv.org/html/2606.19116#S7.SS3.p4.1),[§8\.1](https://arxiv.org/html/2606.19116#S8.SS1.p2.1)\.
- \[34\]P\. Henderson, X\. Li, D\. Jurafsky, T\. Hashimoto, M\. A\. Lemley, and P\. Liang\(2023\)Foundation models and fair use\.InarXiv preprint arXiv:2303\.15715,Cited by:[§8\.4](https://arxiv.org/html/2606.19116#S8.SS4.p2.1)\.
- \[35\]J\. Howard\(2024\)A proposed standard for LLMs: llms\.txt\.Note:[https://llmstxt\.org](https://llmstxt.org/)Cited by:[§3\.3\.3](https://arxiv.org/html/2606.19116#S3.SS3.SSS3.p2.1),[§7\.3](https://arxiv.org/html/2606.19116#S7.SS3.p2.1)\.
- \[36\]S\. Kapooret al\.\(2025\)Infrastructure for AI agents\.Technical reportTechnical Report2501\.10114,arXiv\.Cited by:[§2\.2](https://arxiv.org/html/2606.19116#S2.SS2.p2.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.7.5.5.5.5.2)\.
- \[37\]A\. Karpathy\(2025\)HTML is the language of the agentic web\.Note:[https://twitter\.com/karpathy](https://twitter.com/karpathy)Cited by:[§6\.2\.1](https://arxiv.org/html/2606.19116#S6.SS2.SSS1.p1.1)\.
- \[38\]J\. Kirchenbauer, J\. Geiping, Y\. Wen, J\. Katz, I\. Miers, and T\. Goldstein\(2023\)A watermark for large language models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p4.1),[§2\.4](https://arxiv.org/html/2606.19116#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.14.12.12.12.12.3)\.
- \[39\]M\. Koster\(1994\)A standard for robot exclusion\.Note:[https://www\.robotstxt\.org/orig\.html](https://www.robotstxt.org/orig.html)Cited by:[§3\.1\.1](https://arxiv.org/html/2606.19116#S3.SS1.SSS1.p2.1)\.
- \[40\]Y\. Liet al\.\(2025\)Agentic web: weaving the next web with AI agents\.Technical reportTechnical Report2507\.21206,arXiv\.Cited by:[§2\.2](https://arxiv.org/html/2606.19116#S2.SS2.p2.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.8.6.6.6.6.2)\.
- \[41\]E\. Marcotte\(2011\)Responsive web design\.A Book Apart\.Cited by:[§7\.2](https://arxiv.org/html/2606.19116#S7.SS2.p1.1)\.
- \[42\]M\. Martiniet al\.\(2024\)Provenance requirements for agentic content pipelines\.Note:arXiv preprintCited by:[§2\.4](https://arxiv.org/html/2606.19116#S2.SS4.p2.1)\.
- \[43\]Microsoft\(2025\)NLWeb: natural language interface for the web\.Note:[https://github\.com/microsoft/NLWeb](https://github.com/microsoft/NLWeb)Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p4.1),[§2\.2](https://arxiv.org/html/2606.19116#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.15.13.13.13.17.3.1),[§7\.5](https://arxiv.org/html/2606.19116#S7.SS5.p1.1)\.
- \[44\]T\. Nguyenet al\.\(2025\)Economic models for AI agent content access\.Note:arXiv preprintCited by:[§2\.3](https://arxiv.org/html/2606.19116#S2.SS3.p3.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.15.13.13.13.18.4.1)\.
- \[45\]Pew Research Center\(2025\)User behavior with AI overviews in search\.Note:[https://www\.pewresearch\.org](https://www.pewresearch.org/)Cited by:[§3\.2\.2](https://arxiv.org/html/2606.19116#S3.SS2.SSS2.p1.1)\.
- \[46\]D\. Raggett, A\. Le Hors, and I\. Jacobs\(1999\)HTML 4\.01 specification\.W3C Recommendation\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p1.1),[§3\.3\.1](https://arxiv.org/html/2606.19116#S3.SS3.SSS1.p1.1),[§8\.1](https://arxiv.org/html/2606.19116#S8.SS1.p2.1)\.
- \[47\]E\. S\. Raymond\(1999\)The cathedral and the bazaar: musings on linux and open source by an accidental revolutionary\.O’Reilly Media\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p6.1),[§5\.4](https://arxiv.org/html/2606.19116#S5.SS4.p2.1)\.
- \[48\]I\. Research\(2025\)Agent communication protocol \(ACP\)\.Technical reportIBM\.Note:[https://agentcommunicationprotocol\.dev](https://agentcommunicationprotocol.dev/)Cited by:[§2\.2](https://arxiv.org/html/2606.19116#S2.SS2.p1.1)\.
- \[49\]Semrush\(2025\)Zero\-click searches in AI mode\.Note:[https://www\.semrush\.com](https://www.semrush.com/)Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p3.1),[§2\.3](https://arxiv.org/html/2606.19116#S2.SS3.p2.1),[§3\.2\.2](https://arxiv.org/html/2606.19116#S3.SS2.SSS2.p1.1),[§3\.2\.2](https://arxiv.org/html/2606.19116#S3.SS2.SSS2.p2.1),[Table 2](https://arxiv.org/html/2606.19116#S3.T2.1.1.1.1.3.1.3),[§5\.1](https://arxiv.org/html/2606.19116#S5.SS1.p2.1)\.
- \[50\]I\. Shumailov, Z\. Shumaylov, Y\. Zhao, Y\. Gal, N\. Papernot, and R\. Anderson\(2023\)The curse of recursion: training on generated data makes models forget\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p3.1),[§2\.4](https://arxiv.org/html/2606.19116#S2.SS4.p2.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.15.13.13.13.19.5.1),[Figure 2](https://arxiv.org/html/2606.19116#S3.F2),[§3\.3\.2](https://arxiv.org/html/2606.19116#S3.SS3.SSS2.p2.1),[Table 2](https://arxiv.org/html/2606.19116#S3.T2.1.1.1.1.4.2.3.2.1.2.1)\.
- \[51\]SISTRIX\(2026\)Click\-through rate analysis: AI overviews impact\.Note:[https://www\.sistrix\.com](https://www.sistrix.com/)Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p3.1),[§2\.3](https://arxiv.org/html/2606.19116#S2.SS3.p2.1),[§3\.2\.2](https://arxiv.org/html/2606.19116#S3.SS2.SSS2.p1.1),[§3\.2\.2](https://arxiv.org/html/2606.19116#S3.SS2.SSS2.p2.1),[§5\.1](https://arxiv.org/html/2606.19116#S5.SS1.p2.1)\.
- \[52\]L\. von Ahn, M\. Blum, N\. J\. Hopper, and J\. Langford\(2003\)CAPTCHA: using hard AI problems for security\.InAdvances in Cryptology – EUROCRYPT 2003,pp\. 294–311\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p1.1),[§3\.1\.1](https://arxiv.org/html/2606.19116#S3.SS1.SSS1.p1.1)\.
- \[53\]W3C Agent Web Community Group\(2025\)Agent web: architecture and challenges\.Note:[https://www\.w3\.org/community/agent\-web](https://www.w3.org/community/agent-web)Cited by:[§2\.2](https://arxiv.org/html/2606.19116#S2.SS2.p2.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.9.7.7.7.7.2)\.
- \[54\]L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin, W\. X\. Zhao, Z\. Wei, and J\. Wen\(2024\)A survey on large language model based autonomous agents\.Frontiers of Computer Science18\.Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.19116#S2.SS1.p2.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.6.4.4.4.4.2)\.
- \[55\]webMCP Authors\(2025\)webMCP: a client\-side standard for agent\-ready web design\.Technical reportTechnical Report2508\.09171,arXiv\.Cited by:[§3\.3\.2](https://arxiv.org/html/2606.19116#S3.SS3.SSS2.p1.1),[Table 2](https://arxiv.org/html/2606.19116#S3.T2.1.1.1.1.4.2.3.2.1.1.1),[Figure 10](https://arxiv.org/html/2606.19116#S6.F10),[§6\.1](https://arxiv.org/html/2606.19116#S6.SS1.p5.1),[§6\.2\.1](https://arxiv.org/html/2606.19116#S6.SS2.SSS1.p2.1)\.
- \[56\]T\. Wu\(2016\)The attention merchants: the epic scramble to get inside our heads\.Knopf\.Cited by:[§2\.3](https://arxiv.org/html/2606.19116#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.15.13.13.13.21.7.1),[§3\.2\.1](https://arxiv.org/html/2606.19116#S3.SS2.SSS1.p1.1),[§5\.1](https://arxiv.org/html/2606.19116#S5.SS1.p1.1)\.
- \[57\]S\. Yao, H\. Chen, J\. Yang, and K\. Narasimhan\(2022\)WebShop: towards scalable real\-world web interaction with grounded language agents\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2606.19116#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.19116#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.3.1.1.1.1.2)\.
- \[58\]R\. Zellers, A\. Holtzman, H\. Rashkin, Y\. Bisk, A\. Farhadi, F\. Roesner, and Y\. Choi\(2019\)Defending against neural fake news\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.4](https://arxiv.org/html/2606.19116#S2.SS4.p1.1)\.
- \[59\]S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig\(2024\)WebArena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.1](https://arxiv.org/html/2606.19116#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2606.19116#S2.T1.4.2.2.2.2.2)\.
- \[60\]J\. Zittrain\(2008\)The future of the internet – and how to stop it\.Yale University Press\.Cited by:[§7\.1](https://arxiv.org/html/2606.19116#S7.SS1.p1.1)\.