Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

# The Agentic Web Index: AI Bot Traffic Statistics | Known Agents Source: [https://knownagents.com/insights](https://knownagents.com/insights) ## [Overview](https://knownagents.com/insights#overview) Key ecosystem metrics across**5,000\+**websites using[Agent Analytics](https://knownagents.com/products/agent-analytics)and[AI Chat Referral Tracking](https://knownagents.com/products/ai-chat-referral-tracking)\. ### [Bot vs\. Human Traffic](https://knownagents.com/insights#bot-vs-human-traffic) ↓ 2% Compared to the previous 90 days The amount of visits from bots vs\. humans ### [Agentrification](https://knownagents.com/insights#agentrification) ↑ 11% Compared to the previous 90 days The percentage of bot traffic that's AI\-related ### [AI Chat Referral Volume](https://knownagents.com/insights#ai-chat-referral-volume) ↓ 9% Compared to the previous 90 days The percentage of human website visits that come from AI chat [See Referral Trends ↓](https://knownagents.com/insights#ai-chat-referrals) ### [Robots\.txt Effectiveness](https://knownagents.com/insights#robots-txt-effectiveness) How often bots follow robots\.txt rules [See Per Agent ↓](https://knownagents.com/insights#robots-txt-and-compliance) ### [Traffic by Agent Type](https://knownagents.com/insights#traffic-by-agent-type) AI Agent AI Agent Uses an actual web browser to autonomously complete complex tasks on behalf of a human user AI Assistant AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer AI Coding Agent AI Coding Agent Fetches documentation and other resources to help build software AI Data Provider AI Data Provider Crawls websites to supply structured content to AI systems as a third\-party service AI Data Scraper AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs AI Search Crawler AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results Archiver Archiver Captures and stores historical website snapshots for long\-term digital preservation Automated Agent Automated Agent Automates browser interactions programmatically without direct human supervision Developer Helper Developer Helper Assists with testing, debugging, and ensuring website functionality Fetcher Fetcher Retrieves web page metadata to power app features like link previews or feeds Intelligence Gatherer Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting Scraper Scraper Extracts large amounts of web data, often without explicit website permission Search Engine Crawler Search Engine Crawler Systematically scans and indexes web pages to include in search results Security Scanner Security Scanner Scans websites for security vulnerabilities, threats, and configuration weaknesses SEO Crawler SEO Crawler Analyzes website structure and content to identify SEO improvement opportunities Uncategorized Uncategorized Not yet assigned a type Undocumented AI Agent Undocumented AI Agent Crawls websites without disclosing its purpose, collecting data for an unknown AI use case Hover over each agent type for more information about what they do ### [Top Agent Types](https://knownagents.com/insights#top-agent-types) Agent types with the most activity ### [Top Visiting Agents](https://knownagents.com/insights#top-agents) [bingbot](https://knownagents.com/agents/bingbot) SRCH Search Engine Crawler Systematically scans and indexes web pages to include in search results 8\.2% [Googlebot](https://knownagents.com/agents/googlebot) SRCH Search Engine Crawler Systematically scans and indexes web pages to include in search results 7\.9% [AhrefsBot](https://knownagents.com/agents/ahrefsbot) SEO SEO Crawler Analyzes website structure and content to identify SEO improvement opportunities 6\.2% [Known Agent](https://knownagents.com/agents/known-agent) DEV Developer Helper Assists with testing, debugging, and ensuring website functionality 5\.5% [ChatGPT\-User](https://knownagents.com/agents/chatgpt-user) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 3\.3% [ClaudeBot](https://knownagents.com/agents/claudebot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 3\.2% [PetalBot](https://knownagents.com/agents/petalbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 3\.2% [SemrushBot](https://knownagents.com/agents/semrushbot) SEO SEO Crawler Analyzes website structure and content to identify SEO improvement opportunities 3\.0% [facebookexternalhit](https://knownagents.com/agents/facebookexternalhit) FTCH Fetcher Retrieves web page metadata to power app features like link previews or feeds 2\.6% [meta\-externalagent](https://knownagents.com/agents/meta-externalagent) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 2\.3% [Amazonbot](https://knownagents.com/agents/amazonbot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 2\.3% [MJ12bot](https://knownagents.com/agents/mj12bot) SEO SEO Crawler Analyzes website structure and content to identify SEO improvement opportunities 2\.2% [Amzn\-SearchBot](https://knownagents.com/agents/amzn-searchbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 2\.2% [DotBot](https://knownagents.com/agents/dotbot) SEO SEO Crawler Analyzes website structure and content to identify SEO improvement opportunities 2\.1% [Applebot](https://knownagents.com/agents/applebot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 2\.1% Agents with the most activity ### [Top Operators](https://knownagents.com/insights#top-operators) Operators with the most activity ## [AI Scraping Activity](https://knownagents.com/insights#ai-scraping) These bots scrape website content to train AI models\. Some belong to AI companies, while others belong to third\-party services that resell the data\.[Automatic Robots\.txt](https://knownagents.com/products/automatic-robots-txt)can block unwanted scraping\. Included agent types include[AI Data Scrapers](https://knownagents.com/agents?agent_type_url_slug=ai-data-scraper)and[AI Data Providers](https://knownagents.com/agents?agent_type_url_slug=ai-data-provider)\. ### [Traffic by Agent Type](https://knownagents.com/insights#ai-scraping-traffic-by-agent-type) AI Data Provider AI Data Provider Crawls websites to supply structured content to AI systems as a third\-party service AI Data Scraper AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs AI scraping activity by agent type over time ### [Top Visited Website Categories](https://knownagents.com/insights#ai-scraping-top-visited-website-categories) Computers and Electronics 5\.3% Business and Industrial 5\.0% Internet and Telecom 4\.5% Website categories with most activity ### [Top Agents](https://knownagents.com/insights#ai-scraping-top-agents) [ClaudeBot](https://knownagents.com/agents/claudebot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 27\.0% [meta\-externalagent](https://knownagents.com/agents/meta-externalagent) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 19\.5% [Amazonbot](https://knownagents.com/agents/amazonbot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 19\.2% [GPTBot](https://knownagents.com/agents/gptbot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 9\.4% [Bytespider](https://knownagents.com/agents/bytespider) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 7\.3% [ShapBot](https://knownagents.com/agents/shapbot) PVDR AI Data Provider Crawls websites to supply structured content to AI systems as a third\-party service 5\.6% [GoogleOther](https://knownagents.com/agents/googleother) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 3\.9% [CCBot](https://knownagents.com/agents/ccbot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 2\.3% [Timpibot](https://knownagents.com/agents/timpibot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 1\.8% [YouBot](https://knownagents.com/agents/youbot) PVDR AI Data Provider Crawls websites to supply structured content to AI systems as a third\-party service 1\.4% [VelenPublicWebCrawler](https://knownagents.com/agents/velenpublicwebcrawler) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 0\.8% [Diffbot](https://knownagents.com/agents/diffbot) PVDR AI Data Provider Crawls websites to supply structured content to AI systems as a third\-party service 0\.5% [DeepSeekBot](https://knownagents.com/agents/deepseekbot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 0\.4% [TerraCotta](https://knownagents.com/agents/terracotta) PVDR AI Data Provider Crawls websites to supply structured content to AI systems as a third\-party service 0\.3% [AIWebIndex](https://knownagents.com/agents/aiwebindex) PVDR AI Data Provider Crawls websites to supply structured content to AI systems as a third\-party service 0\.2% Agents doing the most AI scraping ### [Top Operators](https://knownagents.com/insights#ai-scraping-top-operators) Operators doing the most AI scraping ## [AI Fetching Activity](https://knownagents.com/insights#ai-fetching) These bots fetch website content in real time to power AI assistants, coding agents, and other retrieval\-augmented generation \(RAG\) tasks\. Pages inform responses on the spot, such as when an assistant summarizes an article or a coding agent references documentation\. Included agent types include[AI Coding Agents](https://knownagents.com/agents?agent_type_url_slug=ai-coding-agent)and[AI Assistants](https://knownagents.com/agents?agent_type_url_slug=ai-assistant)\. ### [Traffic by Agent Type](https://knownagents.com/insights#ai-fetching-traffic-by-agent-type) AI Assistant AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer AI Coding Agent AI Coding Agent Fetches documentation and other resources to help build software AI fetching activity by agent type over time ### [Top Visited Website Categories](https://knownagents.com/insights#ai-fetching-top-visited-website-categories) Computers and Electronics 2\.3% Travel and Transportation 1\.9% Business and Industrial 1\.8% Internet and Telecom 1\.7% Website categories with most activity ### [Top Agents](https://knownagents.com/insights#ai-fetching-top-agents) [ChatGPT\-User](https://knownagents.com/agents/chatgpt-user) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 86\.4% [DuckAssistBot](https://knownagents.com/agents/duckassistbot) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 3\.3% [Perplexity\-User](https://knownagents.com/agents/perplexity-user) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 2\.7% [Claude\-User](https://knownagents.com/agents/claude-user) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 2\.5% [MistralAI\-User](https://knownagents.com/agents/mistralai-user) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 2\.1% [Claude\-Code](https://knownagents.com/agents/claude-code) CODE AI Coding Agent Fetches documentation and other resources to help build software 1\.1% [Shap\-User](https://knownagents.com/agents/shap-user) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 1\.0% [Google\-NotebookLM](https://knownagents.com/agents/google-notebooklm) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 0\.4% [Cursor](https://knownagents.com/agents/cursor) CODE AI Coding Agent Fetches documentation and other resources to help build software 0\.4% [Gemini\-Deep\-Research](https://knownagents.com/agents/gemini-deep-research) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 0\.0% [meta\-externalfetcher](https://knownagents.com/agents/meta-externalfetcher) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 0\.0% [GoogleAgent\-URLContext](https://knownagents.com/agents/googleagent-urlcontext) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 0\.0% [Code](https://knownagents.com/agents/code) CODE AI Coding Agent Fetches documentation and other resources to help build software 0\.0% [kagi\-fetcher](https://knownagents.com/agents/kagi-fetcher) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 0\.0% [QualifiedBot](https://knownagents.com/agents/qualifiedbot) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 0\.0% Agents doing the most AI fetching ### [Top Operators](https://knownagents.com/insights#ai-fetching-top-operators) Operators doing the most AI fetching ## [AI Search Indexing Activity](https://knownagents.com/insights#ai-search-indexing) These bots crawl website content so it can be surfaced in AI search engines and AI\-generated answers\. Those answers often include citations or links back to the source pages\. Included agent types include[AI Search Crawlers](https://knownagents.com/agents?agent_type_url_slug=ai-search-crawler)\. ### [Traffic by Agent Type](https://knownagents.com/insights#ai-search-indexing-traffic-by-agent-type) AI Search Crawler AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results AI search indexing activity by agent type over time ### [Top Visited Website Categories](https://knownagents.com/insights#ai-search-indexing-top-visited-website-categories) Travel and Transportation 6\.8% Books and Literature 4\.9% Computers and Electronics 4\.6% Arts and Entertainment 4\.5% Website categories with most activity ### [Top Agents](https://knownagents.com/insights#ai-search-indexing-top-agents) [PetalBot](https://knownagents.com/agents/petalbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 25\.4% [Amzn\-SearchBot](https://knownagents.com/agents/amzn-searchbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 17\.3% [Applebot](https://knownagents.com/agents/applebot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 16\.7% [OAI\-SearchBot](https://knownagents.com/agents/oai-searchbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 11\.9% [meta\-webindexer](https://knownagents.com/agents/meta-webindexer) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 10\.8% [Claude\-SearchBot](https://knownagents.com/agents/claude-searchbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 10\.4% [PerplexityBot](https://knownagents.com/agents/perplexitybot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 3\.4% [LinkupBot](https://knownagents.com/agents/linkupbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 3\.4% [Google\-CloudVertexBot](https://knownagents.com/agents/google-cloudvertexbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 0\.3% [xAI\-SearchBot](https://knownagents.com/agents/xai-searchbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 0\.2% [AzureAI\-SearchBot](https://knownagents.com/agents/azureai-searchbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 0\.2% [AddSearchBot](https://knownagents.com/agents/addsearchbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 0\.0% [Anomura](https://knownagents.com/agents/anomura) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 0\.0% [MistralAI\-Index](https://knownagents.com/agents/mistralai-index) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 0\.0% [amazon\-kendra](https://knownagents.com/agents/amazon-kendra) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 0\.0% Agents doing the most AI search indexing ### [Top Operators](https://knownagents.com/insights#ai-search-indexing-top-operators) Operators doing the most AI search indexing ## [AI Browsing Activity](https://knownagents.com/insights#ai-browsing) These bots use browsers to autonomously navigate websites, click through pages, and make decisions to complete tasks for people\.[Agentic UX best practices](https://web.dev/articles/ai-agent-site-ux)and[Google PageSpeed Insights](https://pagespeed.web.dev/)help evaluate how well websites support them\. Included agent types include[AI Agents](https://knownagents.com/agents?agent_type_url_slug=ai-agent)\. ### [Average Session Duration](https://knownagents.com/insights#average-session-duration) ↓ 5% Compared to the previous 90 days The average duration of a session ### [Average Pages per Session](https://knownagents.com/insights#average-pages-per-session) ↓ 1% Compared to the previous 90 days The average number of pages visited per session ### [Traffic by Agent Type](https://knownagents.com/insights#ai-browsing-traffic-by-agent-type) AI Agent AI Agent Uses an actual web browser to autonomously complete complex tasks on behalf of a human user AI browsing activity by agent type over time ### [Top Visited Website Categories](https://knownagents.com/insights#ai-browsing-top-visited-website-categories) Computers and Electronics 0\.0% Internet and Telecom 0\.0% Travel and Transportation 0\.0% Business and Industrial 0\.0% Books and Literature 0\.0% Arts and Entertainment 0\.0% Website categories with most activity ### [Top Agents](https://knownagents.com/insights#ai-browsing-top-agents) [Google\-Agent](https://knownagents.com/agents/google-agent) AGNT AI Agent Uses an actual web browser to autonomously complete complex tasks on behalf of a human user 43\.0% [Manus\-User](https://knownagents.com/agents/manus-user) AGNT AI Agent Uses an actual web browser to autonomously complete complex tasks on behalf of a human user 40\.1% [ChatGPT Agent](https://knownagents.com/agents/chatgpt-agent) AGNT AI Agent Uses an actual web browser to autonomously complete complex tasks on behalf of a human user 16\.9% [NovaAct](https://knownagents.com/agents/novaact) AGNT AI Agent Uses an actual web browser to autonomously complete complex tasks on behalf of a human user 0\.0% [GoogleAgent\-Mariner](https://knownagents.com/agents/googleagent-mariner) AGNT AI Agent Uses an actual web browser to autonomously complete complex tasks on behalf of a human user 0\.0% [AmazonBuyForMe](https://knownagents.com/agents/amazonbuyforme) AGNT AI Agent Uses an actual web browser to autonomously complete complex tasks on behalf of a human user 0\.0% [TwinAgent](https://knownagents.com/agents/twinagent) AGNT AI Agent Uses an actual web browser to autonomously complete complex tasks on behalf of a human user 0\.0% Agents doing the most AI browsing ### [Top Operators](https://knownagents.com/insights#ai-browsing-top-operators) Operators doing the most AI browsing ## [Robots\.txt & Compliance](https://knownagents.com/insights#robots-txt-and-compliance) See which robots\.txt rules are set across the web and how well agents follow them\. An agent's[Robots\.txt Effectiveness](https://knownagents.com/insights#measuring-robots-txt-effectiveness)measures the effectiveness of a disallow rule for it by estimating how much the agent reduces its traffic after it's blocked\. ### [Effectiveness \(Overall\)](https://knownagents.com/insights#robots-txt-effectiveness-overall) How often[all agents](https://knownagents.com/agents)across all agent types follow robots\.txt rules ### [Top Rule\-Following Agents](https://knownagents.com/insights#top-rule-following-agents) [LinkupBot](https://knownagents.com/agents/linkupbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 100\.0% [serpstatbot](https://knownagents.com/agents/serpstatbot) SEO SEO Crawler Analyzes website structure and content to identify SEO improvement opportunities 100\.0% [proximic](https://knownagents.com/agents/proximic) INT Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting 100\.0% [ClarityBot](https://knownagents.com/agents/claritybot) SEO SEO Crawler Analyzes website structure and content to identify SEO improvement opportunities 100\.0% [IAS crawler](https://knownagents.com/agents/ias-crawler) INT Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting 100\.0% [Barkrowler](https://knownagents.com/agents/barkrowler) SEO SEO Crawler Analyzes website structure and content to identify SEO improvement opportunities 100\.0% [um\-IC](https://knownagents.com/agents/um-ic) INT Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting 100\.0% [SEOkicks](https://knownagents.com/agents/seokicks) SEO SEO Crawler Analyzes website structure and content to identify SEO improvement opportunities 100\.0% [Dragonfly](https://knownagents.com/agents/dragonfly) INT Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting 100\.0% [AffsignalCrawler](https://knownagents.com/agents/affsignalcrawler) INT Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting 100\.0% [AdsBot\-Google\-Mobile](https://knownagents.com/agents/adsbot-google-mobile) INT Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting 100\.0% [meta\-externalads](https://knownagents.com/agents/meta-externalads) INT Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting 100\.0% [UptimeRobot](https://knownagents.com/agents/uptimerobot) DEV Developer Helper Assists with testing, debugging, and ensuring website functionality 100\.0% [Leikibot](https://knownagents.com/agents/leikibot) INT Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting 100\.0% [TTD\-Content](https://knownagents.com/agents/ttd-content) INT Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting 100\.0% Agents with the best[Robots\.txt Effectiveness](https://knownagents.com/insights#measuring-robots-txt-effectiveness)percentages ### [Top Rule\-Breaking Agents](https://knownagents.com/insights#top-rule-breaking-agents) [Baiduspider](https://knownagents.com/agents/baiduspider) SRCH Search Engine Crawler Systematically scans and indexes web pages to include in search results 82\.6% [SirdataBot](https://knownagents.com/agents/sirdatabot) UNC Uncategorized Not yet assigned a type 84\.5% [ShapBot](https://knownagents.com/agents/shapbot) PVDR AI Data Provider Crawls websites to supply structured content to AI systems as a third\-party service 90\.4% [YandexBot](https://knownagents.com/agents/yandexbot) SRCH Search Engine Crawler Systematically scans and indexes web pages to include in search results 91\.2% [Sogou web spider](https://knownagents.com/agents/sogou-web-spider) SRCH Search Engine Crawler Systematically scans and indexes web pages to include in search results 92\.7% [Mediapartners\-Google](https://knownagents.com/agents/mediapartners-google) INT Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting 93\.2% [Scrapy](https://knownagents.com/agents/scrapy) SCRP Scraper Extracts large amounts of web data, often without explicit website permission 93\.8% [Amazonbot](https://knownagents.com/agents/amazonbot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 94\.7% [HeadlessChrome](https://knownagents.com/agents/headlesschrome) AUTO Automated Agent Automates browser interactions programmatically without direct human supervision 95\.4% [linkfluence](https://knownagents.com/agents/linkfluence) INT Intelligence Gatherer Analyzes web content for brand safety, competitive insights, and ad targeting 95\.9% [DotBot](https://knownagents.com/agents/dotbot) SEO SEO Crawler Analyzes website structure and content to identify SEO improvement opportunities 96\.1% [OAI\-SearchBot](https://knownagents.com/agents/oai-searchbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 96\.3% [ChatGPT\-User](https://knownagents.com/agents/chatgpt-user) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 96\.8% [archive\.org\_bot](https://knownagents.com/agents/archive-org-bot) ARCH Archiver Captures and stores historical website snapshots for long\-term digital preservation 96\.9% Agents with the worst[Robots\.txt Effectiveness](https://knownagents.com/insights#measuring-robots-txt-effectiveness)percentages ### [Top Blocked Agents](https://knownagents.com/insights#top-blocked-bots) ● [GPTBot](https://knownagents.com/agents/gptbot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 24\.6% ● [CCBot](https://knownagents.com/agents/ccbot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 22\.6% ● [ClaudeBot](https://knownagents.com/agents/claudebot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 21\.7% ● [Bytespider](https://knownagents.com/agents/bytespider) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 18\.9% ● [PerplexityBot](https://knownagents.com/agents/perplexitybot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 17\.9% ● [ChatGPT\-User](https://knownagents.com/agents/chatgpt-user) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 16\.1% ● [anthropic\-ai](https://knownagents.com/agents/anthropic-ai) UND Undocumented AI Agent Crawls websites without disclosing its purpose, collecting data for an unknown AI use case 15\.8% ● [meta\-externalagent](https://knownagents.com/agents/meta-externalagent) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 15\.6% ● [Amazonbot](https://knownagents.com/agents/amazonbot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 15\.6% ● [Diffbot](https://knownagents.com/agents/diffbot) PVDR AI Data Provider Crawls websites to supply structured content to AI systems as a third\-party service 14\.0% ● [Claude\-Web](https://knownagents.com/agents/claude-web) UND Undocumented AI Agent Crawls websites without disclosing its purpose, collecting data for an unknown AI use case 14\.0% ● [cohere\-ai](https://knownagents.com/agents/cohere-ai) UND Undocumented AI Agent Crawls websites without disclosing its purpose, collecting data for an unknown AI use case 14\.0% ● [omgili](https://knownagents.com/agents/omgili) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 13\.5% Agents blocked by the most top websites ## [Spoofing & Security](https://knownagents.com/insights#spoofing-and-security) See which agents are most frequently impersonated, and how spoofing activity changes over time\. A visit is considered spoofed when it claims a recognized agent identity but fails that agent's supported authentication method, such as verified IP or Web Bot Auth\. Active Threat: AI Bot Spoofing Campaign We are observing a widespread campaign impersonating AI bots to scan websites for vulnerabilities\. The attacker appears to be targeting credential and configuration paths used by AI coding tools\.[Contact us](https://knownagents.com/insights#)for more information\. ### [Spoofed Traffic by Agent Identity](https://knownagents.com/insights#spoofing-trends-over-time) The percentage of impersonated website traffic for each agent identity over time ### [Top Spoofed Agent Identities](https://knownagents.com/insights#top-spoofed-agent-identities) ● [Googlebot](https://knownagents.com/agents/googlebot) SRCH Search Engine Crawler Systematically scans and indexes web pages to include in search results 0\.5% ● [ChatGPT\-User](https://knownagents.com/agents/chatgpt-user) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 0\.1% ● [GPTBot](https://knownagents.com/agents/gptbot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 0\.1% ● [OAI\-SearchBot](https://knownagents.com/agents/oai-searchbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 0\.1% ● [PerplexityBot](https://knownagents.com/agents/perplexitybot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 0\.1% ● [ClaudeBot](https://knownagents.com/agents/claudebot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 0\.0% ● [Applebot](https://knownagents.com/agents/applebot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 0\.0% ● [bingbot](https://knownagents.com/agents/bingbot) SRCH Search Engine Crawler Systematically scans and indexes web pages to include in search results 0\.0% ● [Perplexity\-User](https://knownagents.com/agents/perplexity-user) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 0\.0% ● [MistralAI\-User](https://knownagents.com/agents/mistralai-user) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 0\.0% ● [GoogleOther](https://knownagents.com/agents/googleother) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 0\.0% ● [AhrefsBot](https://knownagents.com/agents/ahrefsbot) SEO SEO Crawler Analyzes website structure and content to identify SEO improvement opportunities 0\.0% ● [Claude\-User](https://knownagents.com/agents/claude-user) ASST AI Assistant Fetches website content in response to a user prompt, to include in an AI\-generated answer 0\.0% ● [Claude\-SearchBot](https://knownagents.com/agents/claude-searchbot) SRCH AI Search Crawler Indexes website content to possibly include as citations in AI\-powered search results 0\.0% ● [CCBot](https://knownagents.com/agents/ccbot) SCRP AI Data Scraper Downloads website content to include in datasets used for training AI models such as LLMs 0\.0% The most impersonated agent identities ### [Recent Top Targeted Paths](https://knownagents.com/insights#example-paths) /\.config/anthropic/credentials/default\.json /\.claude/settings\.json /\.claude\.json /\.hermes/\.env /\.openclaw/\.env /\.codex/config\.toml /\.continue/config\.json /\.aider\.conf\.yml /service\-account\.json /serviceaccountkey\.json /service\_account\.json /firebase\-adminsdk\.json /firebase\-service\-account\.json /\.aws/credentials /\.aws/config /\.s3cfg /\.boto /\.npmrc /\.env\.example /\.env\.local /\.env\.production /\.env\.backup /\.env\.old /backend/\.env /api/\.env /admin/\.env /dockerfile /docker\-compose\.yaml /\.docker/config\.json /terraform\.tfstate /credentials\.json /secrets\.json /secrets\.yml /key\.json /rclone\.conf Examples of recent top targeted request paths ## [AI Chat Referrals](https://knownagents.com/insights#ai-chat-referrals) See which AI platforms like ChatGPT, Perplexity, and Gemini cite websites and send them human referral traffic\. Citations are estimated\.[Google's guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)explains how websites can optimize their content to be more visible in AI chat responses \(GEO\)\. ### [Traffic by AI Chat Platform](https://knownagents.com/insights#traffic-by-ai-chat-platform) ChatGPT Claude Copilot DeepSeek Gemini Meta AI Mistral Perplexity Referral activity by AI chat platform over time ### [Top Mentioned \(Cited\) Website Categories](https://knownagents.com/insights#top-mentioned-cited-website-categories) Computers and Electronics 2\.3% Travel and Transportation 1\.9% Business and Industrial 1\.8% Internet and Telecom 1\.6% Website categories most frequently cited in AI chat responses ### [Top Clicked \(Referred\) Website Categories](https://knownagents.com/insights#top-clicked-referred-website-categories) Travel and Transportation 0\.3% Computers and Electronics 0\.1% Business and Industrial 0\.1% Internet and Telecom 0\.0% Website categories receiving the most referrals from AI chat ## [Methodology](https://knownagents.com/insights#methodology) ### [Data Scope](https://knownagents.com/insights#data-scope) The Index is updated daily with completed days of traffic, security, and referral data from more than 5,000 websites using[Agent Analytics](https://knownagents.com/products/agent-analytics)and[AI Chat Referral Tracking](https://knownagents.com/products/ai-chat-referral-tracking)\. The current partial day is excluded\. Percentage\-change tags compare the current period with the preceding period of the same duration\. Agent names, operators, and classifications come from the[Agent Directory](https://knownagents.com/agents), which is updated as new agents are discovered or existing agents change\. Website categories follow the taxonomy used by[Google AdSense](https://adsense.google.com/start)\. Participating websites are not a random sample of the entire web, and the qualifying set can change as websites connect, disconnect, or cross activity thresholds\. Results characterize the observed network and broader directional trends; they should not be interpreted as a precise census of global web traffic\. ### [Qualification & Aggregation](https://knownagents.com/insights#qualification-and-aggregation) Only websites meeting minimum activity and data\-quality requirements are included\. Internal, test, incomplete, or anomalous data is excluded\. Bot traffic percentages use total server traffic as their denominator\. AI chat referral percentages use estimated human traffic, calculated by excluding identified bot visits from total server traffic\. Rates are calculated for each qualifying website first, then averaged across websites and completed days\. This gives each website equal weight regardless of traffic volume and prevents a small number of high\-traffic websites from dominating the results\. Daily charts are not smoothed, allowing normal seasonality to remain visible\. ### [Measuring Robots\.txt Effectiveness](https://knownagents.com/insights#measuring-robots-txt-effectiveness) An agent's Robots\.txt Effectiveness estimates the reduction in its request rate associated with a full disallow rule\. For each completed day, Known Agents establishes an agent\-specific baseline from qualifying websites where that agent is allowed, adjusts the baseline for the overall traffic of each website where the agent is disallowed, and compares the expected activity with the activity actually observed\. Only website\-day observations with sufficient site traffic, agent activity, cross\-site coverage, and expected volume qualify\. Scores also require repeated observations across multiple websites and days\. When an agent publishes a supported authentication method, only verified traffic is attributed to it\. Each qualifying website\-day contributes equally\. Scores range from 0%, meaning no measurable reduction, to 100%, meaning no qualifying requests were observed where the agent was disallowed\. The headline Robots\.txt Effectiveness metric gives each qualifying agent equal weight\. Because this is an observational estimate rather than a controlled experiment, it measures an association with robots\.txt rules but does not claim that robots\.txt caused every observed difference\. Top Blocked Bots is calculated separately using daily robots\.txt scans of[Similarweb's top 1,000 websites](https://www.similarweb.com/top-websites)\. ### [Identifying Spoofed Bots](https://knownagents.com/insights#identifying-spoofed-bots) Spoofing statistics measure traffic from visits that claim the identity of a known agent but fail a supported authentication method, such as published IP verification or HTTP message signatures\. Each agent's daily rate is calculated against total server traffic for every qualifying website, then averaged across websites\. A failed check indicates that the visit was likely impersonating the named agent; it does not identify the software or operator that actually made the request\. Agents without a supported authentication method are not included in these measurements\. ### [AI Chat Citations & Referrals](https://knownagents.com/insights#ai-chat-citations-and-referrals) AI chat referral statistics count directly observed human visits carrying a recognized AI platform in the referring URL or campaign source\. Visits without usable referral information cannot be attributed to an AI platform\. Citation statistics are estimates based on requests from agents known to retrieve content for AI platforms\. Those requests indicate that content may have informed a response, but they do not confirm that a source appeared as a citation to a user\. Because AI platforms do not provide a complete public record of their sources, citation results should be interpreted as directional patterns rather than exact citation counts\. ## [Frequently Asked Questions](https://knownagents.com/insights#frequently-asked-questions) ### Can journalists and media organizations use this data? Absolutely\. You may cite The Agentic Web Index with attribution and a link to this page\. For interviews, fact\-checking, background context, or a more specific breakdown for a story,[contact us](https://knownagents.com/insights#)and include your deadline\. --- ### Do you work with researchers? Absolutely\. We welcome thoughtful research into how agents and bots are changing the web\.[Tell us about your research question](https://knownagents.com/insights#), timeframe, and intended use\. Depending on the scope and data constraints, we may be able to provide additional context, compare approaches, or explore a joint analysis\. --- ### Can I request a specific analysis? Yes\. If you need a breakdown by agent, operator, activity type, website category, or time period that is not shown here,[contact us](https://knownagents.com/insights#)\. When the underlying data supports it, we can examine the question and provide a focused analysis\. --- ### How do I see these trends on my own website? [Agent Analytics](https://knownagents.com/products/agent-analytics)shows which agents and bots visit your website, what they access, and how their activity changes over time\.[AI Chat Referral Tracking](https://knownagents.com/products/ai-chat-referral-tracking)measures the human traffic arriving from AI chat platforms\.[Automatic Robots\.txt](https://knownagents.com/products/automatic-robots-txt)helps manage which bots can access your content\.