AI crawlers and user agents
On 26 October 2026, the ledger evaluated 198 AI user agents against the robots.txt of every top 10k site.
Last verified 2026-10-26 21:13 UTC (daily run of 2026-10-26). Proof for that day.
The list comes from the community-maintained ai.robots.txt project and is refreshed every day. Headline agents are fixed so their series stay comparable. Shares: top 10k panel, 2026-10-26.
| User agent | Operator | Blocks | Partly | Names it |
|---|---|---|---|---|
| GPTBot headline | OpenAI Scrapes data to train OpenAI's products. | 14.6% | 3.7% | 16.5% |
| ClaudeBot headline | Anthropic Scrapes data to train Anthropic's AI products. | 13.8% | 3.7% | 14.2% |
| CCBot headline | Common Crawl Foundation Provides open crawl dataset, used for many purposes, including Machine Learning/AI. | 15.4% | 3.6% | 14.1% |
| Google-Extended headline | Google LLM training. | 12.5% | 3.6% | 13.3% |
| Bytespider headline | ByteDance LLM training. | 15.1% | 3.6% | 12.7% |
| PerplexityBot headline | Perplexity Search result generation. | 10.0% | 3.7% | 11.3% |
| ChatGPT-User headline | OpenAI AI Assistants | 8.7% | 3.9% | 10.4% |
| Meta-ExternalAgent headline | Meta Used to train models and improve products. | 12.4% | 3.8% | 10.3% |
| Applebot-Extended headline | Apple Powers features in Siri, Spotlight, Safari, Apple Intelligence, and others. | 12.1% | 3.8% | 10.2% |
| Amazonbot headline | Amazon Service improvement and enabling answers for Alexa users. | 11.9% | 3.7% | 10.1% |
| OAI-SearchBot headline | OpenAI Search result generation. | 7.3% | 4.0% | 9.1% |
| Googlebot headline | – | 2.8% | 3.8% | 8.3% |
| Claude-SearchBot headline | Anthropic Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses. | 7.2% | 3.8% | 5.8% |
| Claude-User headline | Anthropic AI Assistants | 7.2% | 3.9% | 5.7% |
| Perplexity-User headline | Perplexity AI Assistants | 7.5% | 3.8% | 5.6% |
| anthropic-ai | Anthropic Scrapes data to train Anthropic's AI products. | 11.8% | 3.7% | 9.8% |
| cohere-ai | Cohere Retrieves data to provide responses to user-initiated prompts. | 11.9% | 3.7% | 9.2% |
| Diffbot | Diffbot AI Data Providers | 12.2% | 3.6% | 8.7% |
| omgili | Webz.io Data is sold. | 11.7% | 3.7% | 8.0% |
| Claude-Web | Anthropic Undocumented AI Agents | 10.5% | 3.6% | 8.0% |
| PetalBot | Huawei Used to provide recommendations in Hauwei assistant and AI search services. | 10.8% | 3.6% | 7.4% |
| YouBot | You Scrapes data for search engine and LLMs. | 9.5% | 3.8% | 6.6% |
| FacebookBot | Meta/Facebook Training language models | 9.3% | 3.8% | 6.4% |
| facebookexternalhit | Meta/Facebook Ostensibly only for sharing, but likely used as an AI crawler as well | 3.6% | 3.9% | 5.9% |
| omgilibot | Webz.io Data is sold. | 9.5% | 3.7% | 5.8% |
| Timpibot | Timpi Scrapes data for use in training LLMs. | 9.0% | 3.7% | 5.1% |
| ImagesiftBot | ImageSift ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support their suite of web intelligence products | 8.7% | 3.7% | 4.8% |
| DuckAssistBot | Unclear at this time. AI Assistants | 7.7% | 3.8% | 4.8% |
| Google-CloudVertexBot | Google Build and manage AI models for businesses employing Vertex AI | 8.1% | 3.8% | 4.6% |
| meta-externalfetcher | Unclear at this time. AI Assistants | 7.7% | 3.9% | 4.4% |
| Scrapy | Zyte Scrapes data for a variety of uses including training AI. | 8.2% | 3.7% | 4.3% |
| Webzio-Extended | Unclear at this time. AI Data Scrapers | 8.2% | 3.8% | 4.2% |
| GoogleOther | Google Scrapes data. | 7.0% | 3.9% | 4.1% |
| Applebot | Unclear at this time. AI Search Crawlers | 4.8% | 3.8% | 4.0% |
| ICC-Crawler | NICT Scrapes data to train and support AI technologies. | 8.1% | 3.8% | 4.0% |
| cohere-training-data-crawler | Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products AI Data Scrapers | 7.7% | 3.8% | 3.7% |
| MistralAI-User | Mistral AI Assistants | 7.0% | 3.8% | 3.6% |
| Cotoyogi | ROIS AI LLM Scraper. | 7.3% | 3.8% | 3.1% |
| PanguBot | the Chinese company Huawei AI Data Scrapers | 7.1% | 3.8% | 3.0% |
| FriendlyCrawler | Unknown We are using the data from the crawler to build datasets for machine learning experiments. | 7.1% | 3.8% | 3.0% |
| VelenPublicWebCrawler | Velen Crawler Scrapes data for business data sets and machine learning models. | 7.0% | 3.8% | 2.9% |
| DeepSeekBot | DeepSeek Training language models and improving AI products | 6.5% | 3.9% | 2.7% |
| SemrushBot-SWA | Semrush Checks URLs on your site for SEO Writing Assistant. | 6.8% | 3.9% | 2.7% |
| QualifiedBot | Qualified AI Assistants | 6.8% | 3.8% | 2.7% |
| meta-webindexer | Meta AI Assistants | 6.3% | 3.8% | 2.6% |
| Brightbot | Unclear at this time. AI Data Providers | 6.6% | 3.9% | 2.5% |
| atlassian-bot | Atlassian AI search, assistants and agents | 6.3% | 3.9% | 2.2% |
| FirecrawlAgent | Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications AI Data Providers | 6.1% | 3.9% | 2.2% |
| ExaBot | Unclear at this time. AI Data Providers | 5.9% | 3.8% | 2.1% |
| iaskspider | Unclear at this time. Undocumented AI Agents | 6.3% | 3.9% | 2.1% |
| WARDBot | WEBSPARK AI Data Scrapers | 6.1% | 3.9% | 1.9% |
| ShapBot | Parallel AI Data Providers | 6.1% | 3.9% | 1.9% |
| Gemini-Deep-Research | Unclear at this time. AI Assistants | 5.4% | 3.9% | 1.7% |
| ChatGLM-Spider | Unclear at this time. AI Data Scrapers | 5.8% | 3.9% | 1.7% |
| ISSCyberRiskCrawler | ISS-Corporate Scrapes data to train machine learning models. | 5.9% | 3.9% | 1.7% |
| KimiBot | Moonshot AI AI Data Scrapers | 5.9% | 3.9% | 1.6% |
| Amzn-SearchBot | Unclear at this time. AI Search Crawlers | 5.5% | 3.8% | 1.6% |
| MistralAI-Training | Mistral AI AI Data Scrapers | 5.8% | 3.9% | 1.6% |
| SBIntuitionsBot | SB Intuitions Uses data gathered in AI development and information analysis. | 5.7% | 3.9% | 1.6% |
| TikTokSpider | ByteDance LLM training. | 5.5% | 3.8% | 1.6% |
| aiHitBot | aiHit A massive, artificial intelligence/machine learning, automated system. | 5.7% | 3.9% | 1.5% |
| Amzn-User | Amazon, used for fetching web content to answer user queries through Alexa and other Amazon AI services AI Assistants | 5.5% | 3.8% | 1.5% |
| Factset_spyderbot | Factset AI model training. | 5.7% | 3.9% | 1.5% |
| PhindBot | phind AI Assistants | 5.4% | 3.9% | 1.4% |
| Bravebot | https://safe.search.brave.com/help/brave-search-crawler AI Data Providers | 5.3% | 3.9% | 1.4% |
| imageSpider | Unclear at this time. AI Data Scrapers | 5.6% | 3.9% | 1.4% |
| Google-NotebookLM | Unclear at this time. AI Assistants | 5.2% | 3.9% | 1.4% |
| LinerBot | Unclear at this time. AI Assistants | 5.5% | 3.9% | 1.3% |
| iAskBot | Unclear at this time. Undocumented AI Agents | 5.5% | 3.9% | 1.3% |
| Devin | Devin AI AI Coding Agents | 5.3% | 3.9% | 1.2% |
| EchoboxBot | Echobox Data collection to support AI-powered products. | 5.3% | 3.9% | 1.2% |
| LinkupBot | Unclear at this time. AI Search Crawlers | 5.4% | 3.9% | 1.2% |
| SemrushBot-OCOB | Semrush Crawls your site for ContentShake AI tool. | 5.2% | 3.9% | 1.2% |
| YaK | Meltwater According to the [Meltwater Consumer Intelligence page](https://www.meltwater.com/en/suite/consumer-intelligence) 'By applying AI, data science, and market research expertise to a live feed of global data sources, we transform unstructured data into actionable insights allowing better decision-making'. | 5.2% | 3.9% | 1.2% |
| YandexAdditionalBot | Yandex Scrapes/analyzes data for the YandexGPT LLM. | 5.3% | 3.9% | 1.2% |
| Panscient | Panscient Data collection and analysis using machine learning and AI. | 5.3% | 3.9% | 1.1% |
| TavilyBot | Unclear at this time. AI Data Providers | 5.3% | 3.9% | 1.1% |
| GoogleAgent-Mariner | Google AI Agents | 5.2% | 3.9% | 1.1% |
| TerraCotta | Ceramic AI AI Data Providers | 5.3% | 3.9% | 1.1% |
| AddSearchBot | Unclear at this time. AI Search Crawlers | 5.2% | 3.9% | 1.1% |
| Andibot | Andi Search engine using generative AI, AI Search Assistant | 5.1% | 3.9% | 1.1% |
| laion-huggingface-processor | Unclear at this time. AI Data Scrapers | 5.3% | 3.9% | 1.1% |
| YandexAdditional | Yandex Scrapes/analyzes data for the YandexGPT LLM. | 5.1% | 3.9% | 1.0% |
| WRTNBot | Unclear at this time. Undocumented AI Agents | 5.2% | 3.9% | 1.0% |
| NovaAct | Unclear at this time. AI Agents | 5.1% | 3.9% | 1.0% |
| wpbot | QuantumCloud Live chat support and lead generation. | 5.2% | 3.9% | 1.0% |
| amazon-kendra | Amazon Collects data for AI natural language search | 5.1% | 3.9% | 1.0% |
| CloudflareBrowserRenderingCrawler | Cloudflare that returns rendered website content for research, monitoring, and AI data workflows AI Data Providers | 5.1% | 3.9% | 1.0% |
| Crawlspace | Crawlspace AI Data Providers | 5.2% | 3.9% | 1.0% |
| GrokBot | xAI AI Search Crawlers | 5.0% | 3.9% | 1.0% |
| KlaviyoAIBot | Klaviyo AI Assistants | 5.1% | 3.9% | 1.0% |
| KunatoCrawler | Unclear at this time. Undocumented AI Agents | 5.2% | 3.9% | 1.0% |
| Manus-User | Butterfly Effect, a company based in China AI Agents | 5.1% | 3.9% | 0.9% |
| QuillBot | Quillbot Company offers AI detection, writing tools and other services. | 5.2% | 3.9% | 0.9% |
| bedrockbot | Amazon Data scraping for custom AI applications. | 5.1% | 3.9% | 0.9% |
| CloudVertexBot | Unclear at this time. AI Data Scrapers | 5.1% | 3.9% | 0.9% |
| ZanistaBot | Unclear at this time. AI Search Crawlers | 5.1% | 3.9% | 0.9% |
| AmazonBuyForMe | Amazon AI Agents | 5.1% | 3.9% | 0.9% |
| GoogleOther-Image | Google Scrapes data. | 4.8% | 3.9% | 0.9% |
| Poggio-Citations | Poggio, a company that provides AI sales enablement tools for creating tailored narratives, business cases, and account plan… AI Assistants | 5.1% | 3.9% | 0.9% |
| GoogleOther-Video | Google Scrapes data. | 4.8% | 3.9% | 0.9% |
| Thinkbot | Thinkbot Insights on AI integration and automation. | 5.1% | 3.9% | 0.9% |
| Shap-User | Unclear at this time. AI Assistants | 5.1% | 3.9% | 0.9% |
| TwinAgent | Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa… AI Agents | 5.1% | 3.9% | 0.9% |
| Spider | Unclear at this time. AI Data Scrapers | 5.1% | 3.9% | 0.9% |
| Cloudflare-AutoRAG | Cloudflare Collects data for AI search | 5.0% | 3.9% | 0.8% |
| MyCentralAIScraperBot | Unclear at this time. AI data scraper | 4.9% | 3.9% | 0.8% |
| ExaSearchBot | Exa AI Search Crawlers | 5.0% | 3.9% | 0.8% |
| Anomura | Direqt Collects data for AI search | 5.0% | 3.9% | 0.8% |
| ApifyBot | Unclear at this time. AI Data Providers | 4.9% | 3.9% | 0.8% |
| LCC | Unclear at this time. AI Data Scrapers | 4.9% | 3.9% | 0.7% |
| ApifyWebsiteContentCrawler | Unclear at this time. AI Data Providers | 4.9% | 3.9% | 0.7% |
| Google-Firebase | Google Used as part of AI apps developed by users of Google's Firebase AI products. | 4.8% | 3.9% | 0.7% |
| MistralAI-Index | Mistral AI Indexes web content for Mistral AI's search, used to answer questions in Le Chat. Per Mistral, not used for model training. | 4.8% | 3.9% | 0.7% |
| Operator | Unclear at this time. AI Agents | 4.8% | 3.9% | 0.7% |
| xAI-Grok | xAI xAI-Grok is an xAI crawler token associated with Grok content collection. | 4.8% | 3.9% | 0.7% |
| Awario | Awario AI Data Scrapers | 4.7% | 3.9% | 0.6% |
| ERNIEBot | Baidu Collects public web content for Baidu's ERNIE large language models. | 4.8% | 3.9% | 0.6% |
| Google-Agent | Unclear at this time. AI Agents | 4.6% | 3.9% | 0.6% |
| Diffbot-User | Diffbot AI Assistants | 4.8% | 3.9% | 0.6% |
| amazon-QBusiness | Unclear at this time. AI Assistants | 4.8% | 3.9% | 0.6% |
| BuddyBot | BuddyBotLearning AI Learning Companion | 4.8% | 3.9% | 0.6% |
| Kimi-User | Moonshot AI that fetches web content on behalf of users interacting with Kimi AI Assistants | 4.7% | 3.9% | 0.5% |
| OpenAI | OpenAI Unclear at this time. | 4.6% | 3.9% | 0.5% |
| OAI-AdsBot | OpenAI Validates and targets ads on ChatGPT. | 4.4% | 3.9% | 0.5% |
| LAIONDownloader | Large-scale Artificial Intelligence Open Network AI tools and models for machine learning research. | 4.7% | 3.9% | 0.5% |
| AzureAI-SearchBot | Unclear at this time. AI Search Crawlers | 4.6% | 3.9% | 0.4% |
| kagi-fetcher | Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res… AI Assistants | 4.6% | 3.9% | 0.4% |
| Claude-Code | Unclear at this time. AI Coding Agents | 4.5% | 3.9% | 0.4% |
| Code | Unclear at this time. AI Coding Agents | 4.6% | 3.9% | 0.4% |
| Reflectionbot | Reflection Undocumented AI Agents | 4.6% | 3.9% | 0.4% |
| TongyiBot | Alibaba that fetches web content for the Tongyi Qianwen assistant and related Qwen-generated answers AI Assistants | 4.5% | 3.9% | 0.4% |
| GeistHaus-PageFetcher | GeistHaus, a company developing AI systems for therapy and psychological assessment AI Assistants | 4.6% | 3.9% | 0.3% |
| UseAI | Unclear at this time. AI Assistants | 4.6% | 3.9% | 0.3% |
| YiyanBot | Baidu that fetches web content for the yiyan AI Assistants | 4.6% | 3.9% | 0.3% |
| CragCrawler | CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services AI Data Providers | 4.5% | 3.9% | 0.3% |
| Querit-SearchBot | Querit that indexes web content for their search API service, which is designed to provide real-time search results for larg… AI Data Providers | 4.5% | 3.9% | 0.3% |
| QueritBot | Querit, a company providing a search API for large language model integration AI Data Providers | 4.5% | 3.9% | 0.3% |
| QwenBot | Alibaba Collects public web content for Alibaba's Qwen (Tongyi Qianwen) large language models. | 4.5% | 3.9% | 0.3% |
| HenkBot | Unclear at this time. AI Data Providers | 4.5% | 3.9% | 0.3% |
| Kimi-SearchBot | Moonshot AI AI Search Crawlers | 4.5% | 3.9% | 0.3% |
| opencode | Unclear at this time. AI Coding Agents | 4.5% | 3.9% | 0.3% |
| AIWebIndex | Lyrenth AI Search Crawlers | 4.5% | 3.9% | 0.3% |
| Cursor | Unclear at this time. AI Coding Agents | 4.5% | 3.9% | 0.3% |
| Aranet-SearchBot | Unclear at this time. Undocumented AI Agents | 4.5% | 3.9% | 0.3% |
| DoubaoBot | ByteDance AI crawler for ByteDance's Doubao AI assistant. | 4.5% | 3.9% | 0.3% |
| KeenableBot | Keenable AI Search Crawlers | 4.5% | 3.9% | 0.3% |
| MoonshotBot | Kimi MoonshotBot is a training crawler. | 4.5% | 3.9% | 0.3% |
| Trae | Unclear at this time. AI Coding Agents | 4.5% | 3.9% | 0.3% |
| AgentDataBot | Unclear at this time. AI Data Providers | 4.5% | 3.9% | 0.2% |
| BixelBot | Unclear at this time. AI Data Providers | 4.5% | 3.9% | 0.2% |
| Mozilla-Tabstack | Mozilla that performs programmatic, AI-driven interactions with web content through Tabstack AI Data Providers | 4.5% | 3.9% | 0.2% |
| YiBot | 01.ai, an AI company founded by Kai-Fu Lee. This bot crawls the web to gather data and content for indexation and AI model training. | 4.4% | 3.9% | 0.2% |
| Hunyuan | Tencent Hunyuan is a training crawler operated by Tencent. | 4.4% | 3.9% | 0.2% |
| NotebookLM | Unclear at this time. AI Assistants | 4.4% | 3.9% | 0.2% |
| GoogleAgent-URLContext | Google that retrieves web content on behalf of Gemini API users AI Assistants | 4.4% | 3.9% | 0.2% |
| Kimi-Agent | Moonshot AI that browses websites while carrying out user-directed research and other tasks AI Agents | 4.4% | 3.9% | 0.2% |
| qodercli | Qoder that helps developers build, debug, and modify software from the terminal AI Coding Agents | 4.4% | 3.9% | 0.2% |
| NagetBot | Naget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts) AI data scraper | 4.4% | 3.9% | 0.2% |
| Google-Gemini-CLI | Unclear at this time. AI Coding Agents | 4.4% | 3.9% | 0.2% |
| newsai | Unclear at this time. AI data scraper | 4.4% | 3.9% | 0.2% |
| AgentTimes | The Agent Times Data Scraper from RSS Feeds. | 4.4% | 3.9% | 0.1% |
| Lightpanda | Anyone who downloads the Lightpanda client. Possibly being used by a Grok-adjacent organization's botnet. AI Data Scrapers | 4.3% | 3.9% | 0.1% |
| ExaUser | Unclear at this time. AI Assistants | 4.3% | 3.9% | 0.1% |
| Muse | Unclear at this time. AI Agents | 4.3% | 3.9% | 0.1% |
| ElevenlabsBot | Unclear at this time. AI Assistants | 4.3% | 3.9% | 0.0% |
| CledaraBot | Cledara that visits SaaS providers to complete customer-approved administrative tasks, including invoice collection and user managem… AI Agents | 4.3% | 3.9% | 0.0% |
| Codex | Unclear at this time. AI Coding Agents | 4.3% | 3.9% | 0.0% |
| Dot | Unclear at this time. AI Agents | 4.3% | 3.9% | 0.0% |
| HarkBot | Unclear at this time. AI Agents | 4.3% | 3.9% | 0.0% |
| Instinct | Unclear at this time. AI Agents | 4.3% | 3.9% | 0.0% |
| GrokAppAndroid | xAI AI Search Crawlers | 4.3% | 3.9% | 0.0% |
| nbot | NBot AI Search Crawlers | 4.3% | 3.9% | 0.0% |
| AGI Agent | AGI, Inc AI Agents | 4.2% | 3.9% | 0.0% |
| AI2Bot | Ai2 Content is used to train open language models. | 4.2% | 3.9% | 0.0% |
| AI2Bot-DeepResearchEval | Ai2, a non-profit AI research institute AI Assistants | 4.2% | 3.9% | 0.0% |
| Ai2Bot-Dolma | Ai2 Content is used to train open language models. | 4.2% | 3.9% | 0.0% |
| bigsur.ai | Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for busi… AI Assistants | 4.2% | 3.9% | 0.0% |
| Brightbot 1.0 | https://brightdata.com/brightbot LLM/AI training. | 4.2% | 3.9% | 0.0% |
| CartAI Agentic Commerce | CartAI that accesses online stores to support user-directed shopping and checkout workflows AI Agents | 4.2% | 3.9% | 0.0% |
| Channel3Bot | Unclear at this time. AI Search Crawlers | 4.2% | 3.9% | 0.0% |
| ChatGPT Agent | OpenAI AI Agents | 4.2% | 3.9% | 0.0% |
| Crawl4AI | Unclear at this time. Undocumented AI Agents | 4.2% | 3.9% | 0.0% |
| Datenbank Crawler | Datenbank AI Data Scrapers | 4.2% | 3.9% | 0.0% |
| Echobot Bot | Echobox AI Data Scrapers | 4.2% | 3.9% | 0.0% |
| Grok Bot | Unclear at this time. AI Agents | 4.2% | 3.9% | 0.0% |
| iaskspider/2.0 | iAsk Crawls sites to provide answers to user queries. | 4.2% | 3.9% | 0.0% |
| img2dataset | img2dataset Scrapes images for use in LLMs. | 4.2% | 3.9% | 0.0% |
| Kangaroo Bot | Unclear at this time. AI Data Scrapers | 4.2% | 3.9% | 0.0% |
| Linguee Bot | Linguee AI powered translation service | 4.2% | 3.9% | 0.0% |
| MistralAI-User/1.0 | Mistral AI Takes action based on user prompts. | 4.2% | 3.9% | 0.0% |
| netEstate Imprint Crawler | netEstate AI Data Scrapers | 4.2% | 3.9% | 0.0% |
| panscient.com | Panscient Data collection and analysis using machine learning and AI. | 4.2% | 3.9% | 0.0% |
| Payhawk Invoice Fetching Agent | Payhawk that retrieves invoices from supplier websites for users and attaches them to expense records AI Agents | 4.2% | 3.9% | 0.0% |
| Poseidon Research Crawler | Poseidon Research AI research crawler | 4.2% | 3.9% | 0.0% |
| quillbot.com | Quillbot Company offers AI detection, writing tools and other services. | 4.2% | 3.9% | 0.0% |
| Sidetrade indexer bot | Sidetrade Extracts data for a variety of uses including training AI. | 4.2% | 3.9% | 0.0% |
| Terra Cotta | Unclear at this time. AI Data Providers | 4.2% | 3.9% | 0.0% |
An agent no robots.txt names still gets a verdict: the * group's. That is why rarely named agents show the same "blocks" share: it is the share of sites closed to every unnamed crawler.