Which websites block AI crawlers
On 26 October 2026, 958 of 6,561 readable top 10k websites (14.6%; bounds 11.7%–31.8%) fully blocked GPTBot in robots.txt, and 19.3% blocked at least one AI training crawler.
Last verified 2026-10-26 21:13 UTC (daily run of 2026-10-26). Proof for that day.
Share of the top 10k websites blocking each AI bot
Panel: top 10k of the Tranco list W3ZV9, one fixed cohort. Website cohort W = 8,220 sites whose host answered on 2026-10-26 (1,780 panel hosts have no website); U = 1,659 of them could not be read. "Blocks" = robots.txt disallows every content path for that agent (by name or through *), as a share of the W − U readable sites; the bounds assume none / all of the unreadable sites block. "Names it" = robots.txt has a group for that agent. Method.
| Bot | Role | Blocks | Bounds | Names it | 30 days | 2016 reconstructed |
|---|---|---|---|---|---|---|
| GPTBot OpenAI | training · docs | 14.6% 958 of 6,561 | 11.7%–31.8% | 16.5% | – | – |
| OAI-SearchBot OpenAI | search · docs | 7.3% 480 of 6,561 | 5.8%–26.0% | 9.1% | – | – |
| ChatGPT-User OpenAI | user · docs | 8.7% 572 of 6,561 | 7.0%–27.1% | 10.4% | – | – |
| ClaudeBot Anthropic | training · docs | 13.8% 904 of 6,561 | 11.0%–31.2% | 14.2% | – | – |
| Claude-SearchBot Anthropic | search · docs | 7.2% 470 of 6,561 | 5.7%–25.9% | 5.8% | – | – |
| Claude-User Anthropic | user · docs | 7.2% 471 of 6,561 | 5.7%–25.9% | 5.7% | – | – |
| PerplexityBot Perplexity | search · docs | 10.0% 656 of 6,561 | 8.0%–28.2% | 11.3% | – | – |
| Perplexity-User Perplexity | user · docs | 7.5% 493 of 6,561 | 6.0%–26.2% | 5.6% | – | – |
| Googlebot Google | search · docs | 2.8% 185 of 6,561 | 2.3%–22.4% | 8.3% | – | 1.8% |
| Google-Extended Google | training · docs | 12.5% 821 of 6,561 | 10.0%–30.2% | 13.3% | – | – |
| Applebot-Extended Apple | training · docs | 12.1% 791 of 6,561 | 9.6%–29.8% | 10.2% | – | – |
| CCBot Common Crawl | training · docs | 15.4% 1,013 of 6,561 | 12.3%–32.5% | 14.1% | – | 3.3% |
| Bytespider ByteDance | training · docs | 15.1% 990 of 6,561 | 12.0%–32.2% | 12.7% | – | – |
| Meta-ExternalAgent Meta | training · docs | 12.4% 811 of 6,561 | 9.9%–30.0% | 10.3% | – | – |
| Amazonbot Amazon | training · docs | 11.9% 779 of 6,561 | 9.5%–29.7% | 10.1% | – | – |
Training versus search
| Websites that… | Share | Bounds | Readable sites |
|---|---|---|---|
| blocks at least one AI training crawler | 19.3% | 15.4%–35.5% | 6,561 |
| blocks at least one AI search crawler | 10.4% | 8.3%–28.5% | 6,561 |
| closes the whole site to every unnamed crawler (*) | 4.2% | 3.4%–23.6% | 6,561 |
Trend
- GPTBot
- OAI-SearchBot
- ClaudeBot
- Claude-SearchBot
- Google-Extended
- PerplexityBot
- CCBot
Solid lines: daily first-party observations, top 10k of the Tranco list W3ZV9. Dashed: reconstructed from Common Crawl's robots.txt captures (2016-09-05 to 2026-09-21), on the top 10k, sites readable in every crawl; each agent's dashed line starts at its public launch date. The two series are measured differently and are never merged. Data: shares.csv, shares-reconstructed.csv.
Other signals
| Signal | Share of sites | Sites |
|---|---|---|
| Publishes /llms.txt | 12.7% | 799 / 6,286 |
| Publishes /ai.txt | 0.2% | 11 / 6,231 |
| Publishes /.well-known/tdmrep.json | 0.4% | 25 / 5,977 |
| Reserves TDM rights (TDMRep file, header or meta) | 0.8% | 33 / 4,286 |
| Declares Content Signals (robots.txt or header) | 5.2% | 237 / 4,590 |
| Says ai-train=no in a Content Signal | 3.7% | 168 / 4,568 |
| Points to an RSL licence (robots.txt License line or Link header) | 0.2% | 8 / 4,505 |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | 0.0% | 0 / 4,987 |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | 0.0% | 1 / 4,987 |
| Sends noai / noimageai directives | 0.3% | 12 / 4,758 |
| Refused our fetcher (403 / challenge) | 11.2% | 1,121 / 10,000 |
Latest bot policy changes
| Date | Site | What changed |
|---|---|---|
| No changes recorded yet. The first daily run is the baseline; changes appear from the second day on. | ||
All changes · Atom feed · Monthly report
Why this exists
Whether a site opted out of AI training or AI search is a question with legal weight (the EU DSM Directive's machine-readable reservation, the AI Act's copyright duties) and commercial weight (licensing and pay-per-crawl). Policies change quietly. This ledger fetches the same files from the same sites every day, stores every distinct version, hashes each day into a Merkle root and anchors it in Bitcoin with OpenTimestamps, so "what did this site tell GPTBot on 3 March?" has a checkable answer. Methodology.