Claude-Web: share of top websites blocking it

On 26 October 2026, 10.5% of the readable top 10k websites fully blocked Claude-Web in robots.txt (bounds 8.4% to 28.6% over all 8,220 websites, 1,659 of them unreadable); 3.6% blocked it partly and 8.0% named it.

Last verified 2026-10-26 21:13 UTC (daily run of 2026-10-26). Proof for that day.

Operator
Anthropic
What it is
Undocumented AI Agents
Description
Claude-Web is an AI-related agent operated by Anthropic. It's currently unclear exactly what it's used for, since there's no official documentation. If you can provide more detail, please contact us. More info can be found at https://knownagents.com/agents/claude-web
robots.txt token
User-agent: Claude-Web
Listed since
2026-10-26 (in this ledger's registry)

Fully blocks

10.5%

688 sites; bounds 8.4% to 28.6%

Partly blocks

3.6%

239 sites

Name it explicitly

8.0%

522 sites; unreadable: 3,439

Over time

Share of panel sites blocking Claude-Web4 lines, 2016-09-05 to 2026-10-26; values between 2.52 and 10.49 %; latest 10.49 %.1.884.196.508.8111.12Sep 2016Mar 2019Sep 2021Apr 2024Oct 2026blocks, reconstructed (Common Crawl): 2016-09-05 UTC: 3.14 %blocks, reconstructed (Common Crawl): 2017-09-25 UTC: 3.51 %blocks, reconstructed (Common Crawl): 2018-09-24 UTC: 3.61 %blocks, reconstructed (Common Crawl): 2019-09-23 UTC: 2.80 %blocks, reconstructed (Common Crawl): 2020-09-28 UTC: 2.52 %blocks, reconstructed (Common Crawl): 2021-09-27 UTC: 2.61 %blocks, reconstructed (Common Crawl): 2022-10-03 UTC: 2.55 %blocks, reconstructed (Common Crawl): 2024-09-16 UTC: 5.53 %blocks, reconstructed (Common Crawl): 2026-09-21 UTC: 7.46 %partly blocks, reconstructed (Common Crawl): 2016-09-05 UTC: 5.04 %partly blocks, reconstructed (Common Crawl): 2017-09-25 UTC: 4.91 %partly blocks, reconstructed (Common Crawl): 2018-09-24 UTC: 4.82 %partly blocks, reconstructed (Common Crawl): 2019-09-23 UTC: 4.76 %partly blocks, reconstructed (Common Crawl): 2020-09-28 UTC: 4.82 %partly blocks, reconstructed (Common Crawl): 2021-09-27 UTC: 4.98 %partly blocks, reconstructed (Common Crawl): 2022-10-03 UTC: 4.91 %partly blocks, reconstructed (Common Crawl): 2024-09-16 UTC: 4.94 %partly blocks, reconstructed (Common Crawl): 2026-09-21 UTC: 5.35 %blocks, daily observation: 2026-10-26 UTC: 10.49 %partly blocks, daily observation: 2026-10-26 UTC: 3.64 %

"Partly blocks" counts every robots.txt whose rules for Claude-Web disallow some paths, including generic rules for all crawlers (for example /search); it is not a measure of targeted AI policy. Shares are of the readable websites (W − U: 8,220 websites, 1,659 unreadable); the bounds assume none / all of the unreadable sites block. Solid: daily first-party observations (top 10k of the Tranco list W3ZV9, one fixed cohort). Dashed: reconstructed from Common Crawl (top 10k, sites readable in every crawl), a separate series.

Reconstructed values per crawl
CrawlDateBlocksSites read
CC-MAIN-2016-362016-09-053.1%3,216
CC-MAIN-2017-392017-09-253.5%3,216
CC-MAIN-2018-392018-09-243.6%3,216
CC-MAIN-2019-392019-09-232.8%3,216
CC-MAIN-2020-402020-09-282.5%3,216
CC-MAIN-2021-392021-09-272.6%3,216
CC-MAIN-2022-402022-10-032.5%3,216
CC-MAIN-2024-382024-09-165.5%3,216
CC-MAIN-2026-392026-09-217.5%3,216

Recent blockers

Sites that started fully blocking Claude-Web since 2026-07-28, highest-ranked first.

DateSiteWhat changed
ptt.ccnow blocks Claude-Web (was allow)
xnxx.healthnow blocks Claude-Web (was allow)
manhwaweb.comnow blocks Claude-Web (was allow)
xn--12c1ezaww.comnow blocks Claude-Web (was allow)
loteriadehoy.comnow blocks Claude-Web (was allow)
elmundo.esnow blocks Claude-Web (was allow)
filmyzilla56.comnow blocks Claude-Web (was allow)
filmyzilla62.comnow blocks Claude-Web (was allow)
filmyzilla63.comnow blocks Claude-Web (was allow)
bollyflix.afnow blocks Claude-Web (was allow)
vaticannews.vanow blocks Claude-Web (was allow)
rae.esnow blocks Claude-Web (was allow)
fragrantica.comnow blocks Claude-Web (was allow)

Recent unblockers

DateSiteWhat changed
vnexpress.netnow allows Claude-Web (was block)
nova-app.eunow allows Claude-Web (was block)
olhardigital.com.brnow partly blocks Claude-Web (was block)

Sites blocking Claude-Web now

facebook.com (*) · fbcdn.net (*) · instagram.com (*) · twitter.com (*) · googlevideo.com (*) · amazon.com · netflix.com (*) · x.com (*) · pinterest.com (*) · yahoo.com · baidu.com (*) · msn.com · reddit.com (*) · chatgpt.com · skybridge.click (*) · registrar-servers.com (*) · snapchat.com · nytimes.com · flickr.com (*) · cnn.com · meraki.com (*) · bbc.com · bbc.co.uk · t.co (*) · imdb.com (*) · canva.com · reuters.com (*) · nature.com · demdex.net (*) · weibo.com (*) · amazon.co.uk · naver.com (*) · trustpilot.com (*) · netflix.net (*) · nflxvideo.net (*) · washingtonpost.com · bloomberg.com · wsj.com (*) · amazon.de · cookiedatabase.org (*) · aol.com · gcdn.co (*) · kwai.net (*) · amazon.co.jp · statista.com · behance.net · edgecdn.ru (*) · uol.com.br · ft.com · threads.com (*) · netcraze.io (*) · usatoday.com · dailymail.com · telegraph.co.uk · nintendo.com · amazon.fr · amazon.ca · techcrunch.com · sharethrough.com (*) · amazon.in

(*) blocked through the * group, not by name. Full list in current.csv.

Cite this

InfraRabbit AI Crawl Ledger, "Share of top websites blocking Claude-Web", https://infrarabbit.com/crawlers/bot/Claude-Web, retrieved 2026-10-26. CC BY 4.0.