CCBot: share of top websites blocking it

On 26 October 2026, 15.4% of the readable top 10k websites fully blocked CCBot in robots.txt (bounds 12.3% to 32.5% over all 8,220 websites, 1,659 of them unreadable); 3.6% blocked it partly and 14.1% named it.

Last verified 2026-10-26 21:13 UTC (daily run of 2026-10-26). Proof for that day.

Operator
Common Crawl Foundation
What it is
Provides open crawl dataset, used for many purposes, including Machine Learning/AI.
Description
Web archive going back to 2008. Cited in thousands of research papers per year.
robots.txt token
User-agent: CCBot
Listed since
2026-10-26 (in this ledger's registry)

Fully blocks

15.4%

1,013 sites; bounds 12.3% to 32.5%

Partly blocks

3.6%

236 sites

Name it explicitly

14.1%

926 sites; unreadable: 3,439

Over time

Share of panel sites blocking CCBot4 lines, 2016-09-05 to 2026-10-26; values between 2.99 and 15.44 %; latest 15.44 %.1.995.609.2112.8216.44Sep 2016Mar 2019Sep 2021Apr 2024Oct 2026blocks, reconstructed (Common Crawl): 2016-09-05 UTC: 3.33 %blocks, reconstructed (Common Crawl): 2017-09-25 UTC: 3.79 %blocks, reconstructed (Common Crawl): 2018-09-24 UTC: 3.92 %blocks, reconstructed (Common Crawl): 2019-09-23 UTC: 3.20 %blocks, reconstructed (Common Crawl): 2020-09-28 UTC: 2.99 %blocks, reconstructed (Common Crawl): 2021-09-27 UTC: 3.08 %blocks, reconstructed (Common Crawl): 2022-10-03 UTC: 3.05 %blocks, reconstructed (Common Crawl): 2024-09-16 UTC: 9.42 %blocks, reconstructed (Common Crawl): 2026-09-21 UTC: 13.87 %partly blocks, reconstructed (Common Crawl): 2016-09-05 UTC: 5.04 %partly blocks, reconstructed (Common Crawl): 2017-09-25 UTC: 4.91 %partly blocks, reconstructed (Common Crawl): 2018-09-24 UTC: 4.82 %partly blocks, reconstructed (Common Crawl): 2019-09-23 UTC: 4.73 %partly blocks, reconstructed (Common Crawl): 2020-09-28 UTC: 4.79 %partly blocks, reconstructed (Common Crawl): 2021-09-27 UTC: 4.94 %partly blocks, reconstructed (Common Crawl): 2022-10-03 UTC: 4.88 %partly blocks, reconstructed (Common Crawl): 2024-09-16 UTC: 4.85 %partly blocks, reconstructed (Common Crawl): 2026-09-21 UTC: 5.25 %blocks, daily observation: 2026-10-26 UTC: 15.44 %partly blocks, daily observation: 2026-10-26 UTC: 3.60 %

"Partly blocks" counts every robots.txt whose rules for CCBot disallow some paths, including generic rules for all crawlers (for example /search); it is not a measure of targeted AI policy. Shares are of the readable websites (W − U: 8,220 websites, 1,659 unreadable); the bounds assume none / all of the unreadable sites block. Solid: daily first-party observations (top 10k of the Tranco list W3ZV9, one fixed cohort). Dashed: reconstructed from Common Crawl (top 10k, sites readable in every crawl), a separate series.

Reconstructed values per crawl
CrawlDateBlocksSites read
CC-MAIN-2016-362016-09-053.3%3,216
CC-MAIN-2017-392017-09-253.8%3,216
CC-MAIN-2018-392018-09-243.9%3,216
CC-MAIN-2019-392019-09-233.2%3,216
CC-MAIN-2020-402020-09-283.0%3,216
CC-MAIN-2021-392021-09-273.1%3,216
CC-MAIN-2022-402022-10-033.0%3,216
CC-MAIN-2024-382024-09-169.4%3,216
CC-MAIN-2026-392026-09-2113.9%3,216

Recent blockers

Sites that started fully blocking CCBot since 2026-07-28, highest-ranked first.

DateSiteWhat changed
No changes recorded yet. The first daily run is the baseline; changes appear from the second day on.

Recent unblockers

DateSiteWhat changed
No changes recorded yet. The first daily run is the baseline; changes appear from the second day on.

Sites blocking CCBot now

facebook.com (*) · fbcdn.net (*) · instagram.com (*) · twitter.com (*) · googlevideo.com (*) · amazon.com · netflix.com (*) · x.com (*) · pinterest.com (*) · tiktok.com · yahoo.com · baidu.com (*) · amazonvideo.com · vimeo.com · msn.com · reddit.com (*) · chatgpt.com · skybridge.click (*) · registrar-servers.com (*) · t-online.de · snapchat.com · tumblr.com · nytimes.com · gravatar.com · flickr.com (*) · ebay.com · theguardian.com · cnn.com · bbc.com · forbes.com · forter.com · bbc.co.uk · t.co (*) · weather.com · imdb.com (*) · who.int · canva.com · launchpad.net · reuters.com (*) · nature.com · demdex.net (*) · weibo.com (*) · amazon.co.uk · naver.com (*) · netflix.net (*) · nflxvideo.net (*) · espn.com · washingtonpost.com · bloomberg.com · wsj.com (*) · globo.com · amazon.de · cookiedatabase.org (*) · calendly.com · aol.com · gcdn.co (*) · pixabay.com · businessinsider.com · kwai.net (*) · goodreads.com

(*) blocked through the * group, not by name. Full list in current.csv.

Cite this

InfraRabbit AI Crawl Ledger, "Share of top websites blocking CCBot", https://infrarabbit.com/crawlers/bot/CCBot, retrieved 2026-10-26. CC BY 4.0.