Googlebot: share of top websites blocking it

On 26 October 2026, 2.8% of the readable top 10k websites fully blocked Googlebot in robots.txt (bounds 2.3% to 22.4% over all 8,220 websites, 1,659 of them unreadable); 3.8% blocked it partly and 8.3% named it.

Last verified 2026-10-26 21:13 UTC (daily run of 2026-10-26). Proof for that day.

Operator
unclear
What it is
–
robots.txt token
User-agent: Googlebot
Listed since
2026-10-26 (in this ledger's registry)

Fully blocks

2.8%

185 sites; bounds 2.3% to 22.4%

Partly blocks

3.8%

248 sites

Name it explicitly

8.3%

546 sites; unreadable: 3,439

Over time

Share of panel sites blocking Googlebot4 lines, 2016-09-05 to 2026-10-26; values between 1.59 and 5.22 %; latest 2.82 %.1.292.353.404.465.51Sep 2016Mar 2019Sep 2021Apr 2024Oct 2026blocks, reconstructed (Common Crawl): 2016-09-05 UTC: 1.83 %blocks, reconstructed (Common Crawl): 2017-09-25 UTC: 2.46 %blocks, reconstructed (Common Crawl): 2018-09-24 UTC: 2.61 %blocks, reconstructed (Common Crawl): 2019-09-23 UTC: 1.90 %blocks, reconstructed (Common Crawl): 2020-09-28 UTC: 1.80 %blocks, reconstructed (Common Crawl): 2021-09-27 UTC: 1.90 %blocks, reconstructed (Common Crawl): 2022-10-03 UTC: 1.74 %blocks, reconstructed (Common Crawl): 2024-09-16 UTC: 1.59 %blocks, reconstructed (Common Crawl): 2026-09-21 UTC: 1.62 %partly blocks, reconstructed (Common Crawl): 2016-09-05 UTC: 4.88 %partly blocks, reconstructed (Common Crawl): 2017-09-25 UTC: 4.76 %partly blocks, reconstructed (Common Crawl): 2018-09-24 UTC: 4.66 %partly blocks, reconstructed (Common Crawl): 2019-09-23 UTC: 4.54 %partly blocks, reconstructed (Common Crawl): 2020-09-28 UTC: 4.57 %partly blocks, reconstructed (Common Crawl): 2021-09-27 UTC: 4.66 %partly blocks, reconstructed (Common Crawl): 2022-10-03 UTC: 4.76 %partly blocks, reconstructed (Common Crawl): 2024-09-16 UTC: 4.91 %partly blocks, reconstructed (Common Crawl): 2026-09-21 UTC: 5.22 %blocks, daily observation: 2026-10-26 UTC: 2.82 %partly blocks, daily observation: 2026-10-26 UTC: 3.78 %

"Partly blocks" counts every robots.txt whose rules for Googlebot disallow some paths, including generic rules for all crawlers (for example /search); it is not a measure of targeted AI policy. Shares are of the readable websites (W − U: 8,220 websites, 1,659 unreadable); the bounds assume none / all of the unreadable sites block. Solid: daily first-party observations (top 10k of the Tranco list W3ZV9, one fixed cohort). Dashed: reconstructed from Common Crawl (top 10k, sites readable in every crawl), a separate series.

Reconstructed values per crawl
CrawlDateBlocksSites read
CC-MAIN-2016-362016-09-051.8%3,216
CC-MAIN-2017-392017-09-252.5%3,216
CC-MAIN-2018-392018-09-242.6%3,216
CC-MAIN-2019-392019-09-231.9%3,216
CC-MAIN-2020-402020-09-281.8%3,216
CC-MAIN-2021-392021-09-271.9%3,216
CC-MAIN-2022-402022-10-031.7%3,216
CC-MAIN-2024-382024-09-161.6%3,216
CC-MAIN-2026-392026-09-211.6%3,216

Recent blockers

Sites that started fully blocking Googlebot since 2026-07-28, highest-ranked first.

DateSiteWhat changed
kasserver.comnow blocks Googlebot (was allow)

Recent unblockers

DateSiteWhat changed
No changes recorded yet. The first daily run is the baseline; changes appear from the second day on.

Sites blocking Googlebot now

fbcdn.net · googlevideo.com (*) · reddit.com (*) · skybridge.click (*) · registrar-servers.com (*) · demdex.net (*) · naver.com (*) · cookiedatabase.org (*) · gcdn.co (*) · edgecdn.ru (*) · netcraze.io (*) · sharethrough.com · hstgr.net (*) · imgsmail.ru (*) · hihonorcloud.com (*) · quantserve.com · creativecdn.com (*) · crwdcntrl.net (*) · vkontakte.ru (*) · sentinelone.net (*) · adform.net (*) · daum.net (*) · tiktokw.us (*) · xiaohongshu.com · mailchi.mp (*) · btloader.com (*) · namecheaphosting.com (*) · jamfcloud.com (*) · rocketmoney.dev (*) · dextrodedenda.top · namu.wiki (*) · stbid.ru (*) · kakao.com (*) · exhentai.org (*) · aka.ms (*) · firebaseio.com (*) · keirin.jp (*) · sportfilm800.com (*) · feinidi.com · ind.xxx (*) · ad-contents.jp · qcloud.com (*) · adrta.com (*) · weibo.cn (*) · clevertap-prod.com (*) · msappproxy.net (*) · tappx.com (*) · secureserver.net (*) · openssl.org (*) · ad.gt (*) · quantcount.com · vtb.ru (*) · sureserver.com (*) · pxf.io (*) · co.com (*) · _wildcard_.ph (*) · expressapisv2.net (*) · urldefense.com (*) · huntress.io (*) · permutive.app (*)

(*) blocked through the * group, not by name. Full list in current.csv.

Cite this

InfraRabbit AI Crawl Ledger, "Share of top websites blocking Googlebot", https://infrarabbit.com/crawlers/bot/Googlebot, retrieved 2026-10-26. CC BY 4.0.