GPTBot: share of top websites blocking it

On 27 October 2026, 15.0% of the readable top 10k websites fully blocked GPTBot in robots.txt (bounds 12.0% to 32.1% over all 8,219 websites, 1,657 of them unreadable); 3.7% blocked it partly and 16.9% named it.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Operator
OpenAI
What it is
Scrapes data to train OpenAI's products.
Description
Data is used to train current and future models, removed paywalled data, PII and data that violates the company's policies.
robots.txt token
User-agent: GPTBot
Listed since
2026-10-26 (in this ledger's registry)

Fully blocks

15.0%

983 sites; bounds 12.0% to 32.1%

Partly blocks

3.7%

243 sites

Name it explicitly

16.9%

1,111 sites; unreadable: 3,438

Over time

Share of panel sites blocking GPTBot4 lines, 2016-09-05 to 2026-10-27; values between 2.58 and 14.98 %; latest 14.98 %.1.595.188.7812.3815.97Sep 2016Mar 2019Oct 2021Apr 2024Oct 2026blocks, reconstructed (Common Crawl): 2016-09-05 UTC: 3.79 %blocks, reconstructed (Common Crawl): 2016-10-03 UTC: 2.58 %blocks, reconstructed (Common Crawl): 2017-09-25 UTC: 3.66 %blocks, reconstructed (Common Crawl): 2018-09-24 UTC: 3.97 %blocks, reconstructed (Common Crawl): 2019-09-23 UTC: 2.82 %blocks, reconstructed (Common Crawl): 2020-09-28 UTC: 2.68 %blocks, reconstructed (Common Crawl): 2021-09-27 UTC: 2.91 %blocks, reconstructed (Common Crawl): 2022-10-03 UTC: 2.64 %blocks, reconstructed (Common Crawl): 2023-10-02 UTC: 9.83 %blocks, reconstructed (Common Crawl): 2024-09-16 UTC: 12.61 %blocks, reconstructed (Common Crawl): 2025-09-15 UTC: 12.47 %blocks, reconstructed (Common Crawl): 2026-09-21 UTC: 13.91 %partly blocks, reconstructed (Common Crawl): 2016-09-05 UTC: 4.78 %partly blocks, reconstructed (Common Crawl): 2016-10-03 UTC: 4.97 %partly blocks, reconstructed (Common Crawl): 2017-09-25 UTC: 4.63 %partly blocks, reconstructed (Common Crawl): 2018-09-24 UTC: 4.30 %partly blocks, reconstructed (Common Crawl): 2019-09-23 UTC: 4.38 %partly blocks, reconstructed (Common Crawl): 2020-09-28 UTC: 4.28 %partly blocks, reconstructed (Common Crawl): 2021-09-27 UTC: 4.24 %partly blocks, reconstructed (Common Crawl): 2022-10-03 UTC: 4.32 %partly blocks, reconstructed (Common Crawl): 2023-10-02 UTC: 4.10 %partly blocks, reconstructed (Common Crawl): 2024-09-16 UTC: 3.81 %partly blocks, reconstructed (Common Crawl): 2025-09-15 UTC: 4.17 %partly blocks, reconstructed (Common Crawl): 2026-09-21 UTC: 4.12 %blocks, daily observation: 2026-10-26 UTC: 14.60 %blocks, daily observation: 2026-10-27 UTC: 14.98 %partly blocks, daily observation: 2026-10-26 UTC: 3.72 %partly blocks, daily observation: 2026-10-27 UTC: 3.70 %

"Partly blocks" counts every robots.txt whose rules for GPTBot disallow some paths, including generic rules for all crawlers (for example /search); it is not a measure of targeted AI policy. Shares are of the readable websites (W − U: 8,219 websites, 1,657 unreadable); the bounds assume none / all of the unreadable sites block. Solid: daily first-party observations (top 10k of the Tranco list W3ZV9, one fixed cohort). Dashed: reconstructed from Common Crawl (top 10k), a separate series.

Reconstructed values per crawl
CrawlDateBlocksSites read
CC-MAIN-2016-362016-09-053.8%5,355
CC-MAIN-2016-402016-10-032.6%5,235
CC-MAIN-2017-392017-09-253.7%5,355
CC-MAIN-2018-392018-09-244.0%5,768
CC-MAIN-2019-392019-09-232.8%5,773
CC-MAIN-2020-402020-09-282.7%5,860
CC-MAIN-2021-392021-09-272.9%5,914
CC-MAIN-2022-402022-10-032.6%5,692
CC-MAIN-2023-402023-10-029.8%5,930
CC-MAIN-2024-382024-09-1612.6%6,065
CC-MAIN-2025-382025-09-1512.5%6,045
CC-MAIN-2026-392026-09-2113.9%5,868

Recent blockers

Sites that started fully blocking GPTBot since 2026-07-29, highest-ranked first.

DateSiteWhat changed
ptt.ccnow blocks GPTBot (was allow)
xnxx.healthnow blocks GPTBot (was allow)
manhwaweb.comnow blocks GPTBot (was allow)
xn--12c1ezaww.comnow blocks GPTBot (was allow)
loteriadehoy.comnow blocks GPTBot (was allow)
kasserver.comnow blocks GPTBot (was allow)
filmyzilla56.comnow blocks GPTBot (was allow)
filmyzilla62.comnow blocks GPTBot (was allow)
filmyzilla63.comnow blocks GPTBot (was allow)
bollyflix.afnow blocks GPTBot (was allow)
vaticannews.vanow blocks GPTBot (was allow)
swwoittlv.ccnow blocks GPTBot (was allow)
lapresse.canow blocks GPTBot (was allow)
rae.esnow blocks GPTBot (was allow)
dmitriy-mevedev-tor.topnow blocks GPTBot (was allow)
beatport.comnow blocks GPTBot (was allow)
restspan.comnow blocks GPTBot (was allow)
bokepi.comnow blocks GPTBot (was allow)
emojicombos.comnow blocks GPTBot (was allow)
loudecho.ainow blocks GPTBot (was allow)
olympusxyz.comnow blocks GPTBot (was allow)
momvids.comnow blocks GPTBot (was allow)
porn-comic.comnow blocks GPTBot (was allow)
tamilgun.nownow blocks GPTBot (was allow)
cdnfonts.comnow blocks GPTBot (was allow)

Recent unblockers

DateSiteWhat changed
vnexpress.netnow allows GPTBot (was block)
zimbra.comnow allows GPTBot (was block)
nova-app.eunow allows GPTBot (was block)
resetera.comnow allows GPTBot (was block)
olhardigital.com.brnow partly blocks GPTBot (was block)

Sites blocking GPTBot now

facebook.com · fbcdn.net · instagram.com · twitter.com (*) · googlevideo.com (*) · amazon.com · whatsapp.net · x.com (*) · pinterest.com (*) · tiktok.com · whatsapp.com · yahoo.com · baidu.com (*) · msn.com · reddit.com (*) · skybridge.click (*) · registrar-servers.com (*) · t-online.de · snapchat.com · nytimes.com · gravatar.com · flickr.com (*) · ebay.com · cnn.com · bbc.com · forbes.com · bbc.co.uk · t.co (*) · imdb.com (*) · canva.com · launchpad.net · reuters.com (*) · nature.com · demdex.net (*) · alibaba.com · weibo.com (*) · amazon.co.uk · naver.com (*) · espn.com · bloomberg.com · globo.com · amazon.de · cookiedatabase.org (*) · go.com · claude.ai · aol.com · gcdn.co (*) · pixabay.com · kwai.net (*) · goodreads.com · quora.com · amazon.co.jp · statista.com · aliexpress.com · behance.net · edgecdn.ru (*) · tandfonline.com · tripadvisor.com · threads.com · netcraze.io (*)

(*) blocked through the * group, not by name. Full list in current.csv.

Cite this

InfraRabbit AI Crawl Ledger, "Share of top websites blocking GPTBot", https://infrarabbit.com/crawlers/bot/GPTBot, retrieved 2026-10-27. CC BY 4.0.