chatgpt.com: what it tells AI crawlers

On 27 October 2026, chatgpt.com's robots.txt fully blocked Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://chatgpt.com/robots.txt
unchanged since 2026-10-26 · sha256 0676a43da83b5eb1… · view as recorded

The * group

partial applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 22c1e86846f6588b…

# https://www.robotstxt.org/robotstxt.html

User-agent: CCBot
Disallow: /

User-agent: img2dataset
Disallow: /

User-agent: Google-Extended
Disallow: /

# Prevent shared pages and GPTs from being used as Google Ads destinations
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
Disallow: /s/
Disallow: /g/

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: Omgili
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: magpie-crawler
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity‑User
Disallow: /

# General rules for all other bots
User-agent: *

# Place allows first to avoid bots skipping after Disallow: /
# Allow exactly the homepage
Allow: /$
# Allow the homepage with any query parameters
Allow: /?*
Allow: /api/share/og/
Allow: /g/
Allow: /s/
Allow: /gg/v/
Allow: /m/
Allow: /share/
Allow: /maps$
Allow: /maps/
Allow: /*/maps$
Allow: /*/maps/
Allow: /canvas/shared/
Allow: /*/images
Allow: /images
Allow: /*/library
Allow: /library
Allow: /favicon.ico
Allow: /assets/favicon
Allow: /cdn/assets/favicon
Allow: /cdn/assets/
Allow: /auth/
Allow: /gpts$
Allow: /codex
Allow: /*/codex
Allow: /search$
Allow: /backend-anon/
Allow: /public-api/
Allow: /sitemap.xml
Allow: /marketing-sitemap.xml
Allow: /images-sitemap.xml
Allow: /writing-tools-sitemap.xml
Allow: /football-sitemap.xml
Allow: /acquisition-landing-pages-sitemap.xml
Allow: /100chats
Allow: /api/public_content/
Allow: /backend-api/public_content/
Allow: /?ref=dotcom
# Static Landing Pages
Allow: /overview
Allow: /*/overview
Allow: /features
Allow: /*/features
Allow: /health$
Allow: /health/$
Allow: /*/health$
Allow: /*/health/$
Allow: /voice$
Allow: /voice/$
Allow: /*/voice$
Allow: /*/voice/$
Allow: /apps
Allow: /*/apps
Allow: /finances
Allow: /*/finances
Allow: /shopping
Allow: /*/shopping
Allow: /use-cases
Allow: /*/use-cases
Allow: /learn
Allow: /*/learn
Allow: /business
Allow: /*/business
Allow: /pricing
Allow: /*/pricing
Allow: /download
Allow: /*/download
Allow: /students
Allow: /students/2026
Allow: /*/college-students
Allow: /college-students
Allow: /contact-sales
Allow: /*/contact-sales
Allow: /100chats-project
Allow: /*/100chats-project
Allow: /merchants
Allow: /*/merchants
Allow: /parent-resources
Allow: /*/parent-resources
Allow: /plans
Allow: /*/plans
Allow: /plugins
Allow: /translate
Allow: /*/translate
Allow: /writing
Allow: /*/writing
Allow: /ask/
Allow: /futures
Allow: /*/futures
Allow: /football
Allow: /*/football
Allow: /remote
Allow: /*/remote
Allow: /work
Allow: /*/work
Allow: /sites
Allow: /import-to-chatgpt
Allow: /*/import-to-chatgpt
# Exact locale specific homepages
Allow: /am/$
Allow: /ar/$
Allow: /bg-BG/$
Allow: /bn-BD/$
Allow: /bs-BA/$
Allow: /ca-ES/$
Allow: /cs-CZ/$
Allow: /da-DK/$
Allow: /de-DE/$
Allow: /el-GR/$
Allow: /es-ES/$
Allow: /es-419/$
Allow: /et-EE/$
Allow: /fi-FI/$
Allow: /fr-FR/$
Allow: /fr-CA/$
Allow: /gu-IN/$
Allow: /hi-IN/$
Allow: /hr-HR/$
Allow: /hu-HU/$
Allow: /hy-AM/$
Allow: /id-ID/$
Allow: /is-IS/$
Allow: /it-IT/$
Allow: /ja-JP/$
Allow: /ka-GE/$
Allow: /kk/$
Allow: /kn-IN/$
Allow: /ko-KR/$
Allow: /lt/$
Allow: /lv-LV/$
Allow: /mk-MK/$
Allow: /ml/$
Allow: /mn/$
Allow: /mr-IN/$
Allow: /ms-MY/$
Allow: /my-MM/$
Allow: /nb-NO/$
Allow: /nl-NL/$
Allow: /pa/$
Allow: /pl-PL/$
Allow: /pt-BR/$
Allow: /pt-PT/$
Allow: /ro-RO/$
Allow: /ru-RU/$
Allow: /sk-SK/$
Allow: /sl-SI/$
Allow: /so-SO/$
Allow: /sq-AL/$
Allow: /sr-RS/$
Allow: /sv-SE/$
Allow: /sw-TZ/$
Allow: /ta-IN/$
Allow: /te-IN/$
Allow: /th-TH/$
Allow: /tl/$
Allow: /tr-TR/$
Allow: /uk-UA/$
Allow: /ur/$
Allow: /vi-VN/$
Allow: /zh-CN/$
Allow: /zh-TW/$
Allow: /zh-HK/$

# Now block everything else
Disallow: /

# Specific disallows (redundant for some bots, but still useful for those that respect precedence)
Disallow: /auth/logout
Disallow: /auth/login?*
Disallow: /backend-anon/sentinel/*
Disallow: /backend-anon/conversation$
Disallow: /account-link/*

Sitemap: https://chatgpt.com/sitemap.xml
Sitemap: https://chatgpt.com/marketing-sitemap.xml
Sitemap: https://chatgpt.com/images-sitemap.xml
Sitemap: https://chatgpt.com/writing-tools-sitemap.xml
Sitemap: https://chatgpt.com/football-sitemap.xml
Sitemap: https://chatgpt.com/acquisition-landing-pages-sitemap.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of chatgpt.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/chatgpt.com