imdb.com: what it tells AI crawlers

On 27 October 2026, imdb.com's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 1k (Tranco)
robots.txt: ok (HTTP 200) · https://www.imdb.com/robots.txt
unchanged since 2026-10-26 · sha256 8c80a1f3076cb1ea… · view as recorded
The * group: blocks applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	blocks	through the * group
OAI-SearchBot	blocks	through the * group
ChatGPT-User	blocks	through the * group
ClaudeBot	blocks	through the * group
Claude-SearchBot	blocks	through the * group
Claude-User	blocks	through the * group
PerplexityBot	blocks	through the * group
Perplexity-User	blocks	through the * group
Googlebot	allows	named in its own group
Google-Extended	blocks	through the * group
Applebot-Extended	blocks	through the * group
CCBot	blocks	through the * group
Bytespider	blocks	through the * group
Meta-ExternalAgent	blocks	through the * group
Amazonbot	allows	named in its own group
Amzn-SearchBot	allows	named in its own group
Amzn-User	allows	named in its own group
Applebot	allows	named in its own group
DuckAssistBot	allows	named in its own group
facebookexternalhit	allows	named in its own group

Other signals

Publishes /llms.txt	not observed
Publishes /ai.txt	not observed
Publishes /.well-known/tdmrep.json	not observed
Reserves TDM rights (TDMRep file, header or meta)	not observed
Declares Content Signals (robots.txt or header)	not observed
Says ai-train=no in a Content Signal	not observed
Points to an RSL licence (robots.txt License line or Link header)	not observed
Answers with an x402 payment offer (lower bound: only the homepage is probed)	not observed
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	not observed
Sends noai / noimageai directives	not observed
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 476cf8066579839c…

# robots.txt for IMDb properties
# [ images/legacy/robots.txt ]
#
# Limit ScoutJet's crawl rate 
#
User-agent: ScoutJet
Crawl-delay: 3
#
#
# Yahoo!
User-agent: Slurp
Crawl-delay: .1
Disallow: /tvschedule
Disallow: /ActorSearch
Disallow: /ActressSearch
Disallow: /AddRecommendation
Disallow: /ads/
Disallow: /AlternateVersions
Disallow: /AName
Disallow: /Awards
Disallow: /BAgent
Disallow: /Ballot/
Disallow: /BornInYear
Disallow: /BornWhere
Disallow: /BPublicity
Disallow: /BQuotes
Disallow: /BTrivia
Disallow: /BusinessThisDay
Disallow: /BWorks
Disallow: /careers
Disallow: /help/show_leaf?careeratimdb
Disallow: /CommentsAuthor
Disallow: /CommentsEnter
Disallow: /CommentsIndex
Disallow: /Companies
Disallow: /CrazyCredits
Disallow: /Credits
Disallow: /DiedInYear
Disallow: /DiedWhere
Disallow: /DVD
Disallow: /ExciteTitle
Disallow: /Find
Disallow: /FName
Disallow: /GName
Disallow: /Guests
Disallow: /harvest_me
Disallow: /HelpPage
Disallow: /Icons/
Disallow: /JointVentures
Disallow: /Laserdisc
Disallow: /List
Disallow: /Literature
Disallow: /Locations
Disallow: /LocationTree
Disallow: /Lookup
Disallow: /M/
Disallow: /Maltin
Disallow: /MarriedInYear
Disallow: /MetaSearch
Disallow: /Mlinks
Disallow: /More
Disallow: /Movies
Disallow: /Movies/
Disallow: /MyMovies
Disallow: /mymovies/
Disallow: /name_pick_n_mix
Disallow: /Nsearch
Disallow: /NUrls
Disallow: /OnThisDay
Disallow: /Ontv
Disallow: /OnTV
Disallow: /Overlap
Disallow: /Pawards
Disallow: /pick_n_mix
Disallow: /PName
Disallow: /Posters
Disallow: /prepare_data
Disallow: /Psales
Disallow: /Quiz
Disallow: /r/
Disallow: /ra/
Disallow: /Ratings
Disallow: /rd/
Disallow: /Recommendations
Disallow: /register
Disallow: /ReleaseDates
Disallow: /ReleasedInYear
Disallow: /Reviews
Disallow: /rg/
Disallow: /ri/
Disallow: /RName
Disallow: /Sales
Disallow: /SearchAwards
Disallow: /SearchBios
Disallow: /SearchBusiness
Disallow: /SearchCrazy
Disallow: /SearchDVD
Disallow: /SearchGoofs
Disallow: /SearchLaserdisc
Disallow: /SearchLiterature
Disallow: /SearchPlots
Disallow: /SearchPlotWriters
Disallow: /SearchQuotes
Disallow: /SearchRatios
Disallow: /SearchSongs
Disallow: /SearchTaglines
Disallow: /SearchTechnical
Disallow: /SearchTrivia
Disallow: /SearchVersions
Disallow: /ShowAll
Disallow: /Showing
Disallow: /SName
Disallow: /Soundtracks
Disallow: /Taglines
Disallow: /Tawards
Disallow: /Technical
Disallow: /tiger_redirect
Disallow: /Title/ASIN
Disallow: /TitleBrowse
Disallow: /Trailers
Disallow: /Tsearch
Disallow: /TUrls
Disallow: /VName
Disallow: /Vote
Disallow: /WorkedWith
Disallow: /updates
Disallow: /board
Disallow: /boards
Disallow: /name/*/board
Disallow: /title/*/board
Disallow: /character/*/select-*
#
# Everyone else
#
User-agent: *
Disallow: /tvschedule
Disallow: /ActorSearch
Disallow: /ActressSearch
Disallow: /AddRecommendation
Disallow: /ads/
Disallow: /AlternateVersions
Disallow: /AName
Disallow: /Awards
Disallow: /BAgent
Disallow: /Ballot/
Disallow: /BornInYear
Disallow: /BornWhere
Disallow: /BPublicity
Disallow: /BQuotes
Disallow: /BTrivia
Disallow: /BusinessThisDay
Disallow: /BWorks
Disallow: /careers
Disallow: /help/show_leaf?careeratimdb
Disallow: /CommentsAuthor
Disallow: /CommentsEnter
Disallow: /CommentsIndex
Disallow: /Companies
Disallow: /CrazyCredits
Disallow: /Credits
Disallow: /DiedInYear
Disallow: /DiedWhere
Disallow: /DVD
Disallow: /ExciteTitle
Disallow: /Find
Disallow: /FName
Disallow: /GName
Disallow: /Guests
Disallow: /harvest_me
Disallow: /HelpPage
Disallow: /Icons/
Disallow: /JointVentures
Disallow: /Laserdisc
Disallow: /List
Disallow: /Literature
Disallow: /Locations
Disallow: /LocationTree
Disallow: /Lookup
Disallow: /M/
Disallow: /Maltin
Disallow: /MarriedInYear
Disallow: /MetaSearch
Disallow: /Mlinks
Disallow: /More
Disallow: /Movies
Disallow: /Movies/
Disallow: /MyMovies
Disallow: /mymovies/
Disallow: /name_pick_n_mix
Disallow: /Nsearch
Disallow: /NUrls
Disallow: /OnThisDay
Disallow: /Ontv
Disallow: /OnTV
Disallow: /Overlap
Disallow: /Pawards
Disallow: /pick_n_mix
Disallow: /PName
Disallow: /Posters
Disallow: /prepare_data
Disallow: /Psales
Disallow: /Quiz
Disallow: /r/
Disallow: /ra/
Disallow: /Ratings
Disallow: /rd/
Disallow: /Recommendations
Disallow: /register
Disallow: /ReleaseDates
Disallow: /ReleasedInYear
Disallow: /Reviews
Disallow: /rg/
Disallow: /ri/
Disallow: /RName
Disallow: /Sales
Disallow: /SearchAwards
Disallow: /SearchBios
Disallow: /SearchBusiness
Disallow: /SearchCrazy
Disallow: /SearchDVD
Disallow: /SearchGoofs
Disallow: /SearchLaserdisc
Disallow: /SearchLiterature
Disallow: /SearchPlots
Disallow: /SearchPlotWriters
Disallow: /SearchQuotes
Disallow: /SearchRatios
Disallow: /SearchSongs
Disallow: /SearchTaglines
Disallow: /SearchTechnical
Disallow: /SearchTrivia
Disallow: /SearchVersions
Disallow: /ShowAll
Disallow: /Showing
Disallow: /SName
Disallow: /Soundtracks
Disallow: /Taglines
Disallow: /Tawards
Disallow: /Technical
Disallow: /tiger_redirect
Disallow: /Title/ASIN
Disallow: /TitleBrowse
Disallow: /Trailers
Disallow: /Tsearch
Disallow: /TUrls
Disallow: /VName
Disallow: /Vote
Disallow: /WorkedWith
Disallow: /updates
Disallow: /board
Disallow: /boards
Disallow: /name/*/board
Disallow: /title/*/board
Disallow: /user/*/boards
Disallow: /user/*/boards/
Disallow: /lists/tt*
Disallow: /lists/nm*
Disallow: /character/*/select-*
#
#
Sitemap: http://www.imdb.com/sitemap_US_index.xml.gz

robots.txt versions

From	Through	State	Version
2026-10-26	2026-10-27	ok 200	`8c80a1f3076c`

Timeline of changes

Date	What changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	www.imdb.com	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2025-09-15 CC-MAIN-2025-38	www.imdb.com	ok view	blocks	allows	unknown	allows	allows	unknown	allows
2024-09-16 CC-MAIN-2024-38	www.imdb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2023-10-02 CC-MAIN-2023-40	www.imdb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2022-10-03 CC-MAIN-2022-40	www.imdb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2021-09-27 CC-MAIN-2021-39	www.imdb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	www.imdb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2019-09-23 CC-MAIN-2019-39	www.imdb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-09-24 CC-MAIN-2018-39	www.imdb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	www.imdb.com	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2017-01-23 CC-MAIN-2017-04	imdb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-10-03 CC-MAIN-2016-40	imdb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-09-05 CC-MAIN-2016-36	imdb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows

Evidence

Every daily observation of imdb.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/imdb.com