espn.com: what it tells AI crawlers

On 28 October 2026, espn.com's robots.txt fully blocked GPTBot, Google-Extended and CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.espn.com/robots.txt
unchanged since 2026-10-26 · sha256 69e57dadcb6f535e… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-28

Other signals

robots.txt as recorded sha256 a0a09d0c800c6b63…

# robots.txt for www.espn.com

User-agent: *
Disallow: */_/scoreboard/
Disallow: */_/week/
Disallow: */_/*/year/
Disallow: */admin/
Disallow: */boxscore?
Disallow: */calendar/
Disallow: */cat/
Disallow: */conversation?
Disallow: */conversation/
Disallow: */conversations?
Disallow: */conversations/
Disallow: */databaseresults/
Disallow: */date/
Disallow: */deportes/
Disallow: */flash/
Disallow: */login?
Disallow: */login/
Disallow: */order/
Disallow: */photoId/
Disallow: */playbyplay?
Disallow: /watch/*?*lang=
Disallow: /watch/*?*rb=
Disallow: /watch/*?*country=
Disallow: /watch/*?*search=
Disallow: /watch/*?*device=
Disallow: /watch/*?*partner=
Disallow: /watch/*?*redirected=
Disallow: /watch/*?*om-navmethod=
Disallow: */player/advancedstats
Disallow: */player/batvspitch
Disallow: */player/hollinger
Disallow: */player/matches
Disallow: */player/news
Disallow: */player/splits
Disallow: */player/stats
Disallow: */fighter/history
Disallow: */fighter/stats
Disallow: */print?id
Disallow: */printschedule/
Disallow: */search?
Disallow: */search/
Disallow: */season/200
Disallow: */seasontype/
Disallow: */sort/*
Disallow: */split/
Disallow: */start/
Disallow: */swf/
Disallow: */type/
Disallow: */util/
Disallow: */view/
Disallow: */webslices/
Disallow: */year/200
Disallow: /*?ex_cid=
Disallow: /*?innings=
Disallow: /*?user=
Disallow: /*&action=login
Disallow: /*&full=
Disallow: /*&order=
Disallow: /*&photoId=
Disallow: /*&sort=*
Disallow: /ad/
Disallow: /broadband/
Disallow: /cgi
Disallow: /community/
Disallow: /composer/
Disallow: /contests/
Disallow: /espn/now
Disallow: /espnradio/podcast/feeds/easports/
Disallow: /index?sport=*&topId=*
Disallow: /index?sport=*&type=replay
Disallow: /members/
Disallow: /personalization/
Disallow: /sports*/index?topId=*
Disallow: /travel/passport/
Disallow: /video/clipDeportes
Disallow: /video/search
Disallow: /nba/tradeMachine/?tradeId=*
Disallow: */wire?
Disallow: /soccer/*/story/channels/*
Disallow: /load_video_player/*
Disallow: /football/conversation?gameId=*
Disallow: */_/*/channels/
Disallow: */_/channels/
Disallow: /football/*/story/channels/*
Disallow: /football/*/utils/*
Disallow: */crossdomain/*
Disallow: */makecorsrequest/*
Disallow: /soccer/*/utils/*
Disallow: */om-navmethod/*
Disallow: */null/*
Disallow: */undefined/*
Disallow: *ESPNPlusAdobeTracking/
Disallow: */modifier/*
Disallow: /*&xhr=1
Disallow: /*&wsVar=*
Disallow: /*&_slug_=*
Disallow: /*.com
Disallow: /*.net
Disallow: /*.org
Disallow: /*.co.uk
Disallow: */platform/*
Disallow: /*&platform=
Disallow: /*?platform=
Disallow: /football/matchstats
Disallow: /football/lineups
Disallow: /football/commentary
Disallow: /soccer/matchstats
Disallow: /football/video
Disallow: /soccer/lineups
Disallow: /soccer/commentary
Disallow: /soccer/video
Allow: */platform/amp*
Allow: /*&platform=amp
Allow: /*?platform=amp

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of espn.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).

API: https://infrarabbit.com/crawlers/api/v1/policy/espn.com