espn.com: what it tells AI crawlers

On 28 October 2026, espn.com's robots.txt fully blocked GPTBot, Google-Extended and CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.espn.com/robots.txt
unchanged since 2026-10-26 · sha256 69e57dadcb6f535e… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-28

Other signals

robots.txt as recorded sha256 1c1ce2de708a3e71…

# robots.txt for www.espn.com - last updated 20160803

User-agent: *
Disallow: /*admin/
Disallow: /*conversation/
Disallow: /*conversations/
Disallow: /*deportes/
Disallow: /*flash/
Disallow: /*format/
Disallow: /*order/
Disallow: /*search?
Disallow: /*search/
Disallow: /*sort/
Disallow: /*util/
Disallow: /*webslices/
Disallow: /*print?id
Disallow: /ad/
Disallow: /cgi
Disallow: /community/
Disallow: /composer/
Disallow: /contests/
Disallow: /espn/now
Disallow: /espnradio/podcast/feeds/easports/
Disallow: /index?sport=*&topId=*
Disallow: /index?sport=*&type=replay
Disallow: /sports/*/index?topId=*
Disallow: /members/
Disallow: /personalization/
Disallow: /travel/passport/activity
Disallow: /travel/passport/add
Disallow: /travel/passport/event
Disallow: /travel/passport/events
Disallow: /travel/passport/invite
Disallow: /travel/passport/map
Disallow: /travel/passport/photos
Disallow: /travel/passport/rankings
Disallow: /travel/passport/stats
Disallow: /travel/passport/venues
Disallow: /video/search
Disallow: /video/clipDeportes

# Prevents crawl loop issues with incorrectly hyperlinked URLs in posts

Disallow:/*.com
Disallow:/*.net
Disallow:/*.org
Disallow:/*.co.uk
Disallow:/*.com.au
Disallow:/*.es
Disallow:/*.me
Disallow:/*.ly
Disallow:/*.me

# Exclusions for redundant content that consume crawl budget needlessly 

Disallow: /*view/
Disallow: /*start/
Disallow: /*photoId/
Disallow: /*type/
Disallow: /*cat/
Disallow: /*split/
Disallow: /*calendar/
Disallow: /*date/
Disallow: /*seasontype/
Disallow: /*season/200
Disallow: /*year/200
Disallow: /*_/year/20
Disallow: /*_/group/
Disallow: /*_/scoreboard/
Disallow: /*_/week/
Disallow: /*?ex_cid=espnapi_public
Disallow: /*&photoId=
Disallow: /*&full=

# Exclusions for content with little to no value for search engines

Disallow: /*playbyplay?
Disallow: /*boxscore?
Disallow: /*conversation?
Disallow: /*recap?
Disallow: /*preview?
Disallow: /*databaseresults/

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of espn.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).

API: https://infrarabbit.com/crawlers/api/v1/policy/espn.com