britannica.com: what it tells AI crawlers

On 28 October 2026, britannica.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.britannica.com:443/robots.txt
unchanged since 2026-10-26 · sha256 df7371ee298dc17e… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-28

Other signals

robots.txt as recorded sha256 53d7c3076627a7a4…

# /robots.txt file for encyclopaedia britannica 

Sitemap: https://www.britannica.com/bps-ssl-gsm-index.xml 

User-agent: * 
Disallow: /search 
Disallow: /search/magazine 
Disallow: /eb/print 
Disallow: /bps/search 
Disallow: /bps/fulltext 
Disallow: /bps/edit 
Disallow: /bps/logout 
Disallow: /bps/myedits-tool 
Disallow: /bps/store-search 
Disallow: /bps/community 
Disallow: /bps/dashboard 
Disallow: /area 
Disallow: /area=ARTICLES 
Disallow: /ebsco/pdf 
Disallow: /bps/additionalcontent 
Disallow: /facts 
Disallow: /nobelprize/print 
Disallow: /hispanic_heritage/print 
Disallow: /shakespeare/print 
Disallow: /women/print 
Disallow: /presidents/print 
Disallow: /blackhistory/print 
Disallow: /holocaust/print 
Disallow: /oscars/print 
Disallow: /oscar/print 
Disallow: /dday/print 
Disallow: /bps/ebooks-search 
Disallow: /bps/magazine-search 
Disallow: /bps/storeTopicSearch

Disallow: /widgetprofile
Disallow: /user-profile-contributions/ajax
Disallow: /widgetsWebapp
Disallow: /qa
Disallow: /dictionary

User-agent: Mediapartners-Google
Disallow:

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of britannica.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).

API: https://infrarabbit.com/crawlers/api/v1/policy/britannica.com