bbc.com: what it tells AI crawlers

On 27 October 2026, bbc.com's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 1k (Tranco)
robots.txt: ok (HTTP 200) · https://www.bbc.com/robots.txt
unchanged since 2026-10-26 · sha256 37a8567b049d9a34… · view as recorded
The * group: allows applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	blocks	named in its own group
OAI-SearchBot	blocks	named in its own group
ChatGPT-User	blocks	named in its own group
ClaudeBot	blocks	named in its own group
Claude-SearchBot	allows	through the * group
Claude-User	allows	through the * group
PerplexityBot	blocks	named in its own group
Perplexity-User	blocks	named in its own group
Googlebot	allows	through the * group
Google-Extended	blocks	named in its own group
Applebot-Extended	blocks	named in its own group
CCBot	blocks	named in its own group
Bytespider	blocks	named in its own group
Meta-ExternalAgent	blocks	named in its own group
Amazonbot	blocks	named in its own group
Amzn-SearchBot	blocks	named in its own group
Amzn-User	blocks	named in its own group
anthropic-ai	blocks	named in its own group
ApifyBot	blocks	named in its own group
ApifyWebsiteContentCrawler	blocks	named in its own group
Brightbot	blocks	named in its own group
Claude-Web	blocks	named in its own group
CloudflareBrowserRenderingCrawler	blocks	named in its own group
cohere-ai	blocks	named in its own group
Diffbot	blocks	named in its own group
Diffbot-User	blocks	named in its own group
ExaBot	blocks	named in its own group
FirecrawlAgent	blocks	named in its own group
Google-CloudVertexBot	blocks	named in its own group
omgili	blocks	named in its own group
omgilibot	blocks	named in its own group
PetalBot	blocks	named in its own group
Scrapy	blocks	named in its own group
ShapBot	blocks	named in its own group
TavilyBot	blocks	named in its own group
YandexAdditional	blocks	named in its own group
YandexAdditionalBot	blocks	named in its own group
YouBot	blocks	named in its own group

Other signals

Publishes /llms.txt	no
Publishes /ai.txt	no
Publishes /.well-known/tdmrep.json	no
Reserves TDM rights (TDMRep file, header or meta)	no
Declares Content Signals (robots.txt or header)	no
Says ai-train=no in a Content Signal	no
Points to an RSL licence (robots.txt License line or Link header)	no
Answers with an x402 payment offer (lower bound: only the homepage is probed)	no
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	no
Sends noai / noimageai directives	no
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 4a7f5170f3f0a9f3…

# version: 4238e226c2e46c9edc38ed04aa23cc0e6b9a7af6

# HTTP www.bbc.com

User-agent: Googlebot
Sitemap: http://www.bbc.com/sitemap.xml
Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml
Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml
Sitemap: http://www.bbc.com/video_sitemap.xml

Disallow: /iplayer/episode/*?from=r*
Disallow: /iplayer/cy/episode/*?from=r*
Disallow: /iplayer/gd/episode/*?from=r*
Disallow: /_programmes
Disallow: /606/
Disallow: /aboutthebbc/insidethebbc/search
Disallow: /academy/chinese-trad/search
Disallow: /afrique/search
Disallow: /apps/cbbc
Disallow: /apps/flash
Disallow: /apps/ide
Disallow: /apps/ids
Disallow: /apps/ifl
Disallow: /apps/nr
Disallow: /apps/query
Disallow: /apps/t
Disallow: /arabic/search
Disallow: /arts/yourpaintings/artdetective/
Disallow: /arts/yourpaintings/feedback/sent/
Disallow: /arts/yourpaintings/images/guidedtours/
Disallow: /arts/yourpaintings/newsletter/error/
Disallow: /arts/yourpaintings/newsletter/success/
Disallow: /arts/yourpaintings/newsletter/unsubscribe/
Disallow: /arts/yourpaintings/paintings/search*
Disallow: /arts/yourpaintings/search/
Disallow: /azeri/search
Disallow: /bbctrust/search
Disallow: /bengali/search
Disallow: /burmese/search
Disallow: /cbbc/search
Disallow: /cbeebies/search
Disallow: /cgi-bin
Disallow: /cgi-perl
Disallow: /cgi-perlx
Disallow: /cgi-store
Disallow: /corporate2/bbctrust/search
Disallow: /corporate2/insidethebbc/search
Disallow: /corporate2/mediacentre/proginfo/search
Disallow: /corporate2/mediacentre/search
Disallow: /corporate2/safety/search
Disallow: /eastenders/updates/
Disallow: /education$
Disallow: /education/
Disallow: /education/bitesize
Disallow: /education/dev
Disallow: /education/images
Disallow: /education/nav
Disallow: /education/navigation
Disallow: /education/ximages
Disallow: /education/blocks$
Disallow: /education/blocks/
Disallow: /eoltools/
Disallow: /films/gateways
Disallow: /frameworks/fig/
Disallow: /furniture
Disallow: /gahuza/search
Disallow: /h2g2/servers/narthur8
Disallow: /hausa/search
Disallow: /hindi/search
Disallow: /history/domesday/search
Disallow: /includes/
Disallow: /indonesia/search
Disallow: /iplayer/_proxy_
Disallow: /iplayer/bigscreen/
Disallow: /iplayer/cbbc/episodes/
Disallow: /iplayer/cbbc/search
Disallow: /iplayer/cbeebies/episodes/
Disallow: /iplayer/cbeebies/search
Disallow: /iplayer/cy/
Disallow: /iplayer/gd/
Disallow: /iplayer/pagecomponents/
Disallow: /iplayer/playlist/
Disallow: /iplayer/search
Disallow: /iplayer/usercomponents/
Disallow: /kyrgyz/search
Disallow: /learningzone/clips/search
Disallow: /mediacentre/proginfo/search
Disallow: /mediacentre/search
Disallow: /mediaselector
Disallow: /messageboards/
Disallow: /mundo/search
Disallow: /music/releases/
Disallow: /nav/
Disallow: /navigation
Disallow: /nepali/search
Disallow: /news/uk-30888157
Disallow: /news/uk-england-suffolk-20780472
Disallow: /news/0
Disallow: /news/videos/*
Disallow: /partners/
Disallow: /pashto/search
Disallow: /persian/search
Disallow: /portuguese/search
Disallow: /pressoffice/plots/
Disallow: /print/
Disallow: /radio/player
Disallow: /russian/search
Disallow: /safety/search
Disallow: /search$
Disallow: /search/
Disallow: /sinhala/search
Disallow: /skillswise/quickfind/search
Disallow: /somali/search
Disallow: /sport/videos/*
Disallow: /survey/ 
Disallow: /swahili/search
Disallow: /syndication/html/ 
Disallow: /tamil/search
Disallow: /turkce/search
Disallow: /ukchina/simp/search
Disallow: /ukchina/trad/search
Disallow: /ukrainian/search
Disallow: /urdu/search
Disallow: /userinfo/
Disallow: /userinfo
Disallow: /uzbek/search
Disallow: /vietnamese/search
Disallow: /weather/broadband/ 
Disallow: /weather/search
Disallow: /worldservice/images/ 
Disallow: /worldservice/psims/ 
Disallow: /worldservice/ssi/ 
Disallow: /worldservice/survey/
Disallow: /zhongwen/simp/search
Disallow: /zhongwen/trad/search
Disallow: /sounds/search$
Disallow: /sounds/search/
Disallow: /sounds/search?


User-Agent: Yahoo! Slurp
Sitemap: http://www.bbc.com/sitemap.xml
Disallow: /iplayer/episode/*?from=r* 
Disallow: /iplayer/cy/episode/*?from=r* 
Disallow: /iplayer/gd/episode/*?from=r* 
Disallow: /_programmes 
Disallow: /606/ 
Disallow: /aboutthebbc/insidethebbc/search
Disallow: /academy/chinese-trad/search
Disallow: /afrique/search
Disallow: /apps/cbbc 
Disallow: /apps/flash
Disallow: /apps/ide
Disallow: /apps/ids
Disallow: /apps/ifl
Disallow: /apps/nr
Disallow: /apps/query
Disallow: /apps/t
Disallow: /arabic/search
Disallow: /arts/yourpaintings/artdetective/
Disallow: /arts/yourpaintings/feedback/sent/
Disallow: /arts/yourpaintings/images/guidedtours/
Disallow: /arts/yourpaintings/newsletter/error/
Disallow: /arts/yourpaintings/newsletter/success/
Disallow: /arts/yourpaintings/newsletter/unsubscribe/
Disallow: /arts/yourpaintings/paintings/search*
Disallow: /arts/yourpaintings/search/
Disallow: /azeri/search
Disallow: /bbctrust/search
Disallow: /bengali/search
Disallow: /burmese/search
Disallow: /cbbc/search
Disallow: /cbeebies/search
Disallow: /cgi-bin
Disallow: /cgi-perl
Disallow: /cgi-perlx
Disallow: /cgi-store
Disallow: /corporate2/bbctrust/search
Disallow: /corporate2/insidethebbc/search
Disallow: /corporate2/mediacentre/proginfo/search
Disallow: /corporate2/mediacentre/search
Disallow: /corporate2/safety/search
Disallow: /eastenders/updates/
Disallow: /education$
Disallow: /education/
Disallow: /education/bitesize
Disallow: /education/dev
Disallow: /education/images
Disallow: /education/nav
Disallow: /education/navigation
Disallow: /education/ximages
Disallow: /education/blocks$
Disallow: /education/blocks/
Disallow: /eoltools/
Disallow: /films/gateways
Disallow: /food/binder/
Disallow: /food/recipes/search*?*
Disallow: /frameworks/fig/
Disallow: /furniture
Disallow: /gahuza/search
Disallow: /h2g2/servers/narthur8
Disallow: /hausa/search
Disallow: /hindi/search
Disallow: /history/domesday/search
Disallow: /includes/
Disallow: /indonesia/search
Disallow: /iplayer/_proxy_
Disallow: /iplayer/bigscreen/
Disallow: /iplayer/cbbc/episodes/
Disallow: /iplayer/cbbc/search
Disallow: /iplayer/cbeebies/episodes/
Disallow: /iplayer/cbeebies/search
Disallow: /iplayer/cy/
Disallow: /iplayer/gd/
Disallow: /iplayer/pagecomponents/
Disallow: /iplayer/playlist/
Disallow: /iplayer/search
Disallow: /iplayer/usercomponents/
Disallow: /kyrgyz/search
Disallow: /learningzone/clips/search
Disallow: /mediacentre/proginfo/search
Disallow: /mediacentre/search
Disallow: /mediaselector
Disallow: /messageboards/
Disallow: /mundo/search
Disallow: /music/releases/
Disallow: /nav/
Disallow: /navigation
Disallow: /nepali/search
Disallow: /news/uk-30888157
Disallow: /news/uk-england-suffolk-20780472
Disallow: /news/0
Disallow: /news/videos/*
Disallow: /partners/
Disallow: /pashto/search
Disallow: /persian/search
Disallow: /portuguese/search
Disallow: /pressoffice/plots/
Disallow: /print/
Disallow: /radio/player
Disallow: /russian/search
Disallow: /safety/search
Disallow: /search$
Disallow: /search/
Disallow: /sinhala/search
Disallow: /skillswise/quickfind/search
Disallow: /somali/search
Disallow: /survey/
Disallow: /swahili/search
Disallow: /syndication/html/
Disallow: /tamil/search
Disallow: /turkce/search
Disallow: /ukchina/simp/search
Disallow: /ukchina/trad/search
Disallow: /ukrainian/search
Disallow: /urdu/search
Disallow: /uzbek/search
Disallow: /vietnamese/search
Disallow: /weather/broadband/
Disallow: /weather/search
Disallow: /worldservice/images/
Disallow: /worldservice/psims/
Disallow: /worldservice/ssi/
Disallow: /worldservice/survey/
Disallow: /zhongwen/simp/search
Disallow: /zhongwen/trad/search
Disallow: /sounds/search$
Disallow: /sounds/search/
Disallow: /sounds/search?


User-Agent: msnbot
Sitemap: http://www.bbc.com/sitemap.xml
Disallow: /iplayer/episode/*?from=r*
Disallow: /iplayer/cy/episode/*?from=r*
Disallow: /iplayer/gd/episode/*?from=r*
Disallow: /_programmes
Disallow: /606/
Disallow: /aboutthebbc/insidethebbc/search
Disallow: /academy/chinese-trad/search
Disallow: /afrique/search
Disallow: /apps/cbbc
Disallow: /apps/flash
Disallow: /apps/ide
Disallow: /apps/ids
Disallow: /apps/ifl
Disallow: /apps/nr
Disallow: /apps/query
Disallow: /apps/t
Disallow: /arabic/search
Disallow: /arts/yourpaintings/artdetective/
Disallow: /arts/yourpaintings/feedback/sent/
Disallow: /arts/yourpaintings/images/guidedtours/
Disallow: /arts/yourpaintings/newsletter/error/
Disallow: /arts/yourpaintings/newsletter/success/
Disallow: /arts/yourpaintings/newsletter/unsubscribe/
Disallow: /arts/yourpaintings/paintings/search*
Disallow: /arts/yourpaintings/search/
Disallow: /azeri/search
Disallow: /bbctrust/search
Disallow: /bengali/search
Disallow: /burmese/search
Disallow: /cbbc/search
Disallow: /cbeebies/search
Disallow: /cgi-bin
Disallow: /cgi-perl
Disallow: /cgi-perlx
Disallow: /cgi-store
Disallow: /corporate2/bbctrust/search
Disallow: /corporate2/insidethebbc/search
Disallow: /corporate2/mediacentre/proginfo/search
Disallow: /corporate2/mediacentre/search
Disallow: /corporate2/safety/search
Disallow: /eastenders/updates/
Disallow: /education$
Disallow: /education/
Disallow: /education/bitesize
Disallow: /education/dev
Disallow: /education/images
Disallow: /education/nav
Disallow: /education/navigation
Disallow: /education/ximages
Disallow: /education/blocks$
Disallow: /education/blocks/
Disallow: /eoltools/
Disallow: /films/gateways
Disallow: /food/binder/
Disallow: /food/recipes/search*?*
Disallow: /frameworks/fig/
Disallow: /furniture
Disallow: /gahuza/search
Disallow: /h2g2/servers/narthur8
Disallow: /hausa/search
Disallow: /hindi/search
Disallow: /history/domesday/search
Disallow: /includes/
Disallow: /indonesia/search
Disallow: /iplayer/_proxy_
Disallow: /iplayer/bigscreen/
Disallow: /iplayer/cbbc/episodes/
Disallow: /iplayer/cbbc/search
Disallow: /iplayer/cbeebies/episodes/
Disallow: /iplayer/cbeebies/search
Disallow: /iplayer/cy/
Disallow: /iplayer/gd/
Disallow: /iplayer/pagecomponents/
Disallow: /iplayer/playlist/
Disallow: /iplayer/search
Disallow: /iplayer/usercomponents/
Disallow: /kyrgyz/search
Disallow: /learningzone/clips/search
Disallow: /mediacentre/proginfo/search
Disallow: /mediacentre/search
Disallow: /mediaselector
Disallow: /messageboards/
Disallow: /mundo/search
Disallow: /music/releases/
Disallow: /nav/
Disallow: /navigation
Disallow: /nepali/search
Disallow: /news/uk-30888157
Disallow: /news/uk-england-suffolk-20780472
Disallow: /news/0
Disallow: /news/videos/*
Disallow: /partners/
Disallow: /pashto/search
Disallow: /persian/search
Disallow: /portuguese/search
Disallow: /pressoffice/plots/
Disallow: /print/
Disallow: /radio/player
Disallow: /russian/search
Disallow: /safety/search
Disallow: /search$
Disallow: /search/
Disallow: /sinhala/search
Disallow: /skillswise/quickfind/search
Disallow: /somali/search
Disallow: /survey/
Disallow: /swahili/search
Disallow: /syndication/html/
Disallow: /tamil/search
Disallow: /turkce/search
Disallow: /ukchina/simp/search
Disallow: /ukchina/trad/search
Disallow: /ukrainian/search
Disallow: /urdu/search
Disallow: /uzbek/search
Disallow: /vietnamese/search
Disallow: /weather/broadband/
Disallow: /weather/search
Disallow: /worldservice/images/
Disallow: /worldservice/psims/
Disallow: /worldservice/ssi/
Disallow: /worldservice/survey/
Disallow: /zhongwen/simp/search
Disallow: /zhongwen/trad/search
Disallow: /sounds/search$
Disallow: /sounds/search/
Disallow: /sounds/search?


User-agent: Googlebot-Mobile
Disallow: /_programmes
Disallow: /606/
Disallow: /aboutthebbc/insidethebbc/search
Disallow: /academy/chinese-trad/search
Disallow: /afrique/search
Disallow: /apps/cbbc
Disallow: /apps/flash
Disallow: /apps/ide
Disallow: /apps/ids
Disallow: /apps/ifl
Disallow: /apps/nr
Disallow: /apps/query
Disallow: /apps/t
Disallow: /arabic/search
Disallow: /arts/yourpaintings/artdetective/
Disallow: /arts/yourpaintings/feedback/sent/
Disallow: /arts/yourpaintings/images/guidedtours/
Disallow: /arts/yourpaintings/newsletter/error/
Disallow: /arts/yourpaintings/newsletter/success/
Disallow: /arts/yourpaintings/newsletter/unsubscribe/
Disallow: /arts/yourpaintings/paintings/search*
Disallow: /arts/yourpaintings/search/
Disallow: /azeri/search
Disallow: /bbctrust/search
Disallow: /bengali/search
Disallow: /burmese/search
Disallow: /cbbc/search
Disallow: /cbeebies/search
Disallow: /cgi-bin
Disallow: /cgi-perl
Disallow: /cgi-perlx
Disallow: /cgi-store
Disallow: /corporate2/bbctrust/search
Disallow: /corporate2/insidethebbc/search
Disallow: /corporate2/mediacentre/proginfo/search
Disallow: /corporate2/mediacentre/search
Disallow: /corporate2/safety/search
Disallow: /eastenders/updates/
Disallow: /education$
Disallow: /education/
Disallow: /education/bitesize
Disallow: /education/dev
Disallow: /education/images
Disallow: /education/nav
Disallow: /education/navigation
Disallow: /education/ximages
Disallow: /education/blocks$
Disallow: /education/blocks/
Disallow: /eoltools/
Disallow: /films/gateways
Disallow: /food/binder/
Disallow: /food/recipes/search*?*
Disallow: /frameworks/fig/
Disallow: /furniture
Disallow: /gahuza/search
Disallow: /h2g2/servers/narthur8
Disallow: /hausa/search
Disallow: /hindi/search
Disallow: /history/domesday/search
Disallow: /includes/
Disallow: /indonesia/search
Disallow: /iplayer/_proxy_
Disallow: /iplayer/bigscreen/
Disallow: /iplayer/cbbc/episodes/
Disallow: /iplayer/cbbc/search
Disallow: /iplayer/cbeebies/episodes/
Disallow: /iplayer/cbeebies/search
Disallow: /iplayer/cy/
Disallow: /iplayer/gd/
Disallow: /iplayer/pagecomponents/
Disallow: /iplayer/playlist/
Disallow: /iplayer/search
Disallow: /iplayer/usercomponents/
Disallow: /kyrgyz/search
Disallow: /learningzone/clips/search
Disallow: /mediacentre/proginfo/search
Disallow: /mediacentre/search
Disallow: /mediaselector
Disallow: /messageboards/
Disallow: /mundo/search
Disallow: /music/releases/
Disallow: /nav/
Disallow: /navigation
Disallow: /nepali/search
Disallow: /news/uk-30888157
Disallow: /news/uk-england-suffolk-20780472
Disallow: /news/0
Disallow: /news/videos/*
Disallow: /partners/
Disallow: /pashto/search
Disallow: /persian/search
Disallow: /portuguese/search
Disallow: /pressoffice/plots/
Disallow: /print/
Disallow: /radio/player
Disallow: /russian/search
Disallow: /safety/search
Disallow: /search$
Disallow: /search/
Disallow: /sinhala/search
Disallow: /skillswise/quickfind/search
Disallow: /somali/search
Disallow: /survey/
Disallow: /swahili/search
Disallow: /syndication/html/
Disallow: /tamil/search
Disallow: /turkce/search
Disallow: /ukchina/simp/search
Disallow: /ukchina/trad/search
Disallow: /ukrainian/search
Disallow: /urdu/search
Disallow: /uzbek/search
Disallow: /vietnamese/search
Disallow: /weather/broadband/
Disallow: /weather/search
Disallow: /worldservice/images/
Disallow: /worldservice/psims/
Disallow: /worldservice/ssi/
Disallow: /worldservice/survey/
Disallow: /zhongwen/simp/search
Disallow: /zhongwen/trad/search
Disallow: /sounds/search$
Disallow: /sounds/search/
Disallow: /sounds/search?


User-agent: magpie-crawler
Disallow: /


User-agent: *
Sitemap: http://www.bbc.com/sitemap.xml
Disallow: /_programmes
Disallow: /606/
Disallow: /aboutthebbc/insidethebbc/search
Disallow: /academy/chinese-trad/search
Disallow: /afrique/search
Disallow: /apps/cbbc
Disallow: /apps/flash
Disallow: /apps/ide
Disallow: /apps/ids
Disallow: /apps/ifl
Disallow: /apps/nr
Disallow: /apps/query
Disallow: /apps/t
Disallow: /arabic/search
Disallow: /arts/yourpaintings/artdetective/
Disallow: /arts/yourpaintings/feedback/sent/
Disallow: /arts/yourpaintings/images/guidedtours/
Disallow: /arts/yourpaintings/newsletter/error/
Disallow: /arts/yourpaintings/newsletter/success/
Disallow: /arts/yourpaintings/newsletter/unsubscribe/
Disallow: /arts/yourpaintings/paintings/search*
Disallow: /arts/yourpaintings/search/
Disallow: /azeri/search
Disallow: /bbctrust/search
Disallow: /bengali/search
Disallow: /bitesize/my$
Disallow: /bitesize/my/
Disallow: /bitesize/search$
Disallow: /bitesize/search/
Disallow: /bitesize/search?
Disallow: /burmese/search
Disallow: /cbbc/search/
Disallow: /cbbc/search$
Disallow: /cbbc/search?
Disallow: /cbeebies/search/
Disallow: /cbeebies/search$
Disallow: /cbeebies/search?
Disallow: /chwilio/
Disallow: /chwilio$
Disallow: /chwilio?
Disallow: /cgi-bin
Disallow: /cgi-perl
Disallow: /cgi-perlx
Disallow: /cgi-store
Disallow: /corporate2/bbctrust/search
Disallow: /corporate2/insidethebbc/search
Disallow: /corporate2/mediacentre/proginfo/search
Disallow: /corporate2/mediacentre/search
Disallow: /corporate2/safety/search
Disallow: /eastenders/updates/
Disallow: /education$
Disallow: /education/
Disallow: /education/bitesize
Disallow: /education/dev
Disallow: /education/images
Disallow: /education/my$
Disallow: /education/my/
Disallow: /education/nav
Disallow: /education/navigation
Disallow: /education/ximages
Disallow: /education/blocks$
Disallow: /education/blocks/
Disallow: /eoltools/
Disallow: /films/gateways
Disallow: /food/binder/
Disallow: /food/recipes/search*?*
Disallow: /frameworks/fig/
Disallow: /furniture
Disallow: /gahuza/search
Disallow: /h2g2/servers/narthur8
Disallow: /hausa/search
Disallow: /hindi/search
Disallow: /history/domesday/search
Disallow: /includes/
Disallow: /indonesia/search
Disallow: /iplayer/_proxy_
Disallow: /iplayer/bigscreen/
Disallow: /iplayer/cbbc/episodes/
Disallow: /iplayer/cbbc/search
Disallow: /iplayer/cbeebies/episodes/
Disallow: /iplayer/cbeebies/search
Disallow: /iplayer/cy/
Disallow: /iplayer/gd/
Disallow: /iplayer/pagecomponents/
Disallow: /iplayer/playlist/
Disallow: /iplayer/search
Disallow: /iplayer/usercomponents/
Disallow: /kyrgyz/search
Disallow: /learningzone/clips/search
Disallow: /mediacentre/proginfo/search
Disallow: /mediacentre/search
Disallow: /mediaselector
Disallow: /messageboards/
Disallow: /mundo/search
Disallow: /music/releases/
Disallow: /nav/
Disallow: /navigation
Disallow: /nepali/search
Disallow: /news/uk-30888157
Disallow: /news/uk-england-suffolk-20780472
Disallow: /newsround
Disallow: /news/0
Disallow: /news/videos/*
Disallow: /partners/
Disallow: /pashto/search
Disallow: /persian/search
Disallow: /portuguese/search
Disallow: /pressoffice/plots/
Disallow: /print/
Disallow: /radio/player
Disallow: /radio/imda
Disallow: /russian/search
Disallow: /safety/search
Disallow: /search$
Disallow: /search/
Disallow: /search?
Disallow: /sinhala/search
Disallow: /skillswise/quickfind/search
Disallow: /somali/search
Disallow: /survey/
Disallow: /swahili/search
Disallow: /syndication/html/
Disallow: /tamil/search
Disallow: /turkce/search
Disallow: /ukchina/simp/search
Disallow: /ukchina/trad/search
Disallow: /ukrainian/search
Disallow: /urdu/search
Disallow: /userinfo/
Disallow: /uzbek/search
Disallow: /vietnamese/search
Disallow: /weather/broadband/
Disallow: /weather/search
Disallow: /worldservice/images/
Disallow: /worldservice/psims/
Disallow: /worldservice/ssi/
Disallow: /worldservice/survey/
Disallow: /ws/includes
Disallow: /zhongwen/simp/search
Disallow: /zhongwen/trad/search
Disallow: /education/my/activities$
Disallow: /education/my/activities/
Disallow: /bitesize/my/activities$
Disallow: /bitesize/my/activities/
Disallow: /bitesize/topics/
Disallow: /bitesize/subjects/
Disallow: /bitesize/examspecs/
Disallow: /bitesize/levels/
Disallow: /bitesize/guides/
Disallow: /bitesize/decks/
Disallow: /bitesize/flashcards/
Disallow: /bitesize/clips/
Disallow: /bitesize$
Disallow: /sounds/search$
Disallow: /sounds/search/
Disallow: /sounds/search?

Disallow: /u5llnop$
Disallow: /u5llnop/

Allow: /ideas$
Allow: /ideas/
Allow: /scotland$
Allow: /scotland/
Allow: /wales$
Allow: /wales/
Allow: /northernireland$
Allow: /northernireland/
Allow: /alba$
Allow: /alba/
Allow: /cymru$
Allow: /cymru/

robots.txt versions

From	Through	State	Version
2026-10-26	2026-10-27	ok 200	`37a8567b049d`

Timeline of changes

Date	What changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2024-09-16 CC-MAIN-2024-38	www.bbc.com	ok view	blocks	allows	unknown	blocks	allows	unknown	blocks
2023-10-02 CC-MAIN-2023-40	www.bbc.com	ok view	blocks	allows	unknown	allows	allows	unknown	allows
2022-10-03 CC-MAIN-2022-40	www.bbc.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2021-09-27 CC-MAIN-2021-39	www.bbc.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	www.bbc.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2019-09-23 CC-MAIN-2019-39	www.bbc.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-09-24 CC-MAIN-2018-39	www.bbc.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-04-23 CC-MAIN-2018-17	www.bbc.com	ok view	partial	partial	unknown	partial	partial	unknown	partial
2018-01-29 CC-MAIN-2018-05	www.bbc.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-10-23 CC-MAIN-2017-43	www.bbc.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	www.bbc.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-07-24 CC-MAIN-2017-30	www.bbc.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-04-24 CC-MAIN-2017-17	bbc.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-01-23 CC-MAIN-2017-04	www.bbc.com	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2016-10-03 CC-MAIN-2016-40	bbc.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-09-05 CC-MAIN-2016-36	bbc.com	ok view	allows	allows	unknown	allows	allows	unknown	allows

Evidence

Every daily observation of bbc.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/bbc.com