theguardian.com: what it tells AI crawlers

On 27 October 2026, theguardian.com's robots.txt fully blocked ClaudeBot, Claude-SearchBot, PerplexityBot and CCBot; it also referenced an RSL licence.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 1k (Tranco)
robots.txt: ok (HTTP 200) · https://www.theguardian.com/robots.txt
unchanged since 2026-10-26 · sha256 3387cd8d393a0c25… · view as recorded
The * group: allows applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	allows	through the * group
OAI-SearchBot	allows	through the * group
ChatGPT-User	allows	through the * group
ClaudeBot	blocks	named in its own group
Claude-SearchBot	blocks	named in its own group
Claude-User	blocks	named in its own group
PerplexityBot	blocks	named in its own group
Perplexity-User	allows	through the * group
Googlebot	allows	through the * group
Google-Extended	allows	through the * group
Applebot-Extended	blocks	named in its own group
CCBot	blocks	named in its own group
Bytespider	blocks	named in its own group
Meta-ExternalAgent	blocks	named in its own group
Amazonbot	blocks	named in its own group
amazon-QBusiness	blocks	named in its own group
Amzn-SearchBot	blocks	named in its own group
anthropic-ai	blocks	named in its own group
DuckAssistBot	blocks	named in its own group
FacebookBot	blocks	named in its own group
Google-CloudVertexBot	blocks	named in its own group
PetalBot	blocks	named in its own group
YandexAdditional	blocks	named in its own group
YandexAdditionalBot	blocks	named in its own group
YouBot	blocks	named in its own group

Other signals

Publishes /llms.txt	no
Publishes /ai.txt	no
Publishes /.well-known/tdmrep.json	no
Reserves TDM rights (TDMRep file, header or meta)	no
Declares Content Signals (robots.txt or header)	no
Says ai-train=no in a Content Signal	no
Points to an RSL licence (robots.txt License line or Link header)	yes
Answers with an x402 payment offer (lower bound: only the homepage is probed)	no
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	no
Sends noai / noimageai directives	no
Refused our fetcher (403 / challenge)	no

RSL licence: https://theguardian.com/license.xml

robots.txt as recorded sha256 7fefa38288310505…

# This is the robots.txt file for theguardian.com

# Guardian content is made available under our terms and conditions of use.
# Any other uses are not permitted, incl. but not limited to: for large language
# models (LLMs), machine learning and/or artificial intelligence-related
# purposes; with any of the aforementioned technologies; and/or for any
# commercial purposes. Contact licensing@theguardian.com for assistance

User-agent: *
Disallow: /sendarticle/
Disallow: /Users/
Disallow: /users/
Disallow: /*/print$
Disallow: /email/
Disallow: /contactus/
Disallow: /share/
Disallow: /websearch
Disallow: /*?commentpage=
Disallow: /whsmiths/
Disallow: /external/overture/
Disallow: /discussion/report-abuse/*
Disallow: /discussion/report-abuse-ajax/*
Disallow: /discussion/comment-permalink/*
Disallow: /discussion/report-abuse/*
Disallow: /discussion/user-report-abuse/*
Disallow: /discussion/handlers/*
Disallow: /discussion/your-profile
Disallow: /discussion/your-comments
Disallow: /discussion/edit-profile
Disallow: /discussion/search/comments
Disallow: /discussion/*
Disallow: /search
Disallow: /music/artist/*
Disallow: /music/album/*
Disallow: /books/data/*
Disallow: /settings/
Disallow: /embed/
Disallow: /*styles/js-on.css$
Disallow: /sport/olympics/2008/events/*
Disallow: /sport/olympics/2008/medals/*
Disallow: /f/healthcheck
Disallow: /sections
Disallow: /top-stories
Disallow: /most-read/sport
Disallow: /articles
Disallow: /global$
Disallow: /*/feedarticle/*
Disallow: /travel/2013/aug/22/been-there-readers-competition?*
Disallow: /preference/*
Disallow: /59666047/
Disallow: /print/
Disallow: /info/tech-feedback
Disallow: /production-monitoring/
Disallow: *.emailjson
Disallow: *.emailtxt
Disallow: /headline.txt
Disallow: *?*dcr=apps*

User-agent: Mediapartners-Google
Disallow:

Sitemap: http://www.theguardian.com/sitemaps/news.xml
Sitemap: http://www.theguardian.com/sitemaps/video.xml

User-agent: NewsNow
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: MoodleBot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: https://hada.news
Disallow: /

User-agent: https://www.imediaethics.org
Disallow: /

User-agent: Mojeek
Disallow: /

User-agent: JenkersBot
Disallow: /

User-agent: Seekr
Disallow: /

User-agent: Turnitin
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: Arquivo-web-crawler
Disallow: /

User-agent: coccocbot-web
Disallow: /

User-agent: SeznamBot
Disallow: /

User-Agent: PerplexityBot
Disallow: /

User-Agent: yacy
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: claudebot
Disallow: /

User-agent: bingbot
Disallow: /

User-agent: AwarioRssBot
Disallow: /

User-agent: AwarioSmartBot
Disallow: /

User-agent: Netvibes
Disallow: /

User-agent: SentiOne
Disallow: /

User-agent: UptimeRobot
Disallow: /

User-agent: ImageSift
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: YandexAdditional
Disallow: /

User-agent: YandexAdditionalBot
Disallow: /

robots.txt versions

From	Through	State	Version
2026-10-26	2026-10-27	ok 200	`3387cd8d393a`

Timeline of changes

Date	What changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	www.theguardian.com	ok view	allows	allows	unknown	blocks	blocks	unknown	blocks
2025-09-15 CC-MAIN-2025-38	www.theguardian.com	ok view	allows	allows	unknown	blocks	blocks	unknown	blocks
2024-09-16 CC-MAIN-2024-38	www.theguardian.com	ok view	blocks	allows	unknown	blocks	allows	unknown	blocks
2023-10-02 CC-MAIN-2023-40	www.theguardian.com	ok view	blocks	allows	unknown	allows	allows	unknown	allows
2022-10-03 CC-MAIN-2022-40	www.theguardian.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2021-09-27 CC-MAIN-2021-39	www.theguardian.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	www.theguardian.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2019-09-23 CC-MAIN-2019-39	www.theguardian.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-09-24 CC-MAIN-2018-39	www.theguardian.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	www.theguardian.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-07-24 CC-MAIN-2017-30	www.theguardian.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-04-24 CC-MAIN-2017-17	theguardian.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-01-23 CC-MAIN-2017-04	www.theguardian.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-10-03 CC-MAIN-2016-40	theguardian.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-09-05 CC-MAIN-2016-36	www.theguardian.com	ok view	allows	allows	unknown	allows	allows	unknown	allows

Evidence

Every daily observation of theguardian.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/theguardian.com