dailymail.co.uk: what it tells AI crawlers

On 27 October 2026, dailymail.co.uk's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 1k (Tranco)
robots.txt: ok (HTTP 200) · https://www.dailymail.com/robots.txt
unchanged since 2026-10-26 · sha256 4d3c6789fa319d2a… · view as recorded
The * group: allows applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	blocks	named in its own group
OAI-SearchBot	blocks	named in its own group
ChatGPT-User	blocks	named in its own group
ClaudeBot	blocks	named in its own group
Claude-SearchBot	blocks	named in its own group
Claude-User	blocks	named in its own group
PerplexityBot	blocks	named in its own group
Perplexity-User	blocks	named in its own group
Googlebot	allows	through the * group
Google-Extended	blocks	named in its own group
Applebot-Extended	blocks	named in its own group
CCBot	blocks	named in its own group
Bytespider	blocks	named in its own group
Meta-ExternalAgent	blocks	named in its own group
Amazonbot	blocks	named in its own group
anthropic-ai	blocks	named in its own group
bedrockbot	blocks	named in its own group
ChatGLM-Spider	blocks	named in its own group
Claude-Web	blocks	named in its own group
CloudflareBrowserRenderingCrawler	blocks	named in its own group
cohere-ai	blocks	named in its own group
Cotoyogi	blocks	named in its own group
DeepSeekBot	blocks	named in its own group
Diffbot	blocks	named in its own group
DuckAssistBot	blocks	named in its own group
EchoboxBot	blocks	named in its own group
FacebookBot	blocks	named in its own group
FriendlyCrawler	blocks	named in its own group
Gemini-Deep-Research	blocks	named in its own group
Google-CloudVertexBot	blocks	named in its own group
GoogleOther	allows	named in its own group
GoogleOther-Image	blocks	named in its own group
GoogleOther-Video	blocks	named in its own group
iaskspider	blocks	named in its own group
ICC-Crawler	blocks	named in its own group
ImagesiftBot	blocks	named in its own group
ISSCyberRiskCrawler	blocks	named in its own group
KunatoCrawler	blocks	named in its own group
LinerBot	blocks	named in its own group
meta-externalfetcher	blocks	named in its own group
MistralAI-User	blocks	named in its own group
omgili	blocks	named in its own group
omgilibot	blocks	named in its own group
PanguBot	blocks	named in its own group
PetalBot	blocks	named in its own group
QualifiedBot	blocks	named in its own group
Scrapy	blocks	named in its own group
TikTokSpider	blocks	named in its own group
Timpibot	blocks	named in its own group
VelenPublicWebCrawler	blocks	named in its own group
WARDBot	blocks	named in its own group
Webzio-Extended	blocks	named in its own group
wpbot	blocks	named in its own group
WRTNBot	blocks	named in its own group
YouBot	blocks	named in its own group

Other signals

Publishes /llms.txt	no
Publishes /ai.txt	no
Publishes /.well-known/tdmrep.json	no
Reserves TDM rights (TDMRep file, header or meta)	not observed
Declares Content Signals (robots.txt or header)	not observed
Says ai-train=no in a Content Signal	not observed
Points to an RSL licence (robots.txt License line or Link header)	not observed
Answers with an x402 payment offer (lower bound: only the homepage is probed)	not observed
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	not observed
Sends noai / noimageai directives	not observed
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 ba6db3b7941f0b96…

# Robots.txt for https://www.dailymail.co.uk/ updated 27/08/2025
#
#
# All Robots
User-agent: *
#
# Begin Standard Rules
Disallow: /*readcommentshtml*
Disallow: /*login?redirectPath=
Disallow: /*logout?redirectPath=
Disallow: /*nextThread.html
Disallow: /*previousThread.html
Disallow: /*questionId*
Disallow: /*selectedImage*
Disallow: /*threadIndex=*
Disallow: /*topGallery*
Disallow: /ce/item.cms*
Disallow: /guide/*
Disallow: /registration/*
Disallow: /best-buys/clickout/*
Disallow: /best-buys/m/clickout/*
Disallow: /weather.html?latitude=*
Disallow: /weather.html?old=*
Disallow: /home/search.html*
Disallow: /best-buys/mercury/out*
Disallow: /api/infinite-list.html

#
# Disallow Money for Google News
User-agent: Googlebot-News
Disallow: /tmoney/*
#
# Allow Adsense
User-agent: Mediapartners-Google
Disallow:
#
User-agent: CrystalSemanticsBot
Disallow: /
#
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: bedrockbot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGLM-Spider
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Claude-Web
User-agent: cohere-ai
User-agent: Cotoyogi
User-agent: DeepSeekBot
User-agent: Diffbot
User-agent: DuckAssistBot
User-agent: FacebookBot
User-agent: FriendlyCrawler
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: Grok
User-agent: iaskspider/2.0
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: img2dataset
User-agent: ISSCyberRiskCrawler
User-agent: Kangaroo Bot
User-agent: KunatoCrawler
User-agent: Meltwater
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: MistralAI-User
User-agent: OAI-Operator
User-agent: OAI-SearchBot
User-agent: omgili
User-agent: omgilibot
User-agent: PanguBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: PetalBot
User-agent: QualifiedBot
User-agent: Scrapy
User-agent: Seekr
User-agent: Sidetrade indexer bot
User-agent: TikTokSpider
User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: WARDBot
User-agent: Webzio-Extended
User-agent: WRTNBot
User-agent: YouBot
Allow: /buyline/
Disallow: /
#
#Allow Google sandbox privacy testing
User-agent: GoogleOther
Allow: /.well-known/privacy-sandbox-attestations.json
#
# Sitemap Files
Sitemap: https://www.dailymail.co.uk/google-news-sitemap.xml
Sitemap: https://www.dailymail.co.uk/sitemap-channels.xml
Sitemap: https://www.dailymail.co.uk/sitemap-columnist.xml
Sitemap: https://www.dailymail.co.uk/sitemap-topics.xml
Sitemap: https://www.dailymail.co.uk/sitemap-videos-year~2021.xml
Sitemap: https://www.dailymail.co.uk/sitemap-facts-year~2021.xml
Sitemap: https://www.dailymail.co.uk/sitemap-articles-year~2021.xml
Sitemap: https://www.dailymail.co.uk/sitemap-videos-year~2022.xml
Sitemap: https://www.dailymail.co.uk/sitemap-articles-year~2022.xml
Sitemap: https://www.dailymail.co.uk/sitemap-facts-year~2022.xml
Sitemap: https://www.dailymail.co.uk/sitemap-videos-year~2023.xml
Sitemap: https://www.dailymail.co.uk/sitemap-articles-year~2023.xml
Sitemap: https://www.dailymail.co.uk/sitemap-facts-year~2023.xml
Sitemap: https://www.dailymail.co.uk/sitemap-videos-year~2024.xml
Sitemap: https://www.dailymail.co.uk/sitemap-articles-year~2024.xml
Sitemap: https://www.dailymail.co.uk/sitemap-facts-year~2024.xml
Sitemap: https://www.dailymail.co.uk/sitemap-videos-year~2025.xml
Sitemap: https://www.dailymail.co.uk/sitemap-articles-year~2025.xml
Sitemap: https://www.dailymail.co.uk/sitemap-facts-year~2025.xml
Sitemap: https://www.dailymail.co.uk/sitemap-authors.xml

robots.txt versions

From	Through	State	Version
2026-10-26	2026-10-27	ok 200	`4d3c6789fa31`

Timeline of changes

Date	What changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2025-09-15 CC-MAIN-2025-38	www.dailymail.co.uk	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2024-09-16 CC-MAIN-2024-38	www.dailymail.co.uk	ok view	blocks	blocks	unknown	blocks	allows	unknown	blocks
2021-09-27 CC-MAIN-2021-39	www.dailymail.co.uk	ok view	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	www.dailymail.co.uk	ok view	allows	allows	unknown	allows	allows	unknown	allows
2019-09-23 CC-MAIN-2019-39	www.dailymail.co.uk	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-09-24 CC-MAIN-2018-39	www.dailymail.co.uk	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	www.dailymail.co.uk	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-07-24 CC-MAIN-2017-30	www.dailymail.co.uk	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-04-24 CC-MAIN-2017-17	dailymail.co.uk	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-01-23 CC-MAIN-2017-04	dailymail.co.uk	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-10-03 CC-MAIN-2016-40	dailymail.co.uk	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-09-05 CC-MAIN-2016-36	dailymail.co.uk	ok view	allows	allows	unknown	allows	allows	unknown	allows

Evidence

Every daily observation of dailymail.co.uk is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/dailymail.co.uk