dailymail.co.uk: what it tells AI crawlers

On 27 October 2026, dailymail.co.uk's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.dailymail.com/robots.txt
unchanged since 2026-10-26 · sha256 4d3c6789fa319d2a… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 bad020738fd4a075…

# Robots.txt for http://www.dailymail.co.uk/ updated 23/02/2017
# TS-5682
#
# All Robots
User-agent: *
#
# Begin Standard Rules
Disallow: /*readcommentshtml*
Disallow: /home/christmas/*
Disallow: /home/scotland/*
Disallow: /*jsessionid*
Disallow: /*createThread.html
Disallow: /*debateSearchResults.html
Disallow: /*debateTagSearch.html
Disallow: /*debateUserSearch.html
Disallow: /*login?redirectPath=
Disallow: /*logout?redirectPath=
Disallow: /*myStories.html$
Disallow: /*nextThread.html
Disallow: /*previousThread.html
Disallow: /*questionId*
Disallow: /*selectedImage*
Disallow: /*textbased/channel
Disallow: /*threadIndex=*
Disallow: /*topGallery*
Disallow: /ce/item.cms*
Disallow: /dailymail/
Disallow: /debate/polls/poll.html$
Disallow: /destinations/
Disallow: /guide/*
Disallow: /home/sitemaparchive/static$
Disallow: /link/index.html*
Disallow: /mailshopQA/*
Disallow: /SITE=DM/
Disallow: /sport/columnist-1000543/hatchet-man.rss
Disallow: /sport/columnist-1031773/andy-roddick.rss
Disallow: /sport/columnist-1046072/glenn-hoddle.rss
Disallow: /sport/columnist-1060209/jake-humphrey.rss
Disallow: /sport/columnist-353/sir-bobby-robson.rss
Disallow: /sport/columnist-354/ian-ridley--mail-sunday-sports-reporter.rss
Disallow: /sport/columnist-1043711/mark-jeffreys.rss
Disallow: /sport/columnist-423/dan-king.rss
Disallow: /sport/columnist-352/monty-panesar.rss
Disallow: /sudoku*
Disallow: /travel/enjoyengland/index.rss
Disallow: /travel/visitcornwall/index.html
Disallow: /tvshowbiz/tvlistings/*
Disallow: /weather.html*
Disallow: /weather/weather_landing_page.html*
Disallow: /ukplus/*
Disallow: /*/chat/*
Disallow: /app/
Disallow: /home/prmts/*
Disallow: /home/sweepstakes/*
Disallow: /beta/*
Disallow: /home/canneslions/index.html
Disallow: /cms/sites/all/modules/ckeditor_link/proxy.php*
Disallow: /%20user/view.php*
Disallow: /worldhome/*
Disallow: /home/search.html*
Disallow: /registration/*/profile.html
Disallow: /home/article-4252192/*
Disallow: /home/article-1388040/Privacy-Policy-Cookies.html
Disallow: /news/article-2663638/Mother-two-60-bled-death-surgeon-cut-vein-botched-hip-operation.html
Disallow: /news/article-2634104/Transgender-woman-landmark-legal-battle-force-government-forget-identity-man-sex-change.html 
#
# Disallow Money for Google News
User-agent: Googlebot-News
Disallow: /money/*
Disallow: /tmoney/*
#
# Allow Adsense
User-agent: Mediapartners-Google 
Disallow:
#
# Disallow Specific Robots
User-agent: MnoGoSearch/*
Disallow: /
#
User-agent: omgilibot/0.3
Disallow: /
#
User-agent: WebVac
Disallow: /
#
User-agent: WebZip
Disallow: /
#
User-agent: psbot
Disallow: /
#
User-agent: daumoa
Disallow: /
#
user-agent: AhrefsBot
disallow: /
#
User-agent: CrystalSemanticsBot
Disallow: /
#
user-agent: AhrefsBot
disallow: /
#
User-agent: ia_archiver
Disallow: /*/article-*
#
User-agent: Nutch
Disallow: /
#
# Don't crawl profile pages
#Disallow: /registration/*/profile.html
#
# Sitemap Files
Sitemap: http://www.dailymail.co.uk/newssitemap.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2017.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2016.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2015.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2014.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2013.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2012.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2011.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2010.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2009.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2008.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2007.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2006.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2005.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2004.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2003.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2002.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2001.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~2000.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~1999.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~1998.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~1997.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~1996.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~1994.xml
Sitemap: http://www.dailymail.co.uk/sitemap-articles-year~1993.xml
Sitemap: http://www.dailymail.co.uk/videositemap.xml
Sitemap: http://www.dailymail.co.uk/sitemap-channels.xml
Sitemap: http://www.dailymail.co.uk/sitemap-columnist.xml
Sitemap: http://www.dailymail.co.uk/sitemap-topics.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of dailymail.co.uk is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/dailymail.co.uk