telegraph.co.uk: what it tells AI crawlers

On 27 October 2026, telegraph.co.uk's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.telegraph.co.uk/robots.txt
unchanged since 2026-10-26 · sha256 2aff8d5f0e77abba… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 2aff8d5f0e77abba…

# Production Robots.txt file

User-agent: *
User-agent: Google-Extended
# Special parameters
Disallow: /etc.clientlibs/settings/wcm/designs/telegraph/core/clientlibs/save-article.*.js
Disallow: /etc.clientlibs/settings/wcm/designs/telegraph/core/clientlibs/page-refresh.*.js
Disallow: /*?mobile=true
Disallow: /*?mobile=basic
Disallow: /*?ModPagespeed=noscript
Disallow: /*_jcr_content*
Disallow: /*?source=rss
Disallow: /puzzles/puzzle/*?source=

# Internal Search
Disallow: /search/
Allow: /search/$

# Special areas
Disallow: /news/main.jhtml
Disallow: /p/*/embed/
Disallow: /secure/login/*
Disallow: /content/telegraph/
Disallow: /customer/secure/checkout/tesco/
Disallow: /customer/secure/reset-password/
Disallow: /telegraph/*
Disallow: /news-app/*
Disallow: /amp$
Disallow: /application/*
Disallow: /ixale/
Disallow: /core/Content/
Disallow: /r/
Disallow: /travel/hotel/e/*
Disallow: /travel/hotels/hotel-finder/
Disallow: /podcasts-more/
Disallow: /secure/register/
Allow: /travel/hotels/hotel-finder/$
Disallow: /martech/js/
Disallow: /martech/css/
Disallow: /martech-content/
Disallow: /bin/telegraph/recombee-config
Disallow: /*&p=
Disallow: /customer/subscription/*?
Disallow: /edge-redirection/


# Bots which make unnecessary bot traffic
User-agent: endeca
Disallow: /archive/
Disallow: /search/*

User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: bedrockbot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGLM-Spider
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: cohere-ai
User-agent: Cotoyogi
User-agent: DeepSeekBot
User-agent: Diffbot
User-agent: FacebookBot
User-agent: FriendlyCrawler
User-agent: Google-CloudVertexBot
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: Grok
User-agent: iaskspider/2.0
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: img2dataset
User-agent: ISSCyberRiskCrawler
User-agent: Kangaroo Bot
User-agent: KunatoCrawler
User-agent: Meltwater
User-agent: Meta-ExternalAgent
User-agent: omgili
User-agent: omgilibot
User-agent: PanguBot
User-agent: PetalBot
User-agent: QualifiedBot
User-agent: Scrapy
User-agent: Seekr
User-agent: Sidetrade indexer bot
User-agent: TikTokSpider
User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: WARDBot
User-agent: Webzio-Extended
User-agent: WRTNBot
User-agent: Claude-SearchBot
User-agent: DuckAssistBot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Meta-ExternalFetcher
User-agent: MistralAI-User
User-agent: OAI-Operator
User-agent: Perplexity-User
Disallow: /
Allow: /travel/explore-dubai/
Allow: /travel/luxury-stays/
Allow: /business/uk-business-growth/
Allow: /ads.txt
Allow: /bah-avis/usa-sports-road-trip-map
Allow: /betting/
Allow: /british-airways-holidays/cyprus-hotel-map
Allow: /business/ai-era-job-skills/
Allow: /business/ai-workflow-transformation/
Allow: /business/future-hr-technology/
Allow: /business/golf-dp-world-tour/
Allow: /business/search-marketing/
Allow: /business/waste-management/
Allow: /deals/
Allow: /environment/woodland-action/
Allow: /fashion/joseph-ribkoff-autumn-collection/
Allow: /financial-services/
Allow: /food-and-drink/m-and-s-seasonal-favourites/
Allow: /food-and-drink/m-and-s-seasonal-favourites/organic-range/
Allow: /food-and-drink/maldon-salt-perfect-pyramid-challenge/
Allow: /food-and-drink/spanish-dining-uk/
Allow: /football/global-football-industry-awards/
Allow: /health-fitness/how-to-choose-private-hospital-hca-healthcare/
Allow: /interiors/next-home-autumn-interiors-trends/
Allow: /junior-golf/
Allow: /lifestyle-solutions/
Allow: /lifestyle/apl-media/
Allow: /branded-content/
Allow: /money/investing/fc-investment-trust/long-term-investing/
Allow: /branded-content/
Allow: /rugby-union/grassroots-rugby-heroes/
Allow: /rugby-union/levelling-the-playing-field/
Allow: /sailgp/inside-f50-catamaran
Allow: /seabourn/explore-the-mediterranean
Allow: /sport/sailgp-championship/
Allow: /travel/active-family-holidays/
Allow: /travel/away-holidays/
Allow: /travel/barbados-holiday-guide/
Allow: /travel/british-airways-american-express/
Allow: /travel/canadian-affair-holidays/rocky-mountaineer-train-route/
Allow: /travel/croatia-holidays/
Allow: /travel/cruises/holland-america-line-luxury-holidays/
Allow: /travel/cruises/seabourn-luxury-cruises/
Allow: /travel/discover-america/
Allow: /travel/exclusiveoffers/
Allow: /branded-content/
Allow: /travel/florida-guide/
Allow: /travel/fly-drive-holidays/
Allow: /travel/grenada-guide/
Allow: /travel/icons-of-the-sky/
Allow: /travel/indian-ocean-escapes/
Allow: /travel/indulge-in-ireland/
Allow: /travel/jamaica-holiday-guide/
Allow: /travel/japan-ninja-heritage/
Allow: /travel/luxury-all-inclusive-resorts/affordable-caribbean-hotels/
Allow: /travel/luxury-holidays/
Allow: /travel/luxury-hotel-getaways/
Allow: /travel/luxury-island-escapes/
Allow: /travel/maldives-holiday-guide/
Allow: /travel/ski-resorts-for-everyone/
Allow: /travel/spark/best-of-switzerland/
Allow: /travel/tailor-made-tours/
Allow: /travel/tenerife-holidays/
Allow: /travel/tips-for-visiting-spain/
Allow: /travel/travel-counsellors/book-a-cruise-with-travel-counsellors/
Allow: /travel/travel-solutions/
Allow: /travel/visit-cayman-islands/
Allow: /travel/why-choose-malta/
Allow: /trusted-partners/
Allow: /brand-usa
Allow: /travel/visit-aarhus/
Allow: /bespoke-content/



# Sitemap URLs
Sitemap: https://www.telegraph.co.uk/sitemap.xml
Sitemap: https://www.telegraph.co.uk/custom/authors/sitemap.xml
Sitemap: https://www.telegraph.co.uk/custom/main-pages/sitemap.xml
Sitemap: https://www.telegraph.co.uk/custom/daily-news/sitemap.xml
Sitemap: https://www.telegraph.co.uk/custom/author-pages/sitemap.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of telegraph.co.uk is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/telegraph.co.uk