n-tv.de: what it tells AI crawlers

On 27 October 2026, n-tv.de's robots.txt fully blocked GPTBot, ClaudeBot, Google-Extended and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 10k (Tranco)
robots.txt: ok (HTTP 200) · https://www.n-tv.de/robots.txt
unchanged since 2026-10-27 · sha256 62e64baa322adee5… · view as recorded
The * group: allows applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	blocks	named in its own group
OAI-SearchBot	allows	through the * group
ChatGPT-User	allows	through the * group
ClaudeBot	blocks	named in its own group
Claude-SearchBot	allows	through the * group
Claude-User	allows	through the * group
PerplexityBot	allows	through the * group
Perplexity-User	allows	through the * group
Googlebot	allows	through the * group
Google-Extended	blocks	named in its own group
Applebot-Extended	blocks	named in its own group
CCBot	blocks	named in its own group
Bytespider	blocks	named in its own group
Meta-ExternalAgent	blocks	named in its own group
Amazonbot	blocks	named in its own group
Awario	blocks	named in its own group
cohere-ai	blocks	named in its own group
cohere-training-data-crawler	blocks	named in its own group
Cotoyogi	blocks	named in its own group
DeepSeekBot	blocks	named in its own group
Diffbot	blocks	named in its own group
LinkupBot	blocks	named in its own group
MistralAI-Training	blocks	named in its own group
Scrapy	blocks	named in its own group
SemrushBot-SWA	blocks	named in its own group
ShapBot	blocks	named in its own group

Other signals

Publishes /llms.txt	no
Publishes /ai.txt	no
Publishes /.well-known/tdmrep.json	no
Reserves TDM rights (TDMRep file, header or meta)	no
Declares Content Signals (robots.txt or header)	no
Says ai-train=no in a Content Signal	no
Points to an RSL licence (robots.txt License line or Link header)	no
Answers with an x402 payment offer (lower bound: only the homepage is probed)	no
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	no
Sends noai / noimageai directives	no
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 62e64baa322adee5…

# robots.txt for n-tv.de

User-agent: *
Disallow: /suche/
Disallow: /boersenkurse/suche/
Disallow: /stat/
Disallow: /archiv/
Disallow: /auth/
Disallow: /mediathek/videos/etf-masterclass/

User-agent: 360Spider
Disallow: /

User-agent: Adagiobot
Disallow: /

User-agent: Amazonbot
Disallow: /
Allow: /broker-vergleich/
Allow: /casino-vergleich/
Allow: /kreditvergleich/
Allow: /geschaeftskonto-vergleich/
Allow: /online-sprachschule/
Allow: /sportwetten/
Allow: /webtools-im-vergleich/
Allow: /wettanbieter-vergleich/
Allow: /ratgeber/So-findest-du-den-Streamingdienst-der-wirklich-zu-deinem-Alltag-passt-id31165054.html
Allow: /ratgeber/Welcher-Streamingdienst-passt-zu-dir-Warum-RTL-fuer-Fans-deutscher-Serien-Reality-und-Live-TV-ideal-ist-id31160570.html
Allow: /ratgeber/Wer-fuer-eine-Waermepumpe-bis-zu-22-400-Euro-Zuschuss-erhalten-kann-id31217878.html
Allow: /ratgeber/Wann-sich-der-Umstieg-finanziell-rechnet-id31358008.html

User-agent: Applebot-Extended
Disallow: /
Allow: /broker-vergleich/
Allow: /casino-vergleich/
Allow: /kreditvergleich/
Allow: /geschaeftskonto-vergleich/
Allow: /online-sprachschule/
Allow: /sportwetten/
Allow: /webtools-im-vergleich/
Allow: /wettanbieter-vergleich/
Allow: /ratgeber/So-findest-du-den-Streamingdienst-der-wirklich-zu-deinem-Alltag-passt-id31165054.html
Allow: /ratgeber/Welcher-Streamingdienst-passt-zu-dir-Warum-RTL-fuer-Fans-deutscher-Serien-Reality-und-Live-TV-ideal-ist-id31160570.html
Allow: /ratgeber/Wer-fuer-eine-Waermepumpe-bis-zu-22-400-Euro-Zuschuss-erhalten-kann-id31217878.html
Allow: /ratgeber/Wann-sich-der-Umstieg-finanziell-rechnet-id31358008.html

User-agent: audisto
Disallow: /

User-agent: audisto-essential
Disallow: /

User-agent: Awario
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /
Allow: /broker-vergleich/
Allow: /casino-vergleich/
Allow: /kreditvergleich/
Allow: /geschaeftskonto-vergleich/
Allow: /online-sprachschule/
Allow: /sportwetten/
Allow: /webtools-im-vergleich/
Allow: /wettanbieter-vergleich/
Allow: /ratgeber/So-findest-du-den-Streamingdienst-der-wirklich-zu-deinem-Alltag-passt-id31165054.html
Allow: /ratgeber/Welcher-Streamingdienst-passt-zu-dir-Warum-RTL-fuer-Fans-deutscher-Serien-Reality-und-Live-TV-ideal-ist-id31160570.html
Allow: /ratgeber/Wer-fuer-eine-Waermepumpe-bis-zu-22-400-Euro-Zuschuss-erhalten-kann-id31217878.html
Allow: /ratgeber/Wann-sich-der-Umstieg-finanziell-rechnet-id31358008.html

User-agent: coccocbot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: cohere-training-data-collector
Disallow: /

User-agent: cohere-training-data-crawler
Disallow: /

User-agent: Cotoyogi
Disallow: /

User-agent: crawler4j
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: deepcrawl
Disallow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: FullStoryBot
Disallow: /

User-agent: Funnelback
Disallow: /

User-agent: Google-Extended
Disallow: /
Allow: /broker-vergleich/
Allow: /casino-vergleich/
Allow: /kreditvergleich/
Allow: /geschaeftskonto-vergleich/
Allow: /online-sprachschule/
Allow: /sportwetten/
Allow: /webtools-im-vergleich/
Allow: /wettanbieter-vergleich/
Allow: /ratgeber/So-findest-du-den-Streamingdienst-der-wirklich-zu-deinem-Alltag-passt-id31165054.html
Allow: /ratgeber/Welcher-Streamingdienst-passt-zu-dir-Warum-RTL-fuer-Fans-deutscher-Serien-Reality-und-Live-TV-ideal-ist-id31160570.html
Allow: /ratgeber/Wer-fuer-eine-Waermepumpe-bis-zu-22-400-Euro-Zuschuss-erhalten-kann-id31217878.html
Allow: /ratgeber/Wann-sich-der-Umstieg-finanziell-rechnet-id31358008.html

User-agent: GPTBot
Disallow: /
Allow: /broker-vergleich/
Allow: /casino-vergleich/
Allow: /kreditvergleich/
Allow: /geschaeftskonto-vergleich/
Allow: /online-sprachschule/
Allow: /sportwetten/
Allow: /webtools-im-vergleich/
Allow: /wettanbieter-vergleich/
Allow: /ratgeber/So-findest-du-den-Streamingdienst-der-wirklich-zu-deinem-Alltag-passt-id31165054.html
Allow: /ratgeber/Welcher-Streamingdienst-passt-zu-dir-Warum-RTL-fuer-Fans-deutscher-Serien-Reality-und-Live-TV-ideal-ist-id31160570.html
Allow: /ratgeber/Wer-fuer-eine-Waermepumpe-bis-zu-22-400-Euro-Zuschuss-erhalten-kann-id31217878.html
Allow: /ratgeber/Wann-sich-der-Umstieg-finanziell-rechnet-id31358008.html

User-agent: Hatena
Disallow: /

User-agent: heritrix
Disallow: /

User-agent: LinkupBot
Disallow: /

User-agent: lumar
Disallow: /

User-agent: MarketGoo
Disallow: /

User-agent: MegaIndex.ru
Disallow: /

User-agent: meta-externalagent
Disallow: /
Allow: /broker-vergleich/
Allow: /casino-vergleich/
Allow: /kreditvergleich/
Allow: /geschaeftskonto-vergleich/
Allow: /online-sprachschule/
Allow: /sportwetten/
Allow: /webtools-im-vergleich/
Allow: /wettanbieter-vergleich/
Allow: /ratgeber/So-findest-du-den-Streamingdienst-der-wirklich-zu-deinem-Alltag-passt-id31165054.html
Allow: /ratgeber/Welcher-Streamingdienst-passt-zu-dir-Warum-RTL-fuer-Fans-deutscher-Serien-Reality-und-Live-TV-ideal-ist-id31160570.html
Allow: /ratgeber/Wer-fuer-eine-Waermepumpe-bis-zu-22-400-Euro-Zuschuss-erhalten-kann-id31217878.html
Allow: /ratgeber/Wann-sich-der-Umstieg-finanziell-rechnet-id31358008.html

User-agent: mirrorweb
Disallow: /

User-agent: MistralAI-Training
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: Monsidobot
Disallow: /

User-agent: mrfCompass
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: OneTrust
Disallow: /

User-agent: rogerbot
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: SEBot-WA
Disallow: /

User-agent: SearchAtlasBot
Disallow: /

User-agent: SeekportBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: SemrushBot-BA
Disallow: /

User-agent: SemrushBot-SI
Disallow: /

User-agent: SemrushBot-SWA
Disallow: /

User-agent: SeobilityBot
Disallow: /

User-agent: SEOkicks
Disallow: /

User-agent: SEOkicks-Robot
Disallow: /

User-agent: SERankingBacklinksBot
Disallow: /

User-agent: SerpstatBot
Disallow: /

User-agent: ShapBot
Disallow: /

User-agent: SiteAuditBot
Disallow: /

User-agent: SiteimproveBot-Crawler
Disallow: /

User-agent: SogouSpider
Disallow: /

User-agent: StatusCake
Disallow: /

User-agent: TermlyBot
Disallow: /

User-agent: theHarvester
Disallow: /

User-agent: trendictionbot
Disallow: /

User-agent: Watchbot
Disallow: /

User-agent: xovi
Disallow: /

User-agent: XoviBot
Disallow: /

Sitemap: https://www.n-tv.de/sitemap.xml
Sitemap: https://www.n-tv.de/sitemap-video.xml
Sitemap: https://www.n-tv.de/sitemap/sitemap-sections.xml.gz
Sitemap: https://www.n-tv.de/news.xml

robots.txt versions

From	Through	State	Version
2026-10-27	2026-10-28	ok 200	`62e64baa322a`	diff with previous
2026-10-26	2026-10-26	ok 200	`de355f459222`

Timeline of changes

Date	What changed
2026-10-27	robots.txt changed
2026-10-27	now blocks Amazonbot (was allow)
2026-10-27	now blocks Applebot-Extended (was allow)
2026-10-27	now blocks Awario (was allow)
2026-10-27	now blocks Bytespider (was allow)
2026-10-27	now blocks ClaudeBot (was allow)
2026-10-27	now blocks cohere-ai (was allow)
2026-10-27	now blocks cohere-training-data-crawler (was allow)
2026-10-27	now blocks Cotoyogi (was allow)
2026-10-27	now blocks DeepSeekBot (was allow)
2026-10-27	now blocks Diffbot (was allow)
2026-10-27	now blocks LinkupBot (was allow)
2026-10-27	now blocks Meta-ExternalAgent (was allow)
2026-10-27	now blocks MistralAI-Training (was allow)
2026-10-27	now blocks Scrapy (was allow)
2026-10-27	now blocks SemrushBot-SWA (was allow)
2026-10-27	now blocks ShapBot (was allow)

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	www.n-tv.de	ok view	blocks	allows	unknown	allows	allows	unknown	allows
2025-09-15 CC-MAIN-2025-38	www.n-tv.de	ok view	blocks	allows	unknown	allows	allows	unknown	allows
2024-09-16 CC-MAIN-2024-38	www.n-tv.de	ok view	blocks	allows	unknown	allows	allows	unknown	allows
2023-10-02 CC-MAIN-2023-40	www.n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2022-10-03 CC-MAIN-2022-40	www.n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2021-09-27 CC-MAIN-2021-39	www.n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	www.n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2019-09-23 CC-MAIN-2019-39	www.n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-09-24 CC-MAIN-2018-39	n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-01-29 CC-MAIN-2018-05	www.n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-10-23 CC-MAIN-2017-43	www.n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	www.n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-07-24 CC-MAIN-2017-30	www.n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-04-24 CC-MAIN-2017-17	www.n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-01-23 CC-MAIN-2017-04	n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-10-03 CC-MAIN-2016-40	www.n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-09-05 CC-MAIN-2016-36	www.n-tv.de	ok view	allows	allows	unknown	allows	allows	unknown	allows

Evidence

Every daily observation of n-tv.de is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/n-tv.de