nu.nl: what it tells AI crawlers

On 27 October 2026, nu.nl's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel
top 10k (Tranco)
robots.txt
ok (HTTP 200) · https://www.nu.nl/robots.txt
unchanged since 2026-10-26 · sha256 4019062370846461… · view as recorded
The * group
allows applies to every agent without its own group

Per AI bot, 2026-10-27

User agentVerdictHow
GPTBotblocksnamed in its own group
OAI-SearchBotblocksnamed in its own group
ChatGPT-Userblocksnamed in its own group
ClaudeBotblocksnamed in its own group
Claude-SearchBotblocksnamed in its own group
Claude-Userblocksnamed in its own group
PerplexityBotblocksnamed in its own group
Perplexity-Userblocksnamed in its own group
Googlebotallowsthrough the * group
Google-Extendedblocksnamed in its own group
Applebot-Extendedblocksnamed in its own group
CCBotblocksnamed in its own group
Bytespiderblocksnamed in its own group
Meta-ExternalAgentblocksnamed in its own group
Amazonbotblocksnamed in its own group
Amzn-SearchBotblocksnamed in its own group
Amzn-Userblocksnamed in its own group
anthropic-aiblocksnamed in its own group
Applebotblocksnamed in its own group
Claude-Webblocksnamed in its own group
cohere-aiblocksnamed in its own group
Cotoyogiblocksnamed in its own group
DeepSeekBotblocksnamed in its own group
Diffbotblocksnamed in its own group
DuckAssistBotblocksnamed in its own group
ExaBotblocksnamed in its own group
Gemini-Deep-Researchblocksnamed in its own group
Google-Agentblocksnamed in its own group
Google-NotebookLMblocksnamed in its own group
iAskBotblocksnamed in its own group
ICC-Crawlerblocksnamed in its own group
meta-externalfetcherblocksnamed in its own group
meta-webindexerblocksnamed in its own group
MistralAI-Userblocksnamed in its own group
PanguBotblocksnamed in its own group
PetalBotblocksnamed in its own group
SemrushBot-OCOBblocksnamed in its own group
SemrushBot-SWAblocksnamed in its own group
TikTokSpiderblocksnamed in its own group
Timpibotblocksnamed in its own group
VelenPublicWebCrawlerblocksnamed in its own group
YandexAdditionalblocksnamed in its own group
YandexAdditionalBotblocksnamed in its own group
YouBotblocksnamed in its own group

Other signals

Publishes /llms.txtno
Publishes /ai.txtno
Publishes /.well-known/tdmrep.jsonno
Reserves TDM rights (TDMRep file, header or meta)no
Declares Content Signals (robots.txt or header)no
Says ai-train=no in a Content Signalno
Points to an RSL licence (robots.txt License line or Link header)no
Answers with an x402 payment offer (lower bound: only the homepage is probed)no
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)no
Sends noai / noimageai directivesno
Refused our fetcher (403 / challenge)no

robots.txt as recorded sha256 44a8266e63c1772f…

User-agent: *

Disallow: /zoeken
Disallow: /search
Disallow: /block/
Disallow: /components/
Disallow: /profile

Sitemap: https://www.nu.nl/sitemap_index.xml
Sitemap: https://www.nu.nl/sitemap_news.xml
Sitemap: https://www.nu.nl/sitemap_video.xml

User-agent: Googlebot
Disallow: /*redirect=1*

User-agent: Googlebot-News
Disallow: /advertorial/

robots.txt versions

FromThroughStateVersion
2026-10-262026-10-27ok 200401906237084

Timeline of changes

DateWhat changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

CrawlHostStateGPTBotOAI-SearchBotChatGPT-UserClaudeBotClaude-SearchBotClaude-UserPerplexityBot
2026-09-21 CC-MAIN-2026-39nu.nlfetch-blockedunknownunknownunknownunknownunknownunknownunknown
2025-09-15 CC-MAIN-2025-38www.nu.nlok viewblocksallowsunknownblocksallowsunknownblocks
2024-09-16 CC-MAIN-2024-38www.nu.nlok viewblocksallowsunknownallowsallowsunknownblocks
2023-10-02 CC-MAIN-2023-40www.nu.nlok viewblocksallowsunknownallowsallowsunknownallows
2022-10-03 CC-MAIN-2022-40www.nu.nlok viewallowsallowsunknownallowsallowsunknownallows
2021-09-27 CC-MAIN-2021-39www.nu.nlok viewallowsallowsunknownallowsallowsunknownallows
2020-09-28 CC-MAIN-2020-40www.nu.nlok viewallowsallowsunknownallowsallowsunknownallows
2019-09-23 CC-MAIN-2019-39www.nu.nlok viewallowsallowsunknownallowsallowsunknownallows
2018-09-24 CC-MAIN-2018-39www.nu.nlok viewallowsallowsunknownallowsallowsunknownallows
2017-09-25 CC-MAIN-2017-39www.nu.nlok viewallowsallowsunknownallowsallowsunknownallows
2016-10-03 CC-MAIN-2016-40www.nu.nlok viewallowsallowsunknownallowsallowsunknownallows
2016-09-05 CC-MAIN-2016-36www.nu.nlok viewallowsallowsunknownallowsallowsunknownallows

Evidence

Every daily observation of nu.nl is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/nu.nl