faz.net: what it tells AI crawlers

On 27 October 2026, faz.net's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 10k (Tranco)
robots.txt: ok (HTTP 200) · https://www.faz.net/robots.txt
unchanged since 2026-10-26 · sha256 e0b697f51248cd0d… · view as recorded
The * group: allows applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	blocks	named in its own group
OAI-SearchBot	blocks	named in its own group
ChatGPT-User	blocks	named in its own group
ClaudeBot	blocks	named in its own group
Claude-SearchBot	blocks	named in its own group
Claude-User	blocks	named in its own group
PerplexityBot	blocks	named in its own group
Perplexity-User	allows	through the * group
Googlebot	allows	through the * group
Google-Extended	blocks	named in its own group
Applebot-Extended	blocks	named in its own group
CCBot	blocks	named in its own group
Bytespider	blocks	named in its own group
Meta-ExternalAgent	blocks	named in its own group
Amazonbot	blocks	named in its own group
anthropic-ai	blocks	named in its own group
Claude-Web	blocks	named in its own group
CloudVertexBot	blocks	named in its own group
cohere-ai	blocks	named in its own group
cohere-training-data-crawler	blocks	named in its own group
DeepSeekBot	blocks	named in its own group
Devin	blocks	named in its own group
Diffbot	blocks	named in its own group
DuckAssistBot	blocks	named in its own group
GoogleOther	allows	named in its own group
LinerBot	blocks	named in its own group
MistralAI-User	blocks	named in its own group
omgili	blocks	named in its own group
omgilibot	blocks	named in its own group
PanguBot	blocks	named in its own group
PetalBot	blocks	named in its own group
Timpibot	blocks	named in its own group
YouBot	blocks	named in its own group

Other signals

Publishes /llms.txt	no
Publishes /ai.txt	no
Publishes /.well-known/tdmrep.json	no
Reserves TDM rights (TDMRep file, header or meta)	no
Declares Content Signals (robots.txt or header)	no
Says ai-train=no in a Content Signal	no
Points to an RSL licence (robots.txt License line or Link header)	no
Answers with an x402 payment offer (lower bound: only the homepage is probed)	no
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	no
Sends noai / noimageai directives	no
Refused our fetcher (403 / challenge)	no

robots.txt versions

From	Through	State	Version
2026-10-28	2026-10-28	ok 200	`ded0c6d3d30c`	diff with previous
2026-10-26	2026-10-27	ok 200	`e0b697f51248`

Timeline of changes

Date	What changed
2026-10-28	robots.txt changed
2026-10-28	now partly blocks AI2Bot (was allow)
2026-10-28	now partly blocks AI2Bot-DeepResearchEval (was allow)
2026-10-28	now partly blocks Ai2Bot-Dolma (was allow)
2026-10-28	now partly blocks Amazonbot (was block)
2026-10-28	now partly blocks anthropic-ai (was block)
2026-10-28	now partly blocks Applebot-Extended (was block)
2026-10-28	now partly blocks bigsur.ai (was allow)
2026-10-28	now partly blocks Bytespider (was block)
2026-10-28	now partly blocks ChatGPT-User (was block)
2026-10-28	now partly blocks Claude-SearchBot (was block)
2026-10-28	now partly blocks Claude-User (was block)
2026-10-28	now partly blocks Claude-Web (was block)
2026-10-28	now partly blocks ClaudeBot (was block)
2026-10-28	now partly blocks CloudVertexBot (was block)
2026-10-28	now partly blocks cohere-ai (was block)
2026-10-28	now partly blocks cohere-training-data-crawler (was block)
2026-10-28	now partly blocks DeepSeekBot (was block)
2026-10-28	now partly blocks Devin (was block)
2026-10-28	now partly blocks Diffbot (was block)
2026-10-28	now partly blocks DuckAssistBot (was block)
2026-10-28	now partly blocks Googlebot (was allow)
2026-10-28	now partly blocks GoogleOther (was allow)
2026-10-28	now partly blocks Kangaroo Bot (was allow)
2026-10-28	now partly blocks LinerBot (was block)
2026-10-28	now partly blocks Meta-ExternalAgent (was block)
2026-10-28	now partly blocks MistralAI-User (was block)
2026-10-28	now partly blocks MistralAI-User/1.0 (was allow)
2026-10-28	now partly blocks OAI-SearchBot (was block)
2026-10-28	now partly blocks PanguBot (was block)
2026-10-28	now partly blocks Perplexity-User (was allow)
2026-10-28	now partly blocks PerplexityBot (was block)
2026-10-28	now partly blocks PetalBot (was block)
2026-10-28	now partly blocks Timpibot (was block)
2026-10-28	now partly blocks YouBot (was block)
2026-10-28	now partly blocks every agent without its own group (*) (was allow)

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	www.faz.net	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2025-09-15 CC-MAIN-2025-38	www.faz.net	ok view	blocks	allows	unknown	blocks	allows	unknown	blocks
2024-09-16 CC-MAIN-2024-38	www.faz.net	ok view	blocks	allows	unknown	allows	allows	unknown	blocks
2023-10-02 CC-MAIN-2023-40	www.faz.net	ok view	blocks	partial	unknown	partial	partial	unknown	partial
2022-10-03 CC-MAIN-2022-40	www.faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial
2021-09-27 CC-MAIN-2021-39	www.faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial
2020-09-28 CC-MAIN-2020-40	www.faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial
2019-09-23 CC-MAIN-2019-39	www.faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial
2018-09-24 CC-MAIN-2018-39	www.faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial
2018-01-29 CC-MAIN-2018-05	www.faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial
2017-10-23 CC-MAIN-2017-43	www.faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial
2017-09-25 CC-MAIN-2017-39	www.faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial
2017-07-24 CC-MAIN-2017-30	www.faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial
2017-04-24 CC-MAIN-2017-17	faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial
2017-01-23 CC-MAIN-2017-04	www.faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial
2016-10-03 CC-MAIN-2016-40	www.faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial
2016-09-05 CC-MAIN-2016-36	faz.net	ok view	partial	partial	unknown	partial	partial	unknown	partial

Evidence

Every daily observation of faz.net is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/faz.net