fb.com: what it tells AI crawlers

On 26 October 2026, fb.com's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-26 21:13 UTC (daily run of 2026-10-26). Proof for that day.

Panel
top 1k (Tranco)
robots.txt
ok (HTTP 200) · https://fb.com/robots.txt
unchanged since 2026-10-26 · sha256 ae34650d8199df7b… · view as recorded
The * group
blocks applies to every agent without its own group

Per AI bot, 2026-10-26

User agentVerdictHow
GPTBotblocksnamed in its own group
OAI-SearchBotblocksthrough the * group
ChatGPT-Userblocksthrough the * group
ClaudeBotblocksnamed in its own group
Claude-SearchBotblocksthrough the * group
Claude-Userblocksthrough the * group
PerplexityBotblocksnamed in its own group
Perplexity-Userblocksthrough the * group
Googlebotallowsnamed in its own group
Google-Extendedblocksnamed in its own group
Applebot-Extendedblocksnamed in its own group
CCBotblocksthrough the * group
Bytespiderblocksthrough the * group
Meta-ExternalAgentblocksthrough the * group
Amazonbotblocksnamed in its own group
Applebotallowsnamed in its own group
Brightbotblocksnamed in its own group
facebookexternalhitallowsnamed in its own group
PetalBotblocksnamed in its own group
Scrapyblocksnamed in its own group
YaKblocksnamed in its own group

Other signals

Publishes /llms.txtnot observed
Publishes /ai.txtnot observed
Publishes /.well-known/tdmrep.jsonnot observed
Reserves TDM rights (TDMRep file, header or meta)not observed
Declares Content Signals (robots.txt or header)not observed
Says ai-train=no in a Content Signalnot observed
Points to an RSL licence (robots.txt License line or Link header)not observed
Answers with an x402 payment offer (lower bound: only the homepage is probed)not observed
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)not observed
Sends noai / noimageai directivesnot observed
Refused our fetcher (403 / challenge)no

robots.txt versions

FromThroughStateVersion
2026-10-262026-10-27ok 200ae34650d8199

Timeline of changes

DateWhat changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

CrawlHostStateGPTBotOAI-SearchBotChatGPT-UserClaudeBotClaude-SearchBotClaude-UserPerplexityBot
2026-09-21 CC-MAIN-2026-39fb.comok viewblocksblocksunknownblocksblocksunknownblocks
2024-09-16 CC-MAIN-2024-38fb.comok viewblocksblocksunknownblocksblocksunknownblocks
2022-10-03 CC-MAIN-2022-40fb.comok viewblocksblocksunknownblocksblocksunknownblocks
2021-09-27 CC-MAIN-2021-39fb.comok viewblocksblocksunknownblocksblocksunknownblocks
2020-09-28 CC-MAIN-2020-40fb.comok viewblocksblocksunknownblocksblocksunknownblocks
2019-09-23 CC-MAIN-2019-39fb.comok viewblocksblocksunknownblocksblocksunknownblocks
2018-09-24 CC-MAIN-2018-39fb.comok viewblocksblocksunknownblocksblocksunknownblocks
2017-09-25 CC-MAIN-2017-39fb.comredirect-onlyunknownunknownunknownunknownunknownunknownunknown
2016-09-05 CC-MAIN-2016-36fb.comok viewblocksblocksunknownblocksblocksunknownblocks

Evidence

Every daily observation of fb.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-26: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973294)).

API: https://infrarabbit.com/crawlers/api/v1/policy/fb.com