cloudflare.com: what it tells AI crawlers

On 27 October 2026, cloudflare.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot; it also published llms.txt.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.cloudflare.com/robots.txt
unchanged since 2026-10-26 · sha256 c6e360cc6e931ea3… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

Content signals: ai-train=yes, search=yes, ai-input=yes for cohere-ai

llms.txt: "Cloudflare", 17,173 bytes, 90 links.

robots.txt as recorded sha256 c6e360cc6e931ea3…

# Robots.txt for www.cloudflare.com

User-agent: *
Allow: /

# Sitemap
Sitemap: https://www.cloudflare.com/sitemap.xml

# AI/LLM friendly content
# See https://llmstxt.org for the llms.txt specification
# llms.txt provides curated content for AI assistants and LLMs

# Allow AI crawlers to access markdown versions of pages
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Anthropic-AI
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: CCBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Cohere-ai
Allow: /

# Content Signals — declare AI content usage preferences
# See https://contentsignals.org/ and https://datatracker.ietf.org/doc/draft-romm-aipref-contentsignals/
Content-Signal: ai-train=yes, search=yes, ai-input=yes

# AI-friendly content locations
# - /llms.txt - Curated overview for AI/LLMs (markdown)
# - /llms-full.txt - Full expanded content for larger context windows
# - /*.md - Markdown versions of all pages (append .md to any URL)
# - /.well-known/agents.json - Agent discovery and capabilities

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of cloudflare.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/cloudflare.com