theconversation.com: what it tells AI crawlers

On 27 October 2026, theconversation.com's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 10k (Tranco)
robots.txt: ok (HTTP 200) · https://theconversation.com/robots.txt
unchanged since 2026-10-26 · sha256 1423b17b4d2b36d7… · view as recorded
The * group: allows applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	blocks	named in its own group
OAI-SearchBot	blocks	named in its own group
ChatGPT-User	blocks	named in its own group
ClaudeBot	blocks	named in its own group
Claude-SearchBot	blocks	named in its own group
Claude-User	blocks	named in its own group
PerplexityBot	blocks	named in its own group
Perplexity-User	blocks	named in its own group
Googlebot	allows	through the * group
Google-Extended	blocks	named in its own group
Applebot-Extended	blocks	named in its own group
CCBot	blocks	named in its own group
Bytespider	blocks	named in its own group
Meta-ExternalAgent	blocks	named in its own group
Amazonbot	blocks	named in its own group
AddSearchBot	blocks	named in its own group
AgentTimes	blocks	named in its own group
aiHitBot	blocks	named in its own group
AIWebIndex	blocks	named in its own group
amazon-kendra	blocks	named in its own group
amazon-QBusiness	blocks	named in its own group
AmazonBuyForMe	blocks	named in its own group
Amzn-SearchBot	blocks	named in its own group
Amzn-User	blocks	named in its own group
Andibot	blocks	named in its own group
Anomura	blocks	named in its own group
anthropic-ai	blocks	named in its own group
ApifyBot	blocks	named in its own group
ApifyWebsiteContentCrawler	blocks	named in its own group
Applebot	blocks	named in its own group
Aranet-SearchBot	blocks	named in its own group
atlassian-bot	blocks	named in its own group
Awario	blocks	named in its own group
AzureAI-SearchBot	blocks	named in its own group
bedrockbot	blocks	named in its own group
Bravebot	blocks	named in its own group
Brightbot	blocks	named in its own group
BuddyBot	blocks	named in its own group
ChatGLM-Spider	blocks	named in its own group
Claude-Code	blocks	named in its own group
Claude-Web	blocks	named in its own group
Cloudflare-AutoRAG	blocks	named in its own group
CloudVertexBot	blocks	named in its own group
Code	blocks	named in its own group
cohere-ai	blocks	named in its own group
cohere-training-data-crawler	blocks	named in its own group
Cotoyogi	blocks	named in its own group
CragCrawler	blocks	named in its own group
Crawlspace	blocks	named in its own group
Cursor	blocks	named in its own group
DeepSeekBot	blocks	named in its own group
Devin	blocks	named in its own group
Diffbot	blocks	named in its own group
DuckAssistBot	blocks	named in its own group
EchoboxBot	blocks	named in its own group
ExaBot	blocks	named in its own group
ExaSearchBot	blocks	named in its own group
FacebookBot	blocks	named in its own group
Factset_spyderbot	blocks	named in its own group
FirecrawlAgent	blocks	named in its own group
FriendlyCrawler	blocks	named in its own group
GeistHaus-PageFetcher	blocks	named in its own group
Gemini-Deep-Research	blocks	named in its own group
Google-Agent	blocks	named in its own group
Google-CloudVertexBot	blocks	named in its own group
Google-Firebase	blocks	named in its own group
Google-Gemini-CLI	blocks	named in its own group
Google-NotebookLM	blocks	named in its own group
GoogleAgent-Mariner	blocks	named in its own group
GoogleAgent-URLContext	blocks	named in its own group
GoogleOther	blocks	named in its own group
GoogleOther-Image	blocks	named in its own group
GoogleOther-Video	blocks	named in its own group
HenkBot	blocks	named in its own group
iAskBot	blocks	named in its own group
iaskspider	blocks	named in its own group
ICC-Crawler	blocks	named in its own group
ImagesiftBot	blocks	named in its own group
imageSpider	blocks	named in its own group
ISSCyberRiskCrawler	blocks	named in its own group
kagi-fetcher	blocks	named in its own group
Kimi-User	blocks	named in its own group
KlaviyoAIBot	blocks	named in its own group
KunatoCrawler	blocks	named in its own group
laion-huggingface-processor	blocks	named in its own group
LAIONDownloader	blocks	named in its own group
LCC	blocks	named in its own group
Lightpanda	blocks	named in its own group
LinerBot	blocks	named in its own group
LinkupBot	blocks	named in its own group
Manus-User	blocks	named in its own group
meta-externalfetcher	blocks	named in its own group
meta-webindexer	blocks	named in its own group
MistralAI-User	blocks	named in its own group
Mozilla-Tabstack	blocks	named in its own group
MyCentralAIScraperBot	blocks	named in its own group
NagetBot	blocks	named in its own group
newsai	blocks	named in its own group
NotebookLM	blocks	named in its own group
NovaAct	blocks	named in its own group
omgili	blocks	named in its own group
omgilibot	blocks	named in its own group
OpenAI	blocks	named in its own group
opencode	blocks	named in its own group
Operator	blocks	named in its own group
PanguBot	blocks	named in its own group
Panscient	blocks	named in its own group
PetalBot	blocks	named in its own group
PhindBot	blocks	named in its own group
Poggio-Citations	blocks	named in its own group
QualifiedBot	blocks	named in its own group
Querit-SearchBot	blocks	named in its own group
QueritBot	blocks	named in its own group
QuillBot	blocks	named in its own group
Reflectionbot	blocks	named in its own group
SBIntuitionsBot	blocks	named in its own group
Scrapy	blocks	named in its own group
SemrushBot-OCOB	blocks	named in its own group
SemrushBot-SWA	blocks	named in its own group
Shap-User	blocks	named in its own group
ShapBot	blocks	named in its own group
Spider	blocks	named in its own group
TavilyBot	blocks	named in its own group
TerraCotta	blocks	named in its own group
Thinkbot	blocks	named in its own group
TikTokSpider	blocks	named in its own group
Timpibot	blocks	named in its own group
TongyiBot	blocks	named in its own group
Trae	blocks	named in its own group
TwinAgent	blocks	named in its own group
UseAI	blocks	named in its own group
VelenPublicWebCrawler	blocks	named in its own group
WARDBot	blocks	named in its own group
Webzio-Extended	blocks	named in its own group
wpbot	blocks	named in its own group
WRTNBot	blocks	named in its own group
YaK	blocks	named in its own group
YandexAdditional	blocks	named in its own group
YandexAdditionalBot	blocks	named in its own group
YiyanBot	blocks	named in its own group
YouBot	blocks	named in its own group
ZanistaBot	blocks	named in its own group

Other signals

Publishes /llms.txt	no
Publishes /ai.txt	no
Publishes /.well-known/tdmrep.json	no
Reserves TDM rights (TDMRep file, header or meta)	no
Declares Content Signals (robots.txt or header)	no
Says ai-train=no in a Content Signal	no
Points to an RSL licence (robots.txt License line or Link header)	no
Answers with an x402 payment offer (lower bound: only the homepage is probed)	no
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	no
Sends noai / noimageai directives	no
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 05086707ea7c0ae7…

User-Agent: *
Disallow: /search?
Disallow: /metrics/
Disallow: /auth/
Disallow: /content/
Disallow: /comments/*
Disallow: /sign_in?
Disallow: /sign_up?
Disallow: /become-an-author?
Disallow: /profiles/*/articles
Disallow: /profiles/*/activities

Sitemap: http://theconversation.com/sitemap.xml
Sitemap: http://theconversation.com/africa/sitemap_news.xml
Sitemap: http://theconversation.com/au/sitemap_news.xml
Sitemap: http://theconversation.com/fr/sitemap_news.xml
Sitemap: http://theconversation.com/global/sitemap_news.xml
Sitemap: http://theconversation.com/uk/sitemap_news.xml
Sitemap: http://theconversation.com/us/sitemap_news.xml

robots.txt versions

From	Through	State	Version
2026-10-26	2026-10-27	ok 200	`1423b17b4d2b`

Timeline of changes

Date	What changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	theconversation.com	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2025-09-15 CC-MAIN-2025-38	theconversation.com	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2024-09-16 CC-MAIN-2024-38	theconversation.com	ok view	blocks	allows	unknown	blocks	allows	unknown	allows
2023-10-02 CC-MAIN-2023-40	theconversation.com	ok view	blocks	allows	unknown	allows	allows	unknown	allows
2022-10-03 CC-MAIN-2022-40	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2021-09-27 CC-MAIN-2021-39	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2019-09-23 CC-MAIN-2019-39	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-09-24 CC-MAIN-2018-39	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-01-29 CC-MAIN-2018-05	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-10-23 CC-MAIN-2017-43	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-07-24 CC-MAIN-2017-30	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-04-24 CC-MAIN-2017-17	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-01-23 CC-MAIN-2017-04	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-10-03 CC-MAIN-2016-40	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-09-05 CC-MAIN-2016-36	theconversation.com	ok view	allows	allows	unknown	allows	allows	unknown	allows

Evidence

Every daily observation of theconversation.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/theconversation.com