elmundo.es: what it tells AI crawlers

On 28 October 2026, elmundo.es's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel
top 10k (Tranco)
robots.txt
ok (HTTP 200) · https://www.elmundo.es/robots.txt
unchanged since 2026-10-27 · sha256 c66e0723179e0a96… · view as recorded
The * group
allows applies to every agent without its own group

Per AI bot, 2026-10-28

User agentVerdictHow
GPTBotblocksnamed in its own group
OAI-SearchBotblocksnamed in its own group
ChatGPT-Userblocksnamed in its own group
ClaudeBotblocksnamed in its own group
Claude-SearchBotblocksnamed in its own group
Claude-Userblocksnamed in its own group
PerplexityBotblocksnamed in its own group
Perplexity-Userblocksnamed in its own group
Googlebotallowsthrough the * group
Google-Extendedblocksnamed in its own group
Applebot-Extendedblocksnamed in its own group
CCBotblocksnamed in its own group
Bytespiderblocksnamed in its own group
Meta-ExternalAgentblocksnamed in its own group
Amazonbotblocksnamed in its own group
anthropic-aiblocksnamed in its own group
Claude-Webblocksnamed in its own group
cohere-aiblocksnamed in its own group
cohere-training-data-crawlerblocksnamed in its own group
Diffbotblocksnamed in its own group
DuckAssistBotblocksnamed in its own group
ExaBotblocksnamed in its own group
Google-CloudVertexBotblocksnamed in its own group
ImagesiftBotblocksnamed in its own group
meta-externalfetcherblocksnamed in its own group
MistralAI-Userblocksnamed in its own group
omgiliblocksnamed in its own group
omgilibotblocksnamed in its own group
PanguBotblocksnamed in its own group
Timpibotblocksnamed in its own group
YouBotblocksnamed in its own group

Other signals

Publishes /llms.txtno
Publishes /ai.txtno
Publishes /.well-known/tdmrep.jsonno
Reserves TDM rights (TDMRep file, header or meta)no
Declares Content Signals (robots.txt or header)no
Says ai-train=no in a Content Signalno
Points to an RSL licence (robots.txt License line or Link header)no
Answers with an x402 payment offer (lower bound: only the homepage is probed)no
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)no
Sends noai / noimageai directivesno
Refused our fetcher (403 / challenge)no

robots.txt as recorded sha256 0a5500b94fd2b521…

User-agent: *
Disallow: /s/
Disallow: /cgi-bin/
Disallow: /perl/
Disallow: /foros/
Disallow: /sv/comprar-vender-piso-casa.html     # suvivienda.es landing page (20080710 - jquiroga@el-mundo.net)
Disallow: /sv/landing/                          # suvivienda.es landing pages (20080909 - jquiroga@el-mundo.net)
Disallow: /elmundo/envia_noticia.html
Disallow: /elmundo/graciasporvalorar.html
Disallow: /elmundo/rectificaciones/enviar.html
Disallow: /participacion/usuario_registrado.html
Disallow: /participacion/normas_participacion.html
Disallow: /yodona/movil/listado/
Disallow: /hacemosesto/
Disallow: /buscador/
Disallow: /mapadelsitio/faq.html
Disallow: /especiales/2008/09/internacional/elecciones_eeuu/noche_electoral/index.html/
Disallow: /especiales/2004/03/espana/14m/resultados/
Disallow: /elmundo/2012/09/03/baleares/1346690156.html
Disallow: /andalucia/2015/05/28/5566292fca4741df5b8b45a8.html
Disallow: /elmundo/2009/02/22/espana/1235328259.html
Disallow: /elmundo/2013/04/24/barcelona/1366793646.html
Disallow: /elmundo/2009/05/31/espana/1243802563.html
Disallow: /loc/2014/06/11/539722c9e2704ecc718b4599.html
Disallow: /loc/2014/08/23/53f761cdca4741c1568b4580.html
Disallow: /loc/2014/11/01/5453de64ca4741583a8b4589.html
Disallow: /loc/2014/11/01/545408b022601dee378b457e.html
Disallow: /america/2011/06/28/colombia/1309270959.html
Disallow: /cataluna/2014/05/03/5364bfff22601d08458b456c.html
Disallow: /america/2012/07/09/noticias/1341866548.html
Disallow: /tecnologia/2016/03/15/56e7f6f146163f97518b4574.html
Disallow: /elmundo/2011/03/11/espana/1299854139.html
Disallow: /espana/2014/02/04/52f0f41c268e3ecd5d8b457b.html
Disallow: /elmundo/2006/04/27/espana/1146124044.html
Disallow: /andalucia/2014/09/09/540eb15422601d8e308b456c.html
Disallow: /elmundo/2011/03/17/cultura/1300374568.html
Disallow: /elmundo/2000/10/4/sociedad/970675851.html
Disallow: /elmundo/2011/05/10/galicia/1305058376.html
Disallow: /accesible/elmundo/2013/04/24/barcelona/1366793646.html
Disallow: /loc/2016/02/27/56d0ac4fe2704e424b8b4698.html
Disallow: /suplementos/cronica/2008/661/1213480803.html
Disallow: /elmundo/2008/11/03/madrid/1225730692.html
Disallow: /elmundo/2009/01/19/madrid/1232393538.html
Disallow: /elmundo/2012/08/18/madrid/1345296850.html
Disallow: /elmundo/2008/05/22/madrid/1211449408.html
Disallow: /elmundo/2009/06/15/madrid/1245065175.html
Disallow: /elmundo/2008/11/13/madrid/1226590842.html
Disallow: /elmundo/2008/07/03/madrid/1215085120.html
Disallow: /elmundo/2008/06/14/valencia/1213430561.html
Disallow: /elmundo/2008/06/10/madrid/1213111777.html
Disallow: /elmundo/2008/06/04/madrid/1212576271.html
Disallow: /elmundo/2012/08/19/madrid/1345402692.html
Disallow: /elmundo/2010/02/08/madrid/1265633177.html
Disallow: /elmundo/2013/06/07/madrid/1370603306.html
Disallow: /andalucia/2014/01/17/52d95289ca4741d5798b4582.html
Disallow: /andalucia/2014/01/10/52cfdc3cca474162218b4573.html
Disallow: /cataluna/2014/05/03/5364bfff22601d08458b456c.html
Disallow: /elmundo/2008/07/24/barcelona/1216878794.html
Disallow: /elmundo/2011/03/09/economia/1299627923.html
Disallow: /elmundo/2011/03/07/economia/1299519600.html
Disallow: /elmundo/2006/11/11/espana/1163285160.html
Disallow: /loc/2015/06/27/558d738e268e3e8e688b458e.html
Disallow: /economia/2015/03/16/5506120ee2704e95468b456c.html
Disallow: /elmundo/2001/04/03/sociedad/986312895.html
Disallow: /elmundo/2012/03/15/barcelona/1331824835.html
Disallow: /cronica/2001/CR286/CR286-01.html
Disallow: /elmundo/2007/03/02/espana/1172866629.html
Disallow: /loc/2016/02/01/56ab90a8e2704ef7558b4597.html
Disallow: /loc/2016/07/22/5790f83e468aeb4b178b45b0.html
Disallow: /cataluna/2015/05/04/55466710ca4741e7348b456c.html
Disallow: /elmundo/hemeroteca/2016/02/27/m/loc.html
Disallow: /andalucia/2015/07/07/559c18d322601dbf408b45ad.html
Disallow: /loc/2016/05/07/572cc8dbe2704ecc6d8b45bd.html
Disallow: /elmundovino/noticia.html?vi_seccion=2&vs_fecha=201112&vs_noticia=1323073889
Disallow: /elmundovino/noticia.html?vi_seccion=8&vs_fecha=201205&vs_noticia=1336118286
Disallow: /elmundo/2011/08/20/paisvasco/1313839269.html
Disallow: /elmundo/cataluna/2016/12/07/584839e146163f490f8b45cd.html
Disallow: /cronica/2016/08/21/57b95b3546163f010c8b4618.html
Disallow: /loc/2014/02/08/52f54acd268e3e9e408b4581.html 
Disallow: /baleares/2016/02/18/56c60537268e3e0f348b4581.html
Disallow: /baleares/2016/02/17/56c43f19268e3e0e048b45ea.html
Disallow: /elmundo/2013/02/25/economia/1361787909.html
Disallow: /suplementos/cronica/2009/696/1234888844.html
Disallow: /elmundo/2010/01/11/baleares/1263201527.html
Disallow: /madrid/2014/10/10/5436de0de2704e062a8b456d.html
Disallow: /suplementos/cronica/2009/696/1234888844.html
Disallow: /espana/2014/08/03/53dd2bf9ca474155388b457f.html
Disallow: /loc-nc.html
Disallow: /yodona-nc.html
Disallow: /papel/firmas/2017/01/15/5878c3d8468aeb226a8b4611.html
Disallow: /andalucia/2016/06/14/575fb86c22601d67298b4581.html
Disallow: /elmundo/2009/10/13/andalucia/1255414645.html




User-agent: 008
Disallow: /

User-agent: UnisterBot
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: sitecheck.internetseer.com
Disallow: /

User-agent: Zealbot
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: WebZIP
Disallow: /
 
User-agent: linko
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: Microsoft.URL.Control
Disallow: /

User-agent: larbin
Disallow: /

User-agent: libwww
Disallow: /

User-agent: ZyBORG
Disallow: /

User-agent: Download Ninja
Disallow: /

User-agent: Maxthon
Disallow: /

User-agent: CNCDialer
Disallow: /

robots.txt versions

FromThroughStateVersion
2026-10-272026-10-28ok 200c66e0723179ediff with previous
2026-10-262026-10-26ok 20094820b263eb2

Timeline of changes

DateWhat changed
2026-10-27robots.txt changed
2026-10-27now blocks Amazonbot (was allow)
2026-10-27now blocks Applebot-Extended (was allow)
2026-10-27now blocks Bytespider (was allow)
2026-10-27now blocks Claude-SearchBot (was allow)
2026-10-27now blocks Claude-User (was allow)
2026-10-27now blocks Claude-Web (was allow)
2026-10-27now blocks ClaudeBot (was allow)
2026-10-27now blocks cohere-ai (was allow)
2026-10-27now blocks cohere-training-data-crawler (was allow)
2026-10-27now blocks Diffbot (was allow)
2026-10-27now blocks DuckAssistBot (was allow)
2026-10-27now blocks Google-CloudVertexBot (was allow)
2026-10-27now blocks ImagesiftBot (was allow)
2026-10-27now blocks Meta-ExternalAgent (was allow)
2026-10-27now blocks meta-externalfetcher (was allow)
2026-10-27now blocks MistralAI-User (was allow)
2026-10-27now blocks omgili (was allow)
2026-10-27now blocks PanguBot (was allow)
2026-10-27now blocks Perplexity-User (was allow)
2026-10-27now blocks Timpibot (was allow)
2026-10-27now blocks YouBot (was allow)

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

CrawlHostStateGPTBotOAI-SearchBotChatGPT-UserClaudeBotClaude-SearchBotClaude-UserPerplexityBot
2026-09-21 CC-MAIN-2026-39www.elmundo.esok viewblocksblocksunknownallowsallowsunknownblocks
2025-09-15 CC-MAIN-2025-38www.elmundo.esok viewblocksblocksunknownallowsallowsunknownallows
2024-09-16 CC-MAIN-2024-38www.elmundo.esok viewblocksblocksunknownallowsallowsunknownallows
2023-10-02 CC-MAIN-2023-40www.elmundo.esok viewallowsallowsunknownallowsallowsunknownallows
2022-10-03 CC-MAIN-2022-40www.elmundo.esok viewallowsallowsunknownallowsallowsunknownallows
2021-09-27 CC-MAIN-2021-39www.elmundo.esok viewallowsallowsunknownallowsallowsunknownallows
2020-09-28 CC-MAIN-2020-40www.elmundo.esok viewallowsallowsunknownallowsallowsunknownallows
2019-09-23 CC-MAIN-2019-39www.elmundo.esok viewallowsallowsunknownallowsallowsunknownallows
2018-10-22 CC-MAIN-2018-43www.elmundo.esok viewallowsallowsunknownallowsallowsunknownallows
2018-09-24 CC-MAIN-2018-39www2.elmundo.esabsentallowsallowsunknownallowsallowsunknownallows
2018-07-23 CC-MAIN-2018-30www2.elmundo.esabsentallowsallowsunknownallowsallowsunknownallows
2018-04-23 CC-MAIN-2018-17www2.elmundo.esabsentallowsallowsunknownallowsallowsunknownallows
2018-01-29 CC-MAIN-2018-05elmundo.esredirect-onlyunknownunknownunknownunknownunknownunknownunknown
2017-10-23 CC-MAIN-2017-43www.elmundo.esabsentallowsallowsunknownallowsallowsunknownallows
2017-09-25 CC-MAIN-2017-39www.elmundo.esabsentallowsallowsunknownallowsallowsunknownallows
2017-07-24 CC-MAIN-2017-30elmundo.esredirect-onlyunknownunknownunknownunknownunknownunknownunknown
2017-04-24 CC-MAIN-2017-17www.elmundo.esok viewallowsallowsunknownallowsallowsunknownallows
2017-01-23 CC-MAIN-2017-04www.elmundo.esok viewallowsallowsunknownallowsallowsunknownallows
2016-10-03 CC-MAIN-2016-40www.elmundo.esok viewallowsallowsunknownallowsallowsunknownallows
2016-09-05 CC-MAIN-2016-36www.elmundo.esok viewallowsallowsunknownallowsallowsunknownallows

Evidence

Every daily observation of elmundo.es is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).

API: https://infrarabbit.com/crawlers/api/v1/policy/elmundo.es