elmundo.es: what it tells AI crawlers
On 28 October 2026, elmundo.es's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.
Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.
- Panel
- top 10k (Tranco)
- robots.txt
- ok (HTTP 200) · https://www.elmundo.es/robots.txt
unchanged since 2026-10-27 · sha256c66e0723179e0a96…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-28
| User agent | Verdict | How |
|---|---|---|
| GPTBot | blocks | named in its own group |
| OAI-SearchBot | blocks | named in its own group |
| ChatGPT-User | blocks | named in its own group |
| ClaudeBot | blocks | named in its own group |
| Claude-SearchBot | blocks | named in its own group |
| Claude-User | blocks | named in its own group |
| PerplexityBot | blocks | named in its own group |
| Perplexity-User | blocks | named in its own group |
| Googlebot | allows | through the * group |
| Google-Extended | blocks | named in its own group |
| Applebot-Extended | blocks | named in its own group |
| CCBot | blocks | named in its own group |
| Bytespider | blocks | named in its own group |
| Meta-ExternalAgent | blocks | named in its own group |
| Amazonbot | blocks | named in its own group |
| anthropic-ai | blocks | named in its own group |
| Claude-Web | blocks | named in its own group |
| cohere-ai | blocks | named in its own group |
| cohere-training-data-crawler | blocks | named in its own group |
| Diffbot | blocks | named in its own group |
| DuckAssistBot | blocks | named in its own group |
| ExaBot | blocks | named in its own group |
| Google-CloudVertexBot | blocks | named in its own group |
| ImagesiftBot | blocks | named in its own group |
| meta-externalfetcher | blocks | named in its own group |
| MistralAI-User | blocks | named in its own group |
| omgili | blocks | named in its own group |
| omgilibot | blocks | named in its own group |
| PanguBot | blocks | named in its own group |
| Timpibot | blocks | named in its own group |
| YouBot | blocks | named in its own group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | no |
| Declares Content Signals (robots.txt or header) | no |
| Says ai-train=no in a Content Signal | no |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
robots.txt as recorded sha256 2903b50f8762f7b5…
User-agent: * Disallow: /s/ Disallow: /cgi-bin/ Disallow: /perl/ Disallow: /foros/ Disallow: /sv/comprar-vender-piso-casa.html # suvivienda.es landing page (20080710 - jquiroga@el-mundo.net) Disallow: /sv/landing/ # suvivienda.es landing pages (20080909 - jquiroga@el-mundo.net) Disallow: /elmundo/envia_noticia.html Disallow: /elmundo/graciasporvalorar.html Disallow: /elmundo/rectificaciones/enviar.html Disallow: /participacion/usuario_registrado.html Disallow: /participacion/normas_participacion.html Disallow: /yodona/movil/listado/ Disallow: /hacemosesto/ Disallow: /buscador/ Disallow: /mapadelsitio/faq.html Disallow: /especiales/2008/09/internacional/elecciones_eeuu/noche_electoral/index.html/ Disallow: /especiales/2004/03/espana/14m/resultados/ Disallow: /elmundo/2012/09/03/baleares/1346690156.html Disallow: /andalucia/2015/05/28/5566292fca4741df5b8b45a8.html Disallow: /elmundo/2009/02/22/espana/1235328259.html Disallow: /elmundo/2013/04/24/barcelona/1366793646.html Disallow: /elmundo/2009/05/31/espana/1243802563.html Disallow: /loc/2014/06/11/539722c9e2704ecc718b4599.html Disallow: /loc/2014/08/23/53f761cdca4741c1568b4580.html Disallow: /loc/2014/11/01/5453de64ca4741583a8b4589.html Disallow: /loc/2014/11/01/545408b022601dee378b457e.html Disallow: /america/2011/06/28/colombia/1309270959.html Disallow: /cataluna/2014/05/03/5364bfff22601d08458b456c.html Disallow: /america/2012/07/09/noticias/1341866548.html Disallow: /tecnologia/2016/03/15/56e7f6f146163f97518b4574.html Disallow: /elmundo/2011/03/11/espana/1299854139.html Disallow: /espana/2014/02/04/52f0f41c268e3ecd5d8b457b.html Disallow: /elmundo/2006/04/27/espana/1146124044.html Disallow: /andalucia/2014/09/09/540eb15422601d8e308b456c.html Disallow: /elmundo/2011/03/17/cultura/1300374568.html Disallow: /elmundo/2000/10/4/sociedad/970675851.html Disallow: /elmundo/2011/05/10/galicia/1305058376.html Disallow: /accesible/elmundo/2013/04/24/barcelona/1366793646.html Disallow: /loc/2016/02/27/56d0ac4fe2704e424b8b4698.html Disallow: /suplementos/cronica/2008/661/1213480803.html Disallow: /elmundo/2008/11/03/madrid/1225730692.html Disallow: /elmundo/2009/01/19/madrid/1232393538.html Disallow: /elmundo/2012/08/18/madrid/1345296850.html Disallow: /elmundo/2008/05/22/madrid/1211449408.html Disallow: /elmundo/2009/06/15/madrid/1245065175.html Disallow: /elmundo/2008/11/13/madrid/1226590842.html Disallow: /elmundo/2008/07/03/madrid/1215085120.html Disallow: /elmundo/2008/06/14/valencia/1213430561.html Disallow: /elmundo/2008/06/10/madrid/1213111777.html Disallow: /elmundo/2008/06/04/madrid/1212576271.html Disallow: /elmundo/2012/08/19/madrid/1345402692.html Disallow: /elmundo/2010/02/08/madrid/1265633177.html Disallow: /elmundo/2013/06/07/madrid/1370603306.html Disallow: /andalucia/2014/01/17/52d95289ca4741d5798b4582.html Disallow: /andalucia/2014/01/10/52cfdc3cca474162218b4573.html Disallow: /cataluna/2014/05/03/5364bfff22601d08458b456c.html Disallow: /elmundo/2008/07/24/barcelona/1216878794.html Disallow: /elmundo/2011/03/09/economia/1299627923.html Disallow: /elmundo/2011/03/07/economia/1299519600.html Disallow: /elmundo/2006/11/11/espana/1163285160.html Disallow: /loc/2015/06/27/558d738e268e3e8e688b458e.html Disallow: /economia/2015/03/16/5506120ee2704e95468b456c.html Disallow: /elmundo/2001/04/03/sociedad/986312895.html Disallow: /elmundo/2012/03/15/barcelona/1331824835.html Disallow: /cronica/2001/CR286/CR286-01.html Disallow: /elmundo/2007/03/02/espana/1172866629.html Disallow: /loc/2016/02/01/56ab90a8e2704ef7558b4597.html Disallow: /loc/2016/07/22/5790f83e468aeb4b178b45b0.html Disallow: /cataluna/2015/05/04/55466710ca4741e7348b456c.html Disallow: /elmundo/hemeroteca/2016/02/27/m/loc.html Disallow: /andalucia/2015/07/07/559c18d322601dbf408b45ad.html Disallow: /loc/2016/05/07/572cc8dbe2704ecc6d8b45bd.html Disallow: /elmundovino/noticia.html?vi_seccion=2&vs_fecha=201112&vs_noticia=1323073889 Disallow: /elmundovino/noticia.html?vi_seccion=8&vs_fecha=201205&vs_noticia=1336118286 Disallow: /elmundo/2011/08/20/paisvasco/1313839269.html Disallow: /elmundo/cataluna/2016/12/07/584839e146163f490f8b45cd.html Disallow: /cronica/2016/08/21/57b95b3546163f010c8b4618.html Disallow: /loc/2014/02/08/52f54acd268e3e9e408b4581.html Disallow: /baleares/2016/02/18/56c60537268e3e0f348b4581.html Disallow: /baleares/2016/02/17/56c43f19268e3e0e048b45ea.html Disallow: /elmundo/2013/02/25/economia/1361787909.html Disallow: /suplementos/cronica/2009/696/1234888844.html Disallow: /elmundo/2010/01/11/baleares/1263201527.html Disallow: /madrid/2014/10/10/5436de0de2704e062a8b456d.html Disallow: /suplementos/cronica/2009/696/1234888844.html Disallow: /espana/2014/08/03/53dd2bf9ca474155388b457f.html Disallow: /loc-nc.html Disallow: /yodona-nc.html Disallow: /papel/firmas/2017/01/15/5878c3d8468aeb226a8b4611.html Disallow: /andalucia/2016/06/14/575fb86c22601d67298b4581.html Disallow: /elmundo/2009/10/13/andalucia/1255414645.html Disallow: /madrid/2015/11/26/5656fe7fca47419a388b467f.html Disallow: /madrid/2015/11/10/5641f1eae2704eca748b4665.html Disallow: /madrid/2015/11/12/5644d1f646163fa32d8b464b.html Disallow: /madrid/2015/11/06/563cac15e2704e59778b45b9.html Disallow: /elmundo/2012/05/01/espana/1335891011.html Disallow: /elmundo/2009/08/12/madrid/1250068823.html Disallow: /elmundo/2004/07/21/cultura/1090428220.html Disallow: /loc/2014/08/02/53dbb15c268e3e05498b4583.html Disallow: /elmundo/2009/05/11/andalucia_sevilla/1242025053.html Disallow: /elmundo/2011/03/09/galicia/1299689292.html Disallow: /mundodinero/2007/06/29/economia/1183131633.html Disallow: /elmundo/2012/02/06/barcelona/1328514332.html Disallow: /elmundo/2011/04/05/andalucia/1301956474.html Disallow: /cronica/2016/11/13/5826f08cca47412c708b457b.html Disallow: /elmundo/2012/06/02/baleares/1338628762.html Disallow: /elmundo/2012/06/05/baleares/1338891185.html Disallow: /elmundo/2012/06/05/baleares/1338897382.html Disallow: /elmundo/2012/06/08/baleares/1339143375.html Disallow: /elmundo/2012/07/29/baleares/1343553248.html Disallow: /elmundo/2012/08/14/baleares/1344937738.html Disallow: /elmundo/2012/08/24/baleares/1345805179.html Disallow: /elmundo/2012/10/20/baleares/1350757047.html Disallow: /elmundo/2013/01/19/baleares/1358622048.html Disallow: /elmundo/2013/03/03/baleares/1362310749.html Disallow: /loc/2015/07/05/5596c74dca474150048b45a5.html Disallow: /blogs/elmundo/momen-tvs/2016/10/12/adara-la-enchufada-de-gh-17.html Disallow: /economia/2015/12/23/567aa33b22601dc7528b458d.html Disallow: /sociedad/2016/05/16/5738a99f468aeb3e068b458a.html Disallow: /elmundo/2011/01/29/alicante/1296303929.html Disallow: /economia/2016/05/08/572caa3722601db9058b4624.html Disallow: /espana/2014/09/08/540cf013268e3eac5f8b457f.html User-agent: 008 Disallow: / User-agent: UnisterBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: Maxthon Disallow: / User-agent: CNCDialer Disallow: /
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-27 | 2026-10-28 | ok 200 | c66e0723179e | diff with previous |
| 2026-10-26 | 2026-10-26 | ok 200 | 94820b263eb2 |
Timeline of changes
| Date | What changed |
|---|---|
| 2026-10-27 | robots.txt changed |
| 2026-10-27 | now blocks Amazonbot (was allow) |
| 2026-10-27 | now blocks Applebot-Extended (was allow) |
| 2026-10-27 | now blocks Bytespider (was allow) |
| 2026-10-27 | now blocks Claude-SearchBot (was allow) |
| 2026-10-27 | now blocks Claude-User (was allow) |
| 2026-10-27 | now blocks Claude-Web (was allow) |
| 2026-10-27 | now blocks ClaudeBot (was allow) |
| 2026-10-27 | now blocks cohere-ai (was allow) |
| 2026-10-27 | now blocks cohere-training-data-crawler (was allow) |
| 2026-10-27 | now blocks Diffbot (was allow) |
| 2026-10-27 | now blocks DuckAssistBot (was allow) |
| 2026-10-27 | now blocks Google-CloudVertexBot (was allow) |
| 2026-10-27 | now blocks ImagesiftBot (was allow) |
| 2026-10-27 | now blocks Meta-ExternalAgent (was allow) |
| 2026-10-27 | now blocks meta-externalfetcher (was allow) |
| 2026-10-27 | now blocks MistralAI-User (was allow) |
| 2026-10-27 | now blocks omgili (was allow) |
| 2026-10-27 | now blocks PanguBot (was allow) |
| 2026-10-27 | now blocks Perplexity-User (was allow) |
| 2026-10-27 | now blocks Timpibot (was allow) |
| 2026-10-27 | now blocks YouBot (was allow) |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | www.elmundo.es | ok view | blocks | blocks | unknown | allows | allows | unknown | blocks |
| 2025-09-15 CC-MAIN-2025-38 | www.elmundo.es | ok view | blocks | blocks | unknown | allows | allows | unknown | allows |
| 2024-09-16 CC-MAIN-2024-38 | www.elmundo.es | ok view | blocks | blocks | unknown | allows | allows | unknown | allows |
| 2023-10-02 CC-MAIN-2023-40 | www.elmundo.es | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2022-10-03 CC-MAIN-2022-40 | www.elmundo.es | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | www.elmundo.es | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | www.elmundo.es | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | www.elmundo.es | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-10-22 CC-MAIN-2018-43 | www.elmundo.es | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | www2.elmundo.es | absent | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-07-23 CC-MAIN-2018-30 | www2.elmundo.es | absent | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-04-23 CC-MAIN-2018-17 | www2.elmundo.es | absent | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-01-29 CC-MAIN-2018-05 | elmundo.es | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2017-10-23 CC-MAIN-2017-43 | www.elmundo.es | absent | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | www.elmundo.es | absent | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-07-24 CC-MAIN-2017-30 | elmundo.es | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2017-04-24 CC-MAIN-2017-17 | www.elmundo.es | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-01-23 CC-MAIN-2017-04 | www.elmundo.es | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-10-03 CC-MAIN-2016-40 | www.elmundo.es | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-09-05 CC-MAIN-2016-36 | www.elmundo.es | ok view | allows | allows | unknown | allows | allows | unknown | allows |
Evidence
Every daily observation of elmundo.es is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).
API: https://infrarabbit.com/crawlers/api/v1/policy/elmundo.es