elpais.com: what it tells AI crawlers

On 27 October 2026, elpais.com's robots.txt fully blocked ClaudeBot, Claude-SearchBot, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 1k (Tranco)
robots.txt: ok (HTTP 200) · https://elpais.com/robots.txt
unchanged since 2026-10-26 · sha256 7f5346154ef69578… · view as recorded
The * group: allows applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	allows	through the * group
OAI-SearchBot	allows	through the * group
ChatGPT-User	allows	through the * group
ClaudeBot	blocks	named in its own group
Claude-SearchBot	blocks	named in its own group
Claude-User	blocks	named in its own group
PerplexityBot	blocks	named in its own group
Perplexity-User	blocks	named in its own group
Googlebot	allows	through the * group
Google-Extended	allows	through the * group
Applebot-Extended	allows	through the * group
CCBot	blocks	named in its own group
Bytespider	blocks	named in its own group
Meta-ExternalAgent	allows	through the * group
Amazonbot	blocks	named in its own group
Amzn-SearchBot	blocks	named in its own group
Amzn-User	blocks	named in its own group
DeepSeekBot	blocks	named in its own group
DuckAssistBot	blocks	named in its own group
ExaBot	blocks	named in its own group
LinerBot	blocks	named in its own group
MistralAI-User	blocks	named in its own group
omgilibot	blocks	named in its own group
PetalBot	blocks	named in its own group

Other signals

Publishes /llms.txt	no
Publishes /ai.txt	no
Publishes /.well-known/tdmrep.json	no
Reserves TDM rights (TDMRep file, header or meta)	not observed
Declares Content Signals (robots.txt or header)	not observed
Says ai-train=no in a Content Signal	not observed
Points to an RSL licence (robots.txt License line or Link header)	not observed
Answers with an x402 payment offer (lower bound: only the homepage is probed)	not observed
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	not observed
Sends noai / noimageai directives	not observed
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 7f5346154ef69578…

User-agent: PetalBot
Disallow: /

# Bloque bots global confirmado 20260505
User-agent: bedrock-agentcore-browser
Disallow: /

User-agent: Amazon-Bot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: AmazonProductDiscoveryBot
Disallow: /

User-agent: Amzn-SearchBot
Disallow: /

User-agent: Amzn-User
Disallow: /

User-agent: Arquivo-web-crawler
Disallow: /

User-agent: bnf.fr_bot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Claude-Extension
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Comet
Disallow: /

User-agent: CloudFlare-AlwaysOnline
Disallow: /

User-agent: Cloudflare-Healthchecks
Disallow: /

User-agent: CloudflareBrowserRendering
Disallow: /

User-agent: Cloudflare-Radar
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: DuckAssistBot
Disallow: /

User-agent: archive.org_bot
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: MistralAI-User
Disallow: /

User-agent: Moreover
Disallow: /

User-agent: Adbeat
Disallow: /

User-agent: Argus
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Baiduspider-image
Disallow: /

User-agent: Bitdefender
Disallow: /

User-agent: Blogtrottr
Disallow: /

User-agent: BloombergBot
Disallow: /

User-agent: BoardReader
Disallow: /

User-agent: checkmy.ws
Disallow: /

User-agent: Clickagy
Disallow: /

User-agent: CookieBot
Disallow: /

User-agent: cXensebot
Disallow: /

User-agent: DataObserver
Disallow: /

User-agent: DomainStatsBot
Disallow: /

User-agent: Embedly
Disallow: /

User-agent: Feedspotbot
Disallow: /

User-agent: FlipboardProxy
Disallow: /

User-agent: Getintent
Disallow: /

User-agent: Google-Apps-Script
Disallow: /

User-agent: GoogleDrive
Disallow: /

User-agent: Grapevine6
Disallow: /

User-agent: Gumgum
Disallow: /

User-agent: HaloBot
Disallow: /

User-agent: HubSpot Crawler
Disallow: /

User-agent: IbouBot
Disallow: /

User-agent: Inout
Disallow: /

User-agent: LinerBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: Manus
Disallow: /

User-agent: Meltwater
Disallow: /

User-agent: Metajob
Disallow: /

User-agent: Muckrack
Disallow: /

User-agent: NetSeer
Disallow: /

User-agent: Netvibes
Disallow: /

User-agent: Onyphe
Disallow: /

User-agent: OpenClaw
Disallow: /

User-agent: Opoint
Disallow: /

User-agent: OutbrainBot
Disallow: /

User-agent: PlumAnalytics
Disallow: /

User-agent: ProSitemaps
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: proximic
Disallow: /

User-agent: PubMatic
Disallow: /

User-agent: Quantcastbot
Disallow: /

User-agent: SearchConcepts
Disallow: /

User-agent: Semantic-Visions
Disallow: /

User-agent: SentiBot
Disallow: /

User-agent: Sindup
Disallow: /

User-agent: Sirdata
Disallow: /

User-agent: SmartologyBot
Disallow: /

User-agent: Superfeedr
Disallow: /

User-agent: surlybot
Disallow: /

User-agent: Synaptive
Disallow: /

User-agent: TinEye
Disallow: /

User-agent: Trendiction
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: Ubermetrics
Disallow: /

User-agent: Vebidoo
Disallow: /

User-agent: Webcorp
Disallow: /

User-agent: Weborama
Disallow: /

User-agent: WebPageTest
Disallow: /

User-agent: YandexAccessibilityBot
Disallow: /

User-agent: YandexDirect
Disallow: /

User-agent: YandexDirectDyn
Disallow: /

User-agent: YaDirectFetcher
Disallow: /

User-agent: YandexFavicons
Disallow: /

User-agent: YandexMetrika
Disallow: /

User-agent: YandexMobileBot
Disallow: /

User-agent: YandexNews
Disallow: /

User-agent: YandexRCA
Disallow: /

User-agent: YandexRenderResourcesBot
Disallow: /

User-agent: YandexUserproxy
Disallow: /

# OAI-AdsBot se mantiene permitido y no se bloquea.

# Parametros transversales
User-agent: *
Disallow: /*core_dmena

User-agent: *
Disallow: /pruebas/
Disallow: /publicidad/
Disallow: /notificarelacionadas
Disallow: /*.swf$
Disallow: /eskupTSUpdate
Disallow: /ThreadeskupSimple
Disallow: /OuteskupSimple
# Anadidos 20250325 por oscarf
Disallow: /Ineskup
Disallow: /Outeskup
Disallow: /OuteskupSimpleMustache
Disallow: /Msgeskup
Disallow: /Profileeskup
Disallow: /Threadeskup

# Disallow: /.well-known/amphtml/
Disallow: /t/
Disallow: *?preview=true$

# Bots bloqueados
User-agent: 008
Disallow: /
User-agent: AddThis.com
Disallow: /
User-agent: admantx
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: BDCbot
Disallow: /
User-agent: Bender
Disallow: /
User-agent: BIXOCRAWLER
Disallow: /
User-agent: bl.uk_lddc_bot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: BUbiNG
Disallow: /
User-agent: Cliqzbot
Disallow: /
User-agent: CNCDialer
Disallow: /
User-agent: crawler4j
Disallow: /
User-agent: CrystalSemanticsBot
Disallow: /
User-agent: CyberAlert
Disallow: /
User-agent: DigExt
Disallow: /
User-agent: discobot
Disallow: /
User-agent: discoverybot
Disallow: /
User-agent: dloader
Disallow: /
User-agent: dloader(NaverRobot)
Disallow: /
User-agent: DOC
Disallow: /
User-agent: Download Ninja
Disallow: /
User-agent: DTS Agent
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: FairShare
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: Flamingo_SearchEngine
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: Heritrix
Disallow: /
User-agent: heritrix/3.3.0
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: integromedb
Disallow: /
User-agent: IstellaBot
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Jyxobot
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: Kimengi
Disallow: /
User-agent: Kimengi/nineconnections.com
Disallow: /
User-agent: larbin
Disallow: /
User-agent: LexxeBot/1.0
Disallow: /
User-agent: libwww
Disallow: /
User-agent: linko
Disallow: /
User-agent: Livelapbot
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: Maxthon
Disallow: /
User-agent: MetaURI
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: Moreoverbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: nabot
Disallow: /
User-agent: NaverBot
Disallow: /
User-agent: NerdByNature.Bot
Disallow: /
User-agent: netEstate NE Crawler
Disallow: /
User-agent: NetSeer crawler
Disallow: /
User-agent: Newscan
Disallow: /
User-agent: NextGenSearchBot
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: nutch
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: Orthogaffe
Disallow: /
User-agent: PiplBot
Disallow: /
User-agent: Pixray-Seeker
Disallow: /
User-agent: psbot
Disallow: /
User-agent: QuerySeekerSpider
Disallow: /
User-agent: seokicks
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: SiteBot
Disallow: /
User-agent: SiteBot/0.1
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: sogou
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: Spinn3r
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: trendictionbot
Disallow: /
User-agent: trovitBot
Disallow: /
User-agent: UbiCrawler
Disallow: /
User-agent: uMBot-LN
Disallow: /
User-agent: UnisterBot
Disallow: /
User-agent: UniversalFeedParser
Disallow: /
User-agent: WBSearchBot
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebZIP
Disallow: /
User-agent: WeSEE:Search
Disallow: /
User-agent: wget
Disallow: /
User-agent: Wotbot
Disallow: /
User-agent: wotbox
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: Yasni
Disallow: /
User-agent: Zao
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: ZyBORG
Disallow: /

robots.txt versions

From	Through	State	Version
2026-10-26	2026-10-27	ok 200	`7f5346154ef6`

Timeline of changes

Date	What changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	elpais.com	ok view	allows	allows	unknown	blocks	blocks	unknown	blocks
2025-09-15 CC-MAIN-2025-38	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2024-09-16 CC-MAIN-2024-38	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2023-10-02 CC-MAIN-2023-40	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2022-10-03 CC-MAIN-2022-40	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2021-09-27 CC-MAIN-2021-39	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2019-09-23 CC-MAIN-2019-39	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-09-24 CC-MAIN-2018-39	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-04-23 CC-MAIN-2018-17	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-01-29 CC-MAIN-2018-05	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-10-23 CC-MAIN-2017-43	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-07-24 CC-MAIN-2017-30	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-04-24 CC-MAIN-2017-17	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-01-23 CC-MAIN-2017-04	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-10-03 CC-MAIN-2016-40	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-09-05 CC-MAIN-2016-36	elpais.com	ok view	allows	allows	unknown	allows	allows	unknown	allows

Evidence

Every daily observation of elpais.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/elpais.com