elpais.com: what it tells AI crawlers

On 27 October 2026, elpais.com's robots.txt fully blocked ClaudeBot, Claude-SearchBot, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://elpais.com/robots.txt
unchanged since 2026-10-26 · sha256 7f5346154ef69578… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 54363a38a572ac30…

User-Agent: *
Disallow: /buscador/
Disallow: /m/buscador/
Disallow: /pruebas/
Disallow: /.well-known/amphtml/
Disallow: /t/
Disallow: /publicidad/
Disallow: /notificarelacionadas
Disallow: /*.swf$
Disallow: /eskupTSUpdate
Disallow: /*=/*x*/


# Bots bloqueados
User-agent: 008 Disallow: /
User-agent: AddThis.com Disallow: /
User-agent: admantx Disallow: /
User-agent: AhrefsBot Disallow: / 
User-agent: BDCbot Disallow: /
User-agent: Bender Disallow: /
User-agent: BIXOCRAWLER Disallow: / 
User-agent: bl.uk_lddc_bot Disallow: /
User-agent: BLEXBot Disallow: /
User-agent: BUbiNG Disallow: / 
User-agent: Cliqzbot Disallow: / 
User-agent: CNCDialer Disallow: / 
User-agent: crawler4j Disallow: / 
User-agent: CrystalSemanticsBot Disallow: /
User-Agent: CyberAlert Disallow: /
User-agent: DigExt Disallow: / 
User-agent: discobot Disallow: /
User-agent: discoverybot Disallow: /
User-agent: dloader Disallow: / 
User-agent: dloader(NaverRobot) Disallow: / 
User-agent: DOC Disallow: / 
User-agent: dotbot Disallow: / 
User-agent: Download Ninja Disallow: /  
User-agent: DTS Agent Disallow: / 
User-agent: Exabot Disallow: / 
User-agent: Ezooms Disallow: /
User-Agent: FairShare Disallow: / 
User-agent: Fetch Disallow: / 
User-agent: Flamingo_SearchEngine Disallow: / 
User-agent: Genieo Disallow: / 
User-agent: Gigabot Disallow: /
User-agent: grub-client Disallow: / 
User-agent: Heritrix Disallow: / 
User-agent: heritrix/3.3.0 Disallow: / 
User-agent: HTTrack Disallow: / 
User-agent: ia_archiver Disallow: / 
User-agent: integromedb Disallow: /
User-agent: IstellaBot Disallow: /
User-agent: JikeSpider Disallow: /
User-agent: Jyxobot Disallow: /
User-agent: k2spider Disallow: / 
User-agent: Kimengi Disallow: / 
User-agent: Kimengi/nineconnections.com Disallow: /
User-agent: larbin Disallow: / 
User-agent: LexxeBot/1.0 Disallow: /
User-agent: libwww Disallow: / 
User-agent: linko Disallow: / 
User-agent: Livelapbot Disallow: / 
User-agent: magpie-crawler Disallow: / 
User-agent: Maxthon Disallow: / 
User-agent: MetaURI Disallow: / 
User-agent: Microsoft.URL.Control Disallow: / 
User-agent: MJ12bot Disallow: / 
User-agent: Moreover Disallow: / 
User-agent: Moreoverbot Disallow: /
User-agent: MSIECrawler Disallow: / 
User-agent: nabot Disallow: / 
User-agent: NaverBot Disallow: / 
User-agent: NerdByNature.Bot Disallow: /
User-agent: netEstate NE Crawler Disallow: /
User-agent: NetSeer crawler Disallow: /
User-agent: Newscan Disallow: /
User-agent: NextGenSearchBot Disallow: /
User-agent: NPBot Disallow: / 
User-agent: nutch Disallow: / 
User-agent: Offline Explorer Disallow: / 
User-agent: omgilibot Disallow: / 
User-agent: Orthogaffe Disallow: / 
User-agent: PiplBot Disallow: / 
User-agent: Pixray-Seeker Disallow: /
User-agent: proximic Disallow: / 
User-agent: psbot Disallow: /
User-agent: QuerySeekerSpider Disallow: / 
User-agent: rogerbot Disallow: / 
User-agent: seokicks Disallow: /
User-agent: SEOkicks-Robot Disallow: / 
User-agent: SiteBot Disallow: /
User-agent: SiteBot/0.1 Disallow: /
User-agent: sitecheck.internetseer.com Disallow: / 
User-agent: SiteSnagger Disallow: / 
User-agent: Slurp Disallow: / 
User-agent: sogou Disallow: /
User-agent: Sosospider Disallow: /
User-agent: spbot Disallow: / 
User-agent: Spinn3r Disallow: /
User-agent: Teleport Disallow: / 
User-agent: TeleportPro Disallow: / 
User-Agent: trendictionbot Disallow: / 
User-agent: trovitBot Disallow: /
User-agent: TurnitinBot Disallow: /
User-agent: UbiCrawler Disallow: / 
User-agent: uMBot-LN Disallow: / 
User-agent: UnisterBot Disallow: /
User-agent: UniversalFeedParser Disallow: / 
User-agent: WBSearchBot Disallow: /
User-agent: WebCopier Disallow: / 
User-agent: WebReaper Disallow: /
User-agent: WebStripper Disallow: / 
User-agent: WebZIP Disallow: / 
User-agent: WeSEE:Search Disallow: /
User-agent: wget Disallow: /
User-agent: Wotbot Disallow: / 
User-agent: wotbox Disallow: /
User-agent: Xenu Disallow: / 
User-agent: Yasni Disallow: /
User-agent: Zao Disallow: / 
User-agent: Zealbot Disallow: / 
User-agent: ZyBORG Disallow: / 

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of elpais.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/elpais.com