europa.eu: what it tells AI crawlers
On 28 October 2026, europa.eu's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.
Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://europa.eu/robots.txt
unchanged since 2026-10-26 · sha256bede8dd1fdd4b10f…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-28
| User agent | Verdict | How |
|---|---|---|
| GPTBot | allows | through the * group |
| OAI-SearchBot | allows | through the * group |
| ChatGPT-User | allows | through the * group |
| ClaudeBot | allows | through the * group |
| Claude-SearchBot | allows | through the * group |
| Claude-User | allows | through the * group |
| PerplexityBot | allows | through the * group |
| Perplexity-User | allows | through the * group |
| Googlebot | allows | through the * group |
| Google-Extended | allows | through the * group |
| Applebot-Extended | allows | through the * group |
| CCBot | allows | through the * group |
| Bytespider | allows | through the * group |
| Meta-ExternalAgent | allows | through the * group |
| Amazonbot | allows | through the * group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | no |
| Declares Content Signals (robots.txt or header) | no |
| Says ai-train=no in a Content Signal | no |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
robots.txt as recorded sha256 9a3717e20cc3de21…
# robots.txt for EUROPA httpd-80 production server # # last update on 20/06/2019 # User-agent: * # match any robot name Disallow: /cgi-bin/ # don't allow robots into cgi-bin Disallow: /eur-lex/ # don't index old Eurlex - 13/09/2006 Request from OPOCE (Mr O. Grossmann) Disallow: /archives/ # don't index the archives # # COMM EUROPA MANAGEMENT - IM0012723685 - 03/04/2014 # Disallow: /youth/dissemination/ Disallow: /youth/archive/ Disallow: /youth/includes/ Disallow: /youth/misc/ Disallow: /youth/modules/ Disallow: /youth/profiles/ Disallow: /youth/themes/ Disallow: /youth/cron.php Disallow: /youth/install.php Disallow: /youth/LICENSE.txt Disallow: /youth/xmlrpc.php Disallow: /youth/admin/ Disallow: /youth/comment/reply/ Disallow: /youth/filter/tips/ Disallow: /youth/node/add/ Disallow: /youth/search/ Disallow: /youth/user/register/ Disallow: /youth/user/password/ Disallow: /youth/user/login/ Disallow: /youth/user/logout/ # # COMM EUROPA MANAGEMENT - IM0017899419 - 20/06/2019 # User-agent: * Crawl-delay: 10 # Directories Disallow: /capacity4dev/includes/ Disallow: /capacity4dev/misc/ Disallow: /capacity4dev/modules/ Disallow: /capacity4dev/profiles/ Disallow: /capacity4dev/scripts/ Disallow: /capacity4dev/themes/ # Files Disallow: /capacity4dev/CHANGELOG.txt Disallow: /capacity4dev/cron.php Disallow: /capacity4dev/INSTALL.mysql.txt Disallow: /capacity4dev/INSTALL.pgsql.txt Disallow: /capacity4dev/INSTALL.sqlite.txt Disallow: /capacity4dev/install.php Disallow: /capacity4dev/INSTALL.txt Disallow: /capacity4dev/LICENSE.txt Disallow: /capacity4dev/MAINTAINERS.txt Disallow: /capacity4dev/update.php Disallow: /capacity4dev/UPGRADE.txt Disallow: /capacity4dev/xmlrpc.php # Paths (clean URLs) Disallow: /capacity4dev/admin/ Disallow: /capacity4dev/comment/reply/ Disallow: /capacity4dev/filter/tips/ Disallow: /capacity4dev/node/add/ Disallow: /capacity4dev/search/ Disallow: /capacity4dev/user/register/ Disallow: /capacity4dev/user/password/ Disallow: /capacity4dev/user/login/ Disallow: /capacity4dev/user/logout/ Disallow: /capacity4dev/results-reporting-eud-or-hq-unit Disallow: /capacity4dev/eu-rfi Disallow: /capacity4dev/results_reporting_2013-14 Disallow: /capacity4dev/results_reporting_2014-15 # Paths (no clean URLs) Disallow: /capacity4dev/?q=admin/ Disallow: /capacity4dev/?q=comment/reply/ Disallow: /capacity4dev/?q=filter/tips/ Disallow: /capacity4dev/?q=node/add/ Disallow: /capacity4dev/?q=search/ Disallow: /capacity4dev/?q=user/password/ Disallow: /capacity4dev/?q=user/register/ Disallow: /capacity4dev/?q=user/login/ Disallow: /capacity4dev/?q=user/logout/ #SUBDIRECTORY ALIASED # Directories Disallow: /capacity4dev/*/includes/ Disallow: /capacity4dev/*/misc/ Disallow: /capacity4dev/*/modules/ Disallow: /capacity4dev/*/profiles/ Disallow: /capacity4dev/*/scripts/ Disallow: /capacity4dev/*/themes/ # Files Disallow: /capacity4dev/*/CHANGELOG.txt Disallow: /capacity4dev/*/cron.php Disallow: /capacity4dev/*/INSTALL.mysql.txt Disallow: /capacity4dev/*/INSTALL.pgsql.txt Disallow: /capacity4dev/*/INSTALL.sqlite.txt Disallow: /capacity4dev/*/install.php Disallow: /capacity4dev/*/INSTALL.txt Disallow: /capacity4dev/*/LICENSE.txt Disallow: /capacity4dev/*/MAINTAINERS.txt Disallow: /capacity4dev/*/update.php Disallow: /capacity4dev/*/UPGRADE.txt Disallow: /capacity4dev/*/xmlrpc.php # Paths (clean URLs) Disallow: /capacity4dev/*/admin/ Disallow: /capacity4dev/*/comment/reply/ Disallow: /capacity4dev/*/filter/tips/ Disallow: /capacity4dev/*/node/add/ Disallow: /capacity4dev/*/search/ Disallow: /capacity4dev/*/user/register/ Disallow: /capacity4dev/*/user/password/ Disallow: /capacity4dev/*/user/login/ Disallow: /capacity4dev/*/user/logout/ # Paths (no clean URLs) Disallow: /capacity4dev/*/?q=admin/ Disallow: /capacity4dev/*/?q=comment/reply/ Disallow: /capacity4dev/*/?q=filter/tips/ Disallow: /capacity4dev/*/?q=node/add/ Disallow: /capacity4dev/*/?q=search/ Disallow: /capacity4dev/*/?q=user/password/ Disallow: /capacity4dev/*/?q=user/register/ Disallow: /capacity4dev/*/?q=user/login/ Disallow: /capacity4dev/*/?q=user/logout/ # Custom rules Disallow: /capacity4dev/flag/ Disallow: /capacity4dev/?q=flag/ Disallow: /capacity4dev/*/flag/ Disallow: /capacity4dev/capacity4dev/flag/ Disallow: /capacity4dev/*/?q=flag/ Disallow: /capacity4dev/*/searchgroup/ Disallow: /capacity4dev/events/ Disallow: /capacity4dev/*/events/ Disallow: /capacity4dev/*/blogs/ Disallow: /capacity4dev/*/news/ Disallow: /capacity4dev/*/discussions/ Disallow: /capacity4dev/*/media/ Disallow: /capacity4dev/*/documents/ Disallow: /capacity4dev/*/library/ Disallow: /capacity4dev/*/calendar/ Disallow: /capacity4dev/*/members/ # Protect user profile data. Disallow: /capacity4dev/people/ Disallow: /capacity4dev/user/ Disallow: /capacity4dev/users/ Disallow: /capacity4dev/members Disallow: /capacity4dev/*/people/ Disallow: /capacity4dev/*/user/ Disallow: /capacity4dev/*/users/ # SMT Ticket Disallow: /youreurope/citizensACC/ Disallow: /youreurope/businessACC/ Disallow: /youreurope/citizensDEV/ Disallow: /youreurope/businessDEV/
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-28 | ok 200 | bede8dd1fdd4 |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2025-09-15 CC-MAIN-2025-38 | europa.eu | fetch-blocked | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2024-09-16 CC-MAIN-2024-38 | europa.eu | fetch-blocked | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2023-10-02 CC-MAIN-2023-40 | europa.eu | fetch-blocked | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2022-10-03 CC-MAIN-2022-40 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-07-23 CC-MAIN-2018-30 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-04-23 CC-MAIN-2018-17 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-01-29 CC-MAIN-2018-05 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-10-23 CC-MAIN-2017-43 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-07-24 CC-MAIN-2017-30 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-04-24 CC-MAIN-2017-17 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-01-23 CC-MAIN-2017-04 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-10-03 CC-MAIN-2016-40 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-09-05 CC-MAIN-2016-36 | europa.eu | ok view | allows | allows | unknown | allows | allows | unknown | allows |
Evidence
Every daily observation of europa.eu is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).
API: https://infrarabbit.com/crawlers/api/v1/policy/europa.eu