corriere.it: what it tells AI crawlers
On 28 October 2026, corriere.it's robots.txt fully blocked GPTBot, ClaudeBot, Google-Extended and CCBot; it also reserved TDM rights and signalled ai-train=no.
Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://www.corriere.it/robots.txt
unchanged since 2026-10-26 · sha25651b3146315baf7dd…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-28
| User agent | Verdict | How |
|---|---|---|
| GPTBot | blocks | named in its own group |
| OAI-SearchBot | allows | named in its own group |
| ChatGPT-User | blocks | named in its own group |
| ClaudeBot | blocks | named in its own group |
| Claude-SearchBot | allows | through the * group |
| Claude-User | allows | through the * group |
| PerplexityBot | allows | named in its own group |
| Perplexity-User | allows | through the * group |
| Googlebot | allows | through the * group |
| Google-Extended | blocks | named in its own group |
| Applebot-Extended | blocks | named in its own group |
| CCBot | blocks | named in its own group |
| Bytespider | blocks | named in its own group |
| Meta-ExternalAgent | blocks | named in its own group |
| Amazonbot | blocks | named in its own group |
| Amzn-SearchBot | allows | named in its own group |
| anthropic-ai | blocks | named in its own group |
| Applebot | blocks | named in its own group |
| Claude-Web | blocks | named in its own group |
| Diffbot-User | blocks | named in its own group |
| Google-CloudVertexBot | blocks | named in its own group |
| PetalBot | blocks | named in its own group |
| Scrapy | blocks | named in its own group |
| Timpibot | blocks | named in its own group |
| YouBot | blocks | named in its own group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | yes |
| Declares Content Signals (robots.txt or header) | yes |
| Says ai-train=no in a Content Signal | yes |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
Content signals: search=yes, ai-train=no, ai-input=no for *
robots.txt as recorded sha256 aaf5d78da64d0ab5…
User-agent: PetalBot Disallow: / User-agent: Yandex Disallow: / # prohibits crawling for the entire site User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Google-Extended Disallow: / User-agent: * Disallow: *(none) Disallow: *?RedirectUrl Disallow: /privacy.shtml Disallow: /abbonati.Suspended/ Disallow: /corcommunitynew/accesso/LogOut.do Disallow: /indexMethodeProve.shtml Disallow: /apw.js Disallow: /salute/cardiologia/sintomi-infartosintomi-infarto Disallow: /salute/cardiologia/sintomi-infartoheart-attack-symptoms Disallow: /salute/cardiologia/sintomi-infartosíntomas-ataque-cardíaco Disallow: /salute/cardiologia/sintomi-infartosymptomes-infarctus Disallow: /salute/cardiologia/sintomi-infartoсимптомы-инфаркта Disallow: /salute/cardiologia/sintomi-infarto心肌梗塞症状 Disallow: /salute/cardiologia/sintomi-infartoطب القلب Disallow: /salute/cardiologia/sintomi-infartosimptome-de-atac-de-cord Disallow: /salute/sportello_cancro/jsp/* Disallow: /corrierecalcio/app/formazione/ Disallow: /preview/ Disallow: /dwc-xalok/ Disallow: /ssi/ Disallow: /tools/ Disallow: /linked_webroots/ Disallow: /english/includes_methode/ Disallow: /includes_methode/ Disallow: /community-corriere/ Disallow: /widget/ Disallow: /dwc/ Disallow: /digital-edition/ Disallow: /correlati/ Disallow: /cinema/icortissimi-ended/ Disallow: /ZCestino/ Disallow: /prova/ Disallow: /test/ Disallow: /landing/wrapped-2023/ Disallow: /test_mf/ Disallow: /piano-511/ Disallow: /b2b/ Disallow: /_template/ Disallow: /_docs/ Disallow: /_temp/ Disallow: /cf/ Disallow: /servizi/ Disallow: /salute/dizionario/dizionarioSearch.jsp Disallow: /sportello-cancro/dizionario/ Disallow: /dilatua/proRecensione.do Disallow: /newsletter/nl_economia.html # vecchia ultima ora Disallow: /ultima_ora/ #Dizionario della Salute Disallow: /salute/dizionario/dizionarioSearch.jsp Disallow: /salute/dizionario/pages/results.jsp Disallow: /cronache/speciali/2010/maturita/test-universita.shtml Disallow: /smallApp/tellafriend/ Disallow: /*_print.html$ Disallow: /*.xml Disallow: /account/ Allow: /account/login$ Disallow: /account/login? Disallow: /gallery/*/vuoto.shtml Disallow: *pwcmp= Disallow: *reader_id= Disallow: /cruciverba_old/ Allow: /salute/il-medico-risponde/sitemap.xml Allow: /salute/il-medico-risponde/news/sitemap.xml Allow: /salute/sitemap-dizionario-corriere-salute.xml Allow: /dynamic-sitemap/sitemap-last-100/*.xml Allow: /dynamic-feed/*.xml Allow: /cook*sitemap*.xml Allow: /guida-autovelox-e-codice-della-strada/sitemap.xml Allow: /radio/$ Allow: /radio/index.shtml Allow: /servizi/$ Allow: /ultima_ora/agrnews.jsp Allow: /showcase/*.xml Allow: /economia/chiedi-esperto/sitemap.xml Allow: /economia/chiedi-esperto/news/sitemap.xml Allow: /sette/speciale-presidenziali-usa-2024-harris-contro-trump/rss/*.xml Allow: /dynamic-sitemap/corriere/video/lezioni*sitemap*.xml # item Disallow: /corrierepedia Disallow: /firme/pierluigi-battista_80 Disallow: /cook/ricette/ricerca Disallow: /CorriereAppNews/ Disallow: /corrierepoint/ Disallow: /index_mobile_tst_hp_mob_b.shtml Disallow: /*app_v2 Allow: /buone-notizie/*app_v2 Disallow: /*app_v1 Allow: /sitemap*.xml # Allow: /notizie-ultima-ora/rss_col.xml Allow: /rss/*.xml Allow: /salute/sitemap-dizionario.xml Allow: /rss/ultimora.xml # Allow: /rss/homepage.xml Allow: /feed-hp/homepage.xml # Allow: /notizie-ultima-ora/sitemap-news.xml #richiesta da Ruggiero BG27112011 Allow: /editorspicks/ Allow: /oroscopo/sitemap.xml Allow: /sport/risultati-live/sitemap.xml Allow: /elezioni/*.xml Disallow: /*?q= Disallow: /*?f= Disallow: /*?s= Disallow: /*?searchType= Disallow: /*articleurl= Disallow: /economia/quiz/ # Sitemap: https://www.corriere.it/rss/homepage.xml Sitemap: https://www.corriere.it/feed-hp/homepage.xml Sitemap: https://www.corriere.it/salute/sitemap-dizionario-corriere-salute.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/video/Corriere.xml Sitemap: https://www.corriere.it/rss/sitemap_v2.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Economia.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Salute.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Scienze.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Interni.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Esteri.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Sport.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Politica.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Salute__Figli__e__Genitori.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Salute__Sportello__Cancro.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Elezioni.xml #CORRIERE-452 2018-10-08 Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Tecnologia.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Offerte__recensioni.xml Sitemap: https://www.corriere.it/dynamic-feed/rss/section/Dataroom.xml Sitemap: https://www.corriere.it/rss/sitemap/Motori.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Spettacoli.xml Sitemap: https://www.corriere.it/rss/sitemap/Cultura.xml Sitemap: https://vivimilano.corriere.it/sitemap_index.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Cook.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Tecnologia.xml Sitemap: https://www.corriere.it/oroscopo/sitemap.xml Sitemap: https://www.corriere.it/elezioni/sitemap/sitemap.xml Sitemap: https://www.corriere.it/sport/risultati-live/sitemap.xml Sitemap: https://www.corriere.it/salute/il-medico-risponde/sitemap.xml Sitemap: https://www.corriere.it/salute/il-medico-risponde/news/sitemap.xml Sitemap: https://www.corriere.it/rss/sitemap/lettere-al-direttore.xml Sitemap: https://www.corriere.it/rss/sitemap/lo-dico-al-corriere.xml Sitemap: https://www.corriere.it/rss/sitemap/Cook-Last.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Scuola.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Animali.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Opinioni.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Caffe-gramellini.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Ultimo-banco.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Letti-da-rifarei.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Piccole-dosi.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/L-angolo.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Padiglione-italia.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Facce-nuove.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Ritorno-in-solferino.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Oriente-occidente.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Sette.xml Sitemap: https://www.corriere.it/economia/chiedi-esperto/sitemap.xml Sitemap: https://www.corriere.it/economia/chiedi-esperto/news/sitemap.xml Sitemap: https://www.corriere.it/studio/sitemap-studio.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Moda.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/BuoneNotizie.xml Sitemap: https://www.corriere.it/dynamic-feed/rss/section/lettere-al-direttore.xml Sitemap: https://www.corriere.it/dynamic-feed/rss/section/lo-dico-al-corriere.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/lettere__al__direttore.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/lo__dico__al__corriere.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Frammenti-ferruccio-de-bortoli.xml Sitemap: https://www.corriere.it/dynamic-feed/rss/section/frammenti-di-ferruccio-de-bortoli.xml Sitemap: https://www.corriere.it/dynamic-sitemap/corriere/video/lezioni/sitemap-aggregation-index.xml Sitemap: https://www.corriere.it/dynamic-sitemap/corriere/le-lezioni-del-corriere/sitemap.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Giochi.xml Sitemap: https://www.corriere.it/guida-autovelox-e-codice-della-strada/sitemap.xml Sitemap: https://www.corriere.it/dynamic-sitemap/sitemap-last-100/Autovelox.xml
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-28 | ok 200 | 51b3146315ba |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | www.corriere.it | ok view | blocks | allows | unknown | blocks | allows | unknown | allows |
| 2025-09-15 CC-MAIN-2025-38 | www.corriere.it | ok view | blocks | allows | unknown | allows | allows | unknown | allows |
| 2024-09-16 CC-MAIN-2024-38 | www.corriere.it | ok view | blocks | allows | unknown | allows | allows | unknown | allows |
| 2023-10-02 CC-MAIN-2023-40 | www.corriere.it | ok view | blocks | allows | unknown | allows | allows | unknown | allows |
| 2022-10-03 CC-MAIN-2022-40 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-07-23 CC-MAIN-2018-30 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-04-23 CC-MAIN-2018-17 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-01-29 CC-MAIN-2018-05 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-10-23 CC-MAIN-2017-43 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-07-24 CC-MAIN-2017-30 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-04-24 CC-MAIN-2017-17 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-01-23 CC-MAIN-2017-04 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-10-03 CC-MAIN-2016-40 | www.corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-09-05 CC-MAIN-2016-36 | corriere.it | ok view | allows | allows | unknown | allows | allows | unknown | allows |
Evidence
Every daily observation of corriere.it is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).
API: https://infrarabbit.com/crawlers/api/v1/policy/corriere.it