lemonde.fr: what it tells AI crawlers
On 27 October 2026, lemonde.fr's robots.txt fully blocked ClaudeBot, Claude-SearchBot, Google-Extended and CCBot.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://www.lemonde.fr/robots.txt
unchanged since 2026-10-26 · sha256843d2d9af6837d4c…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-27
| User agent | Verdict | How |
|---|---|---|
| GPTBot | allows | through the * group |
| OAI-SearchBot | allows | through the * group |
| ChatGPT-User | allows | through the * group |
| ClaudeBot | blocks | named in its own group |
| Claude-SearchBot | blocks | named in its own group |
| Claude-User | blocks | named in its own group |
| PerplexityBot | allows | through the * group |
| Perplexity-User | allows | through the * group |
| Googlebot | allows | through the * group |
| Google-Extended | blocks | named in its own group |
| Applebot-Extended | blocks | named in its own group |
| CCBot | blocks | named in its own group |
| Bytespider | blocks | named in its own group |
| Meta-ExternalAgent | allows | through the * group |
| Amazonbot | blocks | named in its own group |
| AddSearchBot | blocks | named in its own group |
| AIWebIndex | blocks | named in its own group |
| amazon-kendra | blocks | named in its own group |
| amazon-QBusiness | blocks | named in its own group |
| AmazonBuyForMe | blocks | named in its own group |
| Anomura | blocks | named in its own group |
| anthropic-ai | blocks | named in its own group |
| ApifyBot | blocks | named in its own group |
| ApifyWebsiteContentCrawler | blocks | named in its own group |
| Aranet-SearchBot | blocks | named in its own group |
| atlassian-bot | blocks | named in its own group |
| AzureAI-SearchBot | blocks | named in its own group |
| bedrockbot | blocks | named in its own group |
| Bravebot | blocks | named in its own group |
| Brightbot | blocks | named in its own group |
| ChatGLM-Spider | blocks | named in its own group |
| Claude-Web | blocks | named in its own group |
| Cloudflare-AutoRAG | blocks | named in its own group |
| CloudVertexBot | blocks | named in its own group |
| cohere-ai | blocks | named in its own group |
| cohere-training-data-crawler | blocks | named in its own group |
| Cotoyogi | blocks | named in its own group |
| CragCrawler | blocks | named in its own group |
| Crawlspace | blocks | named in its own group |
| DeepSeekBot | blocks | named in its own group |
| Devin | blocks | named in its own group |
| Diffbot | blocks | named in its own group |
| DoubaoBot | blocks | named in its own group |
| DuckAssistBot | blocks | named in its own group |
| ERNIEBot | blocks | named in its own group |
| ExaBot | blocks | named in its own group |
| FacebookBot | blocks | named in its own group |
| Factset_spyderbot | blocks | named in its own group |
| FirecrawlAgent | blocks | named in its own group |
| FriendlyCrawler | blocks | named in its own group |
| GeistHaus-PageFetcher | blocks | named in its own group |
| Gemini-Deep-Research | blocks | named in its own group |
| Google-Agent | blocks | named in its own group |
| Google-CloudVertexBot | blocks | named in its own group |
| Google-NotebookLM | blocks | named in its own group |
| GoogleAgent-Mariner | blocks | named in its own group |
| GoogleAgent-URLContext | blocks | named in its own group |
| GoogleOther | blocks | named in its own group |
| HenkBot | blocks | named in its own group |
| iAskBot | blocks | named in its own group |
| iaskspider | blocks | named in its own group |
| ICC-Crawler | blocks | named in its own group |
| imageSpider | blocks | named in its own group |
| kagi-fetcher | blocks | named in its own group |
| Kimi-SearchBot | blocks | named in its own group |
| Kimi-User | blocks | named in its own group |
| KimiBot | blocks | named in its own group |
| KlaviyoAIBot | blocks | named in its own group |
| KunatoCrawler | blocks | named in its own group |
| laion-huggingface-processor | blocks | named in its own group |
| LCC | blocks | named in its own group |
| LinerBot | blocks | named in its own group |
| LinkupBot | blocks | named in its own group |
| Manus-User | blocks | named in its own group |
| meta-webindexer | blocks | named in its own group |
| MistralAI-Index | blocks | named in its own group |
| MistralAI-User | blocks | named in its own group |
| MoonshotBot | blocks | named in its own group |
| Mozilla-Tabstack | blocks | named in its own group |
| NovaAct | blocks | named in its own group |
| omgili | blocks | named in its own group |
| omgilibot | blocks | named in its own group |
| PanguBot | blocks | named in its own group |
| PetalBot | blocks | named in its own group |
| PhindBot | blocks | named in its own group |
| Poggio-Citations | blocks | named in its own group |
| QualifiedBot | blocks | named in its own group |
| Querit-SearchBot | blocks | named in its own group |
| QueritBot | blocks | named in its own group |
| QwenBot | blocks | named in its own group |
| SBIntuitionsBot | blocks | named in its own group |
| SemrushBot-OCOB | blocks | named in its own group |
| Shap-User | blocks | named in its own group |
| ShapBot | blocks | named in its own group |
| Spider | blocks | named in its own group |
| TavilyBot | blocks | named in its own group |
| TerraCotta | blocks | named in its own group |
| Timpibot | blocks | named in its own group |
| TongyiBot | blocks | named in its own group |
| TwinAgent | blocks | named in its own group |
| UseAI | blocks | named in its own group |
| VelenPublicWebCrawler | blocks | named in its own group |
| Webzio-Extended | blocks | named in its own group |
| WRTNBot | blocks | named in its own group |
| YaK | blocks | named in its own group |
| YandexAdditional | blocks | named in its own group |
| YandexAdditionalBot | blocks | named in its own group |
| YiyanBot | blocks | named in its own group |
| YouBot | blocks | named in its own group |
| ZanistaBot | blocks | named in its own group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | no |
| Declares Content Signals (robots.txt or header) | no |
| Says ai-train=no in a Content Signal | no |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
robots.txt as recorded sha256 843d2d9af6837d4c…
# 16/08/2019 # Il est interdit d'utiliser des robots d'indexation Web ou d'autres méthodes automatiques de feuilletage ou de navigation sur ce site Web. # Nous interdisons de crawler notre site Web en utilisant un agent d'utilisateur volé qui ne correspond pas à votre identité. # « Violation du droit du producteur de base de données - article L 342-1 et suivant le Code de la propriété intellectuelle ». # Nous vous invitons à nous contacter pour contracter une licence d'utilisation. Seuls les partenaires sont habilités à utiliser nos contenus pour un usage autre que strictement individuel. User-agent: * Allow: /ws/1/live/* Allow: /ws/1/related_content/* Disallow: /ajax/ Disallow: /ajah/ Disallow: /api/ Disallow: /beta Disallow: /element/commun/afficher/ Disallow: /petites-annonces/ Disallow: /qui-sommes-nous/ Disallow: /txt/ Disallow: /verification/source/* Disallow: /noscript/ Disallow: /ws/* Disallow: /lemonde-beta/* Disallow: /_rprt/* Disallow: /layout/* Disallow: /cgi-bin/* Disallow: /envoyer-par-email/* Disallow: /lmdgft/* Disallow: /article-offert/* Disallow: /*?s=43260* Disallow: /*?contributions Disallow: */reactions/ Disallow: */mmpub/ # WordPress Disallow: /blog/*/wp-admin/ Disallow: /blog/*/wp-includes/ Disallow: /blog/*/wp-content/plugins/ Disallow: /blog/*/wp-content/themes/ Disallow: /blog/*/wp-login.php Disallow: /blog/*/wp-register.php Disallow: /blog/*/author/admin/ # Recherche Disallow: /recherche/?*search_keywords=* # Sitemaps Sitemap: https://www.lemonde.fr/sitemap_news.xml Sitemap: https://www.lemonde.fr/sitemap_index.xml # Sitemaps EN Sitemap: https://www.lemonde.fr/en/sitemap_news.xml Sitemap: https://www.lemonde.fr/en/sitemap_index.xml User-agent: Googlebot-Image Allow: /image/ User-agent: Googlebot-News Disallow: /archives/ # Robots exclus de toute indexation. User-agent: Cision Disallow: / User-agent: Talkwater Disallow: / User-agent: Jetbot Disallow: / User-agent: kbcrawl Disallow: / User-agent: Newzbin Disallow: / User-agent: Qwam content intelligence Disallow: / User-agent: Youmag Disallow: / User-agent: Synthesio Disallow: / User-agent: trendybuzz Disallow: / User-agent: scoop.it Disallow: / User-agent: linkfluence Disallow: / User-agent: grub-client Disallow: / User-agent: ia_archiver-web.archive.org Allow: /$ Disallow: /* User-agent: k2spider Disallow: / User-agent: libwww Disallow: / User-agent: wget Disallow: / User-agent: 5erue Disallow: / User-agent: adequat Disallow: / User-agent: adequat-systems Disallow: / User-agent: coexel Disallow: / User-agent: leadbox Disallow: / User-agent: mention Disallow: / User-agent: mytwip Disallow: / User-agent: opinion-tracker Disallow: / User-agent: proxem Disallow: / User-agent: score3 Disallow: / User-agent: vecteurplus Disallow: / User-agent: verticalsearch Disallow: / User-agent: vsw Disallow: / User-agent: winello Disallow: / User-agent: Fetch Disallow: / User-agent: infoseek Disallow: / User-agent: MSIECrawler Disallow: / User-agent: Offline Explorer Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebCopier Disallow: / User-agent: WebStripper Disallow: / User-agent: Zealbot Disallow: / User-agent: asknread.com Disallow: / User-agent: omgilibot Disallow: / User-agent: omgili Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: Amazonbot Disallow: / User-agent: Timpibot Disallow: / User-agent: AI2Bot Disallow: / User-agent: cohere-training-data-crawler Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: Kangaroo Bot Disallow: / User-agent: PanguBot Disallow: / User-agent: MistralAI-User Disallow: / # Cas particulier pour les bots de Facebook User-agent: facebookbot Disallow: / User-agent: FacebookBot Disallow: / # LISTEROBOTS1802 User-agent: 5emeRue Disallow: / User-agent: ACQUIRE MEDIA Disallow: / User-agent: ACTIV Financial (CME Group) Disallow: / User-agent: AlphaSense Disallow: / User-agent: AmiSoftware Disallow: / User-agent: archive.org_bot Disallow: / User-agent: Archive-It Disallow: / User-agent: ArgClrInt Disallow: / User-agent: Ask n read Disallow: / User-agent: Augure Disallow: / User-agent: auramundi Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: Barchart.com Disallow: / User-agent: BattleFin Disallow: / User-agent: Bernin IT Disallow: / User-agent: Blackboard Safeassign Disallow: / User-agent: BLP_bbot Disallow: / User-agent: bluematrix Disallow: / User-agent: Brandwatch Disallow: / User-agent: Briefcase.news Disallow: / User-agent: Buck Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBo Disallow: / User-agent: CikisiBot Disallow: / User-agent: Coexel Disallow: / User-agent: cohere-ai Disallow: / User-agent: Comtex News Network Disallow: / User-agent: ConveraCrawler Disallow: / User-agent: Copyright Licensing Agency Disallow: / User-agent: Corporama Disallow: / User-agent: D&B Hoovers Disallow: / User-agent: Data Expression Disallow: / User-agent: Data Observer Disallow: / User-agent: Dataminr Disallow: / User-agent: Dealogic Disallow: / User-agent: Diffbot Disallow: / User-agent: Digimind Disallow: / User-agent: DirectFN Disallow: / User-agent: Dun & Bradstreet Disallow: / User-agent: Dun & Bradstreet - D&B ESG Intelligence Disallow: / User-agent: Dun & Bradstreet Data Marketplace Disallow: / User-agent: Eagle Alpha Disallow: / User-agent: ecoresearch Disallow: / User-agent: ellisphere Disallow: / User-agent: FactSet Disallow: / User-agent: FeedCheck Disallow: / User-agent: FeedReader Disallow: / User-agent: Feedspot Disallow: / User-agent: Fitch Solutions Disallow: / User-agent: Founder Apabi Disallow: / User-agent: Freshbot Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: Gnowit Disallow: / User-agent: GnowitNewsbot Disallow: / User-agent: Ground News Disallow: / User-agent: ia_archiver Disallow: / User-agent: ICE Connect Desktop Solution Disallow: / User-agent: ICE Data Services Disallow: / User-agent: IHS Markit Disallow: / User-agent: ImageSift Disallow: / User-agent: InMédia Technologies Disallow: / User-agent: Innguma Disallow: / User-agent: Inoreader Disallow: / User-agent: ISI Emerging Markets Disallow: / User-agent: KB Crawl SAS Disallow: / User-agent: Knowings Disallow: / User-agent: Koyfin Disallow: / User-agent: Launchmetrics Disallow: / User-agent: LexisNexis Disallow: / User-agent: Liana Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Make/production Disallow: / User-agent: MarketResearch.com Disallow: / User-agent: MarketWatch Disallow: / User-agent: MarketWise Disallow: / User-agent: Markit Digital Disallow: / User-agent: Mediatoolkitbot Disallow: / User-agent: moduleQ Disallow: / User-agent: MONITIO Disallow: / User-agent: MoodleBot Disallow: / User-agent: Moody's Disallow: / User-agent: Moreover Disallow: / User-agent: MORNINGSTAR Disallow: / User-agent: MuckRack Disallow: / User-agent: Netvibes Disallow: / User-agent: news-api.org Disallow: / User-agent: Newslitbot Disallow: / User-agent: NewsNow Disallow: / User-agent: Northern Light Disallow: / User-agent: Opinion-tracker Disallow: / User-agent: Orbis Disallow: / User-agent: Opoint Disallow: / User-agent: Paqlebot Disallow: / User-agent: Press Monitor Europe Disallow: / User-agent: PressEngineBot D Disallow: / User-agent: PriberamBot Disallow: / User-agent: QuoteMedia Disallow: / User-agent: QWAM CONTENT INTELLIGENCE Disallow: / User-agent: RankurBot Disallow: / User-agent: RavenPack Disallow: / User-agent: ReportLinker Disallow: / User-agent: Research & Markets Disallow: / User-agent: S&P Capital IQ Disallow: / User-agent: S&P Global Market Intelligence Disallow: / User-agent: S&P Global Marketplace Disallow: / User-agent: scoopit Disallow: / User-agent: scoopit-crawler Disallow: / User-agent: scpitspi-rs Disallow: / User-agent: semantic-visions.com Disallow: / User-agent: semantic-visions.com crawler Disallow: / User-agent: SemrushBot Disallow: / User-agent: SentiBot Disallow: / User-agent: SentiOne Disallow: / User-agent: Signal Insights Disallow: / User-agent: Sindup Disallow: / User-agent: smartkarma Disallow: / User-agent: Sociallymap Disallow: / User-agent: spotter Disallow: / User-agent: squirrobot Disallow: / User-agent: Statista Disallow: / User-agent: STOCKBOARD Disallow: / User-agent: telpress.it Disallow: / User-agent: Thomson Reuters Disallow: / User-agent: Thomson Reuters WestLaw Disallow: / User-agent: TraderPlanet Disallow: / User-agent: Tradingcharts.com Disallow: / User-agent: TradingView Disallow: / User-agent: Transform Your API Disallow: / User-agent: trendeo Disallow: / User-agent: trendictionbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: um-FC Disallow: / User-agent: um-IC Disallow: / User-agent: um-LN Disallow: / User-agent: UptimeRobot Disallow: / User-agent: Vable Ltd Disallow: / User-agent: VISIBRAIN Disallow: / User-agent: webzio Disallow: / User-agent: YaK Disallow: / User-agent: YCHARTS Disallow: / User-agent: YouBot Disallow: / User-agent: Zite Disallow: / User-agent: Cotoyogi Disallow: / User-agent: Datenbank Crawler Disallow: / User-agent: Devin Disallow: / User-agent: Factset_spyderbot Disallow: / User-agent: ICC-Crawler Disallow: / User-agent: PetalBot Disallow: / User-agent: SemrushBot-OCOB Disallow: / User-agent: AddSearchBot Disallow: / User-agent: bigsur.ai Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / User-agent: CloudVertexBot Disallow: / User-agent: Gemini-Deep-Research Disallow: / User-agent: GoogleOther Disallow: / User-agent: LinerBot Disallow: / User-agent: netEstate Imprint Crawler Disallow: / User-agent: QualifiedBot Disallow: / User-agent: AI2Bot-DeepResearchEval Disallow: / User-agent: Ai2Bot-Dolma Disallow: / User-agent: Anomura Disallow: / User-agent: ApifyBot Disallow: / User-agent: ApifyWebsiteContentCrawler Disallow: / User-agent: Aranet-SearchBot Disallow: / User-agent: atlassian-bot Disallow: / User-agent: AzureAI-SearchBot Disallow: / User-agent: Bravebot Disallow: / User-agent: Channel3Bot Disallow: / User-agent: ChatGLM-Spider Disallow: / User-agent: Cloudflare-AutoRAG Disallow: / User-agent: Crawl4AI Disallow: / User-agent: DeepSeekBot Disallow: / User-agent: ExaBot Disallow: / User-agent: FirecrawlAgent Disallow: / User-agent: Google-CloudVertexBot Disallow: / User-agent: Google-NotebookLM Disallow: / User-agent: iAskBot Disallow: / User-agent: iaskspider Disallow: / User-agent: imageSpider Disallow: / User-agent: kagi-fetcher Disallow: / User-agent: KlaviyoAIBot Disallow: / User-agent: KunatoCrawler Disallow: / User-agent: laion-huggingface-processor Disallow: / User-agent: LCC Disallow: / User-agent: LinkupBot Disallow: / User-agent: meta-webindexer Disallow: / User-agent: PhindBot Disallow: / User-agent: Poggio-Citations Disallow: / User-agent: SBIntuitionsBot Disallow: / User-agent: Spider Disallow: / User-agent: TavilyBot Disallow: / User-agent: TerraCotta Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: webzio-extended Disallow: / User-agent: WRTNBot Disallow: / User-agent: ZanistaBot Disallow: / User-agent: AmazonBuyForMe Disallow: / User-agent: Google-Agent Disallow: / User-agent: GoogleAgent-Mariner Disallow: / User-agent: Manus-User Disallow: / User-agent: NovaAct Disallow: / User-agent: Shap-User Disallow: / User-agent: TwinAgent Disallow: / User-agent: ZenphonyBot Disallow: / User-agent: Brightbot Disallow: / User-agent: HenkBot Disallow: / User-agent: ShapBot Disallow: / User-agent: Terra Cotta Disallow: / User-agent: CohereBot Disallow: / User-agent: CragCrawler Disallow: / User-agent: GeistHaus-PageFetcher Disallow: / User-agent: GoogleAgent-URLContext Disallow: / User-agent: Kimi-SearchBot Disallow: / User-agent: Kimi-User Disallow: / User-agent: KimiBot Disallow: / User-agent: MistralAI-Index Disallow: / User-agent: MistralBot Disallow: / User-agent: Querit-SearchBot Disallow: / User-agent: QueritBot Disallow: / User-agent: UseAI Disallow: / User-agent: xAI-SearchBot Disallow: / User-agent: AIWebIndex Disallow: / User-agent: amazon-kendra Disallow: / User-agent: amazon-QBusiness Disallow: / User-agent: bedrockbot Disallow: / User-agent: Crawlspace Disallow: / User-agent: Doubaobot Disallow: / User-agent: ERNIEBot Disallow: / User-agent: KimiCrawler Disallow: / User-agent: micro-crawl Disallow: / User-agent: MoonshotBot Disallow: / User-agent: MoonSpider Disallow: / User-agent: Mozilla-Tabstack Disallow: / User-agent: QwenBot Disallow: / User-agent: SalamandraVLM Disallow: / User-agent: TongyiBot Disallow: / User-agent: YandexAdditional Disallow: / User-agent: YandexAdditionalBot Disallow: / User-agent: YiyanBot Disallow: /
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-27 | ok 200 | 843d2d9af683 |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2023-10-02 CC-MAIN-2023-40 | www.lemonde.fr | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2022-10-03 CC-MAIN-2022-40 | www.lemonde.fr | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | www.lemonde.fr | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | www.lemonde.fr | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | www.lemonde.fr | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | www.lemonde.fr | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | www.lemonde.fr | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-01-23 CC-MAIN-2017-04 | lemonde.fr | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-10-03 CC-MAIN-2016-40 | www.lemonde.fr | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-09-05 CC-MAIN-2016-36 | lemonde.fr | ok view | allows | allows | unknown | allows | allows | unknown | allows |
Evidence
Every daily observation of lemonde.fr is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).
API: https://infrarabbit.com/crawlers/api/v1/policy/lemonde.fr