gandi.net: what it tells AI crawlers

On 27 October 2026, gandi.net's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 1k (Tranco)
robots.txt: ok (HTTP 200) · https://www.gandi.net/robots.txt
unchanged since 2026-10-26 · sha256 3fd451a6143f6a84… · view as recorded
The * group: allows applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	allows	through the * group
OAI-SearchBot	allows	through the * group
ChatGPT-User	allows	through the * group
ClaudeBot	allows	through the * group
Claude-SearchBot	allows	through the * group
Claude-User	allows	through the * group
PerplexityBot	allows	through the * group
Perplexity-User	allows	through the * group
Googlebot	allows	through the * group
Google-Extended	allows	through the * group
Applebot-Extended	allows	through the * group
CCBot	allows	through the * group
Bytespider	allows	through the * group
Meta-ExternalAgent	allows	through the * group
Amazonbot	allows	through the * group
facebookexternalhit	allows	named in its own group

Other signals

Publishes /llms.txt	no
Publishes /ai.txt	no
Publishes /.well-known/tdmrep.json	no
Reserves TDM rights (TDMRep file, header or meta)	no
Declares Content Signals (robots.txt or header)	no
Says ai-train=no in a Content Signal	no
Points to an RSL licence (robots.txt License line or Link header)	no
Answers with an x402 payment offer (lower bound: only the homepage is probed)	no
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	no
Sends noai / noimageai directives	no
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 3fd451a6143f6a84…

User-agent: *


###### Sitemaps ######
Sitemap: https://www.gandi.net/sitemap.xml
Sitemap: https://news.gandi.net/sitemaps.xml

###### Disalow ######
Disallow: /*redirect=*
Disallow: /*navigation=*
Disallow: /*client_id=*
Disallow: /*locale=*
Disallow: /*tld=*
Disallow: /*bulk=*
Disallow: /*bu_id=*
Disallow: /*opened=*
Disallow: /*type=*
Disallow: /*template=*
Disallow: /*size=*
Disallow: /*options=*
Disallow: /*error=*
Disallow: /*grid=*
Disallow: /*currency=*
Disallow: /*?c=*
Disallow: /*&c=*
Disallow: /*?sw=*
Disallow: /*&sw=*
Disallow: /*taxes=*
Disallow: /pdf/*
Disallow: /*/sample/*
Disallow: */prices$
Disallow: */rules$
Disallow: */dates$
Disallow: */news$
Allow: */tld/news$
Disallow: *?country=*
Disallow: *?l=*
Disallow: *?lang=*
Disallow: /gcart/*


User-Agent: facebookexternalhit/1.1
Disallow:

User-Agent: Facebot
Disallow:

###### Suppression robots ######
User-agent: AhrefsBot
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: Aqua_Products
Disallow: /
User-agent: asterias
Disallow: /
User-agent: b2w/0.1
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: BacklinkCrawler
Disallow: /
User-agent: bdbrandprotect
Disallow: /
User-agent: Black Hole
Disallow: /
User-agent: BlowFish/1.0
Disallow: /
User-agent: boitho.com-dc
Disallow: /
User-agent: Bookmark search tool
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BotRightHere
Disallow: /
User-agent: BPImageWalker
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: CazoodleBot
Disallow: /
User-agent: Cegbfeieh
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: ConveraCrawler
Disallow: /
User-agent: Copernic
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-agent: discobot
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: Download Ninja
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: FairAd Client
Disallow: /
User-agent: Fasterfox
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: findlinks
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: GetRight/4.2
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: grub
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: Harvest/1.5
Disallow: /
User-agent: heritrix
Disallow: /
User-agent: hloader
Disallow: /
User-agent: httplib
Disallow: /
User-agent: HTTrack
Disallow: / 
User-agent: HTTrack 3.0
Disallow: /
User-agent: HuaweiSymantecSpider
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: ichiro
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: IRLbot
Disallow: /
User-agent: Iron33/1.0.2
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: kalooga
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: Keyword Density/0.9
Disallow: /
User-agent: larbin
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: LexxeBot
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: libwww
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: linko
Disallow: /
User-agent: LinkScan/8.1a Unix
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LNSpiderguy
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: lwp-trivial/1.34
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: MIIxpc/4.2
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: MLBot
Disallow: /
User-agent: moget
Disallow: /
User-agent: moget/2.1
Disallow: /
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: NaverBot
Disallow: /
User-agent: NerdyBot
Disallow: /
User-agent: Net\ Vampire
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: Ocelli
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Openbot
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: Openfind data gatherer
Disallow: /
User-agent: Oracle Ultra Search
Disallow: /
User-agent: penthesilea
Disallow: /
User-agent: PerMan
Disallow: /
User-agent: PhpDig
Disallow: /
User-agent: ProPowerBot/2.14
Disallow: /
User-agent: ProWebWalker
Disallow: /
User-agent: psbot
Disallow: /
User-agent: Python-urllib
Disallow: /
User-agent: QueryN Metasearch
Disallow: /
User-agent: Radiation Retriever 1.1
Disallow: /
User-agent: RepoMonkey
Disallow: /
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /
User-agent: RMA
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: SCOUTjET
Disallow: /
User-agent: SearchDaimon.com-dc
Disallow: /
User-agent: searchpreview
Disallow: /
User-agent: SEOENGBot
Disallow: /
User-agent: ShopWiki
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: Snapbot
Disallow: /
User-agent: Sogou\ web\ spider
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: SpankBot
Disallow: /
User-agent: spanner
Disallow: /
User-agent: spbot
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: suzuran
Disallow: /
User-agent: Szukacz/1.4
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: Telesoft
Disallow: /
User-agent: Templeton
Disallow: /
User-agent: The Intraformant
Disallow: /
User-agent: TheNomad
Disallow: /
User-agent: TightTwatBot
Disallow: /
User-agent: Titan
Disallow: /
User-agent: toCrawl/UrlDispatcher
Disallow: /
User-agent: True_Robot
Disallow: /
User-agent: True_Robot/1.0
Disallow: /
User-agent: turingos
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: TurnitinBot/1.5
Disallow: /
User-agent: TwengaBot
Disallow: /
User-agent: Twiceler
Disallow: /
User-agent: Updownerbot
Disallow: /
User-agent: URL Control
Disallow: /
User-agent: URL_Spider_Pro
Disallow: /
User-agent: URLy Warning
Disallow: /
User-agent: VCI
Disallow: /
User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /
User-agent: Web Image Collector
Disallow: /
User-agent: WebAuto
Disallow: /
User-agent: WebBandit
Disallow: /
User-agent: WebBandit/3.50
Disallow: /
User-agent: WebCapture 2.0
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebCopier v.2.2
Disallow: /
User-agent: WebCopier v3.2a
Disallow: /
User-agent: WebEnhancer
Disallow: /
User-agent: WebmasterWorldForumBot
Disallow: /
User-agent: WebSauger
Disallow: /
User-agent: Website Quester
Disallow: /
User-agent: Webster Pro
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebZIP
Disallow: /
User-agent: WebZip/4.0
Disallow: /
User-agent: WebZIP/4.21
Disallow: /
User-agent: WebZIP/5.0
Disallow: /
User-agent: wget
Disallow: /
User-agent: Wget/1.5.3
Disallow: /
User-agent: Wget/1.6
Disallow: /
User-agent: woozweb-monitoring
Disallow: /
User-agent: woriobot
Disallow: /
User-agent: wprocketbot
Allow: /
User-agent: WWW-Collector-E
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: Xenu's
Disallow: /
User-agent: Xenu's Link Sleuth 1.1c
Disallow: /
User-agent: YACYBIT
Disallow: /
User-agent: yacybot
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: YodaoBot
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /
User-agent: Zeus Link Scout
Disallow: /
User-agent: ZyBorg
Disallow: /
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow: /
User-agent: NerdyBot
Disallow: /
User-agent: WebAuto
Disallow: /
User-agent: wprocketbot
Disallow: /
User-agent: DIIbot
Disallow: /
User-agent: eCatch
Disallow: /
User-agent: Jetbot
Disallow: /
User-agent: Openfind data gathere
Disallow: /
User-agent: QuepasaCreep
Disallow: /
User-agent: Roverbot
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: RepoMonkey Bait &amp; Tackle/v1.01
Disallow: /
User-agent: ExtractorPro
Disallow: /

robots.txt versions

From	Through	State	Version
2026-10-26	2026-10-28	ok 200	`3fd451a6143f`

Timeline of changes

Date	What changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2025-09-15 CC-MAIN-2025-38	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2024-09-16 CC-MAIN-2024-38	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2023-10-02 CC-MAIN-2023-40	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2022-10-03 CC-MAIN-2022-40	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2021-09-27 CC-MAIN-2021-39	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2019-09-23 CC-MAIN-2019-39	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-09-24 CC-MAIN-2018-39	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-01-29 CC-MAIN-2018-05	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-10-23 CC-MAIN-2017-43	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-07-24 CC-MAIN-2017-30	gandi.net	redirect-only	unknown	unknown	unknown	unknown	unknown	unknown	unknown
2017-04-24 CC-MAIN-2017-17	gandi.net	redirect-only	unknown	unknown	unknown	unknown	unknown	unknown	unknown
2017-01-23 CC-MAIN-2017-04	gandi.net	redirect-only	unknown	unknown	unknown	unknown	unknown	unknown	unknown
2016-10-03 CC-MAIN-2016-40	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-09-05 CC-MAIN-2016-36	www.gandi.net	ok view	allows	allows	unknown	allows	allows	unknown	allows

Evidence

Every daily observation of gandi.net is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/gandi.net