fragrantica.com: what it tells AI crawlers
On 27 October 2026, fragrantica.com's robots.txt fully blocked GPTBot, ClaudeBot, Google-Extended and CCBot; it also signalled ai-train=no.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Panel
- top 10k (Tranco)
- robots.txt
- ok (HTTP 200) · https://fragrantica.com/robots.txt
unchanged since 2026-10-27 · sha2563073d1e371afb327…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-27
| User agent | Verdict | How |
|---|---|---|
| GPTBot | blocks | named in its own group |
| OAI-SearchBot | allows | through the * group |
| ChatGPT-User | allows | through the * group |
| ClaudeBot | blocks | named in its own group |
| Claude-SearchBot | allows | through the * group |
| Claude-User | allows | through the * group |
| PerplexityBot | allows | through the * group |
| Perplexity-User | allows | through the * group |
| Googlebot | allows | through the * group |
| Google-Extended | blocks | named in its own group |
| Applebot-Extended | blocks | named in its own group |
| CCBot | blocks | named in its own group |
| Bytespider | blocks | named in its own group |
| Meta-ExternalAgent | blocks | named in its own group |
| Amazonbot | blocks | named in its own group |
| anthropic-ai | blocks | named in its own group |
| atlassian-bot | blocks | named in its own group |
| Claude-Web | blocks | named in its own group |
| cohere-ai | blocks | named in its own group |
| Cotoyogi | blocks | named in its own group |
| Diffbot | blocks | named in its own group |
| Google-CloudVertexBot | blocks | named in its own group |
| GoogleOther | blocks | named in its own group |
| ICC-Crawler | blocks | named in its own group |
| KimiBot | blocks | named in its own group |
| MistralAI-Training | blocks | named in its own group |
| omgili | blocks | named in its own group |
| PetalBot | blocks | named in its own group |
| QualifiedBot | blocks | named in its own group |
| SemrushBot-SWA | blocks | named in its own group |
| WARDBot | blocks | named in its own group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | not observed |
| Declares Content Signals (robots.txt or header) | yes |
| Says ai-train=no in a Content Signal | yes |
| Points to an RSL licence (robots.txt License line or Link header) | not observed |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | not observed |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | not observed |
| Sends noai / noimageai directives | not observed |
| Refused our fetcher (403 / challenge) | no |
Content signals: search=yes, ai-train=no, use=reference for *
robots.txt as recorded sha256 e8818c49b9036d3b…
User-agent: * Disallow: /admin Disallow: /ajax Disallow: /goto.php Disallow: /a.php Disallow: /board/extern.php Disallow: /social Disallow: /board/profile.php Disallow: /board/userlist.php Disallow: /board/search.php Disallow: /thumbz User-agent: AdsBot-Google Allow: * User-agent: Mediapartners-Google Allow: * User-agent: bingbot Disallow: /a.php Disallow: /admin Disallow: /ajax Disallow: /board/ Disallow: /chlen/ Disallow: /clanovi/ Disallow: /goto.php Disallow: /lid/ Disallow: /member/ Disallow: /members/ Disallow: /membros/ Disallow: /membru/ Disallow: /miembros/ Disallow: /Mitglied/ Disallow: /social Disallow: /thumbz Disallow: /uzytkownicy/ User-agent: msnbot Disallow: /a.php Disallow: /admin Disallow: /ajax Disallow: /board/ Disallow: /chlen/ Disallow: /clanovi/ Disallow: /goto.php Disallow: /lid/ Disallow: /member/ Disallow: /members/ Disallow: /membros/ Disallow: /membru/ Disallow: /miembros/ Disallow: /Mitglied/ Disallow: /social Disallow: /thumbz Disallow: /uzytkownicy/ User-agent: ia_archiver Disallow: / User-agent: 008 Disallow: / User-agent: 360Spider Disallow: / User-agent: proximic Disallow: / User-agent: TTD-Content Disallow: / User-agent: 80legs Disallow: / User-agent: Abonti Disallow: / User-agent: Aboundex Disallow: / User-agent: Acunetix Disallow: / User-agent: ADmantX Disallow: / user-agent: AhrefsBot Disallow: / User-agent: AIBOT Disallow: / User-agent: AiHitBot Disallow: / User-agent: Aipbot Disallow: / User-agent: Alexibot Disallow: / User-agent: Alligator Disallow: / User-agent: AllSubmitter Disallow: / User-agent: Anarchie Disallow: / User-agent: Apexoo Disallow: / User-agent: ASPSeek Disallow: / User-agent: Asterias Disallow: / User-agent: Attach Disallow: / User-agent: autoemailspider Disallow: / User-agent: BackDoorBot Disallow: / User-agent: Backlink-Ceck Disallow: / User-agent: backlink-check Disallow: / User-agent: BacklinkCrawler Disallow: / User-agent: BackStreet Disallow: / User-agent: BackWeb Disallow: / User-agent: Badass Disallow: / User-agent: Bandit Disallow: / User-agent: BatchFTP Disallow: / User-agent: Battleztar Bazinga Disallow: / User-agent: BBBike Disallow: / User-agent: BDFetch Disallow: / User-agent: BecomeBot Disallow: / User-agent: BetaBot Disallow: / User-agent: Bigfoot Disallow: / User-agent: Bitacle Disallow: / User-agent: Black Hole Disallow: / User-agent: Blackboard Disallow: / User-agent: BlackWidow Disallow: / User-agent: BLEXBot Disallow: / User-agent: Blow Disallow: / User-agent: BlowFish Disallow: / User-agent: Boardreader Disallow: / User-agent: Bolt Disallow: / User-agent: BomboraBot Disallow: / User-agent: BotALot Disallow: / User-agent: Brandprotect Disallow: / User-agent: Brandwatch Disallow: / User-agent: Bubing Disallow: / User-agent: Buddy Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: BuiltWith Disallow: / User-agent: Bullseye Disallow: / User-agent: BunnySlippers Disallow: / User-agent: BuzzSumo Disallow: / User-agent: Calculon Disallow: / User-agent: CATExplorador Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CCBot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: CheeseBot Disallow: / User-agent: CherryPicker Disallow: / User-agent: ChinaClaw Disallow: / User-agent: Chlooe Disallow: / User-agent: Claritybot Disallow: / User-agent: Cliqzbot Disallow: / User-agent: Cogentbot Disallow: / User-agent: cognitiveseo Disallow: / User-agent: Collector Disallow: / User-agent: com.plumanalytics Disallow: / User-agent: Copier Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: Copyscape Disallow: / User-agent: Cosmos Disallow: / User-agent: Craftbot Disallow: / User-agent: CrazyWebCrawler Disallow: / User-agent: CRAZYWEBCRAWLER Disallow: / User-agent: Crescent Disallow: / User-agent: CSHttp Disallow: / User-agent: Curious Disallow: / User-agent: Custo Disallow: / User-agent: DatabaseDriverMysqli Disallow: / User-agent: DataCha0s Disallow: / User-agent: DBLBot Disallow: / User-agent: Demon Disallow: / User-agent: Deusu Disallow: / User-agent: Devil Disallow: / User-agent: DIIbot Disallow: / User-agent: Dirbuster Disallow: / User-agent: Disco Disallow: / User-agent: Discobot Disallow: / User-agent: Discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DJEcoBot Disallow: / User-agent: DomainAppender Disallow: / User-agent: DomainCrawler Disallow: / User-agent: DomainSigmaCrawler Disallow: / User-agent: dotbot Disallow: / User-agent: Dotbot Disallow: / User-agent: Download Demon Disallow: / User-agent: Download Devil Disallow: / User-agent: Download Wonder Disallow: / User-agent: Dragonfly Disallow: / User-agent: Drip Disallow: / User-agent: DTS Agent Disallow: / User-agent: EasyDL Disallow: / User-agent: Ebingbong Disallow: / User-agent: eCatch Disallow: / User-agent: ECCP/1.0 Disallow: / User-agent: Ecxi Disallow: / User-agent: EirGrabber Disallow: / User-agent: EMail Collector Disallow: / User-agent: EMail Extractor Disallow: / User-agent: EMail Siphon Disallow: / User-agent: EMail Wolf Disallow: / User-agent: EroCrawler Disallow: / User-agent: Evil Disallow: / User-agent: Exabot Disallow: / User-agent: Express WebPictures Disallow: / User-agent: Extractor Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Extreme Picture Finder Disallow: / User-agent: EyeNetIE Disallow: / User-agent: Ezooms Disallow: / User-agent: FDM Disallow: / User-agent: FHscan Disallow: / User-agent: Fimap Disallow: / User-agent: FindLinks Disallow: / User-agent: Findxbot Disallow: / User-agent: Firefox/7.0 Disallow: / User-agent: FlashGet Disallow: / User-agent: Flunky Disallow: / User-agent: Foobot Disallow: / User-agent: fq Disallow: / User-agent: Freeuploader Disallow: / User-agent: FrontPage Disallow: / User-agent: fyberspider Disallow: / User-agent: Fyrebot Disallow: / User-agent: GalaxyBot Disallow: / User-agent: Genieo Disallow: / User-agent: Getintent Disallow: / User-agent: GetRight Disallow: / User-agent: GetWeb Disallow: / User-agent: Gigablast Disallow: / User-agent: Gigabot Disallow: / User-agent: Go!Zilla Disallow: / User-agent: Go-Ahead-Got-It Disallow: / User-agent: Gotit Disallow: / User-agent: GoZilla Disallow: / User-agent: Grabber Disallow: / User-agent: GrabNet Disallow: / User-agent: Grafula Disallow: / User-agent: GrapeFX Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: GridBot Disallow: / User-agent: GT::WWW Disallow: / User-agent: HaosouSpider Disallow: / User-agent: Harvest Disallow: / User-agent: Havij Disallow: / User-agent: HEADMasterSEO Disallow: / User-agent: Heritrix Disallow: / User-agent: Hloader Disallow: / User-agent: HMView Disallow: / User-agent: HTMLparser Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: HTTrack Disallow: / User-agent: Humanlinks Disallow: / User-agent: HybridBot Disallow: / User-agent: Iblog Disallow: / User-agent: Id-search Disallow: / User-agent: IDBot Disallow: / User-agent: IlseBot Disallow: / User-agent: Image Fetch Disallow: / User-agent: Image Stripper Disallow: / User-agent: Image Sucker Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoTekies Disallow: / User-agent: instabid Disallow: / User-agent: Intelliseek Disallow: / User-agent: InterGET Disallow: / User-agent: Internet Ninja Disallow: / User-agent: InternetSeer Disallow: / User-agent: internetVista monitor Disallow: / User-agent: Iria Disallow: / User-agent: IRLbot Disallow: / User-agent: Iskanie Disallow: / User-agent: JamesBOT Disallow: / User-agent: Jbrofuzz Disallow: / User-agent: JennyBot Disallow: / User-agent: JetCar Disallow: / User-agent: JikeSpider Disallow: / User-agent: JOC Web Spider Disallow: / User-agent: Joomla Disallow: / User-agent: Jorgee Disallow: / User-agent: JustView Disallow: / User-agent: Jyxobot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density Disallow: / User-agent: Lanshanbot Disallow: / User-agent: Larbin Disallow: / User-agent: LeechFTP Disallow: / User-agent: LeechGet Disallow: / User-agent: LexiBot Disallow: / User-agent: Lftp Disallow: / User-agent: LibWeb Disallow: / User-agent: Libwhisker Disallow: / User-agent: Lightspeedsystems Disallow: / User-agent: Likse Disallow: / User-agent: Linkdexbot Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkpadBot Disallow: / User-agent: LinkScan Disallow: / User-agent: LinksManager Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinqiaMetadataDownloaderBot Disallow: / User-agent: LinqiaRSSBot Disallow: / User-agent: LinqiaScrapeBot Disallow: / User-agent: Lipperhey Disallow: / User-agent: Litemage_walker Disallow: / User-agent: Lmspider Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: Ltx71 Disallow: / User-agent: lwp-request Disallow: / User-agent: lwp-trivial Disallow: / User-agent: LWP::Simple Disallow: / User-agent: Mag-Net Disallow: / User-agent: Magnet Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Mail.ru Disallow: / User-agent: Majestic12 Disallow: / User-agent: MarkMonitor Disallow: / User-agent: MarkWatch Disallow: / User-agent: Mass Downloader Disallow: / User-agent: Masscan Disallow: / User-agent: Mata Hari Disallow: / User-agent: Meanpathbot Disallow: / User-agent: Mediatoolkitbot Disallow: / User-agent: MegaIndex.ru Disallow: / User-agent: metajobbot Disallow: / User-agent: Metauri Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: Microsoft Data Access Disallow: / User-agent: Microsoft URL Control Disallow: / User-agent: MIDown tool Disallow: / User-agent: MIIxpc Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: Mojeek Disallow: / User-agent: MojeekBot Disallow: / User-agent: Morfeus Fucking Scanner Disallow: / User-agent: MS Web Services Client Protocol Disallow: / User-agent: MSFrontPage Disallow: / User-agent: MSIE 6.0 Disallow: / User-agent: MSIECrawler Disallow: / User-agent: Msrabot Disallow: / User-agent: Musobot Disallow: / User-agent: Name Intelligence Disallow: / User-agent: Nameprotect Disallow: / User-agent: Navroad Disallow: / User-agent: NearSite Disallow: / User-agent: Needle Disallow: / User-agent: Nessus Disallow: / User-agent: Net Vampire Disallow: / User-agent: NetAnts Disallow: / User-agent: Netcraft Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetLyzer Disallow: / User-agent: NetMechanic Disallow: / User-agent: NetSpider Disallow: / User-agent: Nettrack Disallow: / User-agent: NetZIP Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: Nibbler Disallow: / User-agent: NICErsPRO Disallow: / User-agent: Niki-bot Disallow: / User-agent: Nikto Disallow: / User-agent: NimbleCrawler Disallow: / User-agent: Ninja Disallow: / User-agent: Nmap Disallow: / User-agent: NPbot Disallow: / User-agent: Nutch Disallow: / User-agent: Octopus Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Navigator Disallow: / User-agent: Openfind Disallow: / User-agent: OpenLinkProfiler Disallow: / User-agent: Openvas Disallow: / User-agent: OrangeBot Disallow: / User-agent: OrangeSpider Disallow: / User-agent: OutfoxBot Disallow: / User-agent: Page Analyzer Disallow: / User-agent: Page Grabber Disallow: / User-agent: page scorer Disallow: / User-agent: PageAnalyzer Disallow: / User-agent: PageGrabber Disallow: / User-agent: PageScorer Disallow: / User-agent: Panscient Disallow: / User-agent: Papa Foto Disallow: / User-agent: Pavuk Disallow: / User-agent: pcBrowser Disallow: / User-agent: PECL::HTTP Disallow: / User-agent: PeoplePal Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Pi-Monster Disallow: / User-agent: Picscout Disallow: / User-agent: Picsearch Disallow: / User-agent: PictureFinder Disallow: / User-agent: Pimonster Disallow: / User-agent: Pixray Disallow: / User-agent: PleaseCrawl Disallow: / User-agent: plumanalytics Disallow: / User-agent: Pockey Disallow: / User-agent: POE-Component-Client-HTTP Disallow: / User-agent: Probethenet Disallow: / User-agent: ProPowerBot Disallow: / User-agent: ProWebWalker Disallow: / User-agent: Psbot Disallow: / User-agent: Pump Disallow: / User-agent: PyCurl Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Qwantify Disallow: / User-agent: RankActive Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: RankFlex Disallow: / User-agent: RankingBot Disallow: / User-agent: RankingBot2 Disallow: / User-agent: Rankivabot Disallow: / User-agent: RankurBot Disallow: / User-agent: RealDownload Disallow: / User-agent: Reaper Disallow: / User-agent: RebelMouse Disallow: / User-agent: Recorder Disallow: / User-agent: RedesScrapy Disallow: / User-agent: ReGet Disallow: / User-agent: RepoMonkey Disallow: / User-agent: Ripper Disallow: / User-agent: RocketCrawler Disallow: / User-agent: rogerbot Disallow: / User-agent: Rogerbot Disallow: / User-agent: SalesIntelligent Disallow: / User-agent: SBIder Disallow: / User-agent: ScanAlert Disallow: / User-agent: Scanbot Disallow: / User-agent: Scrapy Disallow: / User-agent: Screaming Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: ScreenerBot Disallow: / User-agent: Searchestate Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: Semrush Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: SEOkicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: SEOlyticsCrawler Disallow: / User-agent: Seomoz Disallow: / User-agent: SEOprofiler Disallow: / User-agent: seoscanners Disallow: / User-agent: SEOstats Disallow: / User-agent: serpstatbot Disallow: / User-agent: Siphon Disallow: / User-agent: SISTRIX Disallow: / User-agent: SISTRIX Crawler Disallow: / User-agent: Site Sucker Disallow: / User-agent: Sitebeam Disallow: / User-agent: SiteExplorer Disallow: / User-agent: Siteimprove Disallow: / User-agent: SiteLockSpider Disallow: / User-agent: SiteSnagger Disallow: / User-agent: SiteSucker Disallow: / User-agent: Sitevigil Disallow: / User-agent: Slackbot-LinkExpanding Disallow: / User-agent: SlySearch Disallow: / User-agent: SmartDownload Disallow: / User-agent: Snake Disallow: / User-agent: Snapbot Disallow: / User-agent: Snoopy Disallow: / User-agent: SocialRankIOBot Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sosospider Disallow: / User-agent: Sottopop Disallow: / User-agent: SpaceBison Disallow: / User-agent: Spammen Disallow: / User-agent: SpankBot Disallow: / User-agent: Spanner Disallow: / User-agent: Spbot Disallow: / User-agent: Spinn3r Disallow: / User-agent: SputnikBot Disallow: / User-agent: Sqlmap Disallow: / User-agent: Sqlworm Disallow: / User-agent: Sqworm Disallow: / User-agent: Steeler Disallow: / User-agent: Stripper Disallow: / User-agent: Sucker Disallow: / User-agent: Sucuri Disallow: / User-agent: SuperBot Disallow: / User-agent: SuperHTTP Disallow: / User-agent: Surfbot Disallow: / User-agent: SurveyBot Disallow: / User-agent: Suzuran Disallow: / User-agent: Swiftbot Disallow: / User-agent: Szukacz Disallow: / User-agent: T0PHackTeam Disallow: / User-agent: T8Abot Disallow: / User-agent: tAkeOut Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: Telesoft Disallow: / User-agent: Telesphoreo Disallow: / User-agent: Telesphorep Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: Toata Disallow: / User-agent: Toweyabot Disallow: / User-agent: Trendictionbot Disallow: / User-agent: True_Robot Disallow: / User-agent: Turingos Disallow: / User-agent: Turnitin Bot Disallow: / User-agent: Turnitin Robot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TwengaBot Disallow: / User-agent: Twice Disallow: / User-agent: Typhoeus Disallow: / User-agent: UnisterBot Disallow: / User-agent: URLy Warning Disallow: / User-agent: URLy.Warning Disallow: / User-agent: Vacuum Disallow: / User-agent: Vagabondo Disallow: / User-agent: VB Project Disallow: / User-agent: VCI Disallow: / User-agent: VeriCiteCrawler Disallow: / User-agent: VidibleScraper Disallow: / User-agent: Virusdie Disallow: / User-agent: VoidEYE Disallow: / User-agent: Voil Disallow: / User-agent: Voltron Disallow: / User-agent: Wallpapers/3.0 Disallow: / User-agent: WallpapersHD Disallow: / User-agent: WASALive-Bot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: Web Auto Disallow: / User-agent: Web Bandit Disallow: / User-agent: Web Collage Disallow: / User-agent: Web Copier Disallow: / User-agent: Web Enhancer Disallow: / User-agent: Web Fetch Disallow: / User-agent: Web Fuck Disallow: / User-agent: Web Image Collector Disallow: / User-agent: Web Pix Disallow: / User-agent: Web Reaper Disallow: / User-agent: Web Sauger Disallow: / User-agent: Web Stripper Disallow: / User-agent: Web Sucker Disallow: / User-agent: Web Whacker Disallow: / User-agent: Webalta Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebCollage Disallow: / User-agent: WebCopier Disallow: / User-agent: WEBDAV Disallow: / User-agent: WEBDAV Client Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebFetch Disallow: / User-agent: WebFuck Disallow: / User-agent: WebGo IS Disallow: / User-agent: WebImageCollector Disallow: / User-agent: WebLeacher Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmeup-crawler Disallow: / User-agent: WebPix Disallow: / User-agent: WebReaper Disallow: / User-agent: WebSauger Disallow: / User-agent: Webshag Disallow: / User-agent: Website Extractor Disallow: / User-agent: Website Quester Disallow: / User-agent: WebsiteExtractor Disallow: / User-agent: WebsiteQuester Disallow: / User-agent: Webster Disallow: / User-agent: WebStripper Disallow: / User-agent: WebSucker Disallow: / User-agent: WebWhacker Disallow: / User-agent: WebZIP Disallow: / User-agent: WeSEE Disallow: / User-agent: Whack Disallow: / User-agent: Whacker Disallow: / User-agent: Whatweb Disallow: / User-agent: Widow Disallow: / User-agent: WinHTTrack Disallow: / User-agent: WiseGuys Robot Disallow: / User-agent: WISENutbot Disallow: / User-agent: Wonderbot Disallow: / User-agent: Woobot Disallow: / User-agent: Wotbox Disallow: / User-agent: Wprecon Disallow: / User-agent: WPScan Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: WWW-Mechanize Disallow: / User-agent: WWW::Mechanize Disallow: / User-agent: WWWOFFLE Disallow: / User-agent: x22Mozilla Disallow: / User-agent: Xaldon WebSpider Disallow: / User-agent: Xaldon_WebSpider Disallow: / User-agent: Xenu Disallow: / User-agent: yacybot Disallow: / User-agent: YoudaoBot Disallow: / User-agent: Zade Disallow: / User-agent: Zermelo Disallow: / User-agent: Zeus Disallow: / User-agent: Zgrab Disallow: / User-agent: Zitebot Disallow: / User-agent: ZmEu Disallow: / User-agent: ZumBot Disallow: / User-agent: ZyBorg Disallow: /
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-27 | 2026-10-27 | ok 200 | 3073d1e371af | diff with previous |
| 2026-10-26 | 2026-10-26 | ok 200 | 6bd14de3735b |
Timeline of changes
| Date | What changed |
|---|---|
| 2026-10-27 | robots.txt changed |
| 2026-10-27 | now allows aiHitBot (was block) |
| 2026-10-27 | now blocks Amazonbot (was allow) |
| 2026-10-27 | now blocks anthropic-ai (was allow) |
| 2026-10-27 | now blocks Applebot-Extended (was allow) |
| 2026-10-27 | now blocks atlassian-bot (was allow) |
| 2026-10-27 | now blocks Bytespider (was allow) |
| 2026-10-27 | now allows Claude-SearchBot (was block) |
| 2026-10-27 | now blocks Claude-Web (was allow) |
| 2026-10-27 | now blocks cohere-ai (was allow) |
| 2026-10-27 | now blocks Cotoyogi (was allow) |
| 2026-10-27 | now blocks Diffbot (was allow) |
| 2026-10-27 | now allows ExaBot (was block) |
| 2026-10-27 | now blocks Google-CloudVertexBot (was allow) |
| 2026-10-27 | now blocks Google-Extended (was allow) |
| 2026-10-27 | now blocks GoogleOther (was allow) |
| 2026-10-27 | now blocks ICC-Crawler (was allow) |
| 2026-10-27 | now blocks KimiBot (was allow) |
| 2026-10-27 | now blocks Meta-ExternalAgent (was allow) |
| 2026-10-27 | now blocks MistralAI-Training (was allow) |
| 2026-10-27 | now blocks omgili (was allow) |
| 2026-10-27 | now allows Panscient (was block) |
| 2026-10-27 | now allows PerplexityBot (was block) |
| 2026-10-27 | now blocks PetalBot (was allow) |
| 2026-10-27 | now blocks QualifiedBot (was allow) |
| 2026-10-27 | now allows Scrapy (was block) |
| 2026-10-27 | now blocks SemrushBot-SWA (was allow) |
| 2026-10-27 | now blocks WARDBot (was allow) |
| 2026-10-27 | Content-Signal lines changed |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | fragrantica.com | ok view | blocks | allows | unknown | blocks | allows | unknown | allows |
| 2025-09-15 CC-MAIN-2025-38 | www.fragrantica.com | ok view | blocks | allows | unknown | allows | allows | unknown | blocks |
| 2024-09-16 CC-MAIN-2024-38 | www.fragrantica.com | ok view | blocks | allows | unknown | allows | allows | unknown | blocks |
| 2023-10-02 CC-MAIN-2023-40 | www.fragrantica.com | ok view | blocks | allows | unknown | allows | allows | unknown | allows |
| 2022-10-03 CC-MAIN-2022-40 | www.fragrantica.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | www.fragrantica.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | www.fragrantica.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | www.fragrantica.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | www.fragrantica.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | www.fragrantica.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-07-24 CC-MAIN-2017-30 | www.fragrantica.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-04-24 CC-MAIN-2017-17 | fragrantica.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-01-23 CC-MAIN-2017-04 | www.fragrantica.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-10-03 CC-MAIN-2016-40 | www.fragrantica.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-09-05 CC-MAIN-2016-36 | www.fragrantica.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
Evidence
Every daily observation of fragrantica.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).
API: https://infrarabbit.com/crawlers/api/v1/policy/fragrantica.com