unsplash.com: what it tells AI crawlers
On 27 October 2026, unsplash.com's robots.txt fully blocked ClaudeBot and Claude-SearchBot.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://unsplash.com/robots.txt
unchanged since 2026-10-26 · sha25625bb1899e23f96b9…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-27
| User agent | Verdict | How |
|---|---|---|
| GPTBot | allows | through the * group |
| OAI-SearchBot | allows | through the * group |
| ChatGPT-User | allows | through the * group |
| ClaudeBot | blocks | named in its own group |
| Claude-SearchBot | blocks | named in its own group |
| Claude-User | allows | through the * group |
| PerplexityBot | allows | through the * group |
| Perplexity-User | allows | through the * group |
| Googlebot | allows | through the * group |
| Google-Extended | allows | through the * group |
| Applebot-Extended | allows | through the * group |
| CCBot | allows | through the * group |
| Bytespider | blocks | named in its own group |
| Meta-ExternalAgent | allows | through the * group |
| Amazonbot | allows | through the * group |
| ImagesiftBot | blocks | named in its own group |
| SemrushBot-OCOB | blocks | named in its own group |
| SemrushBot-SWA | blocks | named in its own group |
Other signals
| Publishes /llms.txt | not observed |
| Publishes /ai.txt | not observed |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | not observed |
| Declares Content Signals (robots.txt or header) | not observed |
| Says ai-train=no in a Content Signal | not observed |
| Points to an RSL licence (robots.txt License line or Link header) | not observed |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | not observed |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | not observed |
| Sends noai / noimageai directives | not observed |
| Refused our fetcher (403 / challenge) | no |
robots.txt as recorded sha256 25bb1899e23f96b9…
User-Agent: * Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/de/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/de/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/de/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/static/en/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/landing_pages/en/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/collections/en/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/illustration_searches/en/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/en/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/en/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/en/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/es/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/es/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/es/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/fr/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/fr/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/fr/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/id/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/id/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/id/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/it/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/it/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/it/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/ja/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/ja/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/ja/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/ko/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/ko/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/ko/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searches/pt/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/searchable/pt/sitemap.xml Sitemap: https://sitemaps.unsplash.com/sitemaps/promoted/pt/sitemap.xml Allow: /blog Allow: /de/backgrounds Allow: /backgrounds Allow: /es/backgrounds Allow: /fr/backgrounds Allow: /id/backgrounds Allow: /it/backgrounds Allow: /ja/backgrounds Allow: /ko/backgrounds Allow: /pt-br/backgrounds Disallow: /photos/*/download Disallow: /illustrations/*/download Disallow: /ngetty Disallow: /nautocomplete Disallow: /nabc Disallow: /de/account Disallow: /de/account/* Disallow: /de/b Disallow: /de/b/* Disallow: /de/community Disallow: /de/einstellung Disallow: /de/einstellung/job-posts/* Disallow: /de/kollektionen Disallow: /de/kollektionen/* Disallow: /de/lesezeichen Disallow: /de/login Disallow: /de/plus/r/* Disallow: /de/presse Disallow: /de/registrieren Disallow: /de/s/benutzer/* Disallow: /de/s/kollektionen/* Disallow: /de/s/visuell/* Disallow: /de/werben Disallow: /de/werben/produktplatzierung Disallow: /account Disallow: /account/* Disallow: /b Disallow: /b/* Disallow: /bookmarks Disallow: /join Disallow: /login Disallow: /plus/r/* Disallow: /s/collections/* Disallow: /s/users/* Disallow: /s/visual/* Disallow: /es/%C3%BAnete Disallow: /es/account Disallow: /es/account/* Disallow: /es/anunciarse Disallow: /es/anunciarse/colocaci%C3%B3n-de-producto Disallow: /es/b Disallow: /es/b/* Disallow: /es/colecciones Disallow: /es/colecciones/* Disallow: /es/comunidad Disallow: /es/contrataci%C3%B3n Disallow: /es/contrataci%C3%B3n/job-posts/* Disallow: /es/guardados Disallow: /es/inicia-sesi%C3%B3n Disallow: /es/plus/r/* Disallow: /es/prensa Disallow: /es/s/colecciones/* Disallow: /es/s/imagen/* Disallow: /es/s/usuarios/* Disallow: /fr/account Disallow: /fr/account/* Disallow: /fr/b Disallow: /fr/b/* Disallow: /fr/collections Disallow: /fr/collections/* Disallow: /fr/communaut%C3%A9 Disallow: /fr/connexion Disallow: /fr/faire-votre-publicit%C3%A9 Disallow: /fr/faire-votre-publicit%C3%A9/placement-de-produit Disallow: /fr/favoris Disallow: /fr/newsroom Disallow: /fr/plus/r/* Disallow: /fr/recrutement Disallow: /fr/recrutement/job-posts/* Disallow: /fr/s%E2%80%99inscrire Disallow: /fr/s/collections/* Disallow: /fr/s/utilisateurs/* Disallow: /fr/s/visuel/* Disallow: /id/account Disallow: /id/account/* Disallow: /id/ada-lowongan Disallow: /id/ada-lowongan/job-posts/* Disallow: /id/b Disallow: /id/b/* Disallow: /id/berita Disallow: /id/bookmark Disallow: /id/daftar Disallow: /id/iklankan Disallow: /id/iklankan/penempatan-produk Disallow: /id/koleksi Disallow: /id/koleksi/* Disallow: /id/komunitas Disallow: /id/masuk Disallow: /id/plus/r/* Disallow: /id/s/koleksi/* Disallow: /id/s/pengguna/* Disallow: /id/s/visual/* Disallow: /it/accedi Disallow: /it/account Disallow: /it/account/* Disallow: /it/assunzioni Disallow: /it/assunzioni/job-posts/* Disallow: /it/b Disallow: /it/b/* Disallow: /it/collezioni Disallow: /it/collezioni/* Disallow: /it/community Disallow: /it/diventa-inserzionista Disallow: /it/diventa-inserzionista/posizionamento-del-prodotto Disallow: /it/plus/r/* Disallow: /it/press-room Disallow: /it/registrati Disallow: /it/s/collezioni/* Disallow: /it/s/immagini/* Disallow: /it/s/utenti/* Disallow: /it/segnalibri Disallow: /ja/%E3%82%B3%E3%83%9F%E3%83%A5%E3%83%8B%E3%83%86%E3%82%A3 Disallow: /ja/%E3%82%B3%E3%83%AC%E3%82%AF%E3%82%B7%E3%83%A7%E3%83%B3 Disallow: /ja/%E3%82%B3%E3%83%AC%E3%82%AF%E3%82%B7%E3%83%A7%E3%83%B3/* Disallow: /ja/%E3%82%B5%E3%82%A4%E3%83%B3%E3%82%A4%E3%83%B3 Disallow: /ja/%E3%83%96%E3%83%83%E3%82%AF%E3%83%9E%E3%83%BC%E3%82%AF Disallow: /ja/%E3%83%97%E3%83%AC%E3%82%B9 Disallow: /ja/%E5%BA%83%E5%91%8A Disallow: /ja/%E5%BA%83%E5%91%8A/%E3%83%97%E3%83%AD%E3%83%80%E3%82%AF%E3%83%88%E3%83%97%E3%83%AC%E3%82%A4%E3%82%B9%E3%83%A1%E3%83%B3%E3%83%88 Disallow: /ja/%E6%A1%88%E4%BB%B6%E4%BE%9D%E9%A0%BC%E4%B8%AD Disallow: /ja/%E6%A1%88%E4%BB%B6%E4%BE%9D%E9%A0%BC%E4%B8%AD/job-posts/* Disallow: /ja/%E7%99%BB%E9%8C%B2 Disallow: /ja/account Disallow: /ja/account/* Disallow: /ja/b Disallow: /ja/b/* Disallow: /ja/plus/r/* Disallow: /ja/s/%E3%82%B3%E3%83%AC%E3%82%AF%E3%82%B7%E3%83%A7%E3%83%B3/* Disallow: /ja/s/%E3%83%93%E3%82%B8%E3%83%A5%E3%82%A2%E3%83%AB/* Disallow: /ja/s/%E3%83%A6%E3%83%BC%E3%82%B6%E3%83%BC/* Disallow: /ko/%EA%B0%80%EC%9E%85 Disallow: /ko/%EA%B3%A0%EC%9A%A9-%EC%A4%91 Disallow: /ko/%EA%B3%A0%EC%9A%A9-%EC%A4%91/job-posts/* Disallow: /ko/%EA%B4%91%EA%B3%A0 Disallow: /ko/%EA%B4%91%EA%B3%A0/%EA%B0%84%EC%A0%91-%EA%B4%91%EA%B3%A0 Disallow: /ko/%EB%A1%9C%EA%B7%B8%EC%9D%B8 Disallow: /ko/%EB%B3%B4%EB%8F%84%EC%9E%90%EB%A3%8C Disallow: /ko/%EB%B6%81%EB%A7%88%ED%81%AC Disallow: /ko/%EC%BB%A4%EB%AE%A4%EB%8B%88%ED%8B%B0 Disallow: /ko/%EC%BB%AC%EB%A0%89%EC%85%98 Disallow: /ko/%EC%BB%AC%EB%A0%89%EC%85%98/* Disallow: /ko/%ED%94%8C%EB%9F%AC%EC%8A%A4/r/* Disallow: /ko/account Disallow: /ko/account/* Disallow: /ko/b Disallow: /ko/b/* Disallow: /ko/s/%EC%82%AC%EC%9A%A9%EC%9E%90/* Disallow: /ko/s/%EC%8B%9C%EA%B0%81-%EC%9E%90%EB%A3%8C/* Disallow: /ko/s/%EC%BB%AC%EB%A0%89%EC%85%98/* Disallow: /pt-br/account Disallow: /pt-br/account/* Disallow: /pt-br/anunciar Disallow: /pt-br/anunciar/posicionamento-de-produto Disallow: /pt-br/b Disallow: /pt-br/b/* Disallow: /pt-br/cadastre-se Disallow: /pt-br/cole%C3%A7%C3%B5es Disallow: /pt-br/cole%C3%A7%C3%B5es/* Disallow: /pt-br/comunidade Disallow: /pt-br/contratando Disallow: /pt-br/contratando/job-posts/* Disallow: /pt-br/favoritos Disallow: /pt-br/imprensa Disallow: /pt-br/login Disallow: /pt-br/plus/r/* Disallow: /pt-br/s/cole%C3%A7%C3%B5es/* Disallow: /pt-br/s/usu%C3%A1rios/* Disallow: /pt-br/s/visual/* User-Agent: ImagesiftBot Disallow: / User-Agent: Claude-SearchBot User-Agent: ClaudeBot Disallow: / User-Agent: Bytespider Disallow: / User-Agent: AwarioRssBot User-Agent: AwarioSmartBot Disallow: / User-Agent: dotbot Disallow: / User-Agent: AhrefsBot User-Agent: AhrefsSiteAudit Disallow: / User-Agent: MJ12bot User-Agent: SemrushBot-BA User-Agent: SemrushBot-ESI User-Agent: SemrushBot-FT User-Agent: SemrushBot-OCOB User-Agent: SemrushBot-SI User-Agent: SemrushBot-SWA User-Agent: SiteAuditBot Disallow: / User-Agent: BrightEdge Disallow: / User-agent: proximic Disallow: / User-Agent: DataForSeoBot Disallow: / User-Agent: SeobilityBot Disallow: /
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-27 | ok 200 | 25bb1899e23f |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | unsplash.com | ok view | allows | allows | unknown | blocks | blocks | unknown | allows |
| 2025-09-15 CC-MAIN-2025-38 | unsplash.com | fetch-blocked | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2024-09-16 CC-MAIN-2024-38 | unsplash.com | ok view | allows | allows | unknown | blocks | allows | unknown | allows |
| 2023-10-02 CC-MAIN-2023-40 | unsplash.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2022-10-03 CC-MAIN-2022-40 | unsplash.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | unsplash.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | unsplash.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | unsplash.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | unsplash.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | unsplash.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-01-23 CC-MAIN-2017-04 | unsplash.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-10-03 CC-MAIN-2016-40 | unsplash.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-09-05 CC-MAIN-2016-36 | unsplash.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
Evidence
Every daily observation of unsplash.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).
API: https://infrarabbit.com/crawlers/api/v1/policy/unsplash.com