yandex.com: what it tells AI crawlers
On 27 October 2026, yandex.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://yandex.com/robots.txt
unchanged since 2026-10-26 · sha2567afa5219ab35a832…· view as recorded - The * group
- partial applies to every agent without its own group
Per AI bot, 2026-10-27
| User agent | Verdict | How |
|---|---|---|
| GPTBot | partial | through the * group |
| OAI-SearchBot | partial | through the * group |
| ChatGPT-User | partial | through the * group |
| ClaudeBot | partial | through the * group |
| Claude-SearchBot | partial | through the * group |
| Claude-User | partial | through the * group |
| PerplexityBot | partial | through the * group |
| Perplexity-User | partial | through the * group |
| Googlebot | partial | through the * group |
| Google-Extended | partial | through the * group |
| Applebot-Extended | partial | through the * group |
| CCBot | partial | through the * group |
| Bytespider | partial | through the * group |
| Meta-ExternalAgent | partial | through the * group |
| Amazonbot | partial | through the * group |
| Applebot | partial | named in its own group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | no |
| Declares Content Signals (robots.txt or header) | no |
| Says ai-train=no in a Content Signal | no |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
robots.txt as recorded sha256 7224bd2c36fc06a4…
# yandex.com User-agent: * Disallow: /? Disallow: /403.html Disallow: /404.html Disallow: /500.html Disallow: /about.html Disallow: /adddata Disallow: /adresa-segmentator Disallow: /advanced_engl.html Disallow: /advertising Disallow: /ads/ Disallow: /adfox/ Disallow: /an/ Disallow: /all-supported-params Disallow: /articles Disallow: /blog/*?text= Disallow: /blog/*?tag= Disallow: /blog/*/*/* Disallow: /blog/punto Disallow: /activity/ Disallow: /business/widget Disallow: /MAPS/ Disallow: /MaPS/ Disallow: /MapS/ Disallow: /Maps/ Disallow: /mApS/ Disallow: /blogs* Allow: /blogs/$ Allow: /blogs/pad/$ Allow: /blogs/pad$ Disallow: /catalog/?text= Disallow: /chat Disallow: /cgi-bin/ Disallow: /cgi/ Disallow: /chisla.html Disallow: /clck Disallow: /news Disallow: /news/ Disallow: /collections/feed Disallow: /collections/search/ Disallow: /collections/*/search/* Disallow: /collections/iznanka/ Disallow: /collections/*/_followers* Disallow: /collections/*/_followings* Disallow: /collections/share Disallow: /collections/favorites Disallow: /collections/embed Disallow: /collections/picture/ Disallow: /company/*.rss Disallow: /company/search Disallow: /comments/* Disallow: /conflagexp Disallow: /cy Disallow: /cycounter Disallow: /dzen Disallow: /edu/ping Disallow: /edu/tasks Disallow: /edu/teachers Disallow: /edu/test Disallow: /experiments.xml Disallow: /efir Disallow: /formfeedback Disallow: /gorsel/* Allow: /gorsel/$ Allow: /gorsel/?* Allow: /gorsel/smart/$ Allow: /gorsel/touch/$ Allow: /gorsel/touch/?* Disallow: /goto_issue/ Disallow: /goto_rubric/ Disallow: /i/ Disallow: /images-data Disallow: /images.html Disallow: /images/* Disallow: /images-apphost/* Allow: /images/$ Allow: /images/?* Allow: /images/smart/$ Allow: /images/touch/$ Allow: /images/touch/? Disallow: /index_m Disallow: /infected Disallow: /iznanka/ Disallow: /keyboard_qwerty.html Disallow: /logotypes Disallow: /map-constructor/loader* Disallow: /more_samples Disallow: /msearch Disallow: /msearchpart Disallow: /maps/print/* Allow: /maps/*?lang=kk$ Allow: /maps/*?lang=uz$ Disallow: /nmaps/*? Disallow: /mapeditor/*? Disallow: /metro/*?from Disallow: /norobot Disallow: /opensearch.xml Disallow: /padsearch Disallow: /people* Disallow: /person Disallow: /podpiska/login.pl Disallow: /polling Disallow: /promo/diskelement Disallow: /promo/*goodbye* Disallow: /promo/skype*?* Disallow: /promo/skype/*/* Disallow: /promo/launcher/feedback Disallow: /promo/launcher/mgoodbye/* Disallow: /promo/*welcome* Disallow: /promo/yobject/changelog* Disallow: /soft/bm/goodbye Disallow: /soft/chrome/ext-install/* Disallow: /soft/chrome/searchline-install/* Disallow: /soft/*goodbye* Disallow: /soft/punto/mac/uninstall/* Disallow: /soft/punto/win/uninstall/* Disallow: /soft/*welcome* Disallow: /soft/win/?* Disallow: /quotes Disallow: /redir Disallow: /region_map Disallow: /regions_list.xml Disallow: /regions.html? Disallow: /rubric2sport Disallow: /s/ Disallow: /save Disallow: /safety/?* Disallow: /search Disallow: /setup Disallow: /showcaptcha Disallow: /sitesearch Disallow: /skazki Disallow: /sl/*.html Disallow: /soft/extensions/goodbye Disallow: /sportagent Disallow: /storeclick Disallow: /storerequest Disallow: /subscribe/confirm.pl Disallow: /subscribe/view.pl Disallow: /support/direct-images Disallow: /support/direct-tooltips Disallow: /support/distr Disallow: /support/dsp Disallow: /support/fe4be44a295cc679e19bf0b8f133083d Disallow: /support/maps-beta Disallow: /support/market-images Disallow: /support/security Disallow: /support/webmaster-images Disallow: /support/*zout_ Disallow: /support/search-results/ Disallow: /support/praktikum/flow.html Disallow: /support/praktikum-english/ Disallow: /telsearch Disallow: /themes Disallow: /toggle-experiment Disallow: /touchsearch Disallow: /tune*retpath= Disallow: /versions Disallow: /v$ Disallow: /viewconfig$ Disallow: /video/v$ Disallow: /video/viewconfig$ Disallow: /images/v$ Disallow: /images/viewconfig$ Allow: /uslugi/$ Disallow: /uslugi/* Disallow: /video/* Disallow: /video/*filmId=* Disallow: /video/search Disallow: /video/preview Disallow: /video/*/search Disallow: /video/*/preview Allow: /video/$ Allow: /video/? Allow: /video/sitemap Allow: /video/dizi-izle/ Allow: /video/dizi-izle/? Allow: /video/yerli-dizi-izle/ Allow: /video/yabanci-dizi-izle/ Allow: /video/touch/ Allow: /video/pad/ Disallow: /xmlsearch Disallow: /yaca Disallow: /yandsearch Disallow: /yca/cy Disallow: /soft/distribution Disallow: /soft/?* Disallow: /promo/launcher/?* Disallow: /opera/?* Disallow: /firefox/?* Disallow: /soft/bm/?* Disallow: /soft/browsers/?* Disallow: /soft/punto/?* Disallow: /ie/?* Disallow: /element/?* Disallow: /element/*goodbye* Disallow: /soft/*goodbye* Disallow: /adult Disallow: /sport*?*parent-reqid* Disallow: /mirror*?*parent-reqid* Disallow: /turbo*?*parent-reqid* Disallow: /mirror/hide Disallow: /pogoda/0 Disallow: /turbo*?*ajax=1* Disallow: /sport*?*ajax=1* Disallow: /mirror*?*ajax=1* Disallow: /games/*?*reqId* Disallow: /games/*?*pageId* Disallow: /games/preview/ Disallow: /games/install Disallow: /weather/* Disallow: /weather* Disallow: /pogoda/* Disallow: /pogod* Disallow: /hava/* Disallow: /hava* Disallow: *maps/covid19* Disallow: /support2/*zout_ Disallow: /sport Disallow: /support2/sport* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/* Disallow: /video/touch/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/* Disallow: /medicine Allow: /medicine/doctors Allow: /medicine/clinics Disallow: /finance Clean-Param: ncrnd&redircnt&clid&_&win / Clean-Param: random_cgi&sign /turbo* Clean-Param: app-id&game-id&div-rtx-reqid&lang&platform&header&from /games/ Clean-Param: ll&z&sll /maps Clean-Param: appsearch_header /pogoda Sitemap: https://yandex.com/support/sitemap.xml Sitemap: https://yandex.com/blog/sitemap.xml Sitemap: https://yandex.com/turbo/public/sitemap.xml Sitemap: https://yandex.com/games/sitemaps/sitemap.index.xml User-agent: yandex Disallow: /? Disallow: /403.html Disallow: /404.html Disallow: /500.html Disallow: /about.html Disallow: /adddata Disallow: /adresa-segmentator Disallow: /advanced_engl.html Disallow: /advertising Disallow: /all-supported-params Disallow: /articles Disallow: /activity/ Disallow: /blog/*?text= Disallow: /blog/*?tag= Disallow: /blog/*/*/* Disallow: /blog/punto Disallow: /business/widget Disallow: /MAPS/ Disallow: /MaPS/ Disallow: /MapS/ Disallow: /Maps/ Disallow: /mApS/ Disallow: /blogs* Allow: /blogs/$ Allow: /blogs/pad/$ Allow: /blogs/pad$ Disallow: /catalog/?text= Disallow: /chat Disallow: /cgi-bin/ Disallow: /cgi/ Disallow: /chisla.html Disallow: /clck Disallow: /news Disallow: /news/ Disallow: /collections/feed Disallow: /collections/search/ Disallow: /collections/*/search/* Disallow: /collections/iznanka/ Disallow: /collections/*/_followers* Disallow: /collections/*/_followings* Disallow: /collections/share Disallow: /collections/favorites Disallow: /collections/embed Disallow: /collections/picture/ Disallow: /company/*.rss Disallow: /company/search Disallow: /comments/* Disallow: /conflagexp Disallow: /cy Disallow: /cycounter Disallow: /dzen Disallow: /edu/ping Disallow: /edu/tasks Disallow: /edu/teachers Disallow: /edu/test Disallow: /experiments.xml Disallow: /efir Disallow: /formfeedback Disallow: /games/preview/ Disallow: /games/install Disallow: /gorsel/* Allow: /gorsel/$ Allow: /gorsel/?* Allow: /gorsel/smart/$ Allow: /gorsel/touch/$ Allow: /gorsel/touch/?* Disallow: /goto_issue/ Disallow: /goto_rubric/ Disallow: /i/ Disallow: /images-data Disallow: /images.html Disallow: /images/* Disallow: /images-apphost/* Allow: /images/$ Allow: /images/?* Allow: /images/smart/$ Allow: /images/touch/$ Allow: /images/touch/? Disallow: /index_m Disallow: /infected Disallow: /iznanka/ Disallow: /keyboard_qwerty.html Disallow: /logotypes Disallow: /map-constructor/loader* Disallow: /more_samples Disallow: /msearch Disallow: /msearchpart Disallow: /maps/print/* Allow: /maps/*?lang=kk$ Allow: /maps/*?lang=uz$ Disallow: /nmaps/*? Disallow: /mapeditor/*? Disallow: /metro/*?from Disallow: /norobot Disallow: /opensearch.xml Disallow: /padsearch Disallow: /people* Disallow: /person Disallow: /podpiska/login.pl Disallow: /polling Disallow: /pogoda/0 Disallow: /promo/diskelement Disallow: /promo/*goodbye* Disallow: /promo/skype*?* Disallow: /promo/skype/*/* Disallow: /promo/launcher/feedback Disallow: /promo/launcher/mgoodbye/* Disallow: /promo/*welcome* Disallow: /promo/yobject/changelog* Disallow: /soft/bm/goodbye Disallow: /soft/chrome/ext-install/* Disallow: /soft/chrome/searchline-install/* Disallow: /soft/*goodbye* Disallow: /soft/punto/mac/uninstall/* Disallow: /soft/punto/win/uninstall/* Disallow: /soft/*welcome* Disallow: /soft/win/?* Disallow: /quotes Disallow: /redir Disallow: /region_map Disallow: /regions_list.xml Disallow: /regions.html? Disallow: /rubric2sport Disallow: /s/ Disallow: /save Disallow: /safety/?* Disallow: /search Disallow: /setup Disallow: /showcaptcha Disallow: /sitesearch Disallow: /skazki Disallow: /sl/*.html Disallow: /soft/extensions/goodbye Disallow: /sportagent Disallow: /storeclick Disallow: /storerequest Disallow: /subscribe/confirm.pl Disallow: /subscribe/view.pl Disallow: /support/direct-images Disallow: /support/direct-tooltips Disallow: /support/distr Disallow: /support/dsp Disallow: /support/fe4be44a295cc679e19bf0b8f133083d Disallow: /support/maps-beta Disallow: /support/market-images Disallow: /support/security Disallow: /support/webmaster-images Disallow: /support/*zout_ Disallow: /support/search-results/ Disallow: /support/praktikum/flow.html Disallow: /support/praktikum-english/ Disallow: /telsearch Disallow: /themes Disallow: /toggle-experiment Disallow: /touchsearch Disallow: /tune*retpath= Disallow: /versions Disallow: /v$ Disallow: /viewconfig$ Disallow: /video/v$ Disallow: /video/viewconfig$ Disallow: /images/v$ Disallow: /images/viewconfig$ Allow: /uslugi/$ Disallow: /uslugi/* Disallow: /video/* Disallow: /video/*filmId=* Disallow: /video/search Disallow: /video/preview Disallow: /video/*/search Disallow: /video/*/preview Disallow: /video/touch Allow: /video/$ Allow: /video/? Allow: /video/sitemap Allow: /video/dizi-izle/ Allow: /video/dizi-izle/? Allow: /video/yerli-dizi-izle/ Allow: /video/yabanci-dizi-izle/ Allow: /video/pad/ Allow: /video/search?text=video$ Disallow: /xmlsearch Disallow: /yaca Disallow: /yandsearch Disallow: /yca/cy Disallow: /soft/distribution Disallow: /soft/?* Disallow: /promo/launcher/?* Disallow: /opera/?* Disallow: /firefox/?* Disallow: /soft/bm/?* Disallow: /soft/browsers/?* Disallow: /soft/punto/?* Disallow: /ie/?* Disallow: /element/?* Disallow: /element/*goodbye* Disallow: /soft/*goodbye* Disallow: /adult Disallow: /sport*?*parent-reqid* Disallow: /mirror*?*parent-reqid* Disallow: /turbo*?*parent-reqid* Disallow: /mirror/hide Disallow: /turbo*?*ajax=1* Disallow: /sport*?*ajax=1* Disallow: /mirror*?*ajax=1* Disallow: /weather/* Disallow: /weather* Disallow: /pogoda/* Disallow: /pogod* Disallow: /hava/* Disallow: /hava* Disallow: *maps/covid19* Disallow: /support2/*zout_ Disallow: /sport Disallow: /support2/sport* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/* Disallow: /video/touch/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/* Disallow: /medicine Allow: /medicine/doctors Allow: /medicine/clinics Disallow: /finance User-Agent: Twitterbot Allow: /images Allow: /gorsel Allow: /video Allow: /collections/picture/ Disallow: /adult Disallow: /activity/ Disallow: /news Disallow: /news/ Disallow: /games/*?*reqId* Disallow: /games/*?*pageId* Disallow: /games/preview/ Disallow: /games/install Disallow: /weather/* Disallow: /weather* Disallow: /pogoda/* Disallow: /pogod* Disallow: /hava/* Disallow: /hava* Disallow: *maps/covid19* Disallow: /support2/*zout_ Disallow: /sport Disallow: /support2/sport* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/* Disallow: /video/touch/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/* Disallow: /medicine Allow: /medicine/doctors Allow: /medicine/clinics Disallow: /finance Sitemap: https://yandex.com/support/sitemap.xml Sitemap: https://yandex.com/blog/sitemap.xml Sitemap: https://yandex.com/turbo/public/sitemap.xml Sitemap: https://yandex.com/games/sitemaps/sitemap.index.xml User-agent: Applebot Disallow: /? Disallow: /403.html Disallow: /404.html Disallow: /500.html Disallow: /about.html Disallow: /adddata Disallow: /adresa-segmentator Disallow: /advanced_engl.html Disallow: /advertising Disallow: /all-supported-params Disallow: /articles Disallow: /blog/*?text= Disallow: /blog/*?tag= Disallow: /blog/*/*/* Disallow: /blog/punto Disallow: /activity/ Disallow: /news Disallow: /news/ Disallow: /blogs* Allow: /blogs/$ Allow: /blogs/pad/$ Allow: /blogs/pad$ Disallow: /catalog/?text= Disallow: /chat Disallow: /cgi-bin/ Disallow: /cgi/ Disallow: /chisla.html Disallow: /clck Disallow: /collections/feed Disallow: /collections/search/ Disallow: /collections/*/search/* Disallow: /collections/iznanka/ Disallow: /collections/*/_followers* Disallow: /collections/*/_followings* Disallow: /collections/share Disallow: /collections/favorites Disallow: /collections/embed Disallow: /collections/picture/ Disallow: /company/*.rss Disallow: /company/search Disallow: /comments/* Disallow: /conflagexp Disallow: /cy Disallow: /cycounter Disallow: /dzen Disallow: /edu/ping Disallow: /edu/tasks Disallow: /edu/teachers Disallow: /edu/test Disallow: /experiments.xml Disallow: /efir Disallow: /formfeedback Disallow: /gorsel/* Allow: /gorsel/$ Allow: /gorsel/?* Allow: /gorsel/smart/$ Allow: /gorsel/touch/$ Allow: /gorsel/touch/?* Disallow: /goto_issue/ Disallow: /goto_rubric/ Disallow: /i/ Disallow: /images-data Disallow: /images.html Disallow: /images/* Allow: /images/$ Allow: /images/?* Allow: /images/smart/$ Allow: /images/touch/$ Allow: /images/touch/? Disallow: /index_m Disallow: /infected Disallow: /iznanka/ Disallow: /keyboard_qwerty.html Disallow: /logotypes Disallow: /map-constructor/loader* Disallow: /more_samples Disallow: /msearch Disallow: /msearchpart Disallow: /nmaps/*? Disallow: /maps/print/* Allow: /maps/*?ll=* Allow: /maps/org/ Disallow: /mapeditor/*? Disallow: /metro/*?from Disallow: /norobot Disallow: /opensearch.xml Disallow: /padsearch Disallow: /people* Disallow: /person Disallow: /podpiska/login.pl Disallow: /polling Disallow: /promo/diskelement Disallow: /promo/*goodbye* Disallow: /promo/skype*?* Disallow: /promo/skype/*/* Disallow: /promo/launcher/feedback Disallow: /promo/launcher/mgoodbye/* Disallow: /promo/*welcome* Disallow: /promo/yobject/changelog* Disallow: /soft/bm/goodbye Disallow: /soft/chrome/ext-install/* Disallow: /soft/chrome/searchline-install/* Disallow: /soft/*goodbye* Disallow: /soft/punto/mac/uninstall/* Disallow: /soft/punto/win/uninstall/* Disallow: /soft/*welcome* Disallow: /soft/win/?* Disallow: /quotes Disallow: /redir Disallow: /region_map Disallow: /regions_list.xml Disallow: /regions.html? Disallow: /rubric2sport Disallow: /s/ Disallow: /save Disallow: /safety/?* Disallow: /search Disallow: /setup Disallow: /showcaptcha Disallow: /sitesearch Disallow: /skazki Disallow: /sl/*.html Disallow: /soft/extensions/goodbye Disallow: /sportagent Disallow: /storeclick Disallow: /storerequest Disallow: /subscribe/confirm.pl Disallow: /subscribe/view.pl Disallow: /support/direct-images Disallow: /support/direct-tooltips Disallow: /support/distr Disallow: /support/dsp Disallow: /support/fe4be44a295cc679e19bf0b8f133083d Disallow: /support/maps-beta Disallow: /support/market-images Disallow: /support/security Disallow: /support/webmaster-images Disallow: /support/*zout_ Disallow: /support/search-results/ Disallow: /support/praktikum/flow.html Disallow: /support/praktikum-english/ Disallow: /telsearch Disallow: /themes Disallow: /toggle-experiment Disallow: /touchsearch Disallow: /tune*retpath= Disallow: /versions Disallow: /v$ Disallow: /viewconfig$ Disallow: /video/v$ Disallow: /video/viewconfig$ Disallow: /images/v$ Disallow: /images/viewconfig$ Allow: /uslugi/$ Disallow: /uslugi/* Allow: /tutor Disallow: /tutor/admin Disallow: /tutor/search/problems Disallow: /video/* Disallow: /video/*filmId=* Allow: /video/$ Allow: /video/? Allow: /video/sitemap Allow: /video/dizi-izle/ Allow: /video/dizi-izle/? Allow: /video/yerli-dizi-izle/ Allow: /video/yabanci-dizi-izle/ Allow: /video/touch/ Allow: /video/pad/ Disallow: /xmlsearch Disallow: /yaca Disallow: /yandsearch Disallow: /yca/cy Disallow: /soft/distribution Disallow: /soft/?* Disallow: /promo/launcher/?* Disallow: /opera/?* Disallow: /firefox/?* Disallow: /soft/bm/?* Disallow: /soft/browsers/?* Disallow: /soft/punto/?* Disallow: /ie/?* Disallow: /element/?* Disallow: /element/*goodbye* Disallow: /soft/*goodbye* Disallow: /adult Disallow: /games/*?*reqId* Disallow: /games/*?*pageId* Disallow: /games/preview/ Disallow: /games/install Disallow: /weather/* Disallow: /weather* Disallow: /pogoda/* Disallow: /pogod* Disallow: /hava/* Disallow: /hava* Disallow: *maps/covid19* Disallow: /support2/*zout_ Disallow: /sport Disallow: /support2/sport* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/* Disallow: /video/touch/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/* Clean-Param: ncrnd&redircnt&clid&_&win / Clean-Param: random_cgi&sign /turbo* Clean-Param: app-id&game-id&div-rtx-reqid&lang&platform&header&from /games/ Clean-Param: ll&z&sll /maps Clean-param: oll&rtext&pt&source&via&from&origin&um&utm_medium&utm_source&portal&panorama%5Bspan%5D&panorama%5Bpoint%5D&panorama%5Bfull%5D&panorama%5Bdirection%5D&panorama%5Bair%5D&masstransit%5BstopId%5D&ol&poi%5Bpoint%5D&whatshere%5Bpoint%5D&whatshere%5Bzoom%5D&ouri&ruri&l&mode&tab&rtt&rtm&display-text&text&indoorLevel&trfm&trfst&no-distribution&distance&slices&scale&descy&descx&mapY&mapX&mapID&sctx&lay&where&results&map&id&spn&sspn&rtn&ost&profile-mode /maps/ Sitemap: https://yandex.com/support/sitemap.xml Sitemap: https://yandex.com/blog/sitemap.xml Sitemap: https://yandex.com/turbo/public/sitemap.xml Sitemap: https://yandex.com/games/sitemaps/sitemap.index.xml
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-28 | ok 200 | 7afa5219ab35 |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2025-09-15 CC-MAIN-2025-38 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2024-09-16 CC-MAIN-2024-38 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2023-10-02 CC-MAIN-2023-40 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2022-10-03 CC-MAIN-2022-40 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2021-09-27 CC-MAIN-2021-39 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2020-09-28 CC-MAIN-2020-40 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2019-09-23 CC-MAIN-2019-39 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2018-09-24 CC-MAIN-2018-39 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2018-01-29 CC-MAIN-2018-05 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-10-23 CC-MAIN-2017-43 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-09-25 CC-MAIN-2017-39 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-07-24 CC-MAIN-2017-30 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-04-24 CC-MAIN-2017-17 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-01-23 CC-MAIN-2017-04 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2016-10-03 CC-MAIN-2016-40 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2016-09-05 CC-MAIN-2016-36 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
Evidence
Every daily observation of yandex.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).
API: https://infrarabbit.com/crawlers/api/v1/policy/yandex.com