yandex.com: what it tells AI crawlers
On 27 October 2026, yandex.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://yandex.com/robots.txt
unchanged since 2026-10-26 · sha2567afa5219ab35a832…· view as recorded - The * group
- partial applies to every agent without its own group
Per AI bot, 2026-10-27
| User agent | Verdict | How |
|---|---|---|
| GPTBot | partial | through the * group |
| OAI-SearchBot | partial | through the * group |
| ChatGPT-User | partial | through the * group |
| ClaudeBot | partial | through the * group |
| Claude-SearchBot | partial | through the * group |
| Claude-User | partial | through the * group |
| PerplexityBot | partial | through the * group |
| Perplexity-User | partial | through the * group |
| Googlebot | partial | through the * group |
| Google-Extended | partial | through the * group |
| Applebot-Extended | partial | through the * group |
| CCBot | partial | through the * group |
| Bytespider | partial | through the * group |
| Meta-ExternalAgent | partial | through the * group |
| Amazonbot | partial | through the * group |
| Applebot | partial | named in its own group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | no |
| Declares Content Signals (robots.txt or header) | no |
| Says ai-train=no in a Content Signal | no |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
robots.txt as recorded sha256 3afd7baec66d21b5…
# yandex.com User-agent: * Disallow: /? Disallow: /403.html Disallow: /404.html Disallow: /500.html Disallow: /about.html Disallow: /adddata Disallow: /adresa-segmentator Disallow: /advanced_engl.html Disallow: /advertising Disallow: /all-supported-params Disallow: /articles Disallow: /blog/*?text= Disallow: /blog/*?tag= Disallow: /blog/*/*/* Disallow: /blogs* Allow: /blogs/$ Allow: /blogs/pad/$ Allow: /blogs/pad$ Disallow: /catalog/?text= Disallow: /cgi-bin/ Disallow: /cgi/ Disallow: /chisla.html Disallow: /clck Disallow: /collections/feed Disallow: /collections/search/ Disallow: /collections/*/_followers* Disallow: /collections/*/_followings* Disallow: /collections/share Disallow: /collections/favorites Disallow: /collections/embed Disallow: /collections/picture/ Disallow: /company/*.rss Disallow: /company/search Disallow: /cy Disallow: /cycounter Disallow: /dzen Disallow: /edu/ping Disallow: /edu/tasks Disallow: /edu/teachers Disallow: /edu/test Disallow: /experiments.xml Disallow: /formfeedback Disallow: /gorsel/* Allow: /gorsel/$ Allow: /gorsel/?* Allow: /gorsel/smart/$ Allow: /gorsel/touch/$ Allow: /gorsel/touch/?* Disallow: /goto_issue/ Disallow: /goto_rubric/ Disallow: /i/ Disallow: /images-data Disallow: /images.html Disallow: /images/* Allow: /images/$ Allow: /images/?* Allow: /images/smart/$ Allow: /images/touch/$ Allow: /images/touch/? Disallow: /index_m Disallow: /infected Disallow: /keyboard_qwerty.html Disallow: /logotypes Disallow: /map-constructor/loader* Disallow: /more_samples Disallow: /msearch Disallow: /msearchpart Disallow: /maps/*? Disallow: /nmaps/*? Disallow: /mapeditor/*? Disallow: /norobot Disallow: /opensearch.xml Disallow: /padsearch Disallow: /people* Disallow: /person Disallow: /podpiska/login.pl Disallow: /polling Disallow: /promo/diskelement Disallow: /promo/*goodbye* Disallow: /promo/skype*?* Disallow: /promo/skype/*/* Disallow: /promo/launcher/feedback Disallow: /promo/launcher/mgoodbye/* Disallow: /promo/*welcome* Disallow: /promo/yobject/changelog* Disallow: /soft/bm/goodbye Disallow: /soft/chrome/ext-install/* Disallow: /soft/chrome/searchline-install/* Disallow: /soft/*goodbye* Disallow: /soft/punto/mac/uninstall/* Disallow: /soft/punto/win/uninstall/* Disallow: /soft/*welcome* Disallow: /soft/win/?* Disallow: /quotes Disallow: /redir Disallow: /region_map Disallow: /regions_list.xml Disallow: /regions.html? Disallow: /rubric2sport Disallow: /s/ Disallow: /save Disallow: /search Allow: /search/advanced Disallow: /setup Disallow: /showcaptcha Disallow: /sitesearch Disallow: /skazki Disallow: /sl/*.html Disallow: /soft/extensions/goodbye Disallow: /sportagent Disallow: /storeclick Disallow: /storerequest Disallow: /subscribe/confirm.pl Disallow: /subscribe/view.pl Disallow: /support/direct-images Disallow: /support/direct-tooltips Disallow: /support/distr Disallow: /support/dsp Disallow: /support/fe4be44a295cc679e19bf0b8f133083d Disallow: /support/maps-beta Disallow: /support/market-images Disallow: /support/security Disallow: /support/webmaster-images Disallow: /support/*zout_ Disallow: /support/search-results/ Disallow: /telsearch Disallow: /themes Disallow: /toggle-experiment Disallow: /touchsearch Disallow: /versions Disallow: /v$ Disallow: /viewconfig$ Disallow: /video/v$ Disallow: /video/viewconfig$ Disallow: /images/v$ Disallow: /images/viewconfig$ Disallow: /video/* Disallow: /video/*filmId=* Allow: /video/$ Allow: /video/? Allow: /video/sitemap Allow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/ Allow: /video/dizi-izle/ Allow: /video/dizi-izle/? Allow: /video/yerli-dizi-izle/ Allow: /video/yabanci-dizi-izle/ Allow: /video/touch/ Allow: /video/pad/ Disallow: /xmlsearch Disallow: /yaca Disallow: /yandsearch Disallow: /yca/cy Disallow: /soft/distribution Disallow: /soft/?* Disallow: /promo/launcher/?* Disallow: /opera/?* Disallow: /firefox/?* Disallow: /soft/bm/?* Disallow: /soft/browsers/?* Disallow: /soft/punto/?* Disallow: /ie/?* Disallow: /element/?* Disallow: /element/*goodbye* Disallow: /soft/*goodbye* Disallow: /adult Clean-Param: ncrnd&redircnt&clid&_&win / Sitemap: https://yandex.com/support/sitemap.xml Sitemap: https://yandex.com/blog/sitemap.xml Sitemap: https://yandex.com/collections/sitemap/sitemap-index.xml User-Agent: Twitterbot Allow: /images Allow: /gorsel Allow: /video Allow: /collections/picture/ Disallow: /adult Sitemap: https://yandex.com/support/sitemap.xml Sitemap: https://yandex.com/blog/sitemap.xml Sitemap: https://yandex.com/collections/sitemap/sitemap-index.xml User-agent: Applebot Disallow: /? Disallow: /403.html Disallow: /404.html Disallow: /500.html Disallow: /about.html Disallow: /adddata Disallow: /adresa-segmentator Disallow: /advanced_engl.html Disallow: /advertising Disallow: /all-supported-params Disallow: /articles Disallow: /blog/*?text= Disallow: /blog/*?tag= Disallow: /blog/*/*/* Disallow: /blogs* Allow: /blogs/$ Allow: /blogs/pad/$ Allow: /blogs/pad$ Disallow: /catalog/?text= Disallow: /cgi-bin/ Disallow: /cgi/ Disallow: /chisla.html Disallow: /clck Disallow: /collections/feed Disallow: /collections/search/ Disallow: /collections/*/_followers* Disallow: /collections/*/_followings* Disallow: /collections/share Disallow: /collections/favorites Disallow: /collections/embed Disallow: /collections/picture/ Disallow: /company/*.rss Disallow: /company/search Disallow: /cy Disallow: /cycounter Disallow: /dzen Disallow: /edu/ping Disallow: /edu/tasks Disallow: /edu/teachers Disallow: /edu/test Disallow: /experiments.xml Disallow: /formfeedback Disallow: /gorsel/* Allow: /gorsel/$ Allow: /gorsel/?* Allow: /gorsel/smart/$ Allow: /gorsel/touch/$ Allow: /gorsel/touch/?* Disallow: /goto_issue/ Disallow: /goto_rubric/ Disallow: /i/ Disallow: /images-data Disallow: /images.html Disallow: /images/* Allow: /images/$ Allow: /images/?* Allow: /images/smart/$ Allow: /images/touch/$ Allow: /images/touch/? Disallow: /index_m Disallow: /infected Disallow: /keyboard_qwerty.html Disallow: /logotypes Disallow: /map-constructor/loader* Disallow: /more_samples Disallow: /msearch Disallow: /msearchpart Disallow: /nmaps/*? Disallow: /mapeditor/*? Disallow: /norobot Disallow: /opensearch.xml Disallow: /padsearch Disallow: /people* Disallow: /person Disallow: /podpiska/login.pl Disallow: /polling Disallow: /promo/diskelement Disallow: /promo/*goodbye* Disallow: /promo/skype*?* Disallow: /promo/skype/*/* Disallow: /promo/launcher/feedback Disallow: /promo/launcher/mgoodbye/* Disallow: /promo/*welcome* Disallow: /promo/yobject/changelog* Disallow: /soft/bm/goodbye Disallow: /soft/chrome/ext-install/* Disallow: /soft/chrome/searchline-install/* Disallow: /soft/*goodbye* Disallow: /soft/punto/mac/uninstall/* Disallow: /soft/punto/win/uninstall/* Disallow: /soft/*welcome* Disallow: /soft/win/?* Disallow: /quotes Disallow: /redir Disallow: /region_map Disallow: /regions_list.xml Disallow: /regions.html? Disallow: /rubric2sport Disallow: /s/ Disallow: /save Disallow: /search Allow: /search/advanced Disallow: /setup Disallow: /showcaptcha Disallow: /sitesearch Disallow: /skazki Disallow: /sl/*.html Disallow: /soft/extensions/goodbye Disallow: /sportagent Disallow: /storeclick Disallow: /storerequest Disallow: /subscribe/confirm.pl Disallow: /subscribe/view.pl Disallow: /support/direct-images Disallow: /support/direct-tooltips Disallow: /support/distr Disallow: /support/dsp Disallow: /support/fe4be44a295cc679e19bf0b8f133083d Disallow: /support/maps-beta Disallow: /support/market-images Disallow: /support/security Disallow: /support/webmaster-images Disallow: /support/*zout_ Disallow: /support/search-results/ Disallow: /telsearch Disallow: /themes Disallow: /toggle-experiment Disallow: /touchsearch Disallow: /versions Disallow: /v$ Disallow: /viewconfig$ Disallow: /video/v$ Disallow: /video/viewconfig$ Disallow: /images/v$ Disallow: /images/viewconfig$ Disallow: /video/* Disallow: /video/*filmId=* Allow: /video/$ Allow: /video/? Allow: /video/sitemap Allow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/ Allow: /video/dizi-izle/ Allow: /video/dizi-izle/? Allow: /video/yerli-dizi-izle/ Allow: /video/yabanci-dizi-izle/ Allow: /video/touch/ Allow: /video/pad/ Disallow: /xmlsearch Disallow: /yaca Disallow: /yandsearch Disallow: /yca/cy Disallow: /soft/distribution Disallow: /soft/?* Disallow: /promo/launcher/?* Disallow: /opera/?* Disallow: /firefox/?* Disallow: /soft/bm/?* Disallow: /soft/browsers/?* Disallow: /soft/punto/?* Disallow: /ie/?* Disallow: /element/?* Disallow: /element/*goodbye* Disallow: /soft/*goodbye* Disallow: /adult Clean-Param: ncrnd&redircnt&clid&_&win / Sitemap: https://yandex.com/support/sitemap.xml Sitemap: https://yandex.com/blog/sitemap.xml Sitemap: https://yandex.com/collections/sitemap/sitemap-index.xml
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-28 | ok 200 | 7afa5219ab35 |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2025-09-15 CC-MAIN-2025-38 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2024-09-16 CC-MAIN-2024-38 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2023-10-02 CC-MAIN-2023-40 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2022-10-03 CC-MAIN-2022-40 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2021-09-27 CC-MAIN-2021-39 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2020-09-28 CC-MAIN-2020-40 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2019-09-23 CC-MAIN-2019-39 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2018-09-24 CC-MAIN-2018-39 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2018-01-29 CC-MAIN-2018-05 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-10-23 CC-MAIN-2017-43 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-09-25 CC-MAIN-2017-39 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-07-24 CC-MAIN-2017-30 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-04-24 CC-MAIN-2017-17 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-01-23 CC-MAIN-2017-04 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2016-10-03 CC-MAIN-2016-40 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2016-09-05 CC-MAIN-2016-36 | yandex.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
Evidence
Every daily observation of yandex.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).
API: https://infrarabbit.com/crawlers/api/v1/policy/yandex.com