yandex.ru: what it tells AI crawlers
On 27 October 2026, yandex.ru's robots.txt partly blocked OAI-SearchBot.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://yandex.ru/robots.txt
unchanged since 2026-10-26 · sha256c51a208183d07eb7…· view as recorded - The * group
- partial applies to every agent without its own group
Per AI bot, 2026-10-27
| User agent | Verdict | How |
|---|---|---|
| GPTBot | partial | through the * group |
| OAI-SearchBot | partial | named in its own group |
| ChatGPT-User | partial | through the * group |
| ClaudeBot | partial | through the * group |
| Claude-SearchBot | partial | through the * group |
| Claude-User | partial | through the * group |
| PerplexityBot | partial | through the * group |
| Perplexity-User | partial | through the * group |
| Googlebot | partial | named in its own group |
| Google-Extended | partial | through the * group |
| Applebot-Extended | partial | through the * group |
| CCBot | partial | through the * group |
| Bytespider | partial | through the * group |
| Meta-ExternalAgent | partial | through the * group |
| Amazonbot | partial | through the * group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | not observed |
| Declares Content Signals (robots.txt or header) | not observed |
| Says ai-train=no in a Content Signal | not observed |
| Points to an RSL licence (robots.txt License line or Link header) | not observed |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | not observed |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | not observed |
| Sends noai / noimageai directives | not observed |
| Refused our fetcher (403 / challenge) | no |
robots.txt as recorded sha256 b91c546e27a193a7…
# yandex.ru User-agent: * Disallow: /? Disallow: /403.html Disallow: /404.html Disallow: /500.html Disallow: /about.html Disallow: /adddata Disallow: /adresa-segmentator Disallow: /advanced_engl.html Disallow: /advertising Disallow: /all-supported-params Disallow: /articles Disallow: /aura/ Allow: /aura/promo/ Disallow: /blog/*?text= Disallow: /blog/*?tag= Disallow: /blog/*/*/* Disallow: /blogs* Allow: /blogs$ Allow: /blogs/$ Allow: /blogs/pad/$ Allow: /blogs/pad$ Disallow: /bus/orders* Disallow: /bus/api/* Disallow: /catalog/?text= Disallow: /chat Disallow: /cgi-bin/ Disallow: /cgi/ Disallow: /chisla.html Disallow: /clck Disallow: /collections/feed Disallow: /collections/search/ Disallow: /collections/*/search/* Disallow: /collections/iznanka/ Disallow: /collections/*/_followers* Disallow: /collections/*/_followings* Disallow: /collections/share Disallow: /collections/favorites Disallow: /collections/embed Disallow: /collections/picture/ Disallow: /company/*.rss Disallow: /company/search Disallow: /courier Disallow: /conflagexp Disallow: /cy Disallow: /cycounter Disallow: /dzen Disallow: /edu/ping Disallow: /edu/tasks Disallow: /edu/teachers Disallow: /edu/test Disallow: /experiments.xml Disallow: /formfeedback Disallow: /fresh Disallow: /goto_issue/ Disallow: /goto_rubric/ Disallow: /i/ Disallow: /images-data Disallow: /images.html Disallow: /images/* Disallow: /images-apphost/* Allow: /images/$ Allow: /images/smart/$ Allow: /images/touch/$ Allow: /images/touch/? Disallow: /index_m Disallow: /infected Disallow: /iznanka/ Allow: /health/ Disallow: /keyboard_qwerty.html Disallow: /logotypes Allow: /local Disallow: /local/api Disallow: /local/create Disallow: /local/profile Disallow: /local/*?event_id Disallow: /more_samples Disallow: /msearch Disallow: /msearchpart Disallow: /map-constructor/loader* Disallow: /maps/*? Allow: /maps/*?lang=ru$ Allow: /maps/*?lang=uk$ Allow: /maps/org/ Disallow: /nmaps/*? Disallow: /mapeditor/*? Disallow: /norobot Disallow: /opensearch.xml Disallow: /padsearch Disallow: /people/* Allow: /people$ Allow: /people/$ Allow: /people/pad/$ Allow: /people/pad$ Disallow: /person Disallow: /podpiska/login.pl Disallow: /pogoda/search Disallow: /pogoda/404 Disallow: /pogoda/*?*lat=*&*lon=* Disallow: /pogoda/*?*lon=*&*lat=* Disallow: /polling Disallow: /poll/ Disallow: /promo/*goodbye* Disallow: /promo/diskelement Disallow: /promo/launcher/mgoodbye/* Disallow: /promo/skype*?* Disallow: /promo/skype/*/* Disallow: /promo/*welcome* Disallow: /promo/yobject/changelog* Disallow: /soft/bm/goodbye Disallow: /soft/chrome/ext-install/* Disallow: /soft/chrome/searchline-install/* Disallow: /soft/*goodbye* Disallow: /soft/punto/mac/uninstall/* Disallow: /soft/punto/win/uninstall/* Disallow: /soft/*welcome* Disallow: /soft/win/?* Disallow: /quotes Disallow: /redir Disallow: /region_map Disallow: /regions_list.xml Disallow: /regions.html? Disallow: /rubric2sport Disallow: /s/ Disallow: /save Disallow: /safety/?* Disallow: /search Disallow: /setup Disallow: /showcaptcha Disallow: /sitesearch Disallow: /skazki Disallow: /sl/*.html Disallow: /soft/extensions/goodbye Disallow: /sportagent Disallow: /storeclick Disallow: /storerequest Disallow: /subscribe/confirm.pl Disallow: /subscribe/view.pl Allow: /support/ Disallow: /support/direct-images Disallow: /support/direct-tooltips Disallow: /support/distr Disallow: /support/dsp Disallow: /support/fe4be44a295cc679e19bf0b8f133083d Disallow: /support/maps-beta Disallow: /support/market-images Disallow: /support/security Disallow: /support/webmaster-images Disallow: /support/*zout_ Disallow: /support/search-results/ Disallow: /talk/create Disallow: /talk/api Disallow: /talk/user/*/article Allow: /talk Allow: /talk/user Disallow: /telsearch Disallow: /themes Disallow: /toggle-experiment Disallow: /touchsearch Disallow: /tune*retpath= Allow: /ugcpub/ Disallow: /ugcpub/cabinet Disallow: /v$ Disallow: /viewconfig$ Disallow: /video/v$ Disallow: /video/viewconfig$ Disallow: /images/v$ Disallow: /images/viewconfig$ Allow: /uslugi/* Disallow: /uslugi/cab Disallow: /uslugi/*/category/$ Allow: /tutor Disallow: /tutor/admin Disallow: /versions Disallow: /video/* Disallow: /video/*filmId=* Allow: /video/$ Allow: /video/? Allow: /video/sitemap Allow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/ Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/*text= Allow: /video/dizi-izle/ Allow: /video/dizi-izle/? Allow: /video/yerli-dizi-izle/ Allow: /video/yabanci-dizi-izle/ Allow: /video/touch/ Allow: /video/pad/ Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/%D0%B2%D0%BD%D0%B5-%D0%B8%D0%B3%D1%80%D1%8B* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/%D0%B4%D0%BE%D0%BC%D0%B0%D1%88%D0%BD%D0%B8%D0%B9-%D0%B0%D1%80%D0%B5%D1%81%D1%82* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/%D0%BC%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%B8%D0%B5-%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D0%B8* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/%D0%BE%D1%81%D1%82%D0%BE%D1%80%D0%BE%D0%B6%D0%BD%D0%BE-%D0%B7%D0%B5%D0%BC%D0%BB%D1%8F%D0%BD%D0%B5* Disallow: /xmlsearch Disallow: /yaca Allow: /yaca/$ Allow: /yaca/m/$ Allow: /yaca/m/yca/cat/*/$ Allow: /yaca/*.xml$ Allow: /yaca/cat/*/$ Disallow: /yaca/result.xml Disallow: /yaca/*/tungrp/ Disallow: /yaca/*/time/ Disallow: /yaca/*/geo/ Disallow: /yaca/cat/*/geo/* Disallow: /yaca/*/synt2/ Disallow: /yandsearch Disallow: /yca/cy Allow: /yac2014 Disallow: /soft/distribution Disallow: /soft/?* Disallow: /promo/launcher/?* Disallow: /opera/?* Disallow: /firefox/?* Disallow: /soft/bm/?* Disallow: /soft/browsers/?* Disallow: /soft/punto/?* Disallow: /ie/?* Disallow: /element/?* Disallow: /element/*goodbye* Disallow: /soft/*goodbye* Disallow: /adult Clean-Param: ncrnd&redircnt&clid&_&win / Clean-Param: utm_source&utm_medium&utm_campaign&utm_term&utm_content&utm_nooverride&utm_expid&gclid&_openstat&yclid&gsaid&_gs_rmark&_gs_ref&_gs_cttl&iframe /health/ Clean-Param: text&wizextra&displayid&saas_webreqid /uslugi/*category/* Sitemap: https://yandex.ru/pogoda/sitemap_index.xml Sitemap: https://yandex.ru/support/sitemap.xml Sitemap: https://yandex.ru/blog/sitemap.xml Sitemap: https://yandex.ru/collections/sitemap/sitemap-index.xml Sitemap: https://yandex.ru/local/sitemap.xml Sitemap: https://yandex.ru/znatoki/sitemap/sitemap-index.xml Sitemap: https://yandex.ru/ugcpub/siteindex1.xml Sitemap: https://yandex.ru/ugcpub/siteindex2.xml Sitemap: https://frontend.s3.yandex.net/health/sitemap.xml Sitemap: https://ydo.s3.yandex.net/sitemap/sitemap.xml.gz Sitemap: https://yandex.ru/talk/sitemap.xml Sitemap: https://yandex.ru/uslugi/sitemap/sitemap.xml.gz Sitemap: https://yandex.ru/turbo/public/sitemap.xml Sitemap: https://yandex.ru/tutor/sitemap/?sitemap=sitemap.xml Sitemap: https://yandex.ru/turbo/talk-sitemap-index.xml Sitemap: https://yandex.ru/patents/sitemap/sitemap.xml Sitemap: https://games.yandex.ru/console/sitemaps/sitemap.index.xml User-Agent: Twitterbot Allow: /images Allow: /gorsel Allow: /video Allow: /collections/picture/ Disallow: /adult Allow: /ugcpub/ Disallow: /ugcpub/cabinet Sitemap: https://yandex.ru/pogoda/sitemap_index.xml Sitemap: https://yandex.ru/support/sitemap.xml Sitemap: https://yandex.ru/blog/sitemap.xml Sitemap: https://yandex.ru/collections/sitemap/sitemap-index.xml Sitemap: https://yandex.ru/local/sitemap.xml Sitemap: https://yandex.ru/znatoki/sitemap/sitemap-index.xml Sitemap: https://yandex.ru/ugcpub/siteindex1.xml Sitemap: https://yandex.ru/ugcpub/siteindex2.xml Sitemap: https://frontend.s3.yandex.net/health/sitemap.xml Sitemap: https://ydo.s3.yandex.net/sitemap/sitemap.xml.gz Sitemap: https://yandex.ru/talk/sitemap.xml Sitemap: https://yandex.ru/uslugi/sitemap/sitemap.xml.gz Sitemap: https://yandex.ru/turbo/public/sitemap.xml Sitemap: https://yandex.ru/tutor/sitemap/?sitemap=sitemap.xml Sitemap: https://yandex.ru/turbo/talk-sitemap-index.xml Sitemap: https://yandex.ru/patents/sitemap/sitemap.xml Sitemap: https://games.yandex.ru/console/sitemaps/sitemap.index.xml User-agent: Applebot Disallow: /? Disallow: /403.html Disallow: /404.html Disallow: /500.html Disallow: /about.html Disallow: /adddata Disallow: /adresa-segmentator Disallow: /advanced_engl.html Disallow: /advertising Disallow: /all-supported-params Disallow: /articles Disallow: /aura/ Allow: /aura/promo/ Disallow: /blog/*?text= Disallow: /blog/*?tag= Disallow: /blog/*/*/* Disallow: /blogs* Allow: /blogs$ Allow: /blogs/$ Allow: /blogs/pad/$ Allow: /blogs/pad$ Disallow: /bus/orders* Disallow: /bus/api/* Disallow: /catalog/?text= Disallow: /chat Disallow: /cgi-bin/ Disallow: /cgi/ Disallow: /chisla.html Disallow: /clck Disallow: /collections/feed Disallow: /collections/search/ Disallow: /collections/*/search/* Disallow: /collections/iznanka/ Disallow: /collections/*/_followers* Disallow: /collections/*/_followings* Disallow: /collections/share Disallow: /collections/favorites Disallow: /collections/embed Disallow: /collections/picture/ Disallow: /company/*.rss Disallow: /company/search Disallow: /conflagexp Disallow: /cy Disallow: /cycounter Disallow: /dzen Disallow: /edu/ping Disallow: /edu/tasks Disallow: /edu/teachers Disallow: /edu/test Disallow: /experiments.xml Disallow: /formfeedback Disallow: /fresh Disallow: /goto_issue/ Disallow: /goto_rubric/ Disallow: /i/ Disallow: /images-data Disallow: /images.html Disallow: /images/* Allow: /images/$ Allow: /images/smart/$ Allow: /images/touch/$ Allow: /images/touch/? Disallow: /index_m Disallow: /infected Disallow: /iznanka/ Allow: /health/ Disallow: /keyboard_qwerty.html Allow: /local Disallow: /local/api Disallow: /local/create Disallow: /local/*?event_id Disallow: /local/profile Disallow: /logotypes Disallow: /map-constructor/loader* Disallow: /more_samples Disallow: /msearch Disallow: /msearchpart Disallow: /nmaps/*? Allow: /maps/org/ Disallow: /mapeditor/*? Disallow: /norobot Disallow: /opensearch.xml Disallow: /padsearch Disallow: /people/* Allow: /people$ Allow: /people/$ Allow: /people/pad/$ Allow: /people/pad$ Disallow: /person Disallow: /podpiska/login.pl Disallow: /pogoda/search Disallow: /pogoda/404 Disallow: /pogoda/*?*lat=*&*lon=* Disallow: /pogoda/*?*lon=*&*lat=* Disallow: /polling Disallow: /poll/ Disallow: /promo/*goodbye* Disallow: /promo/diskelement Disallow: /promo/launcher/mgoodbye/* Disallow: /promo/skype*?* Disallow: /promo/skype/*/* Disallow: /promo/*welcome* Disallow: /promo/yobject/changelog* Disallow: /soft/bm/goodbye Disallow: /soft/chrome/ext-install/* Disallow: /soft/chrome/searchline-install/* Disallow: /soft/*goodbye* Disallow: /soft/punto/mac/uninstall/* Disallow: /soft/punto/win/uninstall/* Disallow: /soft/*welcome* Disallow: /soft/win/?* Disallow: /quotes Disallow: /redir Disallow: /region_map Disallow: /regions_list.xml Disallow: /regions.html? Disallow: /rubric2sport Disallow: /s/ Disallow: /save Disallow: /safety/?* Disallow: /search Disallow: /set/ Disallow: /setup Disallow: /showcaptcha Disallow: /sitesearch Disallow: /skazki Disallow: /sl/*.html Disallow: /soft/extensions/goodbye Disallow: /sportagent Disallow: /storeclick Disallow: /storerequest Disallow: /subscribe/confirm.pl Disallow: /subscribe/view.pl Allow: /support/ Disallow: /support/direct-images Disallow: /support/direct-tooltips Disallow: /support/distr Disallow: /support/dsp Disallow: /support/fe4be44a295cc679e19bf0b8f133083d Disallow: /support/maps-beta Disallow: /support/market-images Disallow: /support/security Disallow: /support/webmaster-images Disallow: /support/*zout_ Disallow: /support/search-results/ Disallow: /talk/create Disallow: /talk/api Disallow: /talk/user/*/article Allow: /talk Allow: /talk/user Disallow: /telsearch Disallow: /themes Disallow: /toggle-experiment Disallow: /touchsearch Disallow: /tune*retpath= Allow: /tutor Disallow: /tutor/admin Allow: /ugcpub/ Disallow: /ugcpub/cabinet Disallow: /v$ Disallow: /viewconfig$ Disallow: /video/v$ Disallow: /video/viewconfig$ Disallow: /images/v$ Disallow: /images/viewconfig$ Allow: /uslugi/* Disallow: /uslugi/cab Disallow: /uslugi/*/category/$ Disallow: /versions Disallow: /video/* Disallow: /video/*filmId=* Allow: /video/$ Allow: /video/? Allow: /video/sitemap Allow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/ Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/*text= Allow: /video/dizi-izle/ Allow: /video/dizi-izle/? Allow: /video/yerli-dizi-izle/ Allow: /video/yabanci-dizi-izle/ Allow: /video/touch/ Allow: /video/pad/ Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/%D0%B2%D0%BD%D0%B5-%D0%B8%D0%B3%D1%80%D1%8B* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/%D0%B4%D0%BE%D0%BC%D0%B0%D1%88%D0%BD%D0%B8%D0%B9-%D0%B0%D1%80%D0%B5%D1%81%D1%82* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/%D0%BC%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%B8%D0%B5-%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D0%B8* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/%D0%BE%D1%81%D1%82%D0%BE%D1%80%D0%BE%D0%B6%D0%BD%D0%BE-%D0%B7%D0%B5%D0%BC%D0%BB%D1%8F%D0%BD%D0%B5* Disallow: /xmlsearch Disallow: /yaca Allow: /yaca/$ Allow: /yaca/m/$ Allow: /yaca/m/yca/cat/*/$ Allow: /yaca/*.xml$ Allow: /yaca/cat/*/$ Disallow: /yaca/result.xml Disallow: /yaca/*/tungrp/ Disallow: /yaca/*/time/ Disallow: /yaca/*/geo/ Disallow: /yaca/cat/*/geo/* Disallow: /yaca/*/synt2/ Disallow: /yandsearch Disallow: /yca/cy Allow: /yac2014 Disallow: /soft/distribution Disallow: /soft/?* Disallow: /promo/launcher/?* Disallow: /opera/?* Disallow: /firefox/?* Disallow: /soft/bm/?* Disallow: /soft/browsers/?* Disallow: /soft/punto/?* Disallow: /ie/?* Disallow: /element/?* Disallow: /element/*goodbye* Disallow: /soft/*goodbye* Disallow: /adult Clean-Param: ncrnd&redircnt&clid&_&win / Clean-Param: utm_source&utm_medium&utm_campaign&utm_term&utm_content&utm_nooverride&utm_expid&gclid&_openstat&yclid&gsaid&_gs_rmark&_gs_ref&_gs_cttl&iframe /health/ Clean-Param: text&wizextra&displayid&saas_webreqid /uslugi/*category/* Sitemap: https://yandex.ru/pogoda/sitemap_index.xml Sitemap: https://yandex.ru/support/sitemap.xml Sitemap: https://yandex.ru/blog/sitemap.xml Sitemap: https://yandex.ru/collections/sitemap/sitemap-index.xml Sitemap: https://yandex.ru/local/sitemap.xml Sitemap: https://yandex.ru/znatoki/sitemap/sitemap-index.xml Sitemap: https://yandex.ru/ugcpub/siteindex1.xml Sitemap: https://yandex.ru/ugcpub/siteindex2.xml Sitemap: https://frontend.s3.yandex.net/health/sitemap.xml Sitemap: https://ydo.s3.yandex.net/sitemap/sitemap.xml.gz Sitemap: https://yandex.ru/talk/sitemap.xml Sitemap: https://yandex.ru/uslugi/sitemap/sitemap.xml.gz Sitemap: https://yandex.ru/turbo/public/sitemap.xml Sitemap: https://yandex.ru/tutor/sitemap/?sitemap=sitemap.xml Sitemap: https://yandex.ru/turbo/talk-sitemap-index.xml Sitemap: https://yandex.ru/patents/sitemap/sitemap.xml Sitemap: https://games.yandex.ru/console/sitemaps/sitemap.index.xml User-agent: Yandex Disallow: /? Disallow: /403.html Disallow: /404.html Disallow: /500.html Disallow: /about.html Disallow: /adddata Disallow: /adresa-segmentator Disallow: /advanced_engl.html Disallow: /advertising Disallow: /all-supported-params Disallow: /articles Disallow: /aura/ Allow: /aura/promo/ Disallow: /blog/*?text= Disallow: /blog/*?tag= Disallow: /blog/*/*/* Disallow: /blogs* Allow: /blogs$ Allow: /blogs/$ Allow: /blogs/pad/$ Allow: /blogs/pad$ Disallow: /bus/orders* Disallow: /bus/api/* Disallow: /catalog/?text= Disallow: /chat Disallow: /cgi-bin/ Disallow: /cgi/ Disallow: /chisla.html Disallow: /clck Disallow: /collections/feed Disallow: /collections/search/ Disallow: /collections/*/search/* Disallow: /collections/iznanka/ Disallow: /collections/*/_followers* Disallow: /collections/*/_followings* Disallow: /collections/share Disallow: /collections/favorites Disallow: /collections/embed Disallow: /collections/picture/ Disallow: /company/*.rss Disallow: /company/search Disallow: /conflagexp Disallow: /cy Disallow: /cycounter Disallow: /dzen Disallow: /edu/ping Disallow: /edu/tasks Disallow: /edu/teachers Disallow: /edu/test Disallow: /experiments.xml Disallow: /formfeedback Disallow: /fresh Disallow: /goto_issue/ Disallow: /goto_rubric/ Disallow: /i/ Disallow: /images-data Disallow: /images.html Disallow: /images/* Allow: /images/$ Allow: /images/smart/$ Allow: /images/touch/$ Allow: /images/touch/? Disallow: /index_m Disallow: /infected Disallow: /iznanka/ Allow: /health/ Disallow: /keyboard_qwerty.html Disallow: /logotypes Allow: /local Disallow: /local/api Disallow: /local/create Disallow: /local/profile Disallow: /local/*?event_id Disallow: /more_samples Disallow: /msearch Disallow: /msearchpart Disallow: /map-constructor/loader* Disallow: /maps/*? Allow: /maps/*?lang=ru$ Allow: /maps/*?lang=uk$ Allow: /maps/org/ Disallow: /nmaps/*? Disallow: /mapeditor/*? Disallow: /norobot Disallow: /opensearch.xml Disallow: /padsearch Disallow: /people/* Allow: /people$ Allow: /people/$ Allow: /people/pad/$ Allow: /people/pad$ Disallow: /person Disallow: /podpiska/login.pl Disallow: /pogoda/search Disallow: /pogoda/404 Disallow: /pogoda/*?*lat=*&*lon=* Disallow: /pogoda/*?*lon=*&*lat=* Disallow: /polling Disallow: /poll/ Disallow: /promo/*goodbye* Disallow: /promo/diskelement Disallow: /promo/launcher/mgoodbye/* Disallow: /promo/skype*?* Disallow: /promo/skype/*/* Disallow: /promo/*welcome* Disallow: /promo/yobject/changelog* Disallow: /soft/bm/goodbye Disallow: /soft/chrome/ext-install/* Disallow: /soft/chrome/searchline-install/* Disallow: /soft/*goodbye* Disallow: /soft/punto/mac/uninstall/* Disallow: /soft/punto/win/uninstall/* Disallow: /soft/*welcome* Disallow: /soft/win/?* Disallow: /quotes Disallow: /redir Disallow: /region_map Disallow: /regions_list.xml Disallow: /regions.html? Disallow: /rubric2sport Disallow: /s/ Disallow: /save Disallow: /safety/?* Disallow: /search Disallow: /setup Disallow: /showcaptcha Disallow: /sitesearch Disallow: /skazki Disallow: /sl/*.html Disallow: /soft/extensions/goodbye Disallow: /sportagent Disallow: /storeclick Disallow: /storerequest Disallow: /subscribe/confirm.pl Disallow: /subscribe/view.pl Allow: /support/ Disallow: /support/direct-images Disallow: /support/direct-tooltips Disallow: /support/distr Disallow: /support/dsp Disallow: /support/fe4be44a295cc679e19bf0b8f133083d Disallow: /support/maps-beta Disallow: /support/market-images Disallow: /support/security Disallow: /support/webmaster-images Disallow: /support/*zout_ Disallow: /support/search-results/ Disallow: /talk/create Disallow: /talk/api Disallow: /talk/user/*/article Allow: /talk Allow: /talk/user Disallow: /telsearch Disallow: /themes Disallow: /toggle-experiment Disallow: /touchsearch Disallow: /tune*retpath= Allow: /tutor Disallow: /tutor/admin Sitemap: https://yandex.ru/tutor/sitemap/?sitemap=sitemap.xml Allow: /ugcpub/ Disallow: /ugcpub/cabinet Disallow: /v$ Disallow: /viewconfig$ Disallow: /video/v$ Disallow: /video/viewconfig$ Disallow: /images/v$ Disallow: /images/viewconfig$ Allow: /uslugi/* Disallow: /uslugi/cab Disallow: /uslugi/*/category/$ Disallow: /versions Disallow: /video/* Disallow: /video/*filmId=* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/*text= Allow: /video/$ Allow: /video/? Allow: /video/sitemap Allow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/ Allow: /video/dizi-izle/ Allow: /video/dizi-izle/? Allow: /video/yerli-dizi-izle/ Allow: /video/yabanci-dizi-izle/ Allow: /video/touch/ Allow: /video/pad/ Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/%D0%B2%D0%BD%D0%B5-%D0%B8%D0%B3%D1%80%D1%8B* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/%D0%B4%D0%BE%D0%BC%D0%B0%D1%88%D0%BD%D0%B8%D0%B9-%D0%B0%D1%80%D0%B5%D1%81%D1%82* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/%D0%BC%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%B8%D0%B5-%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D0%B8* Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/%D0%BE%D1%81%D1%82%D0%BE%D1%80%D0%BE%D0%B6%D0%BD%D0%BE-%D0%B7%D0%B5%D0%BC%D0%BB%D1%8F%D0%BD%D0%B5* Disallow: /xmlsearch Disallow: /yaca Allow: /yaca/$ Allow: /yaca/m/$ Allow: /yaca/m/yca/cat/*/$ Allow: /yaca/*.xml$ Allow: /yaca/cat/*/$ Disallow: /yaca/result.xml Disallow: /yaca/*/tungrp/ Disallow: /yaca/*/time/ Disallow: /yaca/*/geo/ Disallow: /yaca/cat/*/geo/* Disallow: /yaca/*/synt2/ Disallow: /yandsearch Disallow: /yca/cy Allow: /yac2014 Disallow: /soft/distribution Disallow: /soft/?* Disallow: /promo/launcher/?* Disallow: /opera/?* Disallow: /firefox/?* Disallow: /soft/bm/?* Disallow: /soft/browsers/?* Disallow: /soft/punto/?* Disallow: /ie/?* Disallow: /element/?* Disallow: /element/*goodbye* Disallow: /soft/*goodbye* Disallow: /adult Clean-Param: ncrnd&redircnt&clid&_&win / Clean-Param: utm_source&utm_medium&utm_campaign&utm_term&utm_content&utm_nooverride&utm_expid&gclid&_openstat&yclid&gsaid&_gs_rmark&_gs_ref&_gs_cttl&iframe /health/ Clean-Param: text&wizextra&displayid&saas_webreqid /uslugi/*category/*
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-27 | ok 200 | c51a208183d0 |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2025-09-15 CC-MAIN-2025-38 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2024-09-16 CC-MAIN-2024-38 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2023-10-02 CC-MAIN-2023-40 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2022-10-03 CC-MAIN-2022-40 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2021-09-27 CC-MAIN-2021-39 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2020-09-28 CC-MAIN-2020-40 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2019-09-23 CC-MAIN-2019-39 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2018-09-24 CC-MAIN-2018-39 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2018-01-29 CC-MAIN-2018-05 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-10-23 CC-MAIN-2017-43 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-09-25 CC-MAIN-2017-39 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-07-24 CC-MAIN-2017-30 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-04-24 CC-MAIN-2017-17 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-01-23 CC-MAIN-2017-04 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2016-10-03 CC-MAIN-2016-40 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2016-09-05 CC-MAIN-2016-36 | yandex.ru | ok view | partial | partial | unknown | partial | partial | unknown | partial |
Evidence
Every daily observation of yandex.ru is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).
API: https://infrarabbit.com/crawlers/api/v1/policy/yandex.ru