ya.ru: what it tells AI crawlers

On 28 October 2026, ya.ru's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://ya.ru/robots.txt
unchanged since 2026-10-26 · sha256 e8051b90a93aefd9… · view as recorded

The * group

partial applies to every agent without its own group

Per AI bot, 2026-10-28

Other signals

robots.txt as recorded sha256 4fc47f062ce83502…

User-agent: *
Disallow: /?
Disallow: /404.html
Disallow: /about.html
Disallow: /adddata
Disallow: /advanced_engl.html
Disallow: /advertising
Disallow: /articles
Disallow: /chisla.html
Disallow: /cgi-bin/
Disallow: /cgi/
Disallow: /cy
Disallow: /discounts/
Disallow: /dzen.html
Disallow: /i/
Disallow: /ie3/yandsearch
Disallow: /keyboard_qwerty.html
Disallow: /logotypes
Disallow: /norobot
Disallow: /polling
Disallow: /redir
Disallow: /regions.html?
Disallow: /s/
Disallow: /setup
Disallow: /skazki
Disallow: /subscribe/confirm.pl
Disallow: /subscribe/view.pl
Disallow: /yaca
Disallow: /ya
Disallow: /yandsearch
Disallow: /catalog/?text=
Disallow: /msearch
Disallow: /themes
Disallow: /showcaptcha
Disallow: /sitesearch
Disallow: /sl/*.html
Disallow: /403.html
Disallow: /404.html
Disallow: /500.html
Disallow: /adresa-segmentator
Disallow: /all-supported-params
Disallow: /cgi-bin/hidereferer
Disallow: /cgi-bin/set-intl
Disallow: /cgi-bin/xmlsearch.pl
Disallow: /cgi-bin/yandpage
Disallow: /cgi-bin/yandsearch
Disallow: /click|/cy
Disallow: /clck
Disallow: /cycounter
Disallow: /dzen
Allow: /design-school$
Allow: /edu$
Allow: /edu/$
Disallow: /edu/tasks
Disallow: /edu/teachers
Disallow: /edu/test
Disallow: /edu/ping
Disallow: /experiments.xml
Disallow: /family
Disallow: /familysearch
Disallow: /formfeedback
Disallow: /goto_issue/
Disallow: /goto_rubric/
Disallow: /i/yandex-big.gaf
Disallow: /ie3/yandsearch
Allow: /jobs$
Allow: /jobs/$
Disallow: /images-data
Disallow: /images.html
Disallow: /images/*
Allow: /images/$
Allow: /images/smart/$
Allow: /images/touch/$
Disallow: /index_m
Disallow: /infected
Disallow: /largesearch
Disallow: /map/.+/news.html
Disallow: /more_samples
Disallow: /msearch
Disallow: /msearchpart
Disallow: /norobot
Disallow: /opensearch.xml
Disallow: /padsearch
Disallow: /people
Disallow: /person
Disallow: /podpiska/login.pl
Disallow: /quotes
Disallow: /redir
Disallow: /redir_warning
Disallow: /region_map
Disallow: /regions_list.xml
Disallow: /rubric2sport
Disallow: /save*
Disallow: /promo/skype*
Disallow: /schoolsearch
Disallow: /search/advanced
Disallow: /search/customize
Disallow: /search/extra-snippet
Disallow: /search/inforequest
Disallow: /search
Disallow: /sitesearch
Disallow: /sportagent
Disallow: /storeclick
Disallow: /storerequest
Disallow: /telsearch
Disallow: /toggle-experiment
Disallow: /touchsearch
Disallow: /v
Disallow: /versions
Disallow: /video/*
Allow: /video/$
Allow: /video/touch/$
Disallow: /wpage
Disallow: /xmlsearch
Disallow: /yandpage
Disallow: /yandsearch
Allow: /yac2014
Disallow: /yca/cy

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of ya.ru is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).

API: https://infrarabbit.com/crawlers/api/v1/policy/ya.ru