gosuslugi.ru: what it tells AI crawlers

On 28 October 2026, gosuslugi.ru's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.gosuslugi.ru/robots.txt
unchanged since 2026-10-26 · sha256 ee498570e53c9190… · view as recorded

The * group

partial applies to every agent without its own group

Per AI bot, 2026-10-28

Other signals

robots.txt as recorded sha256 3a422caf76f6c57b…

Host: https://www.gosuslugi.ru

User-agent: SputnikBot
Allow: /$
Disallow: /
Disallow: /paygate
Disallow: /*/result
Disallow: /*/form
Disallow: /*root.betaUrl
Disallow: /*payment
Disallow: /situation/migration
Allow: /foreign
Allow: /1
Allow: /2
Allow: /3
Allow: /4
Allow: /5
Allow: /6
Allow: /7
Allow: /8
Allow: /9
Allow: /structure
Allow: /help
Allow: /sitemap
Allow: /category
Allow: /pay
Allow: /search$
Allow: /legal-entity
Allow: /*landing
Allow: /situation
Allow: /promo
Allow: /entrepreneur
Allow: /information
Allow: /migrant
Allow: /*/sputnik
Allow: /new
Allow: /r
Disallow: *?search=

User-agent: GoogleBot
Allow: /$
Disallow: /
Disallow: /paygate
Disallow: /*/result
Disallow: /*/form
Disallow: /*root.betaUrl
Disallow: /*payment
Disallow: */search?query=
Disallow: *attrs.formUrl*
Disallow: *root.partnersUrl*
Disallow: /situation/migration
Disallow: /user
Disallow: /feedback
Allow: /api/cms/
Allow: /foreign
Allow: /1
Allow: /2
Allow: /3
Allow: /4
Allow: /5
Allow: /6
Allow: /7
Allow: /8
Allow: /9
Allow: /structure
Allow: /help
Allow: /sitemap
Allow: /category
Allow: /pay
Allow: /search$
Allow: /legal-entity
Allow: /*landing
Allow: /situation
Allow: /promo
Allow: /entrepreneur
Allow: /information
Allow: /migrant
Allow: /*/sputnik
Allow: /htdocs
Allow: /api/catalog/v3/passports/
Allow: /api/catalog/v3/departments/
Allow: /api/catalog/v3/services/
Allow: /api/catalog/v3/situations/
Allow: /api/cms/v1/news/
Allow: /api/cms/v1/faq/categories/
Allow: /api/cms/v1/footers/EPGUV3_DESK/
Allow: /new
Allow: /r
Disallow: */info/1$
Disallow: */info/2$
Disallow: */info/3$
Disallow: */info/4$
Allow: *widgets/prod*
Allow: */api/catalog/v3/*
Disallow: *?search=
Allow: /api/nsi/v1/epgu/detectRegion


User-agent: *
Allow: /$
Disallow: /
Disallow: /paygate
Disallow: /*/result
Disallow: /*/form
Disallow: /*root.betaUrl
Disallow: /*payment
Disallow: /foreign/wsapi
Disallow: /*/booking
Disallow: /*/mfc
Disallow: */search?query=
Disallow: *attrs.formUrl*
Disallow: *root.partnersUrl*
Disallow: /situation/migration
Disallow: /news
Disallow: /feedback
Allow: /foreign
Allow: /1
Allow: /2
Allow: /3
Allow: /4
Allow: /5
Allow: /6
Allow: /7
Allow: /8
Allow: /9
Allow: /structure
Allow: /help
Allow: /sitemap
Allow: /category
Allow: /pay
Allow: /search$
Allow: /legal-entity
Allow: /*landing
Allow: /situation
Allow: /promo
Allow: /entrepreneur
Allow: /information
Allow: /migrant
Allow: /new
Allow: /r
Disallow: *?search=

Clean-param: utm_campaign&utm_medium&utm_source&utm_term&utm_content


Sitemap: https://www.gosuslugi.ru/sitemap/sitemap-manual.xml
Sitemap: https://www.gosuslugi.ru/sitemap/all-sitemap.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of gosuslugi.ru is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).

API: https://infrarabbit.com/crawlers/api/v1/policy/gosuslugi.ru