beget.ru: what it tells AI crawlers

On 27 October 2026, beget.ru's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 1k (Tranco)
robots.txt: ok (HTTP 200) · https://beget.com/robots.txt
unchanged since 2026-10-26 · sha256 00d9dbf8050c123d… · view as recorded
The * group: allows applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	allows	through the * group
OAI-SearchBot	allows	through the * group
ChatGPT-User	allows	through the * group
ClaudeBot	allows	through the * group
Claude-SearchBot	allows	through the * group
Claude-User	allows	through the * group
PerplexityBot	allows	through the * group
Perplexity-User	allows	through the * group
Googlebot	allows	through the * group
Google-Extended	allows	through the * group
Applebot-Extended	allows	through the * group
CCBot	allows	through the * group
Bytespider	allows	through the * group
Meta-ExternalAgent	allows	through the * group
Amazonbot	allows	through the * group

Other signals

Publishes /llms.txt	no
Publishes /ai.txt	no
Publishes /.well-known/tdmrep.json	no
Reserves TDM rights (TDMRep file, header or meta)	not observed
Declares Content Signals (robots.txt or header)	not observed
Says ai-train=no in a Content Signal	not observed
Points to an RSL licence (robots.txt License line or Link header)	not observed
Answers with an x402 payment offer (lower bound: only the homepage is probed)	not observed
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	not observed
Sends noai / noimageai directives	not observed
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 00d9dbf8050c123d…

User-agent: *
Allow: /.well-known/
Allow: *.mp4
Allow: *.jpg
Allow: *.png
Allow: *.jpeg
Allow: *.svg
Allow: *.gif
Allow: *.pdf
Allow:  */domains/zone/ru
Allow: */domains/zone/xn--p1ai
Allow: */domains/zone/com
Allow: */domains/zone/online
Allow: */domains/zone/store
Allow: */domains/zone/tech
Allow: */domains/zone/site
Allow: */domains/zone/me
Allow: */domains/zone/info
Allow: */domains/zone/org
Allow: */domains/zone/su
Allow: */domains/zone/net
Allow: */domains/zone/biz
Allow: */domains/zone/io
Allow: */domains/zone/pro
Allow: */domains/zone/in
Allow: */domains/zone/eu
Allow: */domains/zone/fun
Allow: */domains/zone/moscow
Allow: */domains/zone/cc
Allow: */domains/zone/co
Allow: */domains/zone/name
Allow: */domains/zone/one
Allow: */domains/zone/shop
Allow: */domains/zone/my
Allow: */domains/zone/ps
Allow: */domains/zone/app
Allow: */domains/zone/design
Allow: */domains/zone/be
Allow: */domains/zone/agency
Allow: */domains/zone/ae
Allow: */domains/zone/market
Allow: */domains/zone/wtf
Allow: */domains/zone/world
Allow: */domains/zone/wiki
Allow: */domains/zone/team
Allow: */domains/zone/xyz
Allow: */domains/zone/studio
Allow: */domains/zone/tv
Allow: */domains/zone/top
Allow: */domains/zone/email
Allow: */domains/zone/spb.ru
Allow: */domains/zone/ws
Allow: */domains/zone/space
Allow: */domains/zone/pw
Allow: */domains/zone/vc
Allow: */domains/zone/fm 
Allow: */domains/zone/family 
Allow: */domains/zone/%D0%BE%D0%BD%D0%BB%D0%B0%D0%B9%D0%BD
Allow: */domains/zone/pp.ru 
Allow: */domains/zone/ink
Allow: */domains/zone/money
Allow: */domains/zone/nl
Allow: */domains/zone/school
Allow: */domains/zone/support
Allow: */domains/zone/academy
Allow: */domains/zone/cash
Allow: */domains/zone/city
Allow: */domains/zone/it
Allow: */ru/platform-info/modules/s3
Disallow: *?srsltid*
Disallow: *?*
Disallow: */news/2012/*
Disallow: */news/2013/*
Disallow: */news/2014/*
Disallow: */news/2016/*
Disallow: */news/2017/*
Disallow: */news/2018/*
Disallow: */domains/callback/esia
Disallow: */esia/*
Disallow: */domains/zone/*domainZoneData
Disallow: */partners-rules
Disallow: /session_test.php
Disallow: */authcode/*
Disallow: */domains/search/*
Disallow: */domains/search-ai
Disallow: */domains/whois/*
Disallow: */domains/zone
Disallow: */domains/zone/*
Disallow: */domains/confirmation/*
Disallow: */ru/confirm-domain-account/*
Disallow: */kb/search
Disallow: */en/news/
Disallow: */ru/platform-info
Disallow: */ru/platform-info/*
Disallow: /ru/izmenenie-czen-na-dopolnitelnye-opczii
Disallow: */en/confirm-domain-account/*
Disallow: */en/platform-info
Disallow: */en/platform-info/*
Disallow: /en/izmenenie-czen-na-dopolnitelnye-opczii
Disallow: /ru/super-sekretnaya-stranicza
Disallow: /ru/isp-promo
Disallow: /ru/unsubscribe
Disallow: /ru/platform-info-modules-s3
Disallow: /ru/pravila-provedeniya-reklamnoj-akczii-10-keshbeka-na-pervuyu-oplatu-v-beget-oblake

User-agent: Yandex
Allow: *.mp4
Allow: *.jpg
Allow: *.png
Allow: *.jpeg
Allow: *.svg
Allow: *.gif
Allow: *.pdf
Allow: */domains/zone/ru
Allow: */domains/zone/xn--p1ai
Allow: */domains/zone/com
Allow: */domains/zone/online
Allow: */domains/zone/store
Allow: */domains/zone/tech
Allow: */domains/zone/site
Allow: */domains/zone/me
Allow: */domains/zone/info
Allow: */domains/zone/org
Allow: */domains/zone/su
Allow: */domains/zone/net
Allow: */domains/zone/biz
Allow: */domains/zone/io
Allow: */domains/zone/pro
Allow: */domains/zone/in
Allow: */domains/zone/eu
Allow: */domains/zone/fun
Allow: */domains/zone/moscow
Allow: */domains/zone/cc
Allow: */domains/zone/co
Allow: */domains/zone/name
Allow: */domains/zone/one
Allow: */domains/zone/shop
Allow: */domains/zone/my
Allow: */domains/zone/ps
Allow: */domains/zone/app
Allow: */domains/zone/design
Allow: */domains/zone/be
Allow: */domains/zone/agency
Allow: */domains/zone/ae
Allow: */domains/zone/market
Allow: */domains/zone/wtf
Allow: */domains/zone/world
Allow: */domains/zone/wiki
Allow: */domains/zone/team
Allow: */domains/zone/xyz
Allow: */domains/zone/studio
Allow: */domains/zone/tv
Allow: */domains/zone/top
Allow: */domains/zone/email
Allow: */domains/zone/spb.ru
Allow: */domains/zone/ws
Allow: */domains/zone/space
Allow: */domains/zone/pw
Allow: */domains/zone/vc
Allow: */domains/zone/fm 
Allow: */domains/zone/family 
Allow: */domains/zone/%D0%BE%D0%BD%D0%BB%D0%B0%D0%B9%D0%BD
Allow: */domains/zone/pp.ru 
Allow: */domains/zone/ink
Allow: */domains/zone/money
Allow: */domains/zone/nl
Allow: */domains/zone/school
Allow: */domains/zone/support
Allow: */domains/zone/academy
Allow: */domains/zone/cash
Allow: */domains/zone/city
Allow: */domains/zone/it
Allow: */ru/platform-info/modules/s3
Disallow: *?srsltid*
Disallow: *?*
Disallow: */news/2012/*
Disallow: */news/2013/*
Disallow: */news/2014/*
Disallow: */news/2016/*
Disallow: */news/2017/*
Disallow: */news/2018/*
Disallow: */domains/callback/esia
Disallow: */esia/*
Disallow:  */domains/zone/*domainZoneData
Disallow: */partners-rules
Disallow: /session_test.php
Disallow: */authcode/*
Disallow: */domains/search/*
Disallow: */domains/search-ai
Disallow: */domains/whois/*
Disallow: */domains/zone
Disallow: */domains/zone/*
Disallow: */domains/confirmation/*
Disallow: */ru/confirm-domain-account/*
Disallow: */kb/search
Disallow: */en/news/
Disallow: */ru/platform-info
Disallow: */ru/platform-info/*
Disallow: /ru/izmenenie-czen-na-dopolnitelnye-opczii
Disallow: */en/confirm-domain-account/*
Disallow: */en/platform-info
Disallow: */en/platform-info/*
Disallow: /en/izmenenie-czen-na-dopolnitelnye-opczii
Disallow: /ru/super-sekretnaya-stranicza
Disallow: /ru/isp-promo
Disallow: /ru/unsubscribe
Disallow: /ru/platform-info-modules-s3
Disallow: /ru/pravila-provedeniya-reklamnoj-akczii-10-keshbeka-na-pervuyu-oplatu-v-beget-oblake

Clean-param: addt&cat&tariff&dname&sa&goal
Clean-param: source&utm_source&utm_campaign&utm_medium&utm_term&utm_content&type&utm&ref&partner&advert&stat&page
Clean-param: section&tags&cookie-policy&typelic&_gl&device&block&pos&region&region_name&etext

Sitemap: https://beget.com/sitemap.xml

robots.txt versions

From	Through	State	Version
2026-10-26	2026-10-28	ok 200	`00d9dbf8050c`

Timeline of changes

Date	What changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	beget.ru	redirect-only	unknown	unknown	unknown	unknown	unknown	unknown	unknown
2025-09-15 CC-MAIN-2025-38	www.beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2024-09-16 CC-MAIN-2024-38	beget.ru	redirect-only	unknown	unknown	unknown	unknown	unknown	unknown	unknown
2023-10-02 CC-MAIN-2023-40	www.beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2022-10-03 CC-MAIN-2022-40	www.beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2021-09-27 CC-MAIN-2021-39	www.beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	www.beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2019-09-23 CC-MAIN-2019-39	www.beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-09-24 CC-MAIN-2018-39	www.beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-01-29 CC-MAIN-2018-05	www.beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-10-23 CC-MAIN-2017-43	beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-07-24 CC-MAIN-2017-30	beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-04-24 CC-MAIN-2017-17	beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-01-23 CC-MAIN-2017-04	beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-10-03 CC-MAIN-2016-40	beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-09-05 CC-MAIN-2016-36	beget.ru	ok view	allows	allows	unknown	allows	allows	unknown	allows

Evidence

Every daily observation of beget.ru is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/beget.ru