pikabu.ru: what it tells AI crawlers

On 28 October 2026, pikabu.ru's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://pikabu.ru/robots.txt
unchanged since 2026-10-26 · sha256 ffc58bede1a9269f… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-28

Other signals

robots.txt as recorded sha256 81a4e18ef8b4ff45…

#             _________
#            /         \
#           /           \
#          /             \
#          |  ####  #### |
#          | / @ | / @ | |   .---------------------------------.
#          | \__/  \__/  |  (  YOU'RE DEFINITELY NOT A CRAWLER  )
#          |             |   .---------------------------------.
#          \    ___     /  _/
#           \          /
#            \________/
#                |
#               /|\
#              / | \
#             /  |  |
#            /   |  |
#                |

User-Agent: *
Allow: */community/edits/new
Allow: /search?n=16&r=3
Allow: /original-stories
Disallow: /generate_xml.php
Disallow: /restore.php
Disallow: /ajax/*
Disallow: */ajax.php*
Disallow: /api/*
Disallow: /api2/*
Disallow: /assets/*
Disallow: /cron/*
Disallow: /data_vars/*
Disallow: /external/*
Disallow: /files/*
Disallow: /staff/*
Disallow: /pds/*
Disallow: *username=*
Disallow: /offer-redirect/
Disallow: /*?r=
Disallow: /*?q=*
Disallow: */edits/*
Disallow: /*?n=*
Disallow: /admitad/finance/redirect
Disallow: /page/misc/adstat.php*
Disallow: /.well-known/*
Disallow: /search?*
Disallow: /tag/*st=*
Disallow: /tag/*n=*
Disallow: /tag/*et=*
Disallow: /tag/*d=*
Disallow: /tag/*cl=*
Disallow: /tag/*vh=*
Disallow: /@*/*
Disallow: /community/*/*
Disallow: /communities/*
Disallow: /communities?t=*
Disallow: /communities?q=*
Disallow: /add?community=*
Disallow: /information/docs
Disallow: *etm_term=*
Disallow: *ysclid=*
Disallow: *loc=*
Disallow: /ext/*
Disallow: /*?u=*
Disallow: *dv=*
Disallow: *mv=*

User-Agent: Yandex
Allow: */community/edits/new
Allow: /search?n=16&r=3
Allow: /original-stories
Disallow: /generate_xml.php
Disallow: /restore.php
Disallow: /ajax/*
Disallow: */ajax.php*
Disallow: /api/*
Disallow: /api2/*
Disallow: /assets/*
Disallow: /cron/*
Disallow: /data_vars/*
Disallow: /external/*
Disallow: /files/*
Disallow: /rabbit_workers/*
Disallow: /staff/*
Disallow: /pds/*
Disallow: *username=*
Disallow: /offer-redirect/
Disallow: /*?r=
Disallow: /*?q=*
Disallow: */edits/*
Disallow: /*?n=*
Disallow: /admitad/finance/redirect
Disallow: /page/misc/adstat.php*
Disallow: /.well-known/*
Disallow: /search?*
Disallow: /tag/*st=*
Disallow: /tag/*n=*
Disallow: /tag/*et=*
Disallow: /tag/*d=*
Disallow: /tag/*cl=*
Disallow: /tag/*vh=*
Disallow: /@*/*
Disallow: /community/*/*
Disallow: /communities/*
Disallow: /communities?t=*
Disallow: /communities?q=*
Disallow: /add?community=*
Disallow: *dv=*
Disallow: *mv=*
Disallow: /*?u=*
Clean-param: rsid&story_url&vkmain&andpostshare&tgmain&YandexZenSpecial&linkshare&fbmain&viber&turbo_turbo&vshare&vkgame&iospostshare&andlinkshare&st&dv&d&photo&text&longread&is_hot&from&cpm_token&goto

Crawl-Delay: 3

User-Agent: Googlebot
Allow: */community/edits/new
Allow: /search?n=16&r=3
Allow: /original-stories
Disallow: /generate_xml.php
Disallow: /restore.php
Disallow: /ajax/*
Disallow: */ajax.php*
Disallow: /api/*
Disallow: /api2/*
Disallow: /assets/*
Disallow: /cron/*
Disallow: /data_vars/*
Disallow: /external/*
Disallow: /files/*
Disallow: /staff/*
Disallow: /pds/*
Disallow: *username=*
Disallow: /offer-redirect/
Disallow: /*?r=
Disallow: /*?q=*
Disallow: */edits/*
Disallow: /*?n=*
Disallow: /admitad/finance/redirect
Disallow: /page/misc/adstat.php*
Disallow: /.well-known/*
Disallow: /search?*
Disallow: /tag/*st=*
Disallow: /tag/*n=*
Disallow: /tag/*et=*
Disallow: /tag/*d=*
Disallow: /tag/*cl=*
Disallow: /tag/*vh=*
Disallow: /communities/*
Disallow: /communities?t=*
Disallow: /communities?q=*
Disallow: /add?community=*
Disallow: /information/docs
Disallow: *etm_term=*
Disallow: *ysclid=*
Disallow: *loc=*
Disallow: /ext/*
Disallow: /*?u=*

User-Agent: Mail.RU_Bot
Allow: */community/edits/new
Allow: /search?n=16&r=3
Allow: /original-stories
Disallow: /generate_xml.php
Disallow: /restore.php
Disallow: /WT4fgrvixo2F/*
Disallow: /ajax/*
Disallow: */ajax.php*
Disallow: /api/*
Disallow: /api2/*
Disallow: /assets/*
Disallow: /cron/*
Disallow: /data_vars/*
Disallow: /external/*
Disallow: /files/*
Disallow: /rabbit_workers/*
Disallow: /staff/*
Disallow: /pds/*
Disallow: *username=*
Disallow: /offer-redirect/
Disallow: /*?r=
Disallow: /*?q=*
Disallow: */edits/*
Disallow: /*?n=*
Disallow: /admitad/finance/redirect
Disallow: /page/misc/adstat.php*
Disallow: /.well-known/*
Disallow: /search?*
Disallow: /tag/*st=*
Disallow: /tag/*n=*
Disallow: /tag/*et=*
Disallow: /tag/*d=*
Disallow: /tag/*cl=*
Disallow: /tag/*vh=*
Disallow: /@*/*
Disallow: /community/*/*
Disallow: /communities/*
Disallow: /communities?t=*
Disallow: /communities?q=*
Disallow: /add?community=*
Disallow: *dv=*
Disallow: *mv=*
Crawl-Delay: 3

Sitemap: https://pikabu.ru/sitemap_index.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of pikabu.ru is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (pending (4 calendars)).

API: https://infrarabbit.com/crawlers/api/v1/policy/pikabu.ru