jetbrains.com: what it tells AI crawlers

On 27 October 2026, jetbrains.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.jetbrains.com/robots.txt
unchanged since 2026-10-26 · sha256 9372e147d5d59b25… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 8d02072f66004b46…

User-Agent: *
# Global
Disallow: */search/
Disallow: */shop/
Disallow: */eshop*
Disallow: */estore*
Disallow: */unitrun*
Disallow: */languages/
Disallow: */feedback/
# crawling _allowed_: indexing blocked via robots noindex
# Disallow: */promo/
# Disallow: */download-thanks

# Sitemaps
Sitemap: https://www.jetbrains.com/sitemap_index.xml

# Yandex
User-agent: Yandex
Clean-param: q&products&utm_source&productedition&src&var&type&productversion&order&from&search&_ga&_gac&_gl&edition&utm_content&vsversion&hl&u&keyword&utm_medium&dclid&topic&rel&source&build&platform&ref&installedvsversions&utm_code&fbclid&products_id&fromMenu&fromblog&origin&code&fromide&timezone&utm_campaign&gclsrc&sid&page&section&mkt_tok&mode&keymap&rss&s

# Baidu
User-agent: Baidu
Disallow: */promo/
Disallow: */download-thanks
Disallow: ?q
Disallow: ?products
Disallow: ?utm_source
Disallow: ?productedition
Disallow: ?src
Disallow: ?var
Disallow: ?type
Disallow: ?productversion
Disallow: ?order
Disallow: ?from
Disallow: ?search
Disallow: ?_ga
Disallow: ?_gac
Disallow: ?_gl
Disallow: ?edition
Disallow: ?utm_content
Disallow: ?vsversion
Disallow: ?hl
Disallow: ?u
Disallow: ?keyword
Disallow: ?utm_medium
Disallow: ?dclid
Disallow: ?topic
Disallow: ?rel
Disallow: ?source
Disallow: ?build
Disallow: ?platform
Disallow: ?ref
Disallow: ?installedvsversions
Disallow: ?utm_code
Disallow: ?fbclid
Disallow: ?products_id
Disallow: ?fromMenu
Disallow: ?fromFooter
Disallow: ?fromblog
Disallow: ?origin
Disallow: ?code
Disallow: ?fromide
Disallow: ?timezone
Disallow: ?utm_campaign
Disallow: ?gclsrc
Disallow: ?sid
Disallow: ?page
Disallow: ?section
Disallow: ?mkt_tok
Disallow: ?mode
Disallow: ?keymap
Disallow: ?rss
Disallow: ?s

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of jetbrains.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/jetbrains.com