hbr.org: what it tells AI crawlers

On 27 October 2026, hbr.org's robots.txt fully blocked GPTBot, Google-Extended and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://hbr.org/robots.txt
unchanged since 2026-10-26 · sha256 793e323f3d6c3a72… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 d46c39d2f57a6983…

User-Agent: *
Disallow: /resources/
Allow: /resources/js
Allow: /resources/css
Allow: /resources/images/
Allow: /resources/xml/sitemaps/
Disallow: /fastanswers
Disallow: /my-library*
Disallow: /email-colleague/
Disallow: /add-to-cart/
Disallow: /login*
Disallow: /shopping-cart/
Disallow: /shipping-payment
Disallow: /review-order
Disallow: /order/thank-you/
Disallow: /content/ipad/
Disallow: /newsletters*
Disallow: /product/recommended*
Disallow: /webinar-assessment
Disallow: /search*
Disallow: /academic-subscriptions
Disallow: /alumni-subscriptions
Disallow: /*?*language=es
Disallow: /*?*language=pt
Allow: /resources/html/infographics/2018/ceo-activism-desktop/index.html
Disallow: /resources/html/error/404.html
Disallow: /sponsored/2022/10/disruptors-who-are-changing-their-industries
Disallow: /api/recaptcha/enabled
Disallow: /register?*
User-Agent: 008
Disallow: /
User-Agent: FeedBurner/1.0
Allow: /resources/xml/
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of hbr.org is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/hbr.org