businessinsider.com: what it tells AI crawlers

On 27 October 2026, businessinsider.com's robots.txt fully blocked ClaudeBot and CCBot; it also published llms.txt.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.businessinsider.com/robots.txt
unchanged since 2026-10-26 · sha256 89f51cf7e3b532fe… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

llms.txt: "Business Insider", 1,066 bytes, 9 links.

robots.txt as recorded sha256 89f51cf7e3b532fe…

User-agent: *
Disallow: /*?utm_campaign=Monitor&
Disallow: /adframe
Disallow: /afp$
Disallow: /ajax/
Disallow: /answers$
Disallow: /archives
Disallow: /associated-press$
Disallow: /authentication$
Disallow: /author/*/date
Disallow: /author/*/mostread
Disallow: /categories
Disallow: /cms/
Disallow: /comments$
Disallow: /cross-domain$
Disallow: /document/
Disallow: /insider$
Disallow: /partner/
Disallow: /reuters$
Disallow: /reviews/out?
Disallow: /guides/s?
Disallow: /guides/search?
Disallow: /personal-finance/s?
Disallow: /track.gif
Disallow: /uk$
Disallow: /ws/
Disallow: /business-insider$
Disallow: /*/contributor
Disallow: /bi$
Disallow: /news-insider$
Disallow: /*/rss
Disallow: /*.rss

User-agent: AmazonAdBot
Disallow: /adframe
Disallow: /afp$
Disallow: /ajax/
Disallow: /answers$
Disallow: /archives
Disallow: /associated-press$
Disallow: /authentication$
Disallow: /author/*/date
Disallow: /author/*/mostread
Disallow: /categories
Disallow: /cms/
Disallow: /comments$
Disallow: /cross-domain$
Disallow: /document/
Disallow: /insider$
Disallow: /partner/
Disallow: /reuters$
Disallow: /reviews/out?
Disallow: /guides/s?
Disallow: /guides/search?
Disallow: /personal-finance/s?
Disallow: /track.gif
Disallow: /uk$
Disallow: /ws/
Disallow: /business-insider$
Disallow: /*/contributor
Disallow: /bi$
Disallow: /news-insider$
Disallow: /*/rss
Disallow: /*.rss
Allow: /

User-agent: anthropic-ai
Disallow: /

User-agent: AwarioSmartBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: newspaper
Disallow: /

User-agent: peer39_crawler
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: Timpibot
Disallow: /

Sitemap: https://www.businessinsider.com/sitemap/latest.xml
Sitemap: https://www.businessinsider.com/sitemap/google-news.xml
Sitemap: https://www.businessinsider.com/sitemap/index.xml

Sitemap: https://www.businessinsider.com/guides/sitemap/latest.xml
Sitemap: https://www.businessinsider.com/guides/sitemap/google-news.xml
Sitemap: https://www.businessinsider.com/guides/sitemap/index.xml

Sitemap: https://www.businessinsider.com/sitemap/landing-pages.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of businessinsider.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/businessinsider.com