investopedia.com: what it tells AI crawlers

On 27 October 2026, investopedia.com's robots.txt fully blocked ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 10k (Tranco)

robots.txt

ok (HTTP 200) · https://www.investopedia.com/robots.txt
unchanged since 2026-10-26 · sha256 cd45cc09cf213678… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 cd45cc09cf213678…

#People Inc. content is made available for your non-commercial use subject to our
#Terms of Use here: https://www.people.inc/brands-termsofservice.
#Use of any crawler or other tool, device, or process to data mine or scrape the
#content on this website using automated means for any purpose other than
#directing traffic to this website or serving authorized advertisements on this
#website is prohibited without prior written permission from People Inc.
#Prohibited uses include but are not limited to:
#(1) text and data mining activities under Art. 4 of the EU Directive on
#Copyright in the Digital Single Market;
#(2) development or operation of any artificial intelligence, machine learning,
#or large language model (LLM) technology, including by training or fine-tuning
#such technology or using it for retrieval-augmented generation; and
#(3) creating data sets containing People Inc. content or sharing it with others.
#For the avoidance of doubt, the fact that any such tool, device, or process is
#not blocked via this robots.txt file does not constitute a waiver of any of
#People Inc.'s rights under our Terms of Use or relevant law.
#Contact People Inc - Content Licensing contentlicensing@people.inc for assistance.

User-agent: *
Disallow: /embed?
Disallow: /cdn-cgi/

User-agent: Pinterest
User-agent: Pinterestbot
Disallow:

User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: GPTBot
Disallow: /thmb/

User-agent: Google-Extended
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Claude-SearchBot
User-agent: cohere-ai
User-agent: Meta-ExternalAgent
User-agent: meta-webindexer
User-agent: PerplexityBot
Disallow: /
Allow: */presented/
Allow: */integrated/

User-agent: AmazonAdBot
Allow: /

User-agent: amazon-kendra
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: AmazonBuyForMe
User-agent: Applebot-Extended
User-agent: archive.org_bot
User-agent: Arquivo-web-crawler
User-agent: AwarioRssBot
User-agent: AwarioSmartBot
User-agent: Baiduspider
User-agent: Baiduspider-ads
User-agent: Baiduspider-cpro
User-agent: Baiduspider-favo
User-agent: Baiduspider-image
User-agent: Baiduspider-news
User-agent: Baiduspider-video
User-agent: Bravebot
User-agent: Brightbot 1.0
User-agent: Bytespider
User-agent: Claude-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: cohere-training-data-crawler
User-agent: Diffbot-User
User-agent: Diffbot
User-agent: DuckAssistBot
User-agent: ERNIEBot
User-agent: ExaSearchBot
User-agent: FacebookBot
User-agent: FirecrawlAgent
User-agent: FriendlyCrawler
User-agent: Google-CloudVertexBot
User-agent: heritrix
User-agent: iAskBot
User-agent: iaskspider
User-agent: iaskspider/2.0
User-agent: ImagesiftBot
User-agent: imageSpider
User-agent: ISSCyberRiskCrawler
User-agent: Kagibot
User-agent: LinerBot
User-agent: LinkUpBot
User-agent: Mastodon
User-agent: Meltwaterimageservice
User-agent: MeltwaterNews
User-agent: Meta-ExternalFetcher
User-agent: MistralAI-Index
User-agent: MistralAI-user
User-agent: MistralAI-User/1.0
User-agent: Neevabot
User-agent: news-please
User-agent: omgili
User-agent: omgilibot
User-agent: Perplexity-User
User-agent: PetalBot
User-agent: Quora-Bot
User-agent: Shap-User
User-agent: ShapBot
User-agent: TaraGroup Intelligent Bot
User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: VettedAiBot
User-agent: Webzio
User-agent: Webzio-Extended
User-agent: Yisouspider
User-agent: YouBot
Disallow: /

Sitemap: https://www.investopedia.com/sitemap.xml
Sitemap: https://www.investopedia.com/google-news-sitemap.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of investopedia.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/investopedia.com