forbes.com: what it tells AI crawlers

On 27 October 2026, forbes.com's robots.txt fully blocked GPTBot, ClaudeBot, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.forbes.com/robots.txt
unchanged since 2026-10-26 · sha256 29e98ab89e88236d… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 650c54f6dd01eaf1…

User-agent: *
Allow: /ajax/follow_info*
Allow: /ajax/getUniregUserData*

Disallow: /following/
Disallow: /search/
Disallow: /follow/
Disallow: /ajax/
Disallow: /author/following/
Disallow: /blog/following/
Disallow: /json/
Disallow: /header/channels/
Disallow: /find-more/
Disallow: /preview/
Disallow: *wp-admin*
Disallow: /typeahead/
Disallow: /pepe/
Disallow: /media-manager/
User-agent: googlebot-news
Disallow: /sites/adobe/
Disallow: /sites/arizonacommerce/
Disallow: /sites/bmoharrisbank/
Disallow: /sites/businessaviation/
Disallow: /sites/capitalonespark/
Disallow: /sites/cdw/
Disallow: /sites/centurylink/
Disallow: /sites/dell/
Disallow: /sites/emc/
Disallow: /sites/ey/
Disallow: /sites/fedex/
Disallow: /sites/fidelity/
Disallow: /sites/gyro/
Disallow: /sites/hsbc/
Disallow: /sites/houlihanlawrence/
Disallow: /sites/hiscox/
Disallow: /sites/ibm/
Disallow: /sites/infosys/
Disallow: /sites/mdanderson/
Disallow: /sites/medidata/
Disallow: /sites/microsoft/
Disallow: /sites/microsoftdynamics/
Disallow: /sites/netapp/
Disallow: /sites/northwesternmutual/
Disallow: /sites/oppenheimerfunds/
Disallow: /sites/oracle/
Disallow: /sites/pace/
Disallow: /sites/raymondjames/
Disallow: /sites/sage/
Disallow: /sites/samsung/
Disallow: /sites/sap/
Disallow: /sites/sungardas/
Disallow: /sites/taboola/
Disallow: /sites/ups/
Disallow: /sites/delta/
Disallow: /sites/airbus/
Disallow: /sites/symantec/
Disallow: /sites/paypal/
Disallow: /sites/olddominion/
Disallow: /sites/salesforce/
Disallow: /sites/gapinternational/
Disallow: /sites/sprintbusiness/
Disallow: /sites/transamerica/
Disallow: /sites/constantcontact/
Disallow: /sites/robertocoin/
Disallow: /sites/canon/
Disallow: /sites/samsungbusiness/
Disallow: /sites/att/
Disallow: /sites/lasvegas/
Disallow: /sites/iowa/
Disallow: /sites/teradata/
Disallow: /sites/thehartford/
Disallow: /sites/statoil/
Disallow: /sites/cit/
Disallow: /sites/deloitte/
Disallow: /sites/huawei/
Disallow: /sites/relateiq/
Disallow: /sites/cftc/
Disallow: /sites/celebritycruises/
Disallow: /sites/universityofcalifornia/
Disallow: /sites/coxbusiness/
Disallow: /sites/pega/
Disallow: /sites/athenahealth/ 
Disallow: /sites/etrade/
Disallow: /sites/comcastbusiness/
Disallow: /sites/optum/
Disallow: /sites/seagate/
Disallow: /sites/rbcwealthmanagement/
Disallow: /sites/avigilon/
Disallow: /sites/citi/
Disallow: /sites/castlight/
Disallow: /sites/hyatt/
Disallow: /sites/johnhancock/
Disallow: /sites/koreanair/
Disallow: /sites/merrilllynch/
Disallow: /sites/ptc/
Disallow: /sites/toyota/
Disallow: /sites/unify/
Disallow: /sites/xerox/
Disallow: /sites/united/
Disallow: /sites/zenithwatches/
Disallow: /sites/jpmorganchase/
Disallow: /sites/ramcommercial/
Disallow: /sites/puremichigan/
Disallow: /sites/tmfgroup/
Disallow: /sites/usaa/
Disallow: /sites/emerson/
Disallow: /sites/aberdeen/
Disallow: /sites/kellogg/
Disallow: /sites/zurich/
Disallow: /sites/freeenterprise/
Disallow: /sites/allclearid/
Disallow: /sites/accenture/
Disallow: /sites/mercedesbenz/
Disallow: /sites/nice/
Disallow: /sites/riverbed/
Disallow: /sites/pimco/
Disallow: /sites/netjets/
Disallow: /sites/vmware/
Disallow: /sites/visa/
Disallow: /sites/statefarm/
Disallow: /sites/philips/
Disallow: /sites/adp/
Disallow: /sites/zenithwatches/
Disallow: /sites/citrix/ 
Disallow: /sites/gwcic/
Disallow: /sites/cargill/
Disallow: /sites/bankofsingapore/
Disallow: /sites/dbsbank/
Disallow: /sites/gradsoflife/
Disallow: /sites/internationaliotcouncil/
Disallow: /pega/customers-for-life
Disallow: /celebritycruises/guide-to-the-globe
Disallow: /sites/disneyinstitute/
Disallow: /sites/concertocloud/
Disallow: /sites/capitalonefutureedge/
Disallow: /sites/mahindracomviva/
Disallow: /sites/workday/
Disallow: /sites/colorado/
Disallow: /sites/charlesschwab/
Disallow: /sites/chicagobooth/
Disallow: /sites/morganstanley/
Disallow: /sites/opentext/
Disallow: /sites/icims/
Disallow: /sites/thriventmutualfunds/
Disallow: /sites/cbre/
Disallow: /sites/universityofphoenix/#57934e047260
Disallow: /sites/ppg/
Disallow: /sites/americanfunds/
Disallow: /sites/lilly/
Disallow: /sites/norton/
Disallow: /sites/qualityassurance/
Disallow: /sites/qlik/
Disallow: /whats-next/planes-trains-and-analytics
Disallow: /top-50-wealth-managers
Disallow: /toyota/fueled-by-skepticism
Disallow: /toyota/coming-soon
Disallow: /dell/cloud-future-of-medicine
Disallow: /cit/renewable-energy
Disallow: /retail
Disallow: /mobile/cities-transformed 
Disallow: /cit/small-business-mergers-2 
Disallow: /cit/small-business-mergers 
Disallow: /sites/kpmg/
Disallow: /dell/cities-transformed
Disallow: /sites/neustar
Disallow: /sites/ubs
Disallow: /sites/northerntrust
Disallow: /sites/infiniti 
Disallow: /sites/honeywell
Disallow: /sites/charlesschwabadvisor 
Disallow: /sites/frisco 
Disallow: /sites/colehaan 
Disallow: /sites/gapincthiswayahead   
Disallow: /sites/discoverpersonalloans   
Disallow: /sites/delltechnologies
Disallow: /colehaan/daring-to-step-forward
Disallow: /johnhancock/
Disallow: /johnhancock/in-a-lifetime-healthcare
Disallow: /oppenheimerfunds/the-upside

Sitemap: http://www.forbes.com/news_sitemap.xml
Sitemap: http://www.forbes.com/video_sitemap.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of forbes.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/forbes.com