forbes.com: what it tells AI crawlers
On 27 October 2026, forbes.com's robots.txt fully blocked GPTBot, ClaudeBot, PerplexityBot and CCBot.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://www.forbes.com/robots.txt
unchanged since 2026-10-26 · sha25629e98ab89e88236d…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-27
| User agent | Verdict | How |
|---|---|---|
| GPTBot | blocks | named in its own group |
| OAI-SearchBot | allows | named in its own group |
| ChatGPT-User | allows | through the * group |
| ClaudeBot | blocks | named in its own group |
| Claude-SearchBot | allows | named in its own group |
| Claude-User | allows | named in its own group |
| PerplexityBot | blocks | named in its own group |
| Perplexity-User | allows | through the * group |
| Googlebot | allows | through the * group |
| Google-Extended | allows | through the * group |
| Applebot-Extended | blocks | named in its own group |
| CCBot | blocks | named in its own group |
| Bytespider | blocks | named in its own group |
| Meta-ExternalAgent | blocks | named in its own group |
| Amazonbot | blocks | named in its own group |
| anthropic-ai | blocks | named in its own group |
| cohere-ai | blocks | named in its own group |
| Diffbot | blocks | named in its own group |
| FacebookBot | blocks | named in its own group |
| FriendlyCrawler | blocks | named in its own group |
| ImagesiftBot | blocks | named in its own group |
| omgili | blocks | named in its own group |
| omgilibot | blocks | named in its own group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | not observed |
| Declares Content Signals (robots.txt or header) | not observed |
| Says ai-train=no in a Content Signal | not observed |
| Points to an RSL licence (robots.txt License line or Link header) | not observed |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | not observed |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | not observed |
| Sends noai / noimageai directives | not observed |
| Refused our fetcher (403 / challenge) | no |
robots.txt as recorded sha256 29e98ab89e88236d…
User-agent: * Allow: /ajax/follow_info* Allow: /ajax/getUniregUserData* Disallow: /following/ Disallow: /search/ Disallow: /follow/ Disallow: /ajax/ Disallow: /author/following/ Disallow: /blog/following/ Disallow: /json/ Disallow: /header/channels/ Disallow: /find-more/ Disallow: /preview/ Disallow: *wp-admin* Disallow: /typeahead/ Disallow: /pepe/ Disallow: /media-manager/ Disallow: /coupons/visit/* Disallow: /test/ Disallow: /advisor/*zx1fa* Disallow: /advisor/journey/ Disallow: /wui/ Disallow: /research-survey User-agent: googlebot-news Disallow: /sites/adobe/ Disallow: /sites/arizonacommerce/ Disallow: /sites/bmoharrisbank/ Disallow: /sites/businessaviation/ Disallow: /sites/capitalonespark/ Disallow: /sites/cdw/ Disallow: /sites/centurylink/ Disallow: /sites/dell/ Disallow: /sites/emc/ Disallow: /sites/ey/ Disallow: /sites/fedex/ Disallow: /sites/fidelity/ Disallow: /sites/gyro/ Disallow: /sites/hsbc/ Disallow: /sites/houlihanlawrence/ Disallow: /sites/hiscox/ Disallow: /sites/ibm/ Disallow: /sites/infosys/ Disallow: /sites/mdanderson/ Disallow: /sites/medidata/ Disallow: /sites/microsoft/ Disallow: /sites/microsoftdynamics/ Disallow: /sites/netapp/ Disallow: /sites/northwesternmutual/ Disallow: /sites/oppenheimerfunds/ Disallow: /sites/oracle/ Disallow: /sites/pace/ Disallow: /sites/raymondjames/ Disallow: /sites/sage/ Disallow: /sites/samsung/ Disallow: /sites/sap/ Disallow: /sites/sungardas/ Disallow: /sites/taboola/ Disallow: /sites/ups/ Disallow: /sites/delta/ Disallow: /sites/airbus/ Disallow: /sites/symantec/ Disallow: /sites/paypal/ Disallow: /sites/olddominion/ Disallow: /sites/salesforce/ Disallow: /sites/gapinternational/ Disallow: /sites/sprintbusiness/ Disallow: /sites/transamerica/ Disallow: /sites/constantcontact/ Disallow: /sites/robertocoin/ Disallow: /sites/canon/ Disallow: /sites/samsungbusiness/ Disallow: /sites/att/ Disallow: /sites/lasvegas/ Disallow: /sites/iowa/ Disallow: /sites/teradata/ Disallow: /sites/thehartford/ Disallow: /sites/statoil/ Disallow: /sites/cit/ Disallow: /sites/deloitte/ Disallow: /sites/huawei/ Disallow: /sites/relateiq/ Disallow: /sites/cftc/ Disallow: /sites/celebritycruises/ Disallow: /sites/universityofcalifornia/ Disallow: /sites/coxbusiness/ Disallow: /sites/pega/ Disallow: /sites/athenahealth/ Disallow: /sites/etrade/ Disallow: /sites/comcastbusiness/ Disallow: /sites/optum/ Disallow: /sites/seagate/ Disallow: /sites/rbcwealthmanagement/ Disallow: /sites/avigilon/ Disallow: /sites/citi/ Disallow: /sites/castlight/ Disallow: /sites/hyatt/ Disallow: /sites/johnhancock/ Disallow: /sites/koreanair/ Disallow: /sites/merrilllynch/ Disallow: /sites/ptc/ Disallow: /sites/toyota/ Disallow: /sites/unify/ Disallow: /sites/xerox/ Disallow: /sites/united/ Disallow: /sites/zenithwatches/ Disallow: /sites/jpmorganchase/ Disallow: /sites/ramcommercial/ Disallow: /sites/puremichigan/ Disallow: /sites/tmfgroup/ Disallow: /sites/usaa/ Disallow: /sites/emerson/ Disallow: /sites/aberdeen/ Disallow: /sites/kellogg/ Disallow: /sites/zurich/ Disallow: /sites/freeenterprise/ Disallow: /sites/allclearid/ Disallow: /sites/accenture/ Disallow: /sites/mercedesbenz/ Disallow: /sites/nice/ Disallow: /sites/riverbed/ Disallow: /sites/pimco/ Disallow: /sites/netjets/ Disallow: /sites/vmware/ Disallow: /sites/visa/ Disallow: /sites/statefarm/ Disallow: /sites/philips/ Disallow: /sites/adp/ Disallow: /sites/zenithwatches/ Disallow: /sites/citrix/ Disallow: /sites/gwcic/ Disallow: /sites/cargill/ Disallow: /sites/bankofsingapore/ Disallow: /sites/dbsbank/ Disallow: /sites/gradsoflife/ Disallow: /sites/internationaliotcouncil/ Disallow: /pega/customers-for-life Disallow: /celebritycruises/guide-to-the-globe Disallow: /sites/disneyinstitute/ Disallow: /sites/concertocloud/ Disallow: /sites/capitalonefutureedge/ Disallow: /sites/mahindracomviva/ Disallow: /sites/workday/ Disallow: /sites/colorado/ Disallow: /sites/charlesschwab/ Disallow: /sites/chicagobooth/ Disallow: /sites/morganstanley/ Disallow: /sites/opentext/ Disallow: /sites/icims/ Disallow: /sites/thriventmutualfunds/ Disallow: /sites/cbre/ Disallow: /sites/universityofphoenix/#57934e047260 Disallow: /sites/ppg/ Disallow: /sites/americanfunds/ Disallow: /sites/lilly/ Disallow: /sites/norton/ Disallow: /sites/qualityassurance/ Disallow: /sites/qlik/ Disallow: /whats-next/planes-trains-and-analytics Disallow: /top-50-wealth-managers Disallow: /toyota/fueled-by-skepticism Disallow: /toyota/coming-soon Disallow: /dell/cloud-future-of-medicine Disallow: /cit/renewable-energy Disallow: /retail Disallow: /mobile/cities-transformed Disallow: /cit/small-business-mergers-2 Disallow: /cit/small-business-mergers Disallow: /sites/kpmg/ Disallow: /dell/cities-transformed Disallow: /sites/neustar Disallow: /sites/ubs Disallow: /sites/northerntrust Disallow: /sites/infiniti Disallow: /sites/honeywell Disallow: /sites/charlesschwabadvisor Disallow: /sites/frisco Disallow: /sites/colehaan Disallow: /sites/gapincthiswayahead Disallow: /sites/discoverpersonalloans Disallow: /sites/delltechnologies Disallow: /colehaan/daring-to-step-forward Disallow: /johnhancock/ Disallow: /johnhancock/in-a-lifetime-healthcare Disallow: /johnhancock/in-a-lifetime-work Disallow: /oppenheimerfunds/the-upside Disallow: /sites/hpeenterprisenxt Disallow: /sites/rackspace Disallow: /sites/themacallan Disallow: /themacallan/worlds-together Disallow: /sites/inteliotpartnership Disallow: /oppenheimerfunds/the-upside Disallow: /partners/ Disallow: /johnhancock/in-a-lifetime-sustainability Disallow: /sites/domperignon Disallow: /sites/shinola Disallow: /sites/hpinc Disallow: /sites/hp Disallow: /sites/alfresco Disallow: /cbre/built-for-performance Disallow: /sites/mmedric Disallow: /sites/indiana Disallow: /sites/siemenshealthineers Disallow: /sites/rackspace Disallow: /rackspace/elevating-performance Disallow: /sites/suntrust Disallow: /sites/unicefusa Disallow: /sites/mitsubishiheavyindustries Disallow: /best-countries-for-business/list Disallow: /sites/paloaltonetworks Disallow: /sites/f5 Disallow: /sites/samsungtv Disallow: /sites/qa Disallow: /sites/paycom Disallow: /sites/testbrandvoice Disallow: /business/blahblah Disallow: /sites/spdr Disallow: /sites/lincoln Disallow: /sites/abb Disallow: /sites/eycybersecurity Disallow: /sites/civicnation Disallow: /sites/bcg Disallow: /sites/vanguard Disallow: /sites/akamai Disallow: /sites/kyocera Disallow: /sites/socialimpact Disallow: /sites/emirates Disallow: /sites/hpincemea Disallow: /sites/ibmperformancecomputing Disallow: /sites/pfizer Disallow: /sites/ibmpowersystems Disallow: /sites/vanguardcharitable Disallow: /sites/suntrustprivatewealth Disallow: /sites/capitalone Disallow: /sites/juniper Disallow: /sites/wellsfargo Disallow: /sites/bnymellon Disallow: /sites/braintree Disallow: /sites/bnymellonwealthmanagement Disallow: /sites/epson Disallow: /sites/thomsonreuters Disallow: /sites/paysafe Disallow: /sites/josabank Disallow: /sites/hennessyparadisimperial Disallow: /sites/cartier Disallow: /sites/anabluewing Disallow: /sites/eidoo Disallow: /sites/cmstesting Disallow: /sites/adobepartnership Disallow: /sites/cornelljohnson Disallow: /sites/quantcast Disallow: /sites/japan Disallow: /sites/impactpartners Disallow: /sites/nvidia Disallow: /sites/personalcapital Disallow: /sites/microsoft365 Disallow: /sites/providencedigitalinnovation Disallow: /sites/whittiertrust Disallow: /sites/kinetica Disallow: /sites/paulevans Disallow: /sites/ritz-carltonleadershipcenter Disallow: /sites/icoengine Disallow: /sites/tesaro Disallow: /sites/theglenlivet Disallow: /feature/sf-test Disallow: /sites/courtyardmarriott Disallow: /sites/nuveen Disallow: /sites/intelai Disallow: /sites/secret Disallow: /sites/hec Disallow: /retail/old Disallow: /sites/alliantgroup Disallow: /sites/intuitivesurgical Disallow: /sites/sunovion Disallow: /sites/ofx Disallow: /sites/firstrepublicbank Disallow: /sites/1850byfolgers Disallow: /sites/audi Disallow: /sites/nuveencef Disallow: /sites/1850brandcoffee Disallow: /sites/cisco Disallow: /sites/blackstarcapitalltd Disallow: /sites/usbankaltitudereserve Disallow: /sites/dassaultsystemes Disallow: /sites/urogen Disallow: /sites/smartsheet Disallow: /sites/iwc Disallow: /sites/datingcom Disallow: /sites/intuitquickbooks Disallow: /sites/oticon Disallow: /sites/servicenow Disallow: /sites/deloitteriskandfinancialadvisory Disallow: /sites/dbsprivatebank Disallow: /sites/americanexpress Disallow: /sites/tanotofoundation Disallow: /sites/citizensaccess Disallow: /sites/salesforceessentials Disallow: /sites/drexeluniversity Disallow: /sites/microsoftinnovation Disallow: /sites/mypos Disallow: /sites/harvardpilgrimhealthcare Disallow: /sites/appdynamics Disallow: /sites/ministryoftourismindia Disallow: /sites/orlando Disallow: /sites/extrahop Disallow: /sites/ascm Disallow: /sites/crowe Disallow: /sites/peerfit Disallow: /sites/theleukemialymphomasociety Disallow: /sites/hewlettpackardenterprise Disallow: /sites/montblanc Disallow: /sites/meero Disallow: /sites/braze Disallow: /sites/publicissapient Disallow: /sites/flexport User-agent: Amazonbot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot User-agent: CCbot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: cohere-ai User-agent: Cohere-AI Disallow: / User-agent: Diffbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: GPTBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: meta-externalagent User-agent: Meta-ExternalAgent Disallow: / User-Agent: OAI-SearchBot Allow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-Agent: PerplexityBot Disallow: / User-agent: AhrefsBot User-agent: AhrefsSiteAudit Crawl-delay: 2 User-Agent: Claude-User User-Agent: Claude-SearchBot Allow: / User-agent: AmazonAdBot Allow: /following/ Allow: /search/ Allow: /follow/ Allow: /author/following/ Allow: /blog/following/ Allow: /header/channels/ Allow: /find-more/ Allow: /preview/ Allow: /typeahead/ Allow: /pepe/ Allow: /media-manager/ Allow: /coupons/visit/* Allow: /advisor/*zx1fa* Allow: /wui/ User-agent: APSMeridianBot Disallow: /advisor Disallow: /betting/ Disallow: /councils/ Disallow: /financial-services/ Disallow: /sites/_glenfiddich/ Disallow: /sites/1850brandcoffee/ Disallow: /sites/84-lumber/ Disallow: /sites/aarp/ Disallow: /sites/ab-inbev/ Disallow: /sites/abb/ Disallow: /sites/aberdeen/ Disallow: /sites/accenture-india/ Disallow: /sites/accenture/ Disallow: /sites/acoustic/ Disallow: /sites/acronis/ Disallow: /sites/adobe/ Disallow: /sites/adobepartnership/ Disallow: /sites/adp/ Disallow: /sites/adyen/ Disallow: /sites/airbus/ Disallow: /sites/akamai/ Disallow: /sites/alfresco/ Disallow: /sites/alibaba/ Disallow: /sites/alight/ Disallow: /sites/allclearid/ Disallow: /sites/alliantgroup/ Disallow: /sites/almanac/ Disallow: /sites/alpine/ Disallow: /sites/alteryx/ Disallow: /sites/amazon-ads/ Disallow: /sites/amazon-black-business-accelerator/ Disallow: /sites/amazon-web-services-asean/ Disallow: /sites/amazonwebservices/ Disallow: /sites/amd/ Disallow: /sites/american-cancer-society/ Disallow: /sites/american-heart-association/ Disallow: /sites/american-heart-associations-leaders-of-impact/ Disallow: /sites/american-tire-distributors/ Disallow: /sites/americanexpress/ Disallow: /sites/amp-energy/ Disallow: /sites/amtd-group/ Disallow: /sites/anabluewing/ Disallow: /sites/anaplan/ Disallow: /sites/ant-international/ Disallow: /sites/aptpay/ Disallow: /sites/arizona-commerce-authority/ Disallow: /sites/arizonacommerce/ Disallow: /sites/arm/ Disallow: /sites/ascm/ Disallow: /sites/astrazeneca/ Disallow: /sites/athenahealth/ Disallow: /sites/athene/ Disallow: /sites/atlassian/ Disallow: /sites/att/ Disallow: /sites/audemars-piguet/ Disallow: /sites/audemarspiguet/ Disallow: /sites/audi/ Disallow: /sites/avanade/ Disallow: /sites/avigilon/ Disallow: /sites/aviram-family-foundation/ Disallow: /sites/aws-and-pega/ Disallow: /sites/aws-business-transformation/ Disallow: /sites/aws-enterprise-transformation/ Disallow: /sites/aws-transformation/ Disallow: /sites/awscloudmigration/ Disallow: /sites/awsinfrastructuresolutions/ Disallow: /sites/awsmediaandentertainment/ Disallow: /sites/awsstartups/ Disallow: /sites/bacardi/ Disallow: /sites/bankofsingapore/ Disallow: /sites/baume_mercier/ Disallow: /sites/bayer/ Disallow: /sites/bbtsuntrustnowtruist/ Disallow: /sites/bcbsassociation/ Disallow: /sites/bcg/ Disallow: /sites/benbridgejeweler/ Disallow: /sites/beximco-group/ Disallow: /sites/blackstarcapitalltd/ Disallow: /sites/block-advisors/ Disallow: /sites/bloodcancerunited/ Disallow: /sites/bluejeans-by-verizon/ Disallow: /sites/bluetti-power-inc/ Disallow: /sites/bmc/ Disallow: /sites/bmoharrisbank/ Disallow: /sites/bnymellonwealthmanagement/ Disallow: /sites/bold-by-veuve-clicquot/ Disallow: /sites/boston-university-questrom-school-of-business/ Disallow: /sites/bowlero/ Disallow: /sites/braintree/ Disallow: /sites/brandstorytelling/ Disallow: /sites/braze/ Disallow: /sites/bristol-myers-squibb/ Disallow: /sites/bt/ Disallow: /sites/bushmills/ Disallow: /sites/businessaviation/ Disallow: /sites/businessreporter/ Disallow: /sites/cadence/ Disallow: /sites/calm/ Disallow: /sites/cambium-learning-group/ Disallow: /sites/canon-inc/ Disallow: /sites/canon/ Disallow: /sites/canva/ Disallow: /sites/capgemini/ Disallow: /sites/capital-group-american-funds/ Disallow: /sites/capitalone/ Disallow: /sites/capitalonefutureedge/ Disallow: /sites/capitalonespark/ Disallow: /sites/carecredit/ Disallow: /sites/cargill/ Disallow: /sites/cartier/ Disallow: /sites/castlight/ Disallow: /sites/cathay-pacific/ Disallow: /sites/cbre/ Disallow: /sites/cdw/ Disallow: /sites/celebritycruises/ Disallow: /sites/celonis/ Disallow: /sites/centurylink/ Disallow: /sites/cerity-partners/ Disallow: /sites/cftc/ Disallow: /sites/chainalysis/ Disallow: /sites/charlesschwabadvisor/ Disallow: /sites/chase-ink-business-credit-cards/ Disallow: /sites/chase-ink/ Disallow: /sites/chase-sapphire/ Disallow: /sites/chasehomelending/ Disallow: /sites/checkmarx/ Disallow: /sites/chicagobooth/ Disallow: /sites/cigna/ Disallow: /sites/cirocvs/ Disallow: /sites/cisco-webex/ Disallow: /sites/cisco/ Disallow: /sites/ciscoappdynamics/ Disallow: /sites/cit/ Disallow: /sites/citi/ Disallow: /sites/citizensaccess/ Disallow: /sites/citrix/ Disallow: /sites/city-of-hope/ Disallow: /sites/civicnation/ Disallow: /sites/cle-de-peau-beaute/ Disallow: /sites/cloudera/ Disallow: /sites/cloudflare/ Disallow: /sites/cme-group/ Disallow: /sites/coalition-for-faith-and-media/ Disallow: /sites/coconut-software/ Disallow: /sites/cognizant/ Disallow: /sites/coldwellbanker/ Disallow: /sites/colehaan/ Disallow: /sites/colorado/ Disallow: /sites/comarch/ Disallow: /sites/comcastbusiness/ Disallow: /sites/commure/ Disallow: /sites/commvault/ Disallow: /sites/company-of-good/ Disallow: /sites/concertocloud/ Disallow: /sites/connection/ Disallow: /sites/connectly/ Disallow: /sites/constantcontact/ Disallow: /sites/cornelljohnson/ Disallow: /sites/courtyard/ Disallow: /sites/coxbusiness/ Disallow: /sites/crowe/ Disallow: /sites/ctia-wireless-foundation/ Disallow: /sites/dassaultsystemes/ Disallow: /sites/dataiku/ Disallow: /sites/dbs-wealth-management/ Disallow: /sites/dbsbank/ Disallow: /sites/dbsprivatebank/ Disallow: /sites/deep-instinct/ Disallow: /sites/dell-technologies/ Disallow: /sites/dell/ Disallow: /sites/delltechnologies/ Disallow: /sites/deloitte-microsoft-technology-services-practice/ Disallow: /sites/deloitte/ Disallow: /sites/deloitteriskandfinancialadvisory/ Disallow: /sites/delta/ Disallow: /sites/dhl-supply-chain/ Disallow: /sites/diageo-rare--exceptional/ Disallow: /sites/dimopoulos-injury-law/ Disallow: /sites/diners-club-international/ Disallow: /sites/discoverpersonalloans/ Disallow: /sites/disneyinstitute/ Disallow: /sites/dockers/ Disallow: /sites/doctors-without-borders/ Disallow: /sites/docusign/ Disallow: /sites/dollar-general/ Disallow: /sites/dominican-republic-ministry-of-tourism/ Disallow: /sites/domperignon/ Disallow: /sites/dpworld/ Disallow: /sites/drexeluniversity/ Disallow: /sites/dubai-tourism/ Disallow: /sites/dxc-technology/ Disallow: /sites/ebay/ Disallow: /sites/ecolab/ Disallow: /sites/ehealth/ Disallow: /sites/eidoo/ Disallow: /sites/elevance-health/ Disallow: /sites/emc/ Disallow: /sites/emirates/ Disallow: /sites/empower/ Disallow: /sites/energy-asia/ Disallow: /sites/epos/ Disallow: /sites/epson/ Disallow: /sites/epsoncorporate/ Disallow: /sites/eset/ Disallow: /sites/etrade/ Disallow: /sites/everbridge/ Disallow: /sites/exl/ Disallow: /sites/exness/ Disallow: /sites/extrahop/ Disallow: /sites/ey-global-government--infrastructure/ Disallow: /sites/ey/ Disallow: /sites/eycybersecurity/ Disallow: /sites/f45training/ Disallow: /sites/f5/ Disallow: /sites/facebook/ Disallow: /sites/familyofficehk/ Disallow: /sites/fannie-mae/ Disallow: /sites/fedex-express/ Disallow: /sites/fedex/ Disallow: /sites/ferrari-roma/ Disallow: /sites/fidelity/ Disallow: /sites/first-horizon-bank/ Disallow: /sites/first-tee/ Disallow: /sites/firstrepublicbank/ Disallow: /sites/fis-global/ Disallow: /sites/flexport/ Disallow: /sites/forbeseq/ Disallow: /sites/forcepoint/ Disallow: /sites/fortinet/ Disallow: /sites/freeenterprise/ Disallow: /sites/frisco/ Disallow: /sites/frog/ Disallow: /sites/fujitsu-limited/ Disallow: /sites/funnyfuzzy/ Disallow: /sites/gainbridge/ Disallow: /sites/gamuda-berhad/ Disallow: /sites/ganz/ Disallow: /sites/gapincthiswayahead/ Disallow: /sites/gapinternational/ Disallow: /sites/genentech/ Disallow: /sites/genesyscontactcentercloud/ Disallow: /sites/gerber-life-insurance/ Disallow: /sites/gfk/ Disallow: /sites/gigabyte/ Disallow: /sites/gladly/ Disallow: /sites/glenfiddich/ Disallow: /sites/globalizationpartners/ Disallow: /sites/goodrx/ Disallow: /sites/google-ads/ Disallow: /sites/google-maps/ Disallow: /sites/googlecloud/ Disallow: /sites/googlecloudaisolutions/ Disallow: /sites/gradsoflife/ Disallow: /sites/gran-coramino/ Disallow: /sites/guild/ Disallow: /sites/guocoland/ Disallow: /sites/gwcic/ Disallow: /sites/gyro/ Disallow: /sites/hanwha-brandvoice/ Disallow: /sites/harman/ Disallow: /sites/harvard-division-of-continuing-education/ Disallow: /sites/harvardpilgrimhealthcare/ Disallow: /sites/havas-media-network/ Disallow: /sites/hec/ Disallow: /sites/henley--partners/ Disallow: /sites/hennessyparadisimperial/ Disallow: /sites/hewlettpackardenterprise/ Disallow: /sites/hiscox/ Disallow: /sites/hitachi-vantara-perspectives/ Disallow: /sites/hitachienergy/ Disallow: /sites/hola-prime/ Disallow: /sites/hondajet/ Disallow: /sites/honeywell/ Disallow: /sites/hong-kong-talent-engage/ Disallow: /sites/hongkong/ Disallow: /sites/houlihanlawrence/ Disallow: /sites/hp-inc-and-wwf/ Disallow: /sites/hp/ Disallow: /sites/hpeenterprisenxt/ Disallow: /sites/hpincemea/ Disallow: /sites/hsbc/ Disallow: /sites/huawei/ Disallow: /sites/hubinternational/ Disallow: /sites/hyatt/ Disallow: /sites/ibm/ Disallow: /sites/ibmai/ Disallow: /sites/ibmpowersystems/ Disallow: /sites/ibmsecurity/ Disallow: /sites/icims/ Disallow: /sites/icoengine/ Disallow: /sites/iedcagtechplus/ Disallow: /sites/ifast-global-bank/ Disallow: /sites/iintoo/ Disallow: /sites/impactpartners/ Disallow: /sites/indiana/ Disallow: /sites/infiniti/ Disallow: /sites/infocomm-media-development-authority/ Disallow: /sites/informatica/ Disallow: /sites/infosys/ Disallow: /sites/insights-acxiom/ Disallow: /sites/insights-adobe-microsoft/ Disallow: /sites/insights-adobe/ Disallow: /sites/insights-amazon-web-services-public-sector/ Disallow: /sites/insights-amazon-web-services/ Disallow: /sites/insights-american-express/ Disallow: /sites/insights-arculus-by-composecure/ Disallow: /sites/insights-asqexcellence/ Disallow: /sites/insights-bcg/ Disallow: /sites/insights-bmc/ Disallow: /sites/insights-capitalone/ Disallow: /sites/insights-cisco/ Disallow: /sites/insights-cliffordchance/ Disallow: /sites/insights-cloudera/ Disallow: /sites/insights-delltechnologies/ Disallow: /sites/insights-docusign/ Disallow: /sites/insights-ecolab/ Disallow: /sites/insights-engine-by-gen/ Disallow: /sites/insights-ey/ Disallow: /sites/insights-fortinet/ Disallow: /sites/insights-glassbox/ Disallow: /sites/insights-greenhousecom/ Disallow: /sites/insights-hcl-software/ Disallow: /sites/insights-hitachi/ Disallow: /sites/insights-hsbc/ Disallow: /sites/insights-ibmai/ Disallow: /sites/insights-ibmresiliency/ Disallow: /sites/insights-ibmservices/ Disallow: /sites/insights-idex/ Disallow: /sites/insights-intel/ Disallow: /sites/insights-intelai/ Disallow: /sites/insights-inteldatacenter/ Disallow: /sites/insights-inteliot/ Disallow: /sites/insights-jp-morgan-payments-commerce-solutions/ Disallow: /sites/insights-jpmorgan/ Disallow: /sites/insights-klgates/ Disallow: /sites/insights-kpmg/ Disallow: /sites/insights-kyriba/ Disallow: /sites/insights-michiganeconomicdevelopmentcorporation/ Disallow: /sites/insights-microsoft/ Disallow: /sites/insights-microsoftai/ Disallow: /sites/insights-mmgnet/ Disallow: /sites/insights-northerntrust/ Disallow: /sites/insights-northwestern-mutual/ Disallow: /sites/insights-nuveen/ Disallow: /sites/insights-ofx/ Disallow: /sites/insights-pega/ Disallow: /sites/insights-penske/ Disallow: /sites/insights-pmi/ Disallow: /sites/insights-roostify/ Disallow: /sites/insights-sailpoint/ Disallow: /sites/insights-sap-concur/ Disallow: /sites/insights-sap/ Disallow: /sites/insights-scrumalliance/ Disallow: /sites/insights-sentry-insurance/ Disallow: /sites/insights-silkroad/ Disallow: /sites/insights-smartsheet/ Disallow: /sites/insights-sovos/ Disallow: /sites/insights-synchrony-bank/ Disallow: /sites/insights-teamviewer/ Disallow: /sites/insights-temenos/ Disallow: /sites/insights-teradata/ Disallow: /sites/insights-thomsonreuters/ Disallow: /sites/insights-thoughtworks/ Disallow: /sites/insights-treasuredata/ Disallow: /sites/insights-vertiv/ Disallow: /sites/insights-vmwaresecurity/ Disallow: /sites/insights-whittiertrust/ Disallow: /sites/insights-zoom/ Disallow: /sites/intact-insurance-specialty-solutions/ Disallow: /sites/integra-connect/ Disallow: /sites/intel/ Disallow: /sites/intelai/ Disallow: /sites/inteliot/ Disallow: /sites/inteliotpartnership/ Disallow: /sites/intuit-expert/ Disallow: /sites/intuitivesurgical/ Disallow: /sites/intuitquickbooks/ Disallow: /sites/invesco-qqq/ Disallow: /sites/invest-puerto-rico/ Disallow: /sites/ioi-properties/ Disallow: /sites/iowa/ Disallow: /sites/iris-powered-by-generali/ Disallow: /sites/ishares/ Disallow: /sites/itrustcapital/ Disallow: /sites/iwc/ Disallow: /sites/japan/ Disallow: /sites/japanesevisionarywomen/ Disallow: /sites/jbs-usa/ Disallow: /sites/johnhancock/ Disallow: /sites/johnson--johnson/ Disallow: /sites/josabank/ Disallow: /sites/jp-morgan-payments/ Disallow: /sites/jpmorgan-chase-women-on-the-move/ Disallow: /sites/jpmorganchase/ Disallow: /sites/jumio/ Disallow: /sites/jumpcloud/ Disallow: /sites/juniper/ Disallow: /sites/jurin-ai/ Disallow: /sites/kaiserpermanente/ Disallow: /sites/kantar/ Disallow: /sites/kao/ Disallow: /sites/kellogg/ Disallow: /sites/kenvue/ Disallow: /sites/keybanc-capital-markets/ Disallow: /sites/kinaxis/ Disallow: /sites/kinetica/ Disallow: /sites/kpmg-international/ Disallow: /sites/kpmg/ Disallow: /sites/ksac/ Disallow: /sites/kyndryl/ Disallow: /sites/kyocera/ Disallow: /sites/lasvegas/ Disallow: /sites/lenovo-nvidia/ Disallow: /sites/lexus/ Disallow: /sites/lg/ Disallow: /sites/lgt-private-banking-asia-pacific/ Disallow: /sites/lincoln/ Disallow: /sites/linkedin-apac/ Disallow: /sites/logitech/ Disallow: /sites/lopesan-costa-bavaro-resort-spa--casino/ Disallow: /sites/loreal/ Disallow: /sites/lowes-creator/ Disallow: /sites/lumen/ Disallow: /sites/magnit/ Disallow: /sites/mahindra-group/ Disallow: /sites/mahindracomviva/ Disallow: /sites/malaysia-digital-economy-corporation/ Disallow: /sites/manhattan-associates/ Disallow: /sites/manulife-singapore/ Disallow: /sites/manulife/ Disallow: /sites/marcus-by-goldman-sachs/ Disallow: /sites/mars/ Disallow: /sites/maserati/ Disallow: /sites/mastercard/ Disallow: /sites/maxartechnologies/ Disallow: /sites/mcdonalds/ Disallow: /sites/medallia/ Disallow: /sites/medidata/ Disallow: /sites/meero/ Disallow: /sites/meltwater/ Disallow: /sites/mercedesbenz/ Disallow: /sites/merrilllynch/ Disallow: /sites/meta/ Disallow: /sites/metronome/ Disallow: /sites/micron-technology/ Disallow: /sites/microsoft_/ Disallow: /sites/microsoft365/ Disallow: /sites/microsoftdynamics/ Disallow: /sites/midfirst-bank/ Disallow: /sites/migombank/ Disallow: /sites/ministryoftourismindia/ Disallow: /sites/mitsubishi-electric/ Disallow: /sites/mitsubishiheavyindustries/ Disallow: /sites/monogram/ Disallow: /sites/montblanc/ Disallow: /sites/moveworks/ Disallow: /sites/mr-sancho-pancho/ Disallow: /sites/mural/ Disallow: /sites/mypos/ Disallow: /sites/nacc-thailand/ Disallow: /sites/netapp/ Disallow: /sites/netjets/ Disallow: /sites/neustar/ Disallow: /sites/new-cies/ Disallow: /sites/newyorklifeinvestments/ Disallow: /sites/ni/ Disallow: /sites/nice/ Disallow: /sites/nokia-industry-40/ Disallow: /sites/nomos-glashutte/ Disallow: /sites/none/ Disallow: /sites/northerntrust/ Disallow: /sites/northwesternmutual/ Disallow: /sites/novavax/ Disallow: /sites/ntt-data/ Disallow: /sites/nuskin/ Disallow: /sites/nutanix/ Disallow: /sites/nutrien/ Disallow: /sites/nutro/ Disallow: /sites/nuveen/ Disallow: /sites/nvidia/ Disallow: /sites/nx-group/ Disallow: /sites/nyu-langone-health/ Disallow: /sites/officedepotofficemax/ Disallow: /sites/ofx/ Disallow: /sites/olddominion/ Disallow: /sites/omnia-partners/ Disallow: /sites/opentext/ Disallow: /sites/oppenheimerfunds/ Disallow: /sites/oppo/ Disallow: /sites/optum/ Disallow: /sites/oracle/ Disallow: /sites/oraclecloud/ Disallow: /sites/orlando/ Disallow: /sites/osim/ Disallow: /sites/oticon/ Disallow: /sites/otsuka/ Disallow: /sites/outshift-by-cisco/ Disallow: /sites/pace/ Disallow: /sites/paloaltonetworks/ Disallow: /sites/panasonic-energy/ Disallow: /sites/parloa/ Disallow: /sites/pastebywetransfer/ Disallow: /sites/paulevans/ Disallow: /sites/paycom/ Disallow: /sites/paypal/ Disallow: /sites/paysafe/ Disallow: /sites/peerfit/ Disallow: /sites/pega/ Disallow: /sites/peopleconnect/ Disallow: /sites/pepsico/ Disallow: /sites/perennial-holdings/ Disallow: /sites/personalcapital/ Disallow: /sites/pete--gerrys/ Disallow: /sites/peter-millar/ Disallow: /sites/petronas/ Disallow: /sites/pfizer/ Disallow: /sites/pharmamar/ Disallow: /sites/philanthropy-asia-alliance/ Disallow: /sites/philip-morris-international/ Disallow: /sites/philips/ Disallow: /sites/pimco/ Disallow: /sites/plusgrade/ Disallow: /sites/pmi/ Disallow: /sites/polestar/ Disallow: /sites/post-university/ Disallow: /sites/ppg/ Disallow: /sites/premier/ Disallow: /sites/proctergamble/ Disallow: /sites/pros/ Disallow: /sites/providencedigitalinnovation/ Disallow: /sites/ptc/ Disallow: /sites/publicissapient/ Disallow: /sites/purdue-university/ Disallow: /sites/puremichigan/ Disallow: /sites/pwc-and-aws-switch-on-possibility/ Disallow: /sites/pwc-cloud-and-digital-transformation/ Disallow: /sites/pwc-consulting-llc/ Disallow: /sites/pwc-sustainability/ Disallow: /sites/pwc/ Disallow: /sites/qatarfoundation/ Disallow: /sites/qlik/ Disallow: /sites/quad/ Disallow: /sites/qualityassurance/ Disallow: /sites/quantcast/ Disallow: /sites/quest-software/ Disallow: /sites/quest/ Disallow: /sites/quid/ Disallow: /sites/quility/ Disallow: /sites/rackspace/ Disallow: /sites/ramcommercial/ Disallow: /sites/raymondjames/ Disallow: /sites/rbcwealthmanagement/ Disallow: /sites/realme/ Disallow: /sites/recruit-holdings/ Disallow: /sites/refinitiv/ Disallow: /sites/regeneron/ Disallow: /sites/relateiq/ Disallow: /sites/reliaquest/ Disallow: /sites/revolut-business/ Disallow: /sites/richard-mille/ Disallow: /sites/ricoh/ Disallow: /sites/ricoheurope/ Disallow: /sites/riskified/ Disallow: /sites/ritz-carltonleadershipcenter/ Disallow: /sites/riverbed/ Disallow: /sites/robertocoin/ Disallow: /sites/rocketmortgage/ Disallow: /sites/rocky-mountaineer/ Disallow: /sites/rokt/ Disallow: /sites/ronesans-holding/ Disallow: /sites/ryder/ Disallow: /sites/sage/ Disallow: /sites/sailpoint/ Disallow: /sites/salesforce/ Disallow: /sites/salesforceessentials/ Disallow: /sites/samsung-galaxy-for-business/ Disallow: /sites/samsung/ Disallow: /sites/samsungbusiness/ Disallow: /sites/samsungmobilecomputing/ Disallow: /sites/samsungsds/ Disallow: /sites/samsungtv/ Disallow: /sites/sap/ Disallow: /sites/sapconcur/ Disallow: /sites/saskatchewan/ Disallow: /sites/schmidt-futures/ Disallow: /sites/schneiderelectric/ Disallow: /sites/seagate/ Disallow: /sites/securian-financial/ Disallow: /sites/seoul/ Disallow: /sites/servicenow/ Disallow: /sites/shinola/ Disallow: /sites/shoplazza/ Disallow: /sites/siemens-smart-infrastructure/ Disallow: /sites/siemenshealthineers/ Disallow: /sites/sisense/ Disallow: /sites/sk/ Disallow: /sites/smartsheet/ Disallow: /sites/snowflake/ Disallow: /sites/socialimpact/ Disallow: /sites/softserve/ Disallow: /sites/sopra-banking-software/ Disallow: /sites/spdr/ Disallow: /sites/spectrum-business/ Disallow: /sites/splunk/ Disallow: /sites/sprinklr/ Disallow: /sites/sprintbusiness/ Disallow: /sites/square/ Disallow: /sites/stack-overflow-for-teams/ Disallow: /sites/stand-together/ Disallow: /sites/standard-chartered-bank/ Disallow: /sites/staples-business/ Disallow: /sites/starbucks-card-corporate-sales/ Disallow: /sites/starkey/ Disallow: /sites/state-of-maryland/ Disallow: /sites/statefarm/ Disallow: /sites/statoil/ Disallow: /sites/stewardshipasiacentre/ Disallow: /sites/stjude/ Disallow: /sites/sungardas/ Disallow: /sites/sunovion/ Disallow: /sites/suntrust/ Disallow: /sites/suntrustprivatewealth/ Disallow: /sites/surveymonkey/ Disallow: /sites/suse/ Disallow: /sites/svb/ Disallow: /sites/symantec/ Disallow: /sites/t-rowe-price/ Disallow: /sites/tableau/ Disallow: /sites/tableauapac/ Disallow: /sites/taboola/ Disallow: /sites/take-me-fishing/ Disallow: /sites/takeda/ Disallow: /sites/tanium/ Disallow: /sites/tanotofoundation/ Disallow: /sites/tanqueray-gin/ Disallow: /sites/taskus/ Disallow: /sites/tcs/ Disallow: /sites/td-bank/ Disallow: /sites/teladoc-health/ Disallow: /sites/ten-x/ Disallow: /sites/tencent-cloud/ Disallow: /sites/tequilaherradura/ Disallow: /sites/teradata/ Disallow: /sites/tesaro/ Disallow: /sites/textronaviation/ Disallow: /sites/the-allstate-foundation/ Disallow: /sites/the-ascott-limited/ Disallow: /sites/the-guild/ Disallow: /sites/the-macallan/ Disallow: /sites/the-uks-department-for-business-and-trade/ Disallow: /sites/thebalvenieandglenfiddich/ Disallow: /sites/theglenlivet/ Disallow: /sites/thehartford/ Disallow: /sites/theleukemialymphomasociety/ Disallow: /sites/themacallan/ Disallow: /sites/thepowermba/ Disallow: /sites/think-with-google/ Disallow: /sites/thomsonreuters/ Disallow: /sites/threatlocker/ Disallow: /sites/thrivent/ Disallow: /sites/tiaa/ Disallow: /sites/tiffany/ Disallow: /sites/tiktok/ Disallow: /sites/tm-global/ Disallow: /sites/tmfgroup/ Disallow: /sites/tmobile/ Disallow: /sites/todaysmilitary/ Disallow: /sites/totalwireless/ Disallow: /sites/toyota-prius/ Disallow: /sites/toyota/ Disallow: /sites/transamerica/ Disallow: /sites/trello/ Disallow: /sites/turkish-airlines/ Disallow: /sites/twilio/ Disallow: /sites/ubs/ Disallow: /sites/unicefusa/ Disallow: /sites/unicycive-therapeutics/ Disallow: /sites/unify/ Disallow: /sites/unilever-food-solutions/ Disallow: /sites/unique-properties/ Disallow: /sites/unisys/ Disallow: /sites/unit4/ Disallow: /sites/united-cards-from-chase/ Disallow: /sites/united-healthcare/ Disallow: /sites/united/ Disallow: /sites/universityofcalifornia/ Disallow: /sites/universityofphoenix/ Disallow: /sites/ups/ Disallow: /sites/urogen/ Disallow: /sites/us-bank-wealth-management/ Disallow: /sites/usaa/ Disallow: /sites/usbank/ Disallow: /sites/usbankaltitudereserve/ Disallow: /sites/vaneck/ Disallow: /sites/vanguard/ Disallow: /sites/vanguardcharitable/ Disallow: /sites/varicent/ Disallow: /sites/veeam/ Disallow: /sites/verisign/ Disallow: /sites/verizon-media/ Disallow: /sites/visa-infinite-asia-pacific/ Disallow: /sites/visa/ Disallow: /sites/visier/ Disallow: /sites/vision-by-protiviti/ Disallow: /sites/vmware/ Disallow: /sites/walmart-business/ Disallow: /sites/walmart-marketplace/ Disallow: /sites/webroot/ Disallow: /sites/wellington-management/ Disallow: /sites/wellsfargo/ Disallow: /sites/whittiertrust/ Disallow: /sites/windriver/ Disallow: /sites/wm/ Disallow: /sites/workday/ Disallow: /sites/workfront/ Disallow: /sites/xerox/ Disallow: /sites/xfinity/ Disallow: /sites/your-impact-with-unicef/ Disallow: /sites/zendesk/ Disallow: /sites/zenithwatches/ Disallow: /sites/zircongroup/ Disallow: /sites/zscaler/ Disallow: /sites/zurich/ Disallow: /sites/zycus/ Disallow: /uk/advisor/ Disallow: /wheels/ Sitemap: https://www.forbes.com/news_sitemap.xml Sitemap: https://www.forbes.com/video_sitemap.xml Sitemap: https://www.forbes.com/companies_sitemap.xml Sitemap: https://www.forbes.com/health/sitemap.xml Sitemap: https://www.forbes.com/connect/events-sitemap.xml Sitemap: https://www.forbes.com/advisor/sitemap.xml Sitemap: https://www.forbes.com/home-improvement/sitemap.xml Sitemap: https://www.forbes.com/digital-assets/sitemap.xml
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-27 | ok 200 | 29e98ab89e88 |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | www.forbes.com | fetch-blocked | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2025-09-15 CC-MAIN-2025-38 | forbes.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2024-09-16 CC-MAIN-2024-38 | www.forbes.com | ok view | blocks | blocks | unknown | blocks | allows | unknown | blocks |
| 2023-10-02 CC-MAIN-2023-40 | www3.forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2022-10-03 CC-MAIN-2022-40 | www3.forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | www3.forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | www3.forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | www3.forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | www3.forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-01-29 CC-MAIN-2018-05 | www3.forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-10-23 CC-MAIN-2017-43 | www2.forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | www3.forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-07-24 CC-MAIN-2017-30 | www2.forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-04-24 CC-MAIN-2017-17 | www3.forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-01-23 CC-MAIN-2017-04 | www3.forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-10-03 CC-MAIN-2016-40 | forbes.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-09-05 CC-MAIN-2016-36 | www2.forbes.com | ok view | blocks | blocks | unknown | blocks | blocks | unknown | blocks |
Evidence
Every daily observation of forbes.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).
API: https://infrarabbit.com/crawlers/api/v1/policy/forbes.com