cnet.com: what it tells AI crawlers
On 27 October 2026, cnet.com's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and CCBot.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://www.cnet.com/robots.txt
unchanged since 2026-10-27 · sha2566300fd9712a37580…· view as recorded - The * group
- partial applies to every agent without its own group
Per AI bot, 2026-10-27
| User agent | Verdict | How |
|---|---|---|
| GPTBot | blocks | named in its own group |
| OAI-SearchBot | blocks | named in its own group |
| ChatGPT-User | blocks | named in its own group |
| ClaudeBot | blocks | named in its own group |
| Claude-SearchBot | blocks | named in its own group |
| Claude-User | blocks | named in its own group |
| PerplexityBot | blocks | named in its own group |
| Perplexity-User | blocks | named in its own group |
| Googlebot | partial | through the * group |
| Google-Extended | partial | through the * group |
| Applebot-Extended | blocks | named in its own group |
| CCBot | blocks | named in its own group |
| Bytespider | blocks | named in its own group |
| Meta-ExternalAgent | blocks | named in its own group |
| Amazonbot | blocks | named in its own group |
| AddSearchBot | blocks | named in its own group |
| AgentDataBot | blocks | named in its own group |
| AIWebIndex | blocks | named in its own group |
| amazon-QBusiness | blocks | named in its own group |
| AmazonBuyForMe | blocks | named in its own group |
| Amzn-SearchBot | blocks | named in its own group |
| Amzn-User | blocks | named in its own group |
| anthropic-ai | blocks | named in its own group |
| ApifyBot | blocks | named in its own group |
| ApifyWebsiteContentCrawler | blocks | named in its own group |
| Applebot | blocks | named in its own group |
| atlassian-bot | blocks | named in its own group |
| AzureAI-SearchBot | blocks | named in its own group |
| BixelBot | blocks | named in its own group |
| Bravebot | blocks | named in its own group |
| Brightbot | blocks | named in its own group |
| ChatGLM-Spider | blocks | named in its own group |
| Claude-Code | blocks | named in its own group |
| Claude-Web | blocks | named in its own group |
| Cloudflare-AutoRAG | blocks | named in its own group |
| CloudflareBrowserRenderingCrawler | blocks | named in its own group |
| Code | blocks | named in its own group |
| cohere-ai | blocks | named in its own group |
| cohere-training-data-crawler | blocks | named in its own group |
| Cotoyogi | blocks | named in its own group |
| Cursor | blocks | named in its own group |
| DeepSeekBot | blocks | named in its own group |
| Devin | blocks | named in its own group |
| Diffbot | blocks | named in its own group |
| DuckAssistBot | blocks | named in its own group |
| ExaBot | blocks | named in its own group |
| ExaSearchBot | blocks | named in its own group |
| FacebookBot | blocks | named in its own group |
| Factset_spyderbot | blocks | named in its own group |
| FirecrawlAgent | blocks | named in its own group |
| GeistHaus-PageFetcher | blocks | named in its own group |
| GrokBot | blocks | named in its own group |
| Hunyuan | blocks | named in its own group |
| iAskBot | blocks | named in its own group |
| iaskspider | blocks | named in its own group |
| ICC-Crawler | blocks | named in its own group |
| ImagesiftBot | blocks | named in its own group |
| imageSpider | blocks | named in its own group |
| kagi-fetcher | blocks | named in its own group |
| KeenableBot | blocks | named in its own group |
| Kimi-Agent | blocks | named in its own group |
| Kimi-User | blocks | named in its own group |
| KlaviyoAIBot | blocks | named in its own group |
| KunatoCrawler | blocks | named in its own group |
| laion-huggingface-processor | blocks | named in its own group |
| LCC | blocks | named in its own group |
| LinerBot | blocks | named in its own group |
| LinkupBot | blocks | named in its own group |
| Manus-User | blocks | named in its own group |
| meta-externalfetcher | blocks | named in its own group |
| meta-webindexer | blocks | named in its own group |
| MistralAI-User | blocks | named in its own group |
| NovaAct | blocks | named in its own group |
| omgili | blocks | named in its own group |
| omgilibot | blocks | named in its own group |
| opencode | blocks | named in its own group |
| PanguBot | blocks | named in its own group |
| PetalBot | blocks | named in its own group |
| PhindBot | blocks | named in its own group |
| Poggio-Citations | blocks | named in its own group |
| qodercli | blocks | named in its own group |
| QualifiedBot | blocks | named in its own group |
| Reflectionbot | blocks | named in its own group |
| SBIntuitionsBot | blocks | named in its own group |
| Scrapy | blocks | named in its own group |
| SemrushBot-OCOB | blocks | named in its own group |
| Shap-User | blocks | named in its own group |
| ShapBot | blocks | named in its own group |
| Spider | blocks | named in its own group |
| TavilyBot | blocks | named in its own group |
| TerraCotta | blocks | named in its own group |
| Timpibot | blocks | named in its own group |
| TongyiBot | blocks | named in its own group |
| Trae | blocks | named in its own group |
| TwinAgent | blocks | named in its own group |
| UseAI | blocks | named in its own group |
| VelenPublicWebCrawler | blocks | named in its own group |
| Webzio-Extended | blocks | named in its own group |
| WRTNBot | blocks | named in its own group |
| xAI-Grok | blocks | named in its own group |
| YandexAdditionalBot | blocks | named in its own group |
| YiBot | blocks | named in its own group |
| YiyanBot | blocks | named in its own group |
| YouBot | blocks | named in its own group |
| ZanistaBot | blocks | named in its own group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | not observed |
| Reserves TDM rights (TDMRep file, header or meta) | not observed |
| Declares Content Signals (robots.txt or header) | no |
| Says ai-train=no in a Content Signal | no |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
robots.txt as recorded sha256 6300fd9712a37580…
# www.robotstxt.org/ # www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 # # Ziff Davis content is made available for your non-commercial use subject to our # Terms of Use here: https://www.ziffdavis.com/terms-of-use # Use of any robot, crawler, or other tool to scrape, harvest, extract, or retrieve any content on # this website using automated means is prohibited without written permission from Ziff Davis. # Prohibited uses include but are not limited to: # (1) text and data mining under Art. 4 of the EU Directive on Copyright in the Digital Single # Market; # (2) development or operation of artificial intelligence or machine learning software or # databases, including by training, fine-tuning, embedding, and retrieval-augmented generation; # (3) creating data sets containing our content or sharing it with others; and # (4) any commercial purposes. # Contact licensing@ziffdavis.com for assistance. User-agent: * Disallow: */*max-price=* Disallow: */*min-price=* Disallow: */Xhr* Disallow: */account-information/* Disallow: */ad/ad-cookie/* Disallow: */ajax/ Disallow: */alert/* Disallow: */cnet_global_app_redirect.xml Disallow: */component/* Disallow: */edit/ Disallow: */edit/change-email/* Disallow: */edit/change-password/* Disallow: */email/editor/* Disallow: */email/member/* Disallow: */email/newsletter/* Disallow: */error/* Disallow: */ext/make/ingest/* Disallow: */feature/* Disallow: */finder/*/*/?*filter=*_*_*_*_* Disallow: */finder/*/?*filter=*_*_*_*_* Disallow: */forums/* Disallow: */g00/* Disallow: */informacion-de-mi-cuenta/* Disallow: */members/* Disallow: */products/*?*filter=* Disallow: */roadshow/finder/* Disallow: */search/* Disallow: */user/* Disallow: */videos/manifest/* Disallow: */videos/secure/* Disallow: */videos/selector/* Disallow: */wrap-function* Disallow: */wrap-xhr* Disallow: */xhr* Disallow: *?sort=popularity* Disallow: *?sort=price* Disallow: *?sort=user-rating* Disallow: *hub-preview-version=* Disallow: /.well-known/* Disallow: /api/editionurlgenerator/ Disallow: /apple-news/* Disallow: /au/* Disallow: /es/* Disallow: /html/story/* Disallow: /index.php/* Disallow: /products/* Disallow: /uk/* # Ziff Davis bot list user-agent: AddSearchBot user-agent: AgentDataBot user-agent: AI2Bot user-agent: Ai2Bot-DeepResearchEval user-agent: Ai2Bot-Dolma user-agent: AIWebIndex user-agent: AIWebIndex-Agent user-agent: amazon-QBusiness user-agent: Amazonbot user-agent: AmazonBuyForMe user-agent: Amzn-SearchBot user-agent: Amzn-User user-agent: Anchor Browser user-agent: AndiAgent user-agent: anthropic-ai user-agent: ApifyBot user-agent: ApifyWebsiteContentCrawler user-agent: Applebot user-agent: Applebot-extended user-agent: atlassian-bot user-agent: AutoRAG user-agent: AwarioSmartBot user-agent: AzureAI-SearchBot user-agent: BeansLLM-CorpusBot user-agent: Big Sur AI user-agent: bigsur.ai user-agent: BixelBot user-agent: Brandwatch user-agent: Bravebot user-agent: Brightbot user-agent: Bytespider user-agent: CCBot user-agent: Channel3Bot user-agent: ChatGLM-Spider user-agent: ChatGPT-User user-agent: Claude-Code user-agent: Claude-SearchBot user-agent: Claude-User user-agent: Claude-Web user-agent: ClaudeBot user-agent: Cloudflare-AutoRAG user-agent: CloudflareBrowserRenderingCrawler user-agent: Code user-agent: cohere-ai user-agent: cohere-training-data-crawler user-agent: Cotoyogi user-agent: Crawl4AI user-agent: Cursor user-agent: Datenbank Crawler user-agent: Datenbank-Crawler user-agent: DeepSeekBot user-agent: DesearchBot user-agent: Devin user-agent: Diffbot user-agent: Direqt Anomura user-agent: Dola-SearchBot user-agent: DuckAssistBot user-agent: Echobot Bot user-agent: ExaBot user-agent: ExaSearchBot user-agent: FacebookBot user-agent: Factset_spyderbot user-agent: FirecrawlAgent user-agent: GeistHaus-PageFetcher user-agent: GPTBot user-agent: grok-agent user-agent: GrokBot user-agent: Hunyuan user-agent: iAsk user-agent: iAskBot user-agent: iaskspider user-agent: ICC Crawler user-agent: ICC-Crawler user-agent: ImageBot user-agent: ImageSiftBot user-agent: imageSpider user-agent: IsonSearchBot user-agent: kagi-fetcher user-agent: Kangaroo Bot user-agent: Kangaroo-Bot user-agent: Keenable-User user-agent: KeenableBot user-agent: Kimi-Agent user-agent: Kimi-User user-agent: KlaviyoAIBot user-agent: Kunato user-agent: KunatoCrawler user-agent: laion-huggingface-processor user-agent: LCC user-agent: LINER Bot user-agent: LinerBot user-agent: LinkupBot user-agent: Manus-User user-agent: meta-externalagent user-agent: meta-externalfetcher user-agent: meta-webindexer user-agent: mistral.ai user-agent: MistralAI-User user-agent: netEstate Imprint Crawler user-agent: NovaAct user-agent: Novellum user-agent: Novellum AI Crawl user-agent: OAI-SearchBot user-agent: omgili user-agent: opencode user-agent: PanguBot user-agent: PeopleInc-DCipher-Scraper/1.1.0 user-agent: Perplexity-User user-agent: PerplexityBot user-agent: PetalBot user-agent: PhindBot user-agent: Poggio-Citations user-agent: qodercli user-agent: QualifiedBot user-agent: recraft-image-dataset user-agent: Reflectionbot user-agent: SBIntuitionsBot user-agent: ScryBot user-agent: SeekrBot user-agent: SemrushBot-OCOB user-agent: SemrushBotSwa user-agent: Shap-User user-agent: ShapBot user-agent: SofyaBot user-agent: SomethinggBot user-agent: Spider user-agent: TalarionFrontierCrawler user-agent: TalarionSearchBot user-agent: TavilyBot user-agent: TerraCotta user-agent: Timpibot user-agent: TongyiBot user-agent: Trae user-agent: TwinAgent user-agent: UseAI user-agent: Velen Crawler user-agent: VelenPublicWebCrawler user-agent: Webzio-Extended user-agent: Wrtn user-agent: WRTNBot user-agent: xAI-Grok user-agent: YandexAdditionalBot user-agent: YepBot user-agent: YiBot user-agent: YiyanBot user-agent: YouBot user-agent: ZanistaBot Allow: /home/fiber-cable-or-5g-internet-how-to-choose/ Allow: /home/internet/do-you-need-symmetrical-internet-speeds-what-to-know-before-you-upgrade/ Allow: /home/internet/dont-buy-internet-until-you-check-the-cost-per-mbps/ Allow: /home/internet/equipment-for-fast-home-internet/ Allow: /home/internet/fix-wi-fi-dead-spots-how-to-get-internet-throughout-your-home/ Allow: /home/internet/gigabit-internet-is-it-worth-splurging-for-a-faster-internet-plan/ Allow: /home/internet/how-to-spot-hidden-fees-and-save-money/ Allow: /home/internet/internet-data-caps-explained-avoid-extra-charges-and-make-most-of-your-internet-plan/ Allow: /home/internet/is-your-internet-connection-delivering-on-the-speeds-you-pay-for-heres-how-to-tell/ Allow: /home/internet/lag-free-internet-how-to-find-the-right-internet-speed-for-a-family-of-4-or-more/ Allow: /home/internet/what-are-the-different-internet-connection-types/ Allow: /home/internet/why-your-gigabit-internet-plan-might-be-overkill/ Allow: /home/internet/why-your-upload-speed-is-ruining-your-work-calls-and-how-to-fix-it/ Allow: /home/internet/xfinity-vs-att-which-internet-provider-is-best/ Allow: /home/internet/xfinity-vs-verizon-which-internet-provider-is-best/ Allow: /home/internet/your-internet-bill-might-be-booby-trapped-heres-what-to-do/ Allow: /news/ai-automation-changing-how-we-file-taxes/ Allow: /paid-content/air-fry-without-the-countertop-clutter-how-mideas-new-appliances-can-help-your-holiday-cooking/ Allow: /paid-content/cash-app-families-guide-how-to-set-up-a-teen-or-child-account-step-by-step/ Allow: /paid-content/cash-app-for-teens-safety-and-parental-controls/ Allow: /paid-content/cash-apps-new-kids-debit-card-what-parents-need-to-know/ Allow: /paid-content/can-you-use-an-alienware-laptop-for-both-gaming-and-professional-work/ Allow: /paid-content/cricket-wireless-offers-reliable-prepaid-nationwide-5g/ Allow: /paid-content/dji-builds-on-12-years-of-handheld-imaging-innovation-with-the-dji-osmo-360-ii/ Allow: /paid-content/dji-osmo-pocket-4p-is-a-powerful-tool-for-independent-filmmakers/ Allow: /paid-content/do-you-actually-need-a-separate-antivirus-vpn-and-scam-protection-mcafees-all-in-one-plans-have-you-covered/ Allow: /paid-content/enjoy-unlimited-wi-fi-and-predictable-pricing-with-on-your-home-internet/ Allow: /paid-content/from-tacos-to-tvs-doordash-delivers-your-game-day-essentials/ Allow: /paid-content/galaxy-s26-camera-system-the-biggest-questions-answered-before-you-buy/ Allow: /paid-content/get-an-unlimited-5g-phone-plan-for-25-per-month-with-metro-by-t-mobile/ Allow: /paid-content/get-everything-you-need-for-the-world-cup-delivered-with-doordash/ Allow: /paid-content/h-r-blocks-local-tax-pros-help-you-navigate-complex-returns/ Allow: /paid-content/host-the-holidays-with-less-work-sharkninja-gear-for-cooking-treats-and-cleanup/ Allow: /paid-content/how-cash-app-builds-savings-habits-for-kids-and-teens/ Allow: /paid-content/how-cash-app-sponsorship-works-adding-removing-and-managing-parental-controls/ Allow: /paid-content/how-families-get-more-with-t-mobile/ Allow: /paid-content/how-fast-should-your-home-internet-be-for-browsing-and-streaming-t-mobile/ Allow: /paid-content/how-galaxy-ai-camera-tools-help-you-take-better-photos-automatically-on-the-s25/ Allow: /paid-content/how-h-r-block-helps-freelancers-and-gig-workers-stay-ahead-of-tax-season/ Allow: /paid-content/how-h-r-block-helps-you-maximize-freelance-and-gig-worker-write-offs/ Allow: /paid-content/how-h-r-block-simplifies-taxes-for-first-time-filers/ Allow: /paid-content/how-h-r-blocks-experts-can-help-you-avoid-common-filing-mistakes/ Allow: /paid-content/how-samsungs-galaxy-ai-boosts-gaming-performance/ Allow: /paid-content/how-sling-is-disrupting-the-old-live-tv-streaming-model/ Allow: /paid-content/how-to-build-a-solo-video-setup-with-dji-the-camera-stabilizer-and-mic-you-actually-need/ Allow: /paid-content/how-to-choose-a-reliable-home-internet-provider-in-2026/ Allow: /paid-content/how-to-choose-the-right-dell-ai-laptop-for-your-workflow/ Allow: /paid-content/how-to-get-fast-reliable-fiber-internet-at-a-predictable-transparent-price/ Allow: /paid-content/how-to-get-your-backyard-ready-for-summer-with-lights-and-sound/ Allow: /paid-content/how-to-time-your-next-major-appliance-purchase-for-maximum-savings/ Allow: /paid-content/how-to-try-out-ai-glasses-from-meta-at-your-local-best-buy/ Allow: /paid-content/how-to-watch-live-sports-and-avoid-wheres-the-game-chaos-with-xfinity/ Allow: /paid-content/i-tried-internxt-an-encrypted-secure-cloud-storage-built-for-the-post-quantum-era/ Allow: /paid-content/is-diy-tax-filing-right-for-you-heres-what-h-r-blocks-tools-make-possible/ Allow: /paid-content/is-fiber-to-the-home-ftth-actually-better-than-xfinity-heres-what-the-evidence-says/ Allow: /paid-content/is-it-time-to-upgrade-your-iphone-with-t-mobiles-trade-in-offers-the-answer-is-yes/ Allow: /paid-content/is-wi-fi-7-worth-the-upgrade-heres-how-msis-roamii-mesh-systems-fit-in/ Allow: /paid-content/life-changes-and-new-laws-can-affect-your-tax-refund-how-to-estimate/ Allow: /paid-content/maximize-your-refund-with-h-r-blocks-smart-tax-tools/ Allow: /paid-content/maximize-your-wireless-savings-with-the-t-mobile-visa-credit-card/ Allow: /paid-content/meet-the-new-hp-omnibook-5-an-oled-laptop-built-for-long-days-of-work-class-and-streaming/ Allow: /paid-content/meta-ray-ban-wayfarer-ai-glasses-discount-may-2026/ Allow: /paid-content/news/cash-app-families-guide-how-to-set-up-a-teen-or-child-account-step-by-step/ Allow: /paid-content/news/cash-apps-new-kids-debit-card-what-parents-need-to-know/ Allow: /paid-content/news/cricket-wireless-offers-reliable-prepaid-nationwide-5g/ Allow: /paid-content/news/dji-osmo-pocket-4p-is-a-powerful-tool-for-independent-filmmakers/ Allow: /paid-content/news/enjoy-unlimited-wi-fi-and-predictable-pricing-with-on-your-home-internet/ Allow: /paid-content/news/from-tacos-to-tvs-doordash-delivers-your-game-day-essentials/ Allow: /paid-content/news/galaxy-s26-camera-system-the-biggest-questions-answered-before-you-buy/ Allow: /paid-content/news/get-an-unlimited-5g-phone-plan-for-25-per-month-with-metro-by-t-mobile/ Allow: /paid-content/news/get-everything-you-need-for-the-world-cup-delivered-with-doordash/ Allow: /paid-content/news/h-r-blocks-local-tax-pros-help-you-navigate-complex-returns/ Allow: /paid-content/news/how-cash-app-builds-savings-habits-for-kids-and-teens/ Allow: /paid-content/news/how-cash-app-sponsorship-works-adding-removing-and-managing-parental-controls/ Allow: /paid-content/news/how-galaxy-ai-camera-tools-help-you-take-better-photos-automatically-on-the-s25/ Allow: /paid-content/news/how-h-r-block-helps-you-maximize-freelance-and-gig-worker-write-offs/ Allow: /paid-content/news/how-h-r-block-simplifies-taxes-for-first-time-filers/ Allow: /paid-content/news/how-h-r-blocks-experts-can-help-you-avoid-common-filing-mistakes/ Allow: /paid-content/news/how-samsungs-galaxy-ai-boosts-gaming-performance/ Allow: /paid-content/news/how-sling-is-disrupting-the-old-live-tv-streaming-model/ Allow: /paid-content/news/how-t-mobile-can-help-you-save-thousands-on-your-family-wireless-plan/ Allow: /paid-content/news/how-to-choose-a-reliable-home-internet-provider-in-2026/ Allow: /paid-content/news/how-to-choose-the-right-dell-ai-laptop-for-your-workflow/ Allow: /paid-content/news/how-to-get-fast-reliable-fiber-internet-at-a-predictable-transparent-price/ Allow: /paid-content/news/how-to-get-your-backyard-ready-for-summer-with-lights-and-sound/ Allow: /paid-content/news/how-to-time-your-next-major-appliance-purchase-for-maximum-savings/ Allow: /paid-content/news/how-to-watch-live-sports-and-avoid-wheres-the-game-chaos-with-xfinity/ Allow: /paid-content/news/i-tried-internxt-an-encrypted-secure-cloud-storage-built-for-the-post-quantum-era/ Allow: /paid-content/news/is-diy-tax-filing-right-for-you-heres-what-h-r-blocks-tools-make-possible/ Allow: /paid-content/news/is-wi-fi-7-worth-the-upgrade-heres-how-msis-roamii-mesh-systems-fit-in/ Allow: /paid-content/news/life-changes-and-new-laws-can-affect-your-tax-refund-how-to-estimate/ Allow: /paid-content/news/maximize-your-refund-with-h-r-blocks-smart-tax-tools/ Allow: /paid-content/news/maximize-your-wireless-savings-with-the-t-mobile-visa-credit-card/ Allow: /paid-content/news/meta-ray-ban-wayfarer-ai-glasses-discount-may-2026/ Allow: /paid-content/news/must-know-tax-season-tips-for-small-businesses-and-solopreneurs/ Allow: /paid-content/news/nomad-esim-wants-to-give-you-more-than-just-data/ Allow: /paid-content/news/scale-your-small-business-with-fast-funding-via-loanbuilder/ Allow: /paid-content/news/scale-your-small-business-with-fast-funding-via-loanbuilder/ Allow: /paid-content/news/selling-stuff-online-what-you-could-be-on-the-hook-for-tax-wise/ Allow: /paid-content/news/shopping-for-a-major-appliance-let-best-buy-do-the-heavy-lifting/ Allow: /paid-content/news/streamline-your-weeknight-cooking-routine-with-a-multi-speed-air-fryer/ Allow: /paid-content/news/switch-to-t-mobile-in-15-minutes-and-get-up-to-800-dollars/ Allow: /paid-content/news/t-mobile-5g-home-internet-offers-a-solution-to-contracts-cords-and-hidden-fees/ Allow: /paid-content/news/tempos-fresh-chef-crafted-meals-make-your-meal-prep-smarter-and-tastier/ Allow: /paid-content/news/the-dell-xps-13-is-subtle-luxury-in-laptop-form/ Allow: /paid-content/news/the-difference-between-fiber-to-the-home-and-fiber-powered-internet/ Allow: /paid-content/news/upgrade-your-home-on-a-budget-with-these-summer-appliance-rollbacks-from-walmart/ Allow: /paid-content/news/upgrade-your-summer-content-with-a-portable-digital-camera/ Allow: /paid-content/news/what-americans-pay-for-internet-in-2026-and-how-xfinity-helps-you-save/ Allow: /paid-content/news/why-a-reliable-fiber-powered-network-from-xfinity-is-the-ultimate-solution-for-home-wi-fi/ Allow: /paid-content/news/why-tax-filers-are-choosing-h-r-block-this-season/ Allow: /paid-content/news/why-you-need-a-cutting-edge-vpn-to-keep-you-safe-online/ Allow: /paid-content/news/why-you-need-an-internet-provider-with-no-data-caps/ Allow: /paid-content/nomad-esim-wants-to-give-you-more-than-just-data/ Allow: /paid-content/nordictrack-just-out-engineered-its-own-best-treadmill-heres-everything-thats-new-on-the-ultra-1/ Allow: /paid-content/nordictrack-rowing-machine-or-exercise-bike-which-one-will-you-actually-stick-with/ Allow: /paid-content/oled-or-mini-led-how-to-pick-the-right-lg-tv-for-your-room-lighting-and-viewing-habits/ Allow: /paid-content/ready-to-take-better-photos-how-to-pick-the-right-canon-camera-for-beginners/ Allow: /paid-content/samsung-micro-rgb-tvs-show-you-more-colors-than-ever-before/ Allow: /paid-content/scale-your-small-business-with-fast-funding-via-loanbuilder/ Allow: /paid-content/selling-stuff-online-what-you-could-be-on-the-hook-for-tax-wise/ Allow: /paid-content/shop-now-save-now-12-walmart-tech-picks-at-great-prices-no-promo-code-needed/ Allow: /paid-content/shopping-for-a-major-appliance-let-best-buy-do-the-heavy-lifting/ Allow: /paid-content/should-you-buy-a-frigidaire-refrigerator-at-best-buy-what-to-consider/ Allow: /paid-content/should-you-get-the-iphone-18-now-or-wait-what-to-know-about-preorder-and-carrier-deals/ Allow: /paid-content/stop-comparing-home-internet-providers-by-fiber-vs-cable-use-this-checklist-instead/ Allow: /paid-content/streamline-your-weeknight-cooking-routine-with-a-multi-speed-air-fryer/ Allow: /paid-content/studio-pilates-vs-a-home-reformer-the-math-on-how-much-youd-actually-save/ Allow: /paid-content/what-can-a-roborock-robot-vacuum-and-mop-do-5-common-scenarios-plus-models-to-shop/ Allow: /paid-content/roboform-for-everyday-use-ease-of-use-cost-and-getting-set-up/ Allow: /paid-content/switch-to-t-mobile-in-15-minutes-and-get-up-to-800-dollars/ Allow: /paid-content/t-mobile-5g-home-internet-gets-you-online-fast-with-no-technicians-or-contracts/ Allow: /paid-content/t-mobile-5g-home-internet-offers-a-solution-to-contracts-cords-and-hidden-fees/ Allow: /paid-content/t-mobile-fiber-vs-cable-internet-why-ultra-fast-internet-is-worth-the-upgrade/ Allow: /paid-content/tempos-fresh-chef-crafted-meals-make-your-meal-prep-smarter-and-tastier/ Allow: /paid-content/the-dell-xps-13-is-subtle-luxury-in-laptop-form/ Allow: /paid-content/the-dell-14-plus-is-a-copilot-pc-heres-what-that-actually-means/ Allow: /paid-content/the-difference-between-fiber-to-the-home-and-fiber-powered-internet/ Allow: /paid-content/the-nordictrack-commercial-2450-combines-big-speed-with-smart-adjustment-features/ Allow: /paid-content/the-nordictrack-treadmill-you-actually-need-for-the-12-3-30-workout/ Allow: /paid-content/upgrade-your-home-on-a-budget-with-these-summer-appliance-rollbacks-from-walmart/ Allow: /paid-content/upgrade-your-summer-content-with-a-portable-digital-camera/ Allow: /paid-content/what-americans-pay-for-internet-in-2026-and-how-xfinity-helps-you-save/ Allow: /paid-content/what-is-a-wearable-exoskeleton-and-who-actually-needs-one/ Allow: /paid-content/whats-a-good-casetify-case-for-my-next-iphone-a-complete-guide-to-practical-protection/ Allow: /paid-content/when-can-kids-get-a-debit-card-heres-how-cash-app-handles-every-age/ Allow: /paid-content/how-fast-should-your-home-internet-be/ Allow: /paid-content/when-can-kids-get-a-debit-card-heres-how-cash-app-handles-every-age/ Allow: /paid-content/cash-app-for-teens-safety-and-parental-controls/ Allow: /paid-content/what-can-ray-ban-meta-ai-glasses-do-5-use-cases-for-your-everyday-routine/ Allow: /paid-content/what-to-include-in-a-heart-health-daily-regimen-from-supplements-to-home-monitors/ Allow: /paid-content/what-does-an-npu-actually-do-and-does-your-laptop-need-one/ Allow: /paid-content/which-dell-laptops-are-actually-ai-ready-in-canada/ Allow: /paid-content/why-a-reliable-fiber-powered-network-from-xfinity-is-the-ultimate-solution-for-home-wi-fi/ Allow: /paid-content/why-tax-filers-are-choosing-h-r-block-this-season/ Allow: /paid-content/why-the-samsung-galaxy-s26-ultra-feels-like-the-first-phone-truly-built-for-ai/ Allow: /paid-content/how-to-check-if-t-mobile-fiber-is-available-at-your-address/ Allow: /paid-content/why-you-need-a-cutting-edge-vpn-to-keep-you-safe-online/ Allow: /paid-content/why-you-need-an-internet-provider-with-no-data-caps/ Allow: /tech/mobile/apple-iphone-17-event-september-2025-what-to-expect/ Allow: /tech/mobile/i-underestimated-workout-buddy-apples-playing-the-long-game-for-ai-coaching/ Allow: /tech/mobile/new-iphone-buying-guide-carrier-vs-apple-unlocked/ Allow: /tech/mobile/phone-deals-carrier-bill-credits-installment-plan-buying-guide/ Allow: /tech/mobile/phone-plan-home-internet-bundle/ Allow: /tech/mobile/samsung-galaxy-s26-and-s26-plus-hands-on-first-take/ Allow: /tech/mobile/samsung-galaxy-s26-ultra-vs-galaxy-s26-and-s26-plus-flagship-specs-comparison/ Allow: /tech/mobile/tomorrow-we-could-see-the-iphone-17-debut-with-a-long-overdue-update/ Allow: /tech/services-and-software/a-guide-to-data-removal-services-should-you-pay-for-privacy/ Allow: /tech/services-and-software/ai-automation-changing-how-we-file-taxes/ Allow: /tech/services-and-software/complex-income-tax-filing-rules-2026/ Allow: /tech/services-and-software/crypto-new-tax-form-rules/ Allow: /tech/services-and-software/data-brokers-how-your-personal-data-becomes-business/ Allow: /tech/services-and-software/end-year-tax-checklist-maximize-tax-refund/ Allow: /tech/services-and-software/end-year-tax-checklist-maximize-tax-refund/ Allow: /tech/services-and-software/free-tax-software-cost-upgrades/ Allow: /tech/services-and-software/how-h-r-block-helps-freelancers-and-gig-workers-stay-ahead-of-tax-season/ Allow: /tech/services-and-software/how-to-prevent-irs-penalties-and-delays-this-tax-season/ Allow: /tech/services-and-software/ios-27-is-almost-here-but-you-should-check-out-all-these-ios-26-features-first/ Allow: /tech/services-and-software/married-single-or-self-employed-how-your-filing-status-changes-everything/ Allow: /tech/services-and-software/must-know-tax-season-tips-for-small-businesses-and-solopreneurs/ Allow: /tech/services-and-software/proxies-web-scraping-ultimate-guide/ Allow: /tech/services-and-software/qualified-tips-overtime-tax-deduction-rules/ Allow: /tech/services-and-software/self-employment-tax-guide-1099-contractor/ Allow: /tech/services-and-software/w2-vs-1099-difference/ Allow: /tech/services-and-software/what-is-vibe-coding-how-anyone-can-make-apps-with-the-help-of-ai/ Allow: /tech/services-and-software/what-online-retailers-actually-do-with-your-data-after-you-check-out/ Allow: /tech/services-and-software/what-someone-can-do-with-your-phone-number-and-how-to-protect-yourself/ Allow: /tech/services-and-software/why-use-proxy-server-benefits-drawbacks-common-use-cases/ Allow: /tech/services-and-software/why-would-a-business-need-to-use-a-proxy-server/ Allow: /videos/2026-tax-season-checklist-what-you-need-before-you-file/ Allow: /videos/5-finance-and-tax-tips-for-freelancers/ Allow: /videos/navigating-your-tax-filing-options-simple-vs-complex-situations/ Allow: /videos/tax-breaks-you-might-miss-in-2026-and-how-to-find-them/ Disallow: / #CNET restricted bots User-agent: "008" User-agent: AwarioRssBot User-agent: DDM-DCipher/1.0.7 User-agent: HTTrack User-agent: Magpie-crawler User-agent: Nutch User-agent: Offline Explorer User-agent: Omgilibot User-agent: Peer39_crawler/1.0 User-agent: Scrapy Disallow: / Sitemap: https://www.cnet.com/sitemap-authors.xml Sitemap: https://www.cnet.com/sitemap-google-news.xml Sitemap: https://www.cnet.com/sitemaps/article/index.xml Sitemap: https://www.cnet.com/sitemaps/best/index.xml Sitemap: https://www.cnet.com/sitemaps/categories/index.xml Sitemap: https://www.cnet.com/sitemaps/collections/index.xml Sitemap: https://www.cnet.com/sitemaps/deals/index.xml Sitemap: https://www.cnet.com/sitemaps/gallery/index.xml Sitemap: https://www.cnet.com/sitemaps/how-to/index.xml Sitemap: https://www.cnet.com/sitemaps/updating-story/index.xml Sitemap: https://www.cnet.com/sitemaps/news/index.xml Sitemap: https://www.cnet.com/sitemaps/pages/index.xml Sitemap: https://www.cnet.com/sitemaps/review/index.xml Sitemap: https://www.cnet.com/sitemaps/versus/index.xml Sitemap: https://www.cnet.com/sitemaps/video/index.xml
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-27 | 2026-10-28 | ok 200 | 6300fd9712a3 | diff with previous |
| 2026-10-26 | 2026-10-26 | ok 200 | 0117f1d7196e |
Timeline of changes
| Date | What changed |
|---|---|
| 2026-10-27 | robots.txt changed |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2025-09-15 CC-MAIN-2025-38 | www.cnet.com | ok view | blocks | partial | unknown | partial | partial | unknown | partial |
| 2024-09-16 CC-MAIN-2024-38 | www.cnet.com | ok view | blocks | partial | unknown | blocks | partial | unknown | partial |
| 2023-10-02 CC-MAIN-2023-40 | www.cnet.com | ok view | blocks | partial | unknown | partial | partial | unknown | partial |
| 2022-10-03 CC-MAIN-2022-40 | www.cnet.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2021-09-27 CC-MAIN-2021-39 | www.cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | www.cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | www.cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | www.cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-04-23 CC-MAIN-2018-17 | www.cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-01-29 CC-MAIN-2018-05 | www.cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-10-23 CC-MAIN-2017-43 | www.cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | www.cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-07-24 CC-MAIN-2017-30 | www.cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-04-24 CC-MAIN-2017-17 | cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-01-23 CC-MAIN-2017-04 | www.cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-10-03 CC-MAIN-2016-40 | www.cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-09-05 CC-MAIN-2016-36 | cnet.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
Evidence
Every daily observation of cnet.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).
API: https://infrarabbit.com/crawlers/api/v1/policy/cnet.com