sophos.com: what it tells AI crawlers
On 27 October 2026, sophos.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot; it also published llms.txt.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://www.sophos.com/robots.txt
unchanged since 2026-10-26 · sha256332cae6d621808b8…· view as recorded
Per AI bot, 2026-10-27
| User agent | Verdict | How |
|---|---|---|
| GPTBot | allows | named in its own group |
| OAI-SearchBot | allows | named in its own group |
| ChatGPT-User | allows | named in its own group |
| ClaudeBot | allows | named in its own group |
| Claude-SearchBot | allows | no applicable rules |
| Claude-User | allows | no applicable rules |
| PerplexityBot | allows | named in its own group |
| Perplexity-User | allows | no applicable rules |
| Googlebot | allows | named in its own group |
| Google-Extended | allows | named in its own group |
| Applebot-Extended | allows | no applicable rules |
| CCBot | allows | no applicable rules |
| Bytespider | allows | no applicable rules |
| Meta-ExternalAgent | allows | no applicable rules |
| Amazonbot | blocks | named in its own group |
| Amzn-SearchBot | allows | named in its own group |
| Amzn-User | allows | named in its own group |
| Andibot | allows | named in its own group |
| anthropic-ai | allows | named in its own group |
| DeepSeekBot | blocks | named in its own group |
| ExaBot | allows | named in its own group |
| FirecrawlAgent | allows | named in its own group |
| PetalBot | blocks | named in its own group |
| PhindBot | allows | named in its own group |
| YouBot | allows | named in its own group |
Other signals
| Publishes /llms.txt | yes |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | no |
| Declares Content Signals (robots.txt or header) | no |
| Says ai-train=no in a Content Signal | no |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
llms.txt: "Sophos: Defeat Cyberattacks with AI-Native Security", 25,391 bytes, 126 links.
robots.txt as recorded sha256 708527863c76008f…
# # robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources. # # This file will be ignored unless it is at the root of your host: # Used: http://example.com/robots.txt # Ignored: http://example.com/site/robots.txt # # For more information about the robots.txt standard, see: # http://www.robotstxt.org/robotstxt.html User-agent: * # CSS, JS, Images Allow: /core/*.css$ Allow: /core/*.css? Allow: /core/*.js$ Allow: /core/*.js? Allow: /core/*.gif Allow: /core/*.jpg Allow: /core/*.jpeg Allow: /core/*.png Allow: /core/*.svg Allow: /profiles/*.css$ Allow: /profiles/*.css? Allow: /profiles/*.js$ Allow: /profiles/*.js? Allow: /profiles/*.gif Allow: /profiles/*.jpg Allow: /profiles/*.jpeg Allow: /profiles/*.png Allow: /profiles/*.svg # Directories Disallow: /core/ Disallow: /profiles/ # Files Disallow: /README.txt Disallow: /web.config # Paths (clean URLs) Disallow: /admin/ Disallow: /comment/reply/ Disallow: /filter/tips Disallow: /node/add/ Disallow: /search/ Disallow: /user/register/ Disallow: /user/password/ Disallow: /user/login/ Disallow: /user/logout/ # Paths (no clean URLs) Disallow: /index.php/admin/ Disallow: /index.php/comment/reply/ Disallow: /index.php/filter/tips Disallow: /index.php/node/add/ Disallow: /index.php/search/ Disallow: /index.php/user/password/ Disallow: /index.php/user/register/ Disallow: /index.php/user/login/ Disallow: /index.php/user/logout/ # robots.txt for www.sophos.com - 05-23 # Sitemaps 23 https://www.sophos.com/en-us/sitemap.xml https://www.sophos.com/de-de/sitemap.xml https://www.sophos.com/es-es/sitemap.xml https://www.sophos.com/it-it/sitemap.xml https://www.sophos.com/fr-fr/sitemap.xml https://www.sophos.com/ja-jp/sitemap.xml https://www.sophos.com/pt-br/sitemap.xml https://www.sophos.com/zh-cn/sitemap.xml # Requests for Previous Versions Disallow: /includes/ Disallow: /javascript/ Disallow: /errors/ Disallow: /sophos/docs/ Disallow: /downloads/beta/ Disallow: /virusinfo/analyses/svl.cgi Disallow: /_maintenance Disallow: /support/knowledgebase/search-results Disallow: /search/search-results Disallow: /security/analyses/ Disallow: /pressoffice/news/search-results/ Disallow: /partner_locator_down.html Disallow: /sorry.html Disallow: /emergency_errorpage.html Disallow: /e01b77678c922835290b7960d61922ca Disallow: /akamai-sureroute-test-object.htm Disallow: /companyinfo/ Disallow: /support/updates Disallow: /products/enterprise/ Disallow: /support/dif6qe2nac24zn Disallow: /opinion_poll/ Disallow: /legal/utimaco.html Disallow: /products/beta/windows7/index.html Disallow: /products/vem10683 Disallow: /support/knowledgebase/topics Disallow: /companyinfo/ Disallow: /support/knowledgebase/ Disallow: /forums/ Disallow: /users/ Disallow: /utility/ Disallow: /results.aspx Disallow: /*thank* Disallow: /thankyou.aspx Disallow: /*/thank-you.aspx DIsallow: /*/*thank* Disallow: /*/thankyou.aspx Disallow: /productdownload.aspx Disallow: /*/productdownload.aspx Disallow: /threat-center/ Disallow: /*/threat-center/ Disallow: /*ip-lookup* Disallow: /*/page/ Disallow: /page/ Disallow: /*/tag/ Disallow: /tag/ Disallow: /*/error/404/ Disallow: /*/error/404.aspx Disallow: /error/404.aspx Disallow: /error/404/ Disallow: /pp_404.aspx Disallow: /*/pp_404.aspx Disallow: /about us.html Disallow: /about .html Disallow: /about-us.html Disallow: /*/about-us.html Disallow: /*/about-us.aspx Disallow: /*/about-us/ Disallow: /about-us.aspx Disallow: /about-us/ Disallow: /blogs/ Disallow: /virusinfo/ Disallow: /pressoffice/ Disallow: /support/knowledgebase/ Disallow: /about-us/events Disallow: /virusinfo/analyses/ Disallow: /security/analyses/ Disallow: /Pages/ Disallow: /pages/ Disallow: /Sophos/ Disallow: /sophos/ Disallow: /blogs/gc/ Disallow: /virusinfo/ Disallow: /*/virusinfo/ Disallow: /en-us/index.aspx Disallow: /*/index.aspx Disallow: /en-us/index* Disallow: /*/index*.aspx Disallow: /en-us/legal/sophos-api-and-plugin-termsofuser.aspx Disallow: /*/legal/sophos-api-and-plugin-termsofuse.aspx Disallow: /en-us/legal/mdr-eap-agreement.aspx Disallow: /*/legal/mdr-eap-agreement.aspx Disallow: /legal/mdr-eap-agreement.aspx # Requested Disallow: /*.aspx?cmp=* Disallow: /*/*.aspx?cmp=* Disallow: /*/*/*.aspx?cmp=* Disallow: /*/*/*/*.aspx?cmp=* Disallow: /*/*/*/*/*.aspx?cmp=* Disallow: /en-us/content/best-hardware-firewall.aspx?cmp=41285 Disallow: /de-de/content/best-hardware-firewall.aspx?cmp=41285 Disallow: /*/*/best-hardware-firewall.aspx?cmp=41285 Disallow: /*/medialibrary/SophosNext/Images/ Disallow: /*/medialibrary/images/ Disallow: /battlecards/* Disallow: /Battlecards/* Disallow: /*/Battlecards/ Disallow: /*battlecards* Disallow: /*Battlecards* Disallow: /battlecards.aspx Disallow: /*battlecards.aspx Disallow: /PDFs/partners/* Disallow: /PartnerPortal* Disallow: /partner-portal Disallow: /*Battlecards* Disallow: /*battlecards* Disallow: /en-us/medialibrary/PDFs/notindexed/ Disallow: /*/medialibrary/PDFs/notindexed/ Disallow: /*/medialibrary/PDFs/notindexed Disallow: /medialibrary/PDFs/notindexed Disallow: /*notindexed* Disallow: /*not-indexed* Disallow: /*/security-news-trends/ Disallow: /security-news-trends/ Disallow: /Microsites/ Disallow: /microsites/ Disallow: /en-us/microsites/ Disallow: /en-us/Microsites/ Disallow: /microsite/ Disallow: /*/microsite/ Disallow: /Microsite/ Disallow: /*/Microsite/ Disallow: /press-office/ Disallow: /*/press-office/ Disallow: /pressoffice/ Disallow: /*/pressoffice/ # Translated Removals Disallow: /ko-kr* Disallow: /zh-tw* Disallow: /tr-tr* Disallow: /cs-cz* Disallow: /hi-in* Disallow: /hu-hu* Disallow: /pl-pl* Disallow: /ko-kr/ Disallow: /zh-tw/ Disallow: /tr-tr/ Disallow: /cs-cz/ Disallow: /hi-in/ Disallow: /pl-pl/ Disallow: /zh-tw/ Disallow: /yy-XX/ Disallow: /yy-XX* Disallow: /yy-xx* Disallow: /yy-xx/ Disallow: /yy-xx/* # KB Disallow: /support/knowledgebase/ Disallow: /*/support/knowledgebase/ Disallow: /support/kb/ Disallow: /*/support/kb/ Disallow: /support/knowledgebase* Disallow: /support/kb* Disallow: /*/support/knowledgebase* Disallow: /*/support/kb* Disallow: /support/cleaners/ Disallow: /*/support/ # Request Regional Migrations Disallow: /*/legacy-fr/ Disallow: /legacy-fr/ Disallow: /*/legacy-de/ Disallow: /legacy-de/ Disallow: /*/legacy-es/ Disallow: /legacy-es/ Disallow: /*/legacy-it/ Disallow: /legacy-it/ Disallow: /*/legacy-cn/ Disallow: /legacy-cn/ Disallow: /*/legacy-ja/ Disallow: /legacy-ja/ Disallow: /legacy-fr/ Disallow: /legacy-de/ Disallow: /legacy-it/ Disallow: /legacy-es/ Disallow: /*legacy* Disallow: /*/*legacy* Disallow: /*/legacy* Disallow: /*legacy* # Requests Disallow: /security/analyses/ Disallow: /*shared-download-components* Disallow: /shared-download-components/ Disallow: /Pages/DownloadRedirect.aspx Disallow: /*/Pages/ Disallow: /*/pages/ Disallow: /downloads/ide/ Disallow: /downloads/ide Disallow: /*/downloads/ide/ Disallow: /ide/* Disallow: /*/ide/ Disallow: /companyinfo/careers/ Disallow: /security/analyses/ Disallow: /mysophos/loggedout Disallow: /mysophos/loggedout* Disallow: /*loggedout* Disallow: /forgotten-password.aspx Disallow: /en-us/threat-analyses/ Disallow: /*utm_source* Disallow: /*utm_medium* Disallow: /*utm_campaign* Disallow: /*id=* Disallow: /*404* Disallow: /pressoffice/ Disallow: /*.do$ Disallow: /*.ru$ Disallow: /*.dll$ Disallow: /?ID=* Disallow: /sitecore/ Disallow: /Sitecore/ Disallow: /console/ Disallow: /console/* Disallow: /console* Disallow: /Console/ Disallow: /layouts/* Disallow: /*layouts* Disallow: /layouts* Disallow: /layouts/ Disallow: /Layouts/ Disallow: /testing/ Disallow: /*testing* Disallow: /testing/* Disallow: /testing* Disallow: /Testing/ Disallow: /Testing/* Disallow: /testing_dev/* Disallow: /testing_dev* Disallow: /testing_dev Disallow: /testing_dev* Disallow: /Testing_dev/* Disallow: /modules/ Disallow: /modules/* Disallow: /Modules/ Disallow: /Modules/* Disallow: /*landing* Disallow: /*/microsites/ Disallow: /sophosid.aspx Disallow: /*/sophosid.aspx Disallow: /sophosid/ Disallow: /*/sophosid/ Disallow: /*?q=* Disallow: /_ExcludeFromDeployment Disallow: /*_ExcludeFromDeployment* Disallow: /_ExcludeFromDeployment/ Disallow: /_ExcludeFromDeployment/* Disallow: /publish/ Disallow: /*publish* Disallow: /publish/* Disallow: /*/publish/ Disallow: /*/Publish/ Disallow: /sophos/reports/* Disallow: /*/sophos/reports/ Disallow: /layouts/ Disallow: /admin/ Disallow: /admin/* Disallow: /admin* Disallow: /translation/ Disallow: /*/translation/ Disallow: /Translation/ Disallow: /*/Translation/ Disallow: /thank-you* Disallow: /thankyou* Disallow: /thank-you Disallow: /thank-you.aspx Disallow: /thankyou.aspx Disallow: /*thank* Disallow: /productdownload.aspx Disallow: /*/productdownload.aspx Disallow: /*/productdownload* Disallow: /productdownload* Disallow: /formbuilder.aspx Disallow: /*formbuilder* Disallow: /formbuilder.aspx* Disallow: /formbuilder* Disallow: /test/ Disallow: /test/* Disallow: /Redirects/* Disallow: /redirects/* Disallow: /downloads/ide/ Disallow: /downloads/ide/* Disallow: /EntityInfo Disallow: /Enduser Protection Suites/ Disallow: /*/enduser protection suites/ Disallow: /Enduser Protection Suites/* Disallow: /enduser protection suites.aspx Disallow: /endpoint-protection-comparison.aspx Disallow: /*/enduser protection comparison/ Disallow: /*/Enduser Protection Suites/ Disallow: /*/Enduser Protection Suites/* Disallow: /*/enduser protection suites.aspx Disallow: /*/endpoint-protection-comparison.aspx Disallow: /*item=* Disallow: /*topics* Disallow: /about-us/ Disallow: /*/about-us.aspx Disallow: /about-us.aspx Disallow: /*/error.aspx Disallow: /error/ Disallow: /*/error/ Disallow: /error* Disallow: /error/404.aspx Disallow: /*forgot* Disallow: /*/Sophos/ Disallow: /*/sophos/ Disallow: /*/Sophos/Security News Trends/ Disallow: /*/sophos/security news trends/ Disallow: /link.aspx Disallow: /link.aspx* Disallow: /*/link.aspx Disallow: /*/link.aspx* Disallow: /*/products/choose-email.aspx Disallow: /products/choose-email.aspx Disallow: /products/utm-9/ Disallow: /*/products/utm-9/ Disallow: /*/blogs/gc/ Disallow: /blogs/gc/ Disallow: /*/virusinfo/analyses/ Disallow: /virusinfo/analyses/ Disallow: /*/products/enterprise/ Disallow: /products/enterprise/ Disallow: /support/faqs/ Disallow: /*/support/faqs/ Disallow: /*/CompeteSecurityDay/ Disallow: /CompeteSecurityDay/ Disallow: /*/content/offers/ Disallow: /content/offers/ Disallow: /content/cpc/ Disallow: /*/content/cpc/ Disallow: /en-us/products/cloud-.aspx Disallow: /*/products/cloud-.aspx Disallow: /*/invincea.aspx Disallow: /*/partners/partnerlocator* Disallow: /partners/partnerlocator* Disallow: /en-us/partners/partnerlocator.html Disallow: /*/partners/partnerlocator.aspx Disallow: /partners/partnerlocator.aspx Disallow: /en-us/partners/partnerlocator.html Disallow: /*/products/next-gen-firewall/firewall-manager* Disallow: /*/products/next-gen-firewall/firewall-manager.aspx Disallow: /en-us/products/next-gen-firewall/firewall-manager.aspx Disallow: /*/products/next-gen-firewall/firewall-manager.aspx* # ML Disallow: /medialibrary/ Disallow: /*/medialibrary/ Disallow: /en-us/medialibrary/ Disallow: /ja-jp/medialibrary/ Disallow: /fr-fr/medialibrary/ Disallow: /de-de/medialibrary/ Disallow: /es-es/medialibrary/ Disallow: /it-it/medialibrary/ Disallow: /PDFs/ Disallow: /pdfs/ Disallow: /pdf/ Disallow: /*/PDFs/ Disallow: /*/pdfs/ Disallow: /lp/ Disallow: /*/lp/ Disallow: /en-us/lp/ # Search Disallow: /search-results.aspx Disallow: /*/search-results.aspx # Sophos Home Microsites Disallow: /sophos-home-education/ Disallow: /*/sophos-home-education/ Disallow: /sophos-home-education.aspx Disallow: /*/sophos-home-education.aspx Disallow: /*/discover.aspx Disallow: /discover.aspx Disallow: /*/discover/ Disallow: /*/hitmanpro/ Disallow: /*/hitmanpro.aspx Disallow: /*/home.aspx Disallow: /*/home/ # New Requests Disallow: /*/content/cpc/ Disallow: /en-us/content/cpc/ Allow: /en-us/content/ Allow: /*/content/threat-hunting.aspx Disallow: /content/cpc/ Disallow: /content/seo-content-template/ Disallow: /*/content/seo-content-template/ Disallow: /*malarkey* Disallow: /*malarky* Allow: /en-us/lp/ Disallow: /pl-pl/ Disallow: /tr-tr/ Disallow: /ko-kr/ Disallow: /cs-cz/ Disallow: /hu-hu/ Disallow: /zh-tw/ Disallow: /pt-br/lp/ Disallow: /*/about us.html Disallow: /about us.html Disallow: /*/link/ Disallow: /en/* Disallow: /ja-jp/medialibrary/PDFs/japan/support-documents/ Disallow: /*initial* Disallow: /*/brand-guidelines Disallow: /*/brand-guidelines/ Disallow: /*downloadKey* Disallow: /*dl=true$ Disallow: /en-us/_.aspx$ Disallow: /_$ Disallow: /en-us/_$ Disallow: /*/_$ Disallow: /*_.aspx$ Disallow: /en-us/*_.aspx$ Disallow: /*cmp= Disallow: /*subId= Disallow: /*CMP= Disallow: /*cmp Disallow: /*affiliate= Disallow: /*subid= Disallow: /*utility items Disallow: /*utility-items Disallow: /en-us/index.aspx Disallow: /*/index.aspx Disallow: /*/index2.aspx Disallow: /*/index3.aspx # Cleanup Disallow: /threat-center/ Disallow: /*/threat-center/ Disallow: /*/threat-center/* Disallow: /*//threat-center/* Disallow: /*//threat-center/threat-analyses Disallow: /en-us//threat-center/threat-analyses Disallow: /en-us//threat-center/ Disallow: /*//threat-center/ # Bot Requests User-agent: MSNBot-Media Disallow: / User-agent: msnbot-media Disallow: / User-agent: AdIdxBot Disallow: / User-agent: adidxbot Disallow: / User-agent: dotbot Disallow: / User-agent: moget/2.1 Disallow: / User-agent: asterias Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: Black Hole Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BotALot Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: CheeseBot Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: DittoSpyder Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: EroCrawler Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Foobot Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: hloader Disallow: / User-agent: httplib Disallow: / User-agent: humanlinks Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: JennyBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: LexiBot Disallow: / User-agent: libWeb/clsHTTP Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: Mata Hari Disallow: / User-agent: Microsoft URL Control - 5.01.4511 Disallow: / User-agent: Microsoft URL Control - 6.00.8169 Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Mister PiX Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: mozilla/4 Disallow: / User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME) Disallow: / User-agent: mozilla/5 Disallow: / User-agent: NetAnts Disallow: / User-agent: NICErsPRO Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RMA Disallow: / User-agent: SiteSnagger Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: suzuran Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: Telesoft Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: URLy Warning Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: WebCopier Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: WebSauger Disallow: / User-agent: Website Quester Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebZip Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: YandexBot Disallow: / User-agent: YandexDirect Disallow: / User-agent: YandexBlogs Disallow: / User-agent: YaDirectFetcher Disallow: / User-agent: YandexNews Disallow: / User-agent: YandexPagechecker Disallow: / User-agent: YandexMetrika Disallow: / User-agent: YandexMedia Disallow: / User-agent: Slurp Disallow: / User-agent: Yahoo Pipes 1.0 Disallow: / User-agent: Sophos.co.at/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+.NET+CLR+1.1.4322) Disallow: / User-agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) Disallow: / User-agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) Disallow: /*search* Sitemap: https://www.sophos.com/sitemap.xml
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-27 | ok 200 | 332cae6d6218 |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | www.sophos.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2025-09-15 CC-MAIN-2025-38 | www.sophos.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2024-09-16 CC-MAIN-2024-38 | www.sophos.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2023-10-02 CC-MAIN-2023-40 | www.sophos.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2022-10-03 CC-MAIN-2022-40 | www.sophos.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2021-09-27 CC-MAIN-2021-39 | www.sophos.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2020-09-28 CC-MAIN-2020-40 | www.sophos.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2019-09-23 CC-MAIN-2019-39 | www.sophos.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2018-09-24 CC-MAIN-2018-39 | www.sophos.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2017-09-25 CC-MAIN-2017-39 | sophos.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2017-07-24 CC-MAIN-2017-30 | www.sophos.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-04-24 CC-MAIN-2017-17 | sophos.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2017-01-23 CC-MAIN-2017-04 | sophos.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2016-10-03 CC-MAIN-2016-40 | www.sophos.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2016-09-05 CC-MAIN-2016-36 | www.sophos.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
Evidence
Every daily observation of sophos.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).
API: https://infrarabbit.com/crawlers/api/v1/policy/sophos.com