dell.com: what it tells AI crawlers
On 28 October 2026, dell.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot; it also published llms.txt.
Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://www.dell.com/robots.txt
unchanged since 2026-10-26 · sha25666cd4b30830097a4…· view as recorded - The * group
- partial applies to every agent without its own group
Per AI bot, 2026-10-28
| User agent | Verdict | How |
|---|---|---|
| GPTBot | partial | through the * group |
| OAI-SearchBot | partial | through the * group |
| ChatGPT-User | partial | through the * group |
| ClaudeBot | partial | through the * group |
| Claude-SearchBot | partial | through the * group |
| Claude-User | partial | through the * group |
| PerplexityBot | partial | through the * group |
| Perplexity-User | partial | through the * group |
| Googlebot | partial | through the * group |
| Google-Extended | partial | through the * group |
| Applebot-Extended | partial | through the * group |
| CCBot | partial | through the * group |
| Bytespider | allows | named in its own group |
| Meta-ExternalAgent | partial | through the * group |
| Amazonbot | partial | through the * group |
Other signals
| Publishes /llms.txt | yes |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | no |
| Declares Content Signals (robots.txt or header) | no |
| Says ai-train=no in a Content Signal | no |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
llms.txt: "Dell Technologies", 90,245 bytes, 284 links.
robots.txt as recorded sha256 0b7673eddc3306ac…
User-agent: * Disallow: /content/products/compare.aspx/cloudedge?c=us&l=en&s=biz Disallow: /koa/search Disallow: */identity Disallow: */survey.aspx Disallow: */serviceselection Disallow: */Identity Disallow: */Serviceselection Disallow: /en-ai Disallow: /en-ap Disallow: /en-ba Disallow: /en-bg Disallow: /en-ch Disallow: /en-de Disallow: /en-dm Disallow: /en-ed Disallow: /en-ee Disallow: /en-gy Disallow: /en-hr Disallow: /en-is Disallow: /en-jp Disallow: /en-kr Disallow: /en-lt Disallow: /en-mk Disallow: /en-si Disallow: /en-th Disallow: /en-yu Disallow: /fr-uk Disallow: /fr-us Disallow: /en-br Disallow: /en-fr Disallow: /en-dk Disallow: /en-es Disallow: /en-cn Disallow: /en-be Disallow: /en-cd Disallow: /en-mx Disallow: /en-nl Allow: /*/compare.aspx*s=biz* Allow: /*/compare.aspx*s=hied* Allow: /*/compare.aspx*s=hea* Allow: /*/compare.aspx*s=fed* Allow: /*/compare.aspx*s=slg* Allow: /*.js$ Allow: /*.css$ Allow: /support/home/*/*/*/product-support/servicetag Allow: /*/compare.aspx*s=k12* Disallow: */api/ Allow: /api/feeds.svc/rss/newsroom-press-releases/ Disallow: /*cs=45 Disallow: /*cs=04sp1 Disallow: /*cs=04sp2 Disallow: /*cs=04sp3 Disallow: /*cs=04sp4 Disallow: /*cs=04sp5 Disallow: /*cs=04sp6 Disallow: /*cs=04sp7 Disallow: /*cs=04sp8 Disallow: /*cs=04sp9 Disallow: /*cs=04sp10 Disallow: /*cs=04sp11 Disallow: /*cs=04sp12 Disallow: /*cs=04sp13 Disallow: /*cs=04sp14 Disallow: /*cs=04sp15 Disallow: /*cs=04sp16 Disallow: /*cs=04SP1 Disallow: /*cs=04SP2 Disallow: /*cs=04SP3 Disallow: /*cs=04SP4 Disallow: /*cs=04SP5 Disallow: /*cs=04SP6 Disallow: /*cs=04SP7 Disallow: /*cs=04SP8 Disallow: /*cs=04SP9 Disallow: /*cs=04SP10 Disallow: /*cs=04SP11 Disallow: /*cs=04SP12 Disallow: /*cs=04SP13 Disallow: /*cs=04SP14 Disallow: /*cs=04SP15 Disallow: /*cs=04SP16 Disallow: /*cs=OWR08 Disallow: /*cs=08 Disallow: /*cs=owr08 Disallow: /*~f=lg Disallow: /support/home/cassette.axd/* Disallow: /*/compare.aspx Disallow: /*/menu.aspx Disallow: /*/superview.aspx Disallow: /*/productdetails.aspx*s=gen* Disallow: /*/productdetail.aspx*s=gen* Disallow: /*/pd.aspx*s=corp* Disallow: /*/productdetails.aspx*s=corp* Disallow: /*/productdetail.aspx*s=corp* Disallow: /*/cp.aspx*s=corp* Disallow: /*/ct.aspx*s=corp* Disallow: /*/pd.aspx*s=gen* Disallow: /*/cp.aspx*s=gen* Disallow: /*/ct.aspx*s=gen* Disallow: /*/pd.aspx*s=dfo* Disallow: /*/productdetails.aspx*s=dfo* Disallow: /*/productdetail.aspx*s=dfo* Disallow: /*/cp.aspx*s=dfo* Disallow: /*/ct.aspx*s=dfo* Disallow: /*/category.aspx*s=dfo* Disallow: /*/compare.aspx*s=dfo* Disallow: /*/category.aspx*s=gen* Disallow: /*/category.aspx*s=corp* Disallow: /*/brand.aspx*s=corp* Disallow: /*/brand.aspx*s=gen* Disallow: /*/brand.aspx*s=dfo* Disallow: /*/popproductdetails.aspx*s=gen* Disallow: /*/popproductdetail.aspx*s=gen* Disallow: /*/popproductdetails.aspx*s=corp* Disallow: /*/popproductdetail.aspx*s=corp* Disallow: /*/popproductdetails.aspx*s=dfo* Disallow: /*/popproductdetail.aspx*s=dfo* Disallow: /content/topics/segtopic.aspx/contact Disallow: /br/empresa/p/test_buddy_shopno_page_waterk Disallow: /br/p/test_buddy_shopno_page_waterx Disallow: /fr/p/inspiron-duo/pd Disallow: /uk/business/p/*/fs$ Disallow: /uk/p/*/fs$ Disallow: /uk/p/shipsfast-deals?cs=ukdhs1 Disallow: /nl/bedrijven/p/*/fs$ Disallow: /nl/p/*/fs$ Disallow: /us/p/alienware-laptops Disallow: /support/Diagnostics/*/DiagnosticHome/Index/ Disallow: /support/Diagnostics/*/DiagnosticSystemInfo/Index/ Disallow: /support/Diagnostics/*/DiagnosticResultHistory/Index Disallow: /support/Diagnostics/*/DiagnosticResults/ShowResults/ Disallow: /support/Diagnostics/*/Dispatch/ValidateOwner Disallow: /support/Diagnostics/*/Dispatch/ShippingInfo Disallow: /support/Diagnostics/*/Dispatch/ConfirmShippingInfo Disallow: /support/Diagnostics/*/Dispatch/ConfirmDispatch Disallow: /support/my-support/*/*/*/*/servicetag/ Disallow: /support/contents/us/en/19/article/Contact-Information/International-Support-Services/intl-support-services Disallow: /learn/*/*/*/mse-assets/navigation-diagnose Disallow: /support/orderstatus/*/*/*/OrderStatus/* Disallow: /support/home/*/*/*/*/servicetag/* Disallow: /support/home/*/*/*/Diagnostics/* Disallow: /support/home/*/*/*/Drivers/DownloadList/* Disallow: /support/home/*/*/*/Drivers/DriversList/* Disallow: /support/home/*/*/*/Drivers/EmailSubscription/* Disallow: /support/home/*/*/*/Drivers/OneClick/* Disallow: /learn/us/en/rc968571/ Disallow: /*/cart/* Disallow: /*/buy/* Disallow: */csbapi/ Disallow: /us/partner/* Disallow: /us/*/chn/* Disallow: /us/partners/* Disallow: /us/en/partner/* Disallow: /*/ed/* Disallow: /*?c=ed* Disallow: /*&c=ed* Disallow: /*?*svcTag=5MGSLL1* Disallow: /support/manuals/*/*/*/flyout/ Disallow: /support/search/ Disallow: /support/Search/ Disallow: /support/csbapi/ Disallow: /*addScript$ Disallow: *t:cp= Disallow: *t:ac= Disallow: /support/clientconnection/ Disallow: /support/*/tab_mob/ Allow: /ed/p/$ Allow: /ed/fr/p/$ Disallow: /ed/p/ Disallow: /ed/fr/p/ Allow: /ed/business/p/$ Allow: /ed/fr/entreprise/p/$ Disallow: /ed/business/p/* Disallow: /ed/fr/entreprise/p/* Disallow: /support/*/ServiceTag/* Disallow: /*/sr/*?appliedRefinements= Disallow: /*?bvstate= Disallow: /*appliedRefinements= Disallow: /*appliedrefinements= Disallow: /*styles/index.less Disallow: /*quickview-template.html Disallow: /*quickview User-agent: Baiduspider Disallow: / Allow: /zh-cn/ Allow: /cn/ Allow: /tw/ Allow: /support/*/cn/zh/ Allow: /support/*/tw/zh/ Allow: /support/*/hk/zh/ Allow: /learn/hk/zh/ Allow: /learn/tw/zh/ Allow: /*&c=cn&* Allow: /*?c=cn&* Allow: /*&c=tw&* Allow: /*?c=tw&* Allow: /*&c=hk&* Allow: /*?c=hk&* User-agent: Sogou web spider Disallow: / Allow: /zh-cn/ Allow: /cn/ Allow: /tw/ Allow: /support/*/cn/zh/ Allow: /support/*/tw/zh/ Allow: /support/*/hk/zh/ Allow: /learn/hk/zh/ Allow: /learn/tw/zh/ Allow: /*&c=cn&* Allow: /*?c=cn&* Allow: /*&c=tw&* Allow: /*?c=tw&* Allow: /*&c=hk&* Allow: /*?c=hk& User-agent: Sogou inst spider Disallow: / Allow: /zh-cn/ Allow: /cn/ Allow: /tw/ Allow: /support/*/cn/zh/ Allow: /support/*/tw/zh/ Allow: /support/*/hk/zh/ Allow: /learn/hk/zh/ Allow: /learn/tw/zh/ Allow: /*&c=cn&* Allow: /*?c=cn&* Allow: /*&c=tw&* Allow: /*?c=tw&* Allow: /*&c=hk&* Allow: /*?c=hk& User-agent: Sogou spider2 Disallow: / Allow: /zh-cn/ Allow: /cn/ Allow: /tw/ Allow: /support/*/cn/zh/ Allow: /support/*/tw/zh/ Allow: /support/*/hk/zh/ Allow: /learn/hk/zh/ Allow: /learn/tw/zh/ Allow: /*&c=cn&* Allow: /*?c=cn&* Allow: /*&c=tw&* Allow: /*?c=tw&* Allow: /*&c=hk&* Allow: /*?c=hk& User-agent: HaoSouSpider Disallow: / Allow: /zh-cn/ Allow: /cn/ Allow: /tw/ Allow: /support/*/cn/zh/ Allow: /support/*/tw/zh/ Allow: /support/*/hk/zh/ Allow: /learn/hk/zh/ Allow: /learn/tw/zh/ Allow: /*&c=cn&* Allow: /*?c=cn&* Allow: /*&c=tw&* Allow: /*?c=tw&* Allow: /*&c=hk&* Allow: /*?c=hk& Sitemap: https://www.dell.com/index-content-href-sitemap.xml.gz Sitemap: https://www.dell.com/deals-href-index-sitemap.xml.gz Sitemap: https://www.dell.com/index-product-pages-hreflang-sitemap.xml.gz Sitemap: https://www.dell.com/index-sitemap.xml.gz Sitemap: https://www.dell.com/index-snp-apj-sitemap.xml.gz Sitemap: https://www.dell.com/index-support-href-sitemap.xml.gz Sitemap: https://www.dell.com/index-video-xml-sitemap.xml.gz # Last Modified on 23 Sept 2019
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-28 | ok 200 | 66cd4b308300 |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | www.dell.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2025-09-15 CC-MAIN-2025-38 | www.dell.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2024-09-16 CC-MAIN-2024-38 | www1.dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2023-10-02 CC-MAIN-2023-40 | www.dell.com | ok view | partial | partial | unknown | partial | partial | unknown | partial |
| 2022-10-03 CC-MAIN-2022-40 | www2.dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | www.dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | www1.dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | www.dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | www1.dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-07-23 CC-MAIN-2018-30 | www1.dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-04-23 CC-MAIN-2018-17 | www2.dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-01-29 CC-MAIN-2018-05 | www2.dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-10-23 CC-MAIN-2017-43 | www2.dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | www1.dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-07-24 CC-MAIN-2017-30 | www1.dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-04-24 CC-MAIN-2017-17 | dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-01-23 CC-MAIN-2017-04 | dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-10-03 CC-MAIN-2016-40 | dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-09-05 CC-MAIN-2016-36 | dell.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
Evidence
Every daily observation of dell.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).
API: https://infrarabbit.com/crawlers/api/v1/policy/dell.com