amazonaws.com: what it tells AI crawlers
On 28 October 2026, amazonaws.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.
Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://aws.amazon.com:443/robots.txt
unchanged since 2026-10-26 · sha256c1f42540a70f0dcd…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-28
| User agent | Verdict | How |
|---|---|---|
| GPTBot | allows | through the * group |
| OAI-SearchBot | allows | through the * group |
| ChatGPT-User | allows | through the * group |
| ClaudeBot | allows | through the * group |
| Claude-SearchBot | allows | through the * group |
| Claude-User | allows | through the * group |
| PerplexityBot | allows | through the * group |
| Perplexity-User | allows | through the * group |
| Googlebot | allows | through the * group |
| Google-Extended | allows | through the * group |
| Applebot-Extended | allows | through the * group |
| CCBot | allows | through the * group |
| Bytespider | allows | through the * group |
| Meta-ExternalAgent | allows | through the * group |
| Amazonbot | allows | through the * group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | not observed |
| Declares Content Signals (robots.txt or header) | not observed |
| Says ai-train=no in a Content Signal | not observed |
| Points to an RSL licence (robots.txt License line or Link header) | not observed |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | not observed |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | not observed |
| Sends noai / noimageai directives | not observed |
| Refused our fetcher (403 / challenge) | no |
robots.txt as recorded sha256 c1f42540a70f0dcd…
User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile Allow: /pm/ Disallow: /alexaforbusiness/ Disallow: /blogs/ Disallow: /*/blogs/ Disallow: /solutions/case-studies/ Disallow: /*/solutions/case-studies/ Allow: /solutions/case-studies/sustainability-emea/ Allow: /*/solutions/case-studies/sustainability-emea/ Allow: /solutions/case-studies/innovators/innovators-eu/ Allow: /*/solutions/case-studies/innovators/innovators-eu/ Allow: /solutions/case-studies/generative-ai-ekacare/ Allow: /cn/solutions/case-studies/generative-ai-ekacare/ Allow: /solutions/case-studies/generative-ai-mindtickle/ Allow: /cn/solutions/case-studies/generative-ai-mindtickle/ Allow: /solutions/case-studies/generative-ai-invideo/ Allow: /solutions/case-studies/generative-ai-mitsui/ Allow: /solutions/case-studies/generative-ai-national-australia-bank/ Allow: /solutions/case-studies/generative-ai-ntu/ Allow: /solutions/case-studies/generative-ai-qarc/ Allow: /solutions/case-studies/generative-ai-swimming-australia/ Allow: /cn/solutions/case-studies/generative-ai-swimming-australia/ Allow: /solutions/case-studies/generative-ai-thematic/ Allow: /cn/solutions/case-studies/generative-ai-thematic/ Allow: /solutions/case-studies/generative-ai-turing/ Allow: /solutions/case-studies/generative-ai-stockmark/ Allow: /*/solutions/case-studies/generative-ai-turing/ Allow: /*/solutions/case-studies/generative-ai-stockmark/ Allow: /solutions/case-studies/generative-ai-starhub/ Allow: /solutions/case-studies/generative-ai-bolttech/ Allow: /*/solutions/case-studies/generative-ai-starhub/ Allow: /*/solutions/case-studies/generative-ai-bolttech/ User-agent: * # Block old What's New posts to preserve crawl budget for current content Disallow: /about-aws/whats-new/200 Disallow: /about-aws/whats-new/201 Disallow: /about-aws/whats-new/2020 Disallow: /about-aws/whats-new/2021 Disallow: /about-aws/whats-new/2022 Disallow: /about-aws/whats-new/2023 Disallow: /about-aws/whats-new/2024 # Block tag Disallow: /*/blogs/*/tag/ Disallow: /blogs/*/tag/ # Other Disallow: */confirmation/ Disallow: */registration-confirmation/ Disallow: /about-aws/launch-a-website-in-minutes-tutorial/ Disallow: /about-aws/media-coverage/2011/08/17/rhat-support-for-sap-solutions-running-on-rhel-on-amazon-ec2/ Disallow: /activate/accelerators Disallow: /activate/accelerators/* Disallow: /activate/accelerators/confirmation/ Disallow: /activate/china-portfolio-signup/ Disallow: /activate/cloudkata/ Disallow: /activate/event/ Disallow: /activate/event/* Disallow: /activate/events/* Disallow: /activate/events/hact/ Disallow: /activate/hackathons Disallow: /activate/hackathons/* Disallow: /activate/harvard-innovation-lab-students-2013/ Disallow: /activate/harvard-innovation-lab-vip-teams-2013/ Disallow: /activate/hsw/confirmation/ Disallow: /activate/loft/ Disallow: /activate/self-starters/confirmation/ Disallow: /activate/wdm/* Disallow: /alexa/* Disallow: /alexa-top-sites/* Disallow: /alexa-web-search/* Disallow: /apac/aws-credits-confirmation/ Disallow: /apac/aws-credits-request-confirmation/ Disallow: /articles/item/ Disallow: /awis/* Disallow: /aws-summit-2014/enterprise-london-db/ Disallow: /aws-summit-2014/london-db/ Disallow: /blogs/7faebe5debfdd358a46f2452a54eec94fce49269/ Disallow: /campaigns/infor-hub-xt5r5iwdrgxbnuz/ Disallow: /careers/aus-nz-event/ Disallow: /careers/aus-nz-event/ Disallow: /careers/aws-ar-event/ Disallow: /careers/aws-aus-event/ Disallow: /careers/aws-australia-event/ Disallow: /careers/aws-bigdata-hiring/ Disallow: /careers/aws-brisbane-event/ Disallow: /careers/aws-db-event/ Disallow: /careers/aws-ks-event/ Disallow: /careers/aws-london-event/ Disallow: /careers/aws-mexicocity-event/ Disallow: /careers/aws-miami-event/ Disallow: /careers/aws-ny-event/ Disallow: /careers/aws-pittsburgh-event/ Disallow: /careers/aws-portland-event/ Disallow: /careers/aws-sea-event/ Disallow: /careers/aws-singapore-event/ Disallow: /careers/aws-sydney-event/ Disallow: /careers/chattanooga-mobile-event/ Disallow: /careers/database-hiring/ Disallow: /careers/devtools-australia-event/ Disallow: /careers/devtools-australia-event/ Disallow: /careers/ebs-cincinnati-event/ Disallow: /careers/ec2-australia-event/ Disallow: /careers/ec2-chicago-event/ Disallow: /careers/ec2-hiring-events/ Disallow: /careers/ec2-ottawa-event/ Disallow: /careers/ec2-vancouver-event/ Disallow: /careers/indianapolis-event/ Disallow: /careers/mexico-edge-hiring-2014/ Disallow: /careers/milwaukee-edge-hiring-2014/ Disallow: /careers/montreal-edge-hiring/ Disallow: /careers/Phoenix-Hiring-Event/ Disallow: /careers/pittsburgh-platform-hiring-2014/ Disallow: /careers/raleigh-mobile-event/ Disallow: /careers/raleigh-storage-hiring-2014/ Disallow: /careers/santiago-event/ Disallow: /careers/Sydney-Hiring-Event/ Disallow: /careers/texas-event/ Disallow: /careers/toronto-event/ Disallow: /careers/toronto-storage-hiring-2014/ Disallow: /Cloud_Start_01.LandingPage.html Disallow: /Cloud_Start_01-Landing-Page-EDO.html Disallow: /Cloud_Start_02.ConfirmationPage.html Disallow: /code/* Disallow: /compliance/contact-us-confirmation/ Disallow: /compliance/hipaa-eligible-services-reference/ Disallow: /compliance/how-to-open-aws-compliance-reports/ Disallow: /compliance/public-sector-contact-confirmation/ Disallow: /compliance/security-assurance-faq Disallow: /compliance/security-assurance-faq-1 Disallow: /compliance/security-assurance-faq-10 Disallow: /compliance/security-assurance-faq-2 Disallow: /compliance/security-assurance-faq-3 Disallow: /compliance/security-assurance-faq-4 Disallow: /compliance/security-assurance-faq-5 Disallow: /compliance/security-assurance-faq-6 Disallow: /compliance/security-assurance-faq-7 Disallow: /compliance/security-assurance-faq-8 Disallow: /compliance/security-assurance-faq-9 Disallow: /compliance/security-assurance-faq-complete Disallow: /compliance/security-assurance-US-Public-Sector-checklist Disallow: /compliance/security-assurance-US-Public-Sector-checklist-1 Disallow: /compliance/security-assurance-US-Public-Sector-checklist-10 Disallow: /compliance/security-assurance-US-Public-Sector-checklist-2 Disallow: /compliance/security-assurance-US-Public-Sector-checklist-3 Disallow: /compliance/security-assurance-US-Public-Sector-checklist-4 Disallow: /compliance/security-assurance-US-Public-Sector-checklist-5 Disallow: /compliance/security-assurance-US-Public-Sector-checklist-6 Disallow: /compliance/security-assurance-US-Public-Sector-checklist-7 Disallow: /compliance/security-assurance-US-Public-Sector-checklist-8 Disallow: /compliance/security-assurance-US-Public-Sector-checklist-9 Disallow: /compliance/security-assurance-US-Public-Sector-checklist-complete Disallow: /compliance/third-party-access/ Disallow: /contact-us/ami-submission-conf/ Disallow: /contact-us/article-submission-conf/ Disallow: /contact-us/case-contact-confirmation/ Disallow: /contact-us/customer-app-submission-conf/ Disallow: /contact-us/developertool-submission-conf/ Disallow: /contact-us/ec2-email-limit-request-confirmation/ Disallow: /contact-us/elasticache-node-limit-request-confirmation/ Disallow: /contact-us/marketing-unsubscribe-confirmation/ Disallow: /contact-us/publicdataset-conf/ Disallow: /contact-us/sales-contact-confirmation/ Disallow: /contact-us/samplecode-submission-conf/ Disallow: /csds/* Disallow: /customerapps/* Disallow: /datasets/ Disallow: /developertools/2555753788650372 Disallow: /documentation/alexa-top-sites/ Disallow: /documentation/awis/ Disallow: /email-preferences Disallow: /email-preferences/ Disallow: /events/applesummit Disallow: /events/cio-advisory-council/ Disallow: /events/finserv-summit-dublin Disallow: /events/finserv-summit-dublin/confirmation Disallow: /events/finserv-summit-singapore Disallow: /events/finserv-summit-singapore/confirmation Disallow: /events/fsi-cloud-symposium-nyc.html Disallow: /federal/events/earthobservationday2015/ Disallow: /federal/events/earthobservationday2015/ConfirmationPage/ Disallow: /forms/activate-support Disallow: /free/cloud-computing-free-tier/ Disallow: /free/databases-free-tier/ Disallow: /get-cloud/ Disallow: /getting-started/functional-groups/ Disallow: /getting-started/functional-groups/* Disallow: /government-education/nonprofits/aws-serve/ Disallow: /government-education/nonprofits/aws-serve-member/ Disallow: /government-education/nonprofits/credit-program/ Disallow: /government-education/nonprofits/credit-program/confirmation Disallow: /government-education/nonprofits/credit-program/terms Disallow: /government-education/nonprofits/grant-program/ Disallow: /government-education/nonprofits/grant-program/confirmation/ Disallow: /government-education/nonprofits/grant-program/terms/ Disallow: /government-education/nonprofits/leadership-forum-poc Disallow: /government-education/nonprofits/leadership-forum-poc/confirmation Disallow: /government-education/public-sector-cloud-start/public-sector-cloud-start/ Disallow: /government-education/public-sector-cloud-start/public-sector-cloud-start1/ Disallow: /hackathons/reinvent-2013/ Disallow: /legal/3874403/ Disallow: /lms/contact-us/ Disallow: /loft/ Disallow: /marketplace/reviews/reviews-list/*/write-comment/ Disallow: /marketplace/reviews/write-review/ Disallow: /marketplace/search* Disallow: /microsoft/bizspark/ Disallow: /microsoft/bizspark/signon/thankyou/ Disallow: /nav/ Disallow: /nav/side-nav/ Disallow: /nav/side-nav/products-services/ Disallow: /nav/tabs/ Disallow: /nav/tabs/account/ Disallow: /nav/tabs/aws/ Disallow: /nav/tabs/community/ Disallow: /nav/tabs/developers/ Disallow: /nav/tabs/products/ Disallow: /nav/tabs/support/ Disallow: /optin/ Disallow: /optout/ Disallow: /partners/contact-us/partners-confirmation/ Disallow: /partners/overview/partner-newsletter/confirmation/ Disallow: /partners/overview/partner-terms-and-conditions/ Disallow: /partners/webinars/request/ Disallow: /ping Disallow: /pm/ Disallow: /ar/pm/ Disallow: /cn/pm/ Disallow: /de/pm/ Disallow: /es/pm/ Disallow: /fr/pm/ Disallow: /id/pm/ Disallow: /it/pm/ Disallow: /jp/pm/ Disallow: /ko/pm/ Disallow: /pt/pm/ Disallow: /ru/pm/ Disallow: /th/pm/ Disallow: /tr/pm/ Disallow: /tw/pm/ Disallow: /vi/pm/ Allow: /pm/amplify/ Allow: /pm/lambda/ Allow: /pm/sagemaker/ Allow: /pm/textract/ Allow: /pm/transcribe/ Disallow: /premiumsupport/iam/ Disallow: /premiumsupport/knowledge-center/feedback-no/ Disallow: /premiumsupport/knowledge-center/feedback-yes/ Disallow: /pricing/awis/ Disallow: /public-data-sets/applicants/ Disallow: /public-data-sets/terms/ Disallow: /publicdatasets-confirmation/ Disallow: /quickstart/promo Disallow: /quickstart/promo/confirmation Disallow: /reinvent/video-case-studies Disallow: /releasenotes/item/ Disallow: /research-credits/data-science-bowl Disallow: /ru/blogs/497c2c2d305b7dfa351d17bd3b4398dc37b0e0a4/ Disallow: /s/* Disallow: /s/dm/house/editorial/startups/ha_ed_activate_loft_launch_sf_1/ Disallow: /sap/saphana/pilot/ Disallow: /sap/saphana/pilot/offer/ Disallow: /sap/saphana/pilot/offer/confirmation Disallow: /search Disallow: /search/ Allow: /api/dirs/items/search Disallow: /security/aws-pgp-public-key Disallow: /security/ec2-pkgs-public-key Disallow: /security/introducing-s2n-new-open-source-TLS-implementation Disallow: /s-moved/dm/house/editorial/startups/ha_ed_activate_loft_launch_sf_1/ Disallow: /solutions/case-studies/imdb/ Disallow: /solutions/featured-partners/ Disallow: /solutions/global-solution-providers/microsoft/bizspark/ Disallow: /solutions/global-solution-providers/microsoft/bizspark/faq/ Disallow: /solutions/global-solution-providers/microsoft/bizspark/signon/ Disallow: /solutions/global-solution-providers/microsoft/bizspark/signon/ Disallow: /solutions/global-solution-providers/microsoft/bizspark/signon/ Disallow: /solutions/global-solution-providers/microsoft/bizspark/signon/thankyou/ Disallow: /solutions/global-solution-providers/microsoft/bizspark/signon/thankyou/ Disallow: /solutions/solution-providers/program/reseller/adguidelines/ Disallow: /solutions/solution-providers/program/reseller/adguidelines/ Disallow: /solutions/solution-providers/program/reseller/authorized/ Disallow: /solutions/solution-providers/program/reseller/authorized/ Disallow: /solutions/solution-providers/program/reseller/customerterms/ Disallow: /solutions/solution-providers/program/reseller/customerterms/ Disallow: /specialhandling-payments/ Disallow: /start-ups/challenge/ Disallow: /start-ups/loft/hack/ Disallow: /start-ups/loft/hacknight/ Disallow: /start-ups/loft/sf2014/ Disallow: /start-ups/loft/sfff100l3ft Disallow: /start-ups/loft/survey/ Disallow: /start-ups/loft/welcome Disallow: /unable-to-process-request/ Disallow: /use-cases/global-solution-providers/microsoft/bizspark/ Disallow: /use-cases/global-solution-providers/microsoft/bizspark/getting-started/ Disallow: /use-cases/global-solution-providers/microsoft/bizspark/signon/ Disallow: /use-cases/global-solution-providers/microsoft/bizspark/signon/thankyou/ Disallow: /use-cases/global-solution-providers/microsoft/bizspark/thankyou/ Disallow: /vmware/about Disallow: /windows/resources/whitepapers/ Disallow: /windows/server2003/access/ Disallow: /z_dx-marketing-team-prototypes/* Disallow: /community/* Disallow: /community/* Disallow: /community/heroes/ Disallow: /*/community/heroes/ Disallow: /*/search Disallow: /*/search/ Sitemap: https://aws.amazon.com/sitemaps/index/ Sitemap: https://d36a099zv34xie.cloudfront.net/psf-sitemap.xml Sitemap: https://aws.amazon.com/marketplace/sitemap
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-28 | ok 200 | c1f42540a70f |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2025-09-15 CC-MAIN-2025-38 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2024-09-16 CC-MAIN-2024-38 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2023-10-02 CC-MAIN-2023-40 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2022-10-03 CC-MAIN-2022-40 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2021-09-27 CC-MAIN-2021-39 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2020-09-28 CC-MAIN-2020-40 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2019-09-23 CC-MAIN-2019-39 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2018-09-24 CC-MAIN-2018-39 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2018-04-23 CC-MAIN-2018-17 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2018-01-29 CC-MAIN-2018-05 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2017-10-23 CC-MAIN-2017-43 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2017-09-25 CC-MAIN-2017-39 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2017-07-24 CC-MAIN-2017-30 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2017-04-24 CC-MAIN-2017-17 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2017-01-23 CC-MAIN-2017-04 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2016-10-03 CC-MAIN-2016-40 | www.amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2016-09-05 CC-MAIN-2016-36 | amazonaws.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
Evidence
Every daily observation of amazonaws.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).
API: https://infrarabbit.com/crawlers/api/v1/policy/amazonaws.com