amazon-kendra: share of top websites blocking it

On 27 October 2026, 5.1% of the readable top 10k websites fully blocked amazon-kendra in robots.txt (bounds 4.1% to 24.2% over all 8,219 websites, 1,657 of them unreadable); 3.9% blocked it partly and 1.0% named it.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Operator
Amazon
What it is
Collects data for AI natural language search
Description
Amazon Kendra is a highly accurate intelligent search service that enables your users to search unstructured data using natural language. It returns specific answers to questions, giving users an experience that's close to interacting with a human expert. It is highly scalable and capable of meeting performance demands, tightly integrated with other AWS services such as Amazon S3 and Amazon Lex, and offers enterprise-grade security.
robots.txt token
User-agent: amazon-kendra
Listed since
2026-10-26 (in this ledger's registry)

Fully blocks

5.1%

336 sites; bounds 4.1% to 24.2%

Partly blocks

3.9%

253 sites

Name it explicitly

1.0%

65 sites; unreadable: 3,438

Over time

Share of panel sites blocking amazon-kendra4 lines, 2016-09-05 to 2026-10-27; values between 2.58 and 5.12 %; latest 5.12 %.2.383.113.854.595.32Sep 2016Mar 2019Oct 2021Apr 2024Oct 2026blocks, reconstructed (Common Crawl): 2016-09-05 UTC: 3.79 %blocks, reconstructed (Common Crawl): 2016-10-03 UTC: 2.58 %blocks, reconstructed (Common Crawl): 2017-01-23 UTC: 3.96 %blocks, reconstructed (Common Crawl): 2017-04-24 UTC: 3.19 %blocks, reconstructed (Common Crawl): 2017-07-24 UTC: 4.13 %blocks, reconstructed (Common Crawl): 2017-09-25 UTC: 3.66 %blocks, reconstructed (Common Crawl): 2017-10-23 UTC: 3.32 %blocks, reconstructed (Common Crawl): 2018-01-29 UTC: 3.32 %blocks, reconstructed (Common Crawl): 2018-09-24 UTC: 3.97 %blocks, reconstructed (Common Crawl): 2019-09-23 UTC: 2.82 %blocks, reconstructed (Common Crawl): 2020-09-28 UTC: 2.68 %blocks, reconstructed (Common Crawl): 2021-09-27 UTC: 2.91 %blocks, reconstructed (Common Crawl): 2022-10-03 UTC: 2.64 %blocks, reconstructed (Common Crawl): 2023-10-02 UTC: 3.00 %blocks, reconstructed (Common Crawl): 2024-09-16 UTC: 2.90 %blocks, reconstructed (Common Crawl): 2025-09-15 UTC: 2.81 %blocks, reconstructed (Common Crawl): 2026-09-21 UTC: 3.43 %partly blocks, reconstructed (Common Crawl): 2016-09-05 UTC: 4.78 %partly blocks, reconstructed (Common Crawl): 2016-10-03 UTC: 4.97 %partly blocks, reconstructed (Common Crawl): 2017-01-23 UTC: 4.78 %partly blocks, reconstructed (Common Crawl): 2017-04-24 UTC: 4.83 %partly blocks, reconstructed (Common Crawl): 2017-07-24 UTC: 4.67 %partly blocks, reconstructed (Common Crawl): 2017-09-25 UTC: 4.63 %partly blocks, reconstructed (Common Crawl): 2017-10-23 UTC: 4.60 %partly blocks, reconstructed (Common Crawl): 2018-01-29 UTC: 4.72 %partly blocks, reconstructed (Common Crawl): 2018-09-24 UTC: 4.30 %partly blocks, reconstructed (Common Crawl): 2019-09-23 UTC: 4.38 %partly blocks, reconstructed (Common Crawl): 2020-09-28 UTC: 4.28 %partly blocks, reconstructed (Common Crawl): 2021-09-27 UTC: 4.24 %partly blocks, reconstructed (Common Crawl): 2022-10-03 UTC: 4.32 %partly blocks, reconstructed (Common Crawl): 2023-10-02 UTC: 4.42 %partly blocks, reconstructed (Common Crawl): 2024-09-16 UTC: 4.42 %partly blocks, reconstructed (Common Crawl): 2025-09-15 UTC: 4.58 %partly blocks, reconstructed (Common Crawl): 2026-09-21 UTC: 4.40 %blocks, daily observation: 2026-10-26 UTC: 5.11 %blocks, daily observation: 2026-10-27 UTC: 5.12 %partly blocks, daily observation: 2026-10-26 UTC: 3.87 %partly blocks, daily observation: 2026-10-27 UTC: 3.86 %

"Partly blocks" counts every robots.txt whose rules for amazon-kendra disallow some paths, including generic rules for all crawlers (for example /search); it is not a measure of targeted AI policy. Shares are of the readable websites (W − U: 8,219 websites, 1,657 unreadable); the bounds assume none / all of the unreadable sites block. Solid: daily first-party observations (top 10k of the Tranco list W3ZV9, one fixed cohort). Dashed: reconstructed from Common Crawl (top 10k), a separate series.

Reconstructed values per crawl
CrawlDateBlocksSites read
CC-MAIN-2016-362016-09-053.8%5,355
CC-MAIN-2016-402016-10-032.6%5,235
CC-MAIN-2017-042017-01-234.0%5,458
CC-MAIN-2017-172017-04-243.2%5,449
CC-MAIN-2017-302017-07-244.1%5,397
CC-MAIN-2017-392017-09-253.7%5,355
CC-MAIN-2017-432017-10-233.3%5,329
CC-MAIN-2018-052018-01-293.3%5,300
CC-MAIN-2018-392018-09-244.0%5,768
CC-MAIN-2019-392019-09-232.8%5,773
CC-MAIN-2020-402020-09-282.7%5,860
CC-MAIN-2021-392021-09-272.9%5,914
CC-MAIN-2022-402022-10-032.6%5,692
CC-MAIN-2023-402023-10-023.0%5,930
CC-MAIN-2024-382024-09-162.9%6,065
CC-MAIN-2025-382025-09-152.8%6,045
CC-MAIN-2026-392026-09-213.4%5,868

Recent blockers

Sites that started fully blocking amazon-kendra since 2026-07-29, highest-ranked first.

DateSiteWhat changed
techtarget.comnow blocks amazon-kendra (was allow)

Recent unblockers

DateSiteWhat changed
No changes recorded yet. The first daily run is the baseline; changes appear from the second day on.

Sites blocking amazon-kendra now

facebook.com (*) · fbcdn.net (*) · instagram.com (*) · twitter.com (*) · googlevideo.com (*) · netflix.com (*) · x.com (*) · pinterest.com (*) · baidu.com (*) · reddit.com (*) · skybridge.click (*) · registrar-servers.com (*) · flickr.com (*) · t.co (*) · imdb.com (*) · reuters.com (*) · demdex.net (*) · weibo.com (*) · naver.com (*) · trustpilot.com (*) · netflix.net (*) · nflxvideo.net (*) · wsj.com (*) · cookiedatabase.org (*) · gcdn.co (*) · kwai.net (*) · edgecdn.ru (*) · sohu.com (*) · threads.com (*) · netcraze.io (*) · sharethrough.com (*) · hstgr.net (*) · lemonde.fr · imgsmail.ru (*) · fb.com (*) · hihonorcloud.com (*) · pinimg.com (*) · quantserve.com (*) · creativecdn.com (*) · crwdcntrl.net (*) · vkontakte.ru (*) · disneyplus.com (*) · people.com · sentinelone.net (*) · adform.net (*) · academia.edu (*) · daum.net (*) · figma.com · tiktokw.us (*) · xiaohongshu.com (*) · mailchi.mp (*) · btloader.com (*) · namecheaphosting.com (*) · jamfcloud.com (*) · healthline.com · investopedia.com · rocketmoney.dev (*) · ouest-france.fr · nypost.com · namu.wiki (*)

(*) blocked through the * group, not by name. Full list in current.csv.

Cite this

InfraRabbit AI Crawl Ledger, "Share of top websites blocking amazon-kendra", https://infrarabbit.com/crawlers/bot/amazon-kendra, retrieved 2026-10-27. CC BY 4.0.