cohere-ai: share of top websites blocking it
On 26 October 2026, 11.9% of the readable top 10k websites fully blocked cohere-ai in robots.txt (bounds 9.5% to 29.7% over all 8,220 websites, 1,659 of them unreadable); 3.7% blocked it partly and 9.2% named it.
Last verified 2026-10-26 21:13 UTC (daily run of 2026-10-26). Proof for that day.
- Operator
- Cohere
- What it is
- Retrieves data to provide responses to user-initiated prompts.
- Description
- Retrieves data based on user prompts.
- robots.txt token
User-agent: cohere-ai- Listed since
- 2026-10-26 (in this ledger's registry)
Fully blocks
11.9%
784 sites; bounds 9.5% to 29.7%
Partly blocks
3.7%
244 sites
Name it explicitly
9.2%
606 sites; unreadable: 3,439
Over time
- blocks, reconstructed (Common Crawl)
- partly blocks, reconstructed (Common Crawl)
- blocks, daily observation
- partly blocks, daily observation
"Partly blocks" counts every robots.txt whose rules for cohere-ai disallow some paths, including generic rules for all crawlers (for example /search); it is not a measure of targeted AI policy. Shares are of the readable websites (W − U: 8,220 websites, 1,659 unreadable); the bounds assume none / all of the unreadable sites block. Solid: daily first-party observations (top 10k of the Tranco list W3ZV9, one fixed cohort). Dashed: reconstructed from Common Crawl (top 10k, sites readable in every crawl), a separate series.
Reconstructed values per crawl
| Crawl | Date | Blocks | Sites read |
|---|---|---|---|
| CC-MAIN-2016-36 | 2016-09-05 | 3.1% | 3,216 |
| CC-MAIN-2017-39 | 2017-09-25 | 3.5% | 3,216 |
| CC-MAIN-2018-39 | 2018-09-24 | 3.6% | 3,216 |
| CC-MAIN-2019-39 | 2019-09-23 | 2.8% | 3,216 |
| CC-MAIN-2020-40 | 2020-09-28 | 2.5% | 3,216 |
| CC-MAIN-2021-39 | 2021-09-27 | 2.6% | 3,216 |
| CC-MAIN-2022-40 | 2022-10-03 | 2.5% | 3,216 |
| CC-MAIN-2024-38 | 2024-09-16 | 4.8% | 3,216 |
| CC-MAIN-2026-39 | 2026-09-21 | 8.1% | 3,216 |
Recent blockers
Sites that started fully blocking cohere-ai since 2026-07-28, highest-ranked first.
| Date | Site | What changed |
|---|---|---|
| n-tv.de | now blocks cohere-ai (was allow) | |
| ptt.cc | now blocks cohere-ai (was allow) | |
| xnxx.health | now blocks cohere-ai (was allow) | |
| manhwaweb.com | now blocks cohere-ai (was allow) | |
| xn--12c1ezaww.com | now blocks cohere-ai (was allow) | |
| loteriadehoy.com | now blocks cohere-ai (was allow) | |
| elmundo.es | now blocks cohere-ai (was allow) | |
| filmyzilla56.com | now blocks cohere-ai (was allow) | |
| filmyzilla62.com | now blocks cohere-ai (was allow) | |
| filmyzilla63.com | now blocks cohere-ai (was allow) | |
| mastodon.social | now blocks cohere-ai (was allow) | |
| bollyflix.af | now blocks cohere-ai (was allow) |
Recent unblockers
| Date | Site | What changed |
|---|---|---|
| vnexpress.net | now allows cohere-ai (was block) |
Sites blocking cohere-ai now
facebook.com (*) · fbcdn.net (*) · instagram.com (*) · twitter.com (*) · googlevideo.com (*) · amazon.com · netflix.com (*) · x.com (*) · pinterest.com (*) · yahoo.com · baidu.com (*) · amazonvideo.com · msn.com · reddit.com (*) · skybridge.click (*) · registrar-servers.com (*) · snapchat.com · nytimes.com · flickr.com (*) · cnn.com · meraki.com (*) · bbc.com · forbes.com · bbc.co.uk · t.co (*) · linktr.ee (*) · imdb.com (*) · launchpad.net · reuters.com (*) · nature.com · demdex.net (*) · weibo.com (*) · amazon.co.uk · naver.com (*) · ea.com · trustpilot.com · netflix.net (*) · nflxvideo.net (*) · bloomberg.com · wsj.com (*) · amazon.de · cookiedatabase.org (*) · aol.com · gcdn.co (*) · kwai.net (*) · amazon.co.jp · statista.com · edgecdn.ru (*) · uol.com.br · ft.com · wired.com · tripadvisor.com · threads.com (*) · netcraze.io (*) · usatoday.com · dailymail.com · sagepub.com · telegraph.co.uk · nintendo.com · amazon.fr
(*) blocked through the * group, not by name. Full list in current.csv.
Cite this
InfraRabbit AI Crawl Ledger, "Share of top websites blocking cohere-ai", https://infrarabbit.com/crawlers/bot/cohere-ai, retrieved 2026-10-26. CC BY 4.0.