MistralAI-Training: share of top websites blocking it
On 27 October 2026, 6.2% of the readable top 10k websites fully blocked MistralAI-Training in robots.txt (bounds 5.0% to 25.1% over all 8,219 websites, 1,657 of them unreadable); 3.9% blocked it partly and 2.0% named it.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Operator
- Mistral AI
- What it is
- AI Data Scrapers
- Description
- MistralAI-Training crawls web content to build training datasets. Documented by Mistral at https://docs.mistral.ai/robots/
- robots.txt token
User-agent: MistralAI-Training- Listed since
- 2026-10-26 (in this ledger's registry)
Fully blocks
6.2%
410 sites; bounds 5.0% to 25.1%
Partly blocks
3.9%
255 sites
Name it explicitly
2.0%
134 sites; unreadable: 3,438
Over time
- blocks, reconstructed (Common Crawl)
- partly blocks, reconstructed (Common Crawl)
- blocks, daily observation
- partly blocks, daily observation
"Partly blocks" counts every robots.txt whose rules for MistralAI-Training disallow some paths, including generic rules for all crawlers (for example /search); it is not a measure of targeted AI policy. Shares are of the readable websites (W − U: 8,219 websites, 1,657 unreadable); the bounds assume none / all of the unreadable sites block. Solid: daily first-party observations (top 10k of the Tranco list W3ZV9, one fixed cohort). Dashed: reconstructed from Common Crawl (top 10k), a separate series.
Reconstructed values per crawl
| Crawl | Date | Blocks | Sites read |
|---|---|---|---|
| CC-MAIN-2016-36 | 2016-09-05 | 3.8% | 5,355 |
| CC-MAIN-2016-40 | 2016-10-03 | 2.6% | 5,235 |
| CC-MAIN-2017-04 | 2017-01-23 | 4.0% | 5,458 |
| CC-MAIN-2017-17 | 2017-04-24 | 3.2% | 5,449 |
| CC-MAIN-2017-30 | 2017-07-24 | 4.1% | 5,397 |
| CC-MAIN-2017-39 | 2017-09-25 | 3.7% | 5,355 |
| CC-MAIN-2017-43 | 2017-10-23 | 3.3% | 5,329 |
| CC-MAIN-2018-05 | 2018-01-29 | 3.3% | 5,300 |
| CC-MAIN-2018-39 | 2018-09-24 | 4.0% | 5,768 |
| CC-MAIN-2019-39 | 2019-09-23 | 2.8% | 5,773 |
| CC-MAIN-2020-40 | 2020-09-28 | 2.7% | 5,860 |
| CC-MAIN-2021-39 | 2021-09-27 | 2.9% | 5,914 |
| CC-MAIN-2022-40 | 2022-10-03 | 2.6% | 5,692 |
| CC-MAIN-2023-40 | 2023-10-02 | 3.0% | 5,930 |
| CC-MAIN-2024-38 | 2024-09-16 | 2.9% | 6,065 |
| CC-MAIN-2025-38 | 2025-09-15 | 2.7% | 6,045 |
| CC-MAIN-2026-39 | 2026-09-21 | 3.0% | 5,868 |
Recent blockers
Sites that started fully blocking MistralAI-Training since 2026-07-29, highest-ranked first.
Recent unblockers
| Date | Site | What changed |
|---|---|---|
| nova-app.eu | now allows MistralAI-Training (was block) |
Sites blocking MistralAI-Training now
facebook.com (*) · fbcdn.net (*) · instagram.com (*) · twitter.com (*) · googlevideo.com (*) · netflix.com (*) · x.com (*) · pinterest.com (*) · baidu.com (*) · reddit.com (*) · skybridge.click (*) · registrar-servers.com (*) · flickr.com (*) · t.co (*) · linktr.ee (*) · imdb.com (*) · reuters.com (*) · demdex.net (*) · weibo.com (*) · naver.com (*) · trustpilot.com (*) · netflix.net (*) · nflxvideo.net (*) · wsj.com (*) · cookiedatabase.org (*) · gcdn.co (*) · kwai.net (*) · edgecdn.ru (*) · sohu.com (*) · threads.com (*) · netcraze.io (*) · unesco.org · sharethrough.com (*) · hstgr.net (*) · imgsmail.ru (*) · fb.com (*) · hihonorcloud.com (*) · pinimg.com (*) · quantserve.com (*) · creativecdn.com (*) · crwdcntrl.net (*) · airbnb.com · vkontakte.ru (*) · disneyplus.com (*) · sentinelone.net (*) · adform.net (*) · academia.edu (*) · daum.net (*) · tiktokw.us (*) · xiaohongshu.com (*) · mailchi.mp (*) · btloader.com (*) · namecheaphosting.com (*) · n-tv.de · jamfcloud.com (*) · rocketmoney.dev (*) · ouest-france.fr · namu.wiki (*) · fmkorea.com (*) · amap.com (*)
(*) blocked through the * group, not by name. Full list in current.csv.
Cite this
InfraRabbit AI Crawl Ledger, "Share of top websites blocking MistralAI-Training", https://infrarabbit.com/crawlers/bot/MistralAI-Training, retrieved 2026-10-27. CC BY 4.0.