trustpilot.com: what it tells AI crawlers

On 28 October 2026, trustpilot.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel: top 1k (Tranco)
robots.txt: ok (HTTP 200) · https://www.trustpilot.com/robots.txt
unchanged since 2026-10-26 · sha256 96ad98c97cd960d5… · view as recorded
The * group: blocks applies to every agent without its own group

Per AI bot, 2026-10-28

User agent	Verdict	How
GPTBot	allows	named in its own group
OAI-SearchBot	allows	named in its own group
ChatGPT-User	allows	named in its own group
ClaudeBot	allows	named in its own group
Claude-SearchBot	allows	named in its own group
Claude-User	allows	named in its own group
PerplexityBot	allows	named in its own group
Perplexity-User	blocks	through the * group
Googlebot	allows	named in its own group
Google-Extended	allows	named in its own group
Applebot-Extended	blocks	named in its own group
CCBot	allows	named in its own group
Bytespider	blocks	named in its own group
Meta-ExternalAgent	allows	named in its own group
Amazonbot	blocks	through the * group
anthropic-ai	blocks	named in its own group
cohere-ai	blocks	named in its own group
Diffbot	blocks	named in its own group
FacebookBot	allows	named in its own group
facebookexternalhit	allows	named in its own group
ImagesiftBot	blocks	named in its own group
meta-externalfetcher	allows	named in its own group
meta-webindexer	allows	named in its own group
omgili	blocks	named in its own group
omgilibot	blocks	named in its own group
Timpibot	blocks	named in its own group
Webzio-Extended	blocks	named in its own group
YouBot	blocks	named in its own group

Other signals

Publishes /llms.txt	not observed
Publishes /ai.txt	not observed
Publishes /.well-known/tdmrep.json	not observed
Reserves TDM rights (TDMRep file, header or meta)	not observed
Declares Content Signals (robots.txt or header)	not observed
Says ai-train=no in a Content Signal	not observed
Points to an RSL licence (robots.txt License line or Link header)	not observed
Answers with an x402 payment offer (lower bound: only the homepage is probed)	not observed
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	not observed
Sends noai / noimageai directives	not observed
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 a46cc7d4dbe9fd96…

User-agent: Googlebot
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: Mediapartners-Google
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: Googlebot-Mobile
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: AdsBot-Google
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: bingbot
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: msnbot
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: ia_archiver
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: Alexabot
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: ScoutJet
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: Yandex
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: Slurp
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: Twitterbot
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: Baiduspider
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: Facebot
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: DuckDuckBot
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: SearchmetricsBot
Disallow: /location
Disallow: /error
Disallow: /sessions
Disallow: /pingdom
Disallow: /evaluate-link
Disallow: /users/activate
Disallow: /users/create

User-agent: *
Disallow: /

Sitemap: http://sitemaps.trustpilot.com/index_en-us.xml

robots.txt versions

From	Through	State	Version
2026-10-26	2026-10-28	ok 200	`96ad98c97cd9`

Timeline of changes

Date	What changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	www.trustpilot.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2025-09-15 CC-MAIN-2025-38	www.trustpilot.com	ok view	blocks	allows	unknown	blocks	allows	unknown	allows
2024-09-16 CC-MAIN-2024-38	www.trustpilot.com	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2023-10-02 CC-MAIN-2023-40	www.trustpilot.com	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2022-10-03 CC-MAIN-2022-40	trustpilot.com	redirect-only	unknown	unknown	unknown	unknown	unknown	unknown	unknown
2021-09-27 CC-MAIN-2021-39	www.trustpilot.com	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2020-09-28 CC-MAIN-2020-40	www.trustpilot.com	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2019-09-23 CC-MAIN-2019-39	www.trustpilot.com	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2018-09-24 CC-MAIN-2018-39	trustpilot.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-07-23 CC-MAIN-2018-30	trustpilot.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-04-23 CC-MAIN-2018-17	trustpilot.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-01-29 CC-MAIN-2018-05	trustpilot.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-10-23 CC-MAIN-2017-43	trustpilot.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	trustpilot.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-07-24 CC-MAIN-2017-30	trustpilot.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-04-24 CC-MAIN-2017-17	trustpilot.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-01-23 CC-MAIN-2017-04	trustpilot.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-10-03 CC-MAIN-2016-40	www.trustpilot.com	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2016-09-05 CC-MAIN-2016-36	www.trustpilot.com	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks

Evidence

Every daily observation of trustpilot.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).

API: https://infrarabbit.com/crawlers/api/v1/policy/trustpilot.com