criteo.com: what it tells AI crawlers

On 27 October 2026, criteo.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://criteo.com/robots.txt
unchanged since 2026-10-26 · sha256 ed40002530b0bd8b… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 ed40002530b0bd8b…

User-agent: *

#Dynamic URLs

Disallow: /*?industry=
Disallow: /*?region=
Disallow: /*?business=
Disallow: /*?solution=
Disallow: /*sortby=
Disallow: /*?action=
Disallow: /*?year=

#Paths & URls
# -----------------

Disallow: /privacy/embed/
Disallow: */media_category
Disallow: */email-preference-center/
Disallow: */process/*
Disallow: /wp-json/
Disallow: */criteo2017/fonts/*
Disallow: */privacy/disable-criteo-services-on-internet-browsers/
Disallow: /blocked-sitemap.xml

# Prevents issues with GWT
# -----------------

Allow: /*.css$
Allow: /*.js$

#Trackbacks
# -----------------

Disallow: /trackback
Disallow: /*trackback
Disallow: /*trackback*
Disallow: /*/trackback

#Block feeds for crawlers
# -----------------

Allow: /feed/$
Disallow: /feed/
Disallow: /comments/feed/
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$


# Sitemap
# -----------------

Sitemap: https://www.criteo.com/sitemap_index.xml
Sitemap: https://www.criteo.com/de/sitemap_index.xml
Sitemap: https://www.criteo.com/fr/sitemap_index.xml
Sitemap: https://www.criteo.com/br/sitemap_index.xml
Sitemap: https://www.criteo.com/jp/sitemap_index.xml
Sitemap: https://www.criteo.com/kr/sitemap_index.xml
Sitemap: https://www.criteo.com/es/sitemap_index.xml
Sitemap: https://www.criteo.com/it/sitemap_index.xml
Sitemap: https://www.criteo.com/ru/sitemap_index.xml
Sitemap: https://www.criteo.com/cn/sitemap_index.xml

# Bots list to exclude
# --------------------------------

User-agent: MSIECrawler
User-agent: WebCopier
User-agent: HTTrack
User-agent: Microsoft.URL.Control
User-agent: libwww
User-agent: Nuclei
User-agent: WikiDo
User-agent: Riddler
User-agent: PetalBot
User-agent: Zoominfobot
User-agent: Go-http-client
User-agent: Node/simplecrawler
User-agent: CazoodleBot
User-agent: dotbot/1.0
User-agent: Gigabot
User-agent: Barkrowler
User-agent: BLEXBot
User-agent: magpie-crawler
Disallow: /

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of criteo.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/criteo.com