xhamster.com: what it tells AI crawlers

On 27 October 2026, xhamster.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://xhamster.com/robots.txt
unchanged since 2026-10-26 · sha256 c31e8b8421bb8c64… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 2a672496a60655f5…

User-agent: *
disallow: *light=*
disallow: /movies/*/download/
disallow: /premium/
disallow: /users/
disallow: /posts/
allow: /users/creators
allow: /user/xHamster/blog/*
allow: /user/xHamster/posts/*
disallow: /stories/*/print
disallow: *embedded_video*
disallow: /photos/gallery/*/
allow: /photos/gallery/*/2$
allow: /photos/gallery/*/3$
allow: /photos/gallery/*/4$
allow: /photos/gallery/*/5$
allow: /photos/gallery/*.html
disallow: /ab.log
disallow: /hot-7
disallow: /hot-30
disallow: /hot-7-pro
disallow: /hot-30-pro
disallow: /switch
disallow: *?max-duration=
disallow: *?min-duration=
disallow: *&max-duration=
disallow: *&min-duration=
disallow: *?geo=
disallow: */categories/*/best/daily
disallow: */categories/*/best/monthly
disallow: */categories/*/best/weekly
disallow: */categories/*/best/year-20
disallow: */categories/*/most-commented
disallow: */categories/*/most-viewed
disallow: */channels/*/newest
disallow: */channels/*/best
disallow: */channels/*/most-commented
disallow: */channels/*/most-viewed
disallow: */pornstars/*/newest
disallow: */pornstars/*/best
disallow: */pornstars/*/most-commented
disallow: */pornstars/*/most-viewed
disallow: */amateurs/*/newest
disallow: */amateurs/*/best
disallow: */amateurs/*/most-commented
disallow: */amateurs/*/most-viewed
disallow: */celebrities/*/newest
disallow: */celebrities/*/best
disallow: */celebrities/*/most-commented
disallow: */celebrities/*/most-viewed
disallow: */tags/*/newest
disallow: */tags/*/best
disallow: */tags/*/most-commented
disallow: */tags/*/most-viewed
allow: */categories/*/best
allow: */pornstars/all/*/best
allow: */amateurs/all/*/best
allow: */celebrities/all/*/best
allow: */channels/all/*/best
allow: */channels/all/best
disallow: */categories/*/best*?max-duration=
disallow: */categories/*/best*?min-duration=
disallow: */categories/*/best*&max-duration=
disallow: */categories/*/best*&min-duration=
disallow: /exit.php
disallow: */*/*/vr?*dimension=
disallow: */*/*/vr?*angle=
disallow: */*/*/vr?*fps=
disallow: /signup?
disallow: /api/
disallow: */search/*cats=
disallow: */search/*date=
disallow: */search/*sort=views
disallow: */search/*sort=longest
disallow: */search/*prod=
disallow: */search/*correction=
disallow: *?skip=
disallow: *&skip=

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of xhamster.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/xhamster.com