xhamster.com: what it tells AI crawlers
On 27 October 2026, xhamster.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://xhamster.com/robots.txt
unchanged since 2026-10-26 · sha256c31e8b8421bb8c64…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-27
| User agent | Verdict | How |
|---|---|---|
| GPTBot | allows | through the * group |
| OAI-SearchBot | allows | through the * group |
| ChatGPT-User | allows | through the * group |
| ClaudeBot | allows | through the * group |
| Claude-SearchBot | allows | through the * group |
| Claude-User | allows | through the * group |
| PerplexityBot | allows | through the * group |
| Perplexity-User | allows | through the * group |
| Googlebot | allows | through the * group |
| Google-Extended | allows | through the * group |
| Applebot-Extended | allows | through the * group |
| CCBot | allows | through the * group |
| Bytespider | allows | through the * group |
| Meta-ExternalAgent | allows | through the * group |
| Amazonbot | allows | through the * group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | no |
| Declares Content Signals (robots.txt or header) | no |
| Says ai-train=no in a Content Signal | no |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
robots.txt as recorded sha256 c31e8b8421bb8c64…
User-agent: * disallow: *light=* disallow: /movies/*/download/ disallow: /premium/ allow: /users/ disallow: /users/*/friends disallow: /users/*/favorites disallow: /users/profiles/ disallow: /playlists/ disallow: /posts/ disallow: /embed/ allow: /users/creators allow: /user/xHamster/blog/* allow: /user/xHamster/posts/* disallow: /stories/*/print disallow: *embedded_video* disallow: /photos/gallery/*/ allow: /photos/gallery/*/2$ allow: /photos/gallery/*/3$ allow: /photos/gallery/*/4$ allow: /photos/gallery/*/5$ allow: /photos/gallery/*.html disallow: /ab.log disallow: /hot-7 disallow: /hot-30 disallow: /hot-7-pro disallow: /hot-30-pro disallow: /switch disallow: *?max-duration= disallow: *?min-duration= disallow: *&max-duration= disallow: *&min-duration= disallow: *?geo= disallow: */gifs/best/alltime disallow: */gifs/best/daily disallow: */gifs/best/monthly disallow: */gifs/best/year-20 disallow: */categories/*/best/daily disallow: */categories/*/best/monthly disallow: */categories/*/best/weekly disallow: */categories/*/best/year-20 disallow: */categories/*/most-commented disallow: */categories/*/most-viewed disallow: */categories/*/producer disallow: */categories/*/creator disallow: */channels/*/newest disallow: */channels/*/best disallow: */channels/*/most-commented disallow: */channels/*/most-viewed disallow: */pornstars/*/newest disallow: */pornstars/*/best disallow: */pornstars/*/most-commented disallow: */pornstars/*/most-viewed disallow: */pornstars/*/exclusive disallow: */creators/*/newest disallow: */creators/*/best disallow: */creators/*/most-commented disallow: */creators/*/most-viewed disallow: */creators/*/exclusive disallow: */creators/*/full-length disallow: */celebrities/*/newest disallow: */celebrities/*/best disallow: */celebrities/*/most-commented disallow: */celebrities/*/most-viewed disallow: */tags/*/newest disallow: */tags/*/best disallow: */tags/*/most-commented disallow: */tags/*/most-viewed disallow: */tags/*/producer disallow: */tags/*/creator allow: */categories/*/best allow: */pornstars/all/*/best allow: */creators/all/*/best allow: */celebrities/all/*/best disallow: */categories/*/best*?max-duration= disallow: */categories/*/best*?min-duration= disallow: */categories/*/best*&max-duration= disallow: */categories/*/best*&min-duration= disallow: /exit.php disallow: */*/*/vr?*dimension= disallow: */*/*/vr?*angle= disallow: */categories/*fps= disallow: */tags/*fps= disallow: */channels/*fps= disallow: */pornstars/*fps= disallow: */creators/*fps= disallow: */celebrities/*fps= disallow: /videos/*?t= # Disallow pornstar deep filters (3-4 components) disallow: */pornstars/*/4k/full-length?* disallow: */pornstars/*/hd?quality=*&* disallow: */pornstars/*/hd/full-length?* disallow: */pornstars/*/vr?quality=*&* disallow: */pornstars/*/vr/full-length?* # Disallow pornstar paginations on deep filters disallow: */pornstars/*/4k/full-length/*?* disallow: */pornstars/*/hd/*?quality=*&* disallow: */pornstars/*/hd/full-length/*?* disallow: */pornstars/*/vr/*?quality=*&* disallow: */pornstars/*/vr/full-length/*?* disallow: *=producer disallow: *=creator disallow: /signup? disallow: /api/ disallow: */search/*cats= disallow: */search/*correction= disallow: */search/*date= disallow: */search/*length= disallow: */search/*orientations= disallow: */search/*prod= disallow: */search/*revert= disallow: */search/*hidewatched= disallow: */search/*sort=date disallow: */search/*sort=longest disallow: */search/*sort=views disallow: */search/*angle= disallow: */search/*dimension= disallow: */search/*exclude= disallow: */search/*fps= disallow: */search/*sort=relevance disallow: */search/*sort=rjib disallow: */search/*quality=781 disallow: */search/*quality=hd disallow: */search/*format=px # Disallow search filters Disallow: */search/*format=vr # Disallow all pagination Disallow: */search/*page= # Allow search pages from 2 to 10 Allow: */search/*?page=2$ Allow: */search/*?page=3$ Allow: */search/*?page=4$ Allow: */search/*?page=5$ Allow: */search/*?page=6$ Allow: */search/*?page=7$ Allow: */search/*?page=8$ Allow: */search/*?page=9$ Allow: */search/*?page=10$ # Allow best search pages from 2 to 10 Allow: */search/*?sort=best&page=2$ Allow: */search/*?sort=best&page=3$ Allow: */search/*?sort=best&page=4$ Allow: */search/*?sort=best&page=5$ Allow: */search/*?sort=best&page=6$ Allow: */search/*?sort=best&page=7$ Allow: */search/*?sort=best&page=8$ Allow: */search/*?sort=best&page=9$ Allow: */search/*?sort=best&page=10$ # Disallow channels pagination 6-50 Disallow: */channels/*/6$ Disallow: */channels/*/7$ Disallow: */channels/*/8$ Disallow: */channels/*/9$ Disallow: */channels/*/10$ Disallow: */channels/*/11$ Disallow: */channels/*/12$ Disallow: */channels/*/13$ Disallow: */channels/*/14$ Disallow: */channels/*/15$ Disallow: */channels/*/16$ Disallow: */channels/*/17$ Disallow: */channels/*/18$ Disallow: */channels/*/19$ Disallow: */channels/*/20$ Disallow: */channels/*/21$ Disallow: */channels/*/22$ Disallow: */channels/*/23$ Disallow: */channels/*/24$ Disallow: */channels/*/25$ Disallow: */channels/*/26$ Disallow: */channels/*/27$ Disallow: */channels/*/28$ Disallow: */channels/*/29$ Disallow: */channels/*/30$ Disallow: */channels/*/31$ Disallow: */channels/*/32$ Disallow: */channels/*/33$ Disallow: */channels/*/34$ Disallow: */channels/*/35$ Disallow: */channels/*/36$ Disallow: */channels/*/37$ Disallow: */channels/*/38$ Disallow: */channels/*/39$ Disallow: */channels/*/40$ Disallow: */channels/*/41$ Disallow: */channels/*/42$ Disallow: */channels/*/43$ Disallow: */channels/*/44$ Disallow: */channels/*/45$ Disallow: */channels/*/46$ Disallow: */channels/*/47$ Disallow: */channels/*/48$ Disallow: */channels/*/49$ Disallow: */channels/*/50$ # Disallow creators pagination 6-50 Disallow: */creators/*/6$ Disallow: */creators/*/7$ Disallow: */creators/*/8$ Disallow: */creators/*/9$ Disallow: */creators/*/10$ Disallow: */creators/*/11$ Disallow: */creators/*/12$ Disallow: */creators/*/13$ Disallow: */creators/*/14$ Disallow: */creators/*/15$ Disallow: */creators/*/16$ Disallow: */creators/*/17$ Disallow: */creators/*/18$ Disallow: */creators/*/19$ Disallow: */creators/*/20$ Disallow: */creators/*/21$ Disallow: */creators/*/22$ Disallow: */creators/*/23$ Disallow: */creators/*/24$ Disallow: */creators/*/25$ Disallow: */creators/*/26$ Disallow: */creators/*/27$ Disallow: */creators/*/28$ Disallow: */creators/*/29$ Disallow: */creators/*/30$ Disallow: */creators/*/31$ Disallow: */creators/*/32$ Disallow: */creators/*/33$ Disallow: */creators/*/34$ Disallow: */creators/*/35$ Disallow: */creators/*/36$ Disallow: */creators/*/37$ Disallow: */creators/*/38$ Disallow: */creators/*/39$ Disallow: */creators/*/40$ Disallow: */creators/*/41$ Disallow: */creators/*/42$ Disallow: */creators/*/43$ Disallow: */creators/*/44$ Disallow: */creators/*/45$ Disallow: */creators/*/46$ Disallow: */creators/*/47$ Disallow: */creators/*/48$ Disallow: */creators/*/49$ Disallow: */creators/*/50$ # Disallow pornstars pagination 6-50 Disallow: */pornstars/*/6$ Disallow: */pornstars/*/7$ Disallow: */pornstars/*/8$ Disallow: */pornstars/*/9$ Disallow: */pornstars/*/10$ Disallow: */pornstars/*/11$ Disallow: */pornstars/*/12$ Disallow: */pornstars/*/13$ Disallow: */pornstars/*/14$ Disallow: */pornstars/*/15$ Disallow: */pornstars/*/16$ Disallow: */pornstars/*/17$ Disallow: */pornstars/*/18$ Disallow: */pornstars/*/19$ Disallow: */pornstars/*/20$ Disallow: */pornstars/*/21$ Disallow: */pornstars/*/22$ Disallow: */pornstars/*/23$ Disallow: */pornstars/*/24$ Disallow: */pornstars/*/25$ Disallow: */pornstars/*/26$ Disallow: */pornstars/*/27$ Disallow: */pornstars/*/28$ Disallow: */pornstars/*/29$ Disallow: */pornstars/*/30$ Disallow: */pornstars/*/31$ Disallow: */pornstars/*/32$ Disallow: */pornstars/*/33$ Disallow: */pornstars/*/34$ Disallow: */pornstars/*/35$ Disallow: */pornstars/*/36$ Disallow: */pornstars/*/37$ Disallow: */pornstars/*/38$ Disallow: */pornstars/*/39$ Disallow: */pornstars/*/40$ Disallow: */pornstars/*/41$ Disallow: */pornstars/*/42$ Disallow: */pornstars/*/43$ Disallow: */pornstars/*/44$ Disallow: */pornstars/*/45$ Disallow: */pornstars/*/46$ Disallow: */pornstars/*/47$ Disallow: */pornstars/*/48$ Disallow: */pornstars/*/49$ Disallow: */pornstars/*/50$ # Disallow tags pagination 11-200 Disallow: */tags/*/11$ Disallow: */tags/*/12$ Disallow: */tags/*/13$ Disallow: */tags/*/14$ Disallow: */tags/*/15$ Disallow: */tags/*/16$ Disallow: */tags/*/17$ Disallow: */tags/*/18$ Disallow: */tags/*/19$ Disallow: */tags/*/20$ Disallow: */tags/*/21$ Disallow: */tags/*/22$ Disallow: */tags/*/23$ Disallow: */tags/*/24$ Disallow: */tags/*/25$ Disallow: */tags/*/26$ Disallow: */tags/*/27$ Disallow: */tags/*/28$ Disallow: */tags/*/29$ Disallow: */tags/*/30$ Disallow: */tags/*/31$ Disallow: */tags/*/32$ Disallow: */tags/*/33$ Disallow: */tags/*/34$ Disallow: */tags/*/35$ Disallow: */tags/*/36$ Disallow: */tags/*/37$ Disallow: */tags/*/38$ Disallow: */tags/*/39$ Disallow: */tags/*/40$ Disallow: */tags/*/41$ Disallow: */tags/*/42$ Disallow: */tags/*/43$ Disallow: */tags/*/44$ Disallow: */tags/*/45$ Disallow: */tags/*/46$ Disallow: */tags/*/47$ Disallow: */tags/*/48$ Disallow: */tags/*/49$ Disallow: */tags/*/50$ Disallow: */tags/*/51$ Disallow: */tags/*/52$ Disallow: */tags/*/53$ Disallow: */tags/*/54$ Disallow: */tags/*/55$ Disallow: */tags/*/56$ Disallow: */tags/*/57$ Disallow: */tags/*/58$ Disallow: */tags/*/59$ Disallow: */tags/*/60$ Disallow: */tags/*/61$ Disallow: */tags/*/62$ Disallow: */tags/*/63$ Disallow: */tags/*/64$ Disallow: */tags/*/65$ Disallow: */tags/*/66$ Disallow: */tags/*/67$ Disallow: */tags/*/68$ Disallow: */tags/*/69$ Disallow: */tags/*/70$ Disallow: */tags/*/71$ Disallow: */tags/*/72$ Disallow: */tags/*/73$ Disallow: */tags/*/74$ Disallow: */tags/*/75$ Disallow: */tags/*/76$ Disallow: */tags/*/77$ Disallow: */tags/*/78$ Disallow: */tags/*/79$ Disallow: */tags/*/80$ Disallow: */tags/*/81$ Disallow: */tags/*/82$ Disallow: */tags/*/83$ Disallow: */tags/*/84$ Disallow: */tags/*/85$ Disallow: */tags/*/86$ Disallow: */tags/*/87$ Disallow: */tags/*/88$ Disallow: */tags/*/89$ Disallow: */tags/*/90$ Disallow: */tags/*/91$ Disallow: */tags/*/92$ Disallow: */tags/*/93$ Disallow: */tags/*/94$ Disallow: */tags/*/95$ Disallow: */tags/*/96$ Disallow: */tags/*/97$ Disallow: */tags/*/98$ Disallow: */tags/*/99$ Disallow: */tags/*/100$ Disallow: */tags/*/101$ Disallow: */tags/*/102$ Disallow: */tags/*/103$ Disallow: */tags/*/104$ Disallow: */tags/*/105$ Disallow: */tags/*/106$ Disallow: */tags/*/107$ Disallow: */tags/*/108$ Disallow: */tags/*/109$ Disallow: */tags/*/110$ Disallow: */tags/*/111$ Disallow: */tags/*/112$ Disallow: */tags/*/113$ Disallow: */tags/*/114$ Disallow: */tags/*/115$ Disallow: */tags/*/116$ Disallow: */tags/*/117$ Disallow: */tags/*/118$ Disallow: */tags/*/119$ Disallow: */tags/*/120$ Disallow: */tags/*/121$ Disallow: */tags/*/122$ Disallow: */tags/*/123$ Disallow: */tags/*/124$ Disallow: */tags/*/125$ Disallow: */tags/*/126$ Disallow: */tags/*/127$ Disallow: */tags/*/128$ Disallow: */tags/*/129$ Disallow: */tags/*/130$ Disallow: */tags/*/131$ Disallow: */tags/*/132$ Disallow: */tags/*/133$ Disallow: */tags/*/134$ Disallow: */tags/*/135$ Disallow: */tags/*/136$ Disallow: */tags/*/137$ Disallow: */tags/*/138$ Disallow: */tags/*/139$ Disallow: */tags/*/140$ Disallow: */tags/*/141$ Disallow: */tags/*/142$ Disallow: */tags/*/143$ Disallow: */tags/*/144$ Disallow: */tags/*/145$ Disallow: */tags/*/146$ Disallow: */tags/*/147$ Disallow: */tags/*/148$ Disallow: */tags/*/149$ Disallow: */tags/*/150$ Disallow: */tags/*/151$ Disallow: */tags/*/152$ Disallow: */tags/*/153$ Disallow: */tags/*/154$ Disallow: */tags/*/155$ Disallow: */tags/*/156$ Disallow: */tags/*/157$ Disallow: */tags/*/158$ Disallow: */tags/*/159$ Disallow: */tags/*/160$ Disallow: */tags/*/161$ Disallow: */tags/*/162$ Disallow: */tags/*/163$ Disallow: */tags/*/164$ Disallow: */tags/*/165$ Disallow: */tags/*/166$ Disallow: */tags/*/167$ Disallow: */tags/*/168$ Disallow: */tags/*/169$ Disallow: */tags/*/170$ Disallow: */tags/*/171$ Disallow: */tags/*/172$ Disallow: */tags/*/173$ Disallow: */tags/*/174$ Disallow: */tags/*/175$ Disallow: */tags/*/176$ Disallow: */tags/*/177$ Disallow: */tags/*/178$ Disallow: */tags/*/179$ Disallow: */tags/*/180$ Disallow: */tags/*/181$ Disallow: */tags/*/182$ Disallow: */tags/*/183$ Disallow: */tags/*/184$ Disallow: */tags/*/185$ Disallow: */tags/*/186$ Disallow: */tags/*/187$ Disallow: */tags/*/188$ Disallow: */tags/*/189$ Disallow: */tags/*/190$ Disallow: */tags/*/191$ Disallow: */tags/*/192$ Disallow: */tags/*/193$ Disallow: */tags/*/194$ Disallow: */tags/*/195$ Disallow: */tags/*/196$ Disallow: */tags/*/197$ Disallow: */tags/*/198$ Disallow: */tags/*/199$ Disallow: */tags/*/200$ disallow: *?skip= disallow: *&skip= disallow: /webcams/ disallow: /tracking? disallow: *?x_platform_switch=* disallow: /xembed.php disallow: /livevideoamp/ disallow: /fhwidgetamp/ disallow: /info/list-of-processors disallow: /info/dsa-transparency-report-2024 disallow: /info/dsa-transparency-report-2025 disallow: /live User-agent: ia_archiver Disallow: /
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-28 | ok 200 | c31e8b8421bb |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2025-09-15 CC-MAIN-2025-38 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2024-09-16 CC-MAIN-2024-38 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2023-10-02 CC-MAIN-2023-40 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2022-10-03 CC-MAIN-2022-40 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-01-29 CC-MAIN-2018-05 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-10-23 CC-MAIN-2017-43 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-07-24 CC-MAIN-2017-30 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-04-24 CC-MAIN-2017-17 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-01-23 CC-MAIN-2017-04 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-10-03 CC-MAIN-2016-40 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-09-05 CC-MAIN-2016-36 | xhamster.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
Evidence
Every daily observation of xhamster.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).
API: https://infrarabbit.com/crawlers/api/v1/policy/xhamster.com