pornhub.com: what it tells AI crawlers

On 28 October 2026, pornhub.com's robots.txt partly blocked CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.pornhub.com/robots.txt
unchanged since 2026-10-28 · sha256 1ec08abd0b386f0a… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-28

Other signals

robots.txt as recorded sha256 4493f13121a4160b…

User-agent: *
Sitemap: https://www.pornhub.com/sitemaps.xml
Disallow: /*7urren*
Disallow: /*cc=*
Disallow: /*correction=*
Disallow: /*embed_player*
Disallow: /*exclude_category=*
Disallow: /*hd=1*
Disallow: /*gifs/search*o=
Disallow: /*incategories*p=*
Disallow: /*login=*
Disallow: /*max_duration=*
Disallow: /*min_duration=*
Disallow: /*orientation=*
Disallow: /*page*page*
Disallow: /*pkey=*
Disallow: /*player.php
Disallow: /*player_v1.6b.swf
Disallow: /*redirect=*
Disallow: /*redirectTo=*
Disallow: /*redirectto=*
Disallow: /*search*filter_category=*
Disallow: /*search*filter_duration=*
Disallow: /*search*o=*
Disallow: /*o=*search*
Disallow: /*showDuration=*
Disallow: /*video_related*
Disallow: /*watch_player*
Disallow: /album/
Disallow: /albums/*-*
Disallow: /albums/*/*
Disallow: /channels/search*
Disallow: /chat/index*
Disallow: /chronopop/*
Disallow: /front/
Disallow: /gay/gifs/search
Disallow: /gifgenerator?viewkey=*
Disallow: /iframe.php*
Disallow: /live*
Disallow: /photo/
Disallow: /phubapi*
Disallow: /players/
Disallow: /playlist/rate*
Disallow: /pornstars*=*=*
Disallow: /pornstars*age=*
Disallow: /pornstars*cup=*
Disallow: /pornstars/search*
Disallow: /pornstar/*/playlists*
Disallow: /pornstar/*/gifs*
Disallow: /pornstar/*/comments*
Disallow: /pornstar/*/similar*
Disallow: /pornstar/*/costars*
Disallow: /pornstar/*/videos/paid
Disallow: /rlock/
Disallow: /select/video/*
Disallow: /shemale/gifs/search
Disallow: /svvt*
Disallow: /*token=*
Disallow: /uploading/gifgenerator*
Disallow: /user/
Disallow: /users/
Disallow: /video/ajax_search_related*
Disallow: /video/ajax_video_related*
Disallow: /video/player_related_datas*
Disallow: /video/pornstar_suggestion_list*
Disallow: /video/privates*
Disallow: /video/propose_pornstar*
Disallow: /video/rate*
Disallow: /video/search_autocomplete*
Disallow: /video/search*p=*o=*
Disallow: /wall_player_v2.php
Disallow: /webmasters*
Disallow: /xmoov_flv/player/index.php
Disallow: /video/search?search=*the+hun+
Disallow: /video/search?search=*the+hun$
Disallow: /video/search?search=*thehun+
Disallow: /video/search?search=*thehun$
Disallow: /model/*/playlists*
Disallow: /model/*/gifs*
Disallow: /model/*/comments*
Disallow: /model/*/similar*
Disallow: /model/*/costars*
Disallow: /model/*/videos/paid
Disallow: */iframe-mobile*

Allow: /live$
Allow: /live?track=*
Allow: /pornstar/videos*
Allow: /model/videos*

User-Agent: bingbot
User-agent: YandexBot
User-agent: YandexVideo
Allow: /embed*

User-agent: Twitterbot
Disallow:

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of pornhub.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).

API: https://infrarabbit.com/crawlers/api/v1/policy/pornhub.com