pornhub.com: what it tells AI crawlers

On 27 October 2026, pornhub.com's robots.txt partly blocked CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.pornhub.com/robots.txt
unchanged since 2026-10-26 · sha256 7e466730c24cb68a… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 9eb02499f47c4a6a…

User-agent: *
Sitemap: https://www.pornhub.com/sitemaps.xml
Disallow: */iframe-mobile*
Disallow: */incategories*t=*
Disallow: */pornstars/search*
Disallow: */videos/fanonly*
Disallow: */videos/private*
Disallow: *ajax=1*
Disallow: *hd=01*
Disallow: *join=*
Disallow: *premium=1*
Disallow: *promo=*
Disallow: *search=*p=*
Disallow: *search=*p=homemade*o=mr$
Disallow: *search=*p=professional*o=mr$
Disallow: *token=*
Disallow: /_xa/ads*
Disallow: /*7urren*
Disallow: /*cc=*
Disallow: /*correction=*
Disallow: /*embed_player*
Disallow: /*exclude_category=*
Disallow: /*gifs/search*o=
Disallow: /*hd=1*
Disallow: /*incategories*p=*
Disallow: /*login=*
Disallow: /*max_duration=*
Disallow: /*min_duration=*
Disallow: /*o=*search*
Disallow: /*orientation=*
Disallow: /*page*page*
Disallow: /*pkey=*
Disallow: /*player.php
Disallow: /*player_v1.6b.swf
Disallow: /*redirect=*
Disallow: /*redirectTo=*
Disallow: /*redirectto=*
Disallow: /*search*filter_category=*
Disallow: /*search*filter_duration=*
Disallow: /*search*o=*
Disallow: /*showDuration=*
Disallow: /*token=*
Disallow: /*video_related*
Disallow: /*watch_player*
Disallow: /album/
Disallow: /albums*o=*
Disallow: /albums*page=*
Disallow: /albums/*-*
Disallow: /albums/*/*
Disallow: /albums/gay?search=*
Disallow: /albums/private*
Disallow: /channels/search*
Disallow: /chat/index*
Disallow: /chronopop/*
Disallow: /embedgif/
Disallow: /front/
Disallow: /gay/gifs/search
Disallow: /gifgenerator?viewkey=*
Disallow: /iframe.php*
Disallow: /interstitial*
Disallow: /language*&p=*
Disallow: /language*?p=*
Disallow: /live*c=*
Disallow: /live*f=*
Disallow: /live*l=*
Disallow: /live*n=*
Disallow: /live*page=*
Disallow: /live*r=*
Disallow: /live*streamate=*
Disallow: /live/*
Disallow: /live*
Disallow: /model/*/*
Disallow: /model/*/live*
Disallow: /model/*/videos/paid
Disallow: /photo/
Disallow: /phubapi*
Disallow: /players/
Disallow: /playlist/rate*
Disallow: /pornstar/*/*achievements*
Disallow: /pornstar/*/*photo*=*
Disallow: /pornstar/*/*subscription*
Disallow: /pornstar/*/costars*
Disallow: /pornstar/*/friends*
Disallow: /pornstar/*/gifs*
Disallow: /pornstar/*/live*
Disallow: /pornstar/*/playlists*
Disallow: /pornstar/*/similar*
Disallow: /pornstar/*/stream*
Disallow: /pornstar/*/subscribers*
Disallow: /pornstar/*/videos/paid
Disallow: /pornstars*&age=*
Disallow: /pornstars*?age=*=*
Disallow: /pornstars*breasttype=*=*
Disallow: /pornstars*cup=*=*
Disallow: /pornstars*ethnicity=*=*
Disallow: /pornstars*gender=*=*
Disallow: /pornstars*hair=*=*
Disallow: /pornstars*piercings=*=*
Disallow: /pornstars*tattoos=*=*
Disallow: */pornstars*=*performerType=*
Disallow: */pornstars*t=*
Disallow: */pornstars*l=*
Disallow: /rlock/
Disallow: /select/video/*
Disallow: /shemale/gifs/search
Disallow: /shemale/gifs/search
Disallow: /svvt*
Disallow: /transgender/gifs/search
Disallow: /uploading/gifgenerator*
Disallow: /user/
Disallow: /users/
Disallow: /video/ajax_search_related*
Disallow: /video/ajax_video_related*
Disallow: */video*id=*
Disallow: /video/manage*
Disallow: /video/player_related_datas*
Disallow: /video/pornstar_suggestion_list*
Disallow: /video/privates*
Disallow: /video/propose_pornstar*
Disallow: /video/rate*
Disallow: /video/search*p=*o=*
Disallow: /video/search?search=*the+hun$
Disallow: /video/search?search=*the+hun+
Disallow: /video/search?search=*thehun$
Disallow: /video/search?search=*thehun+
Disallow: /video/search_autocomplete*
Disallow: /wall_player_v2.php
Disallow: /watch_stream/*
Disallow: /webmasters*
Disallow: /xmoov_flv/player/index.php
Disallow: */fanclub*
Disallow: /*/*/photos/fan*
Disallow: /*/*/photos/private

Allow: *video*search=*o=mr$
Allow: /model/*/videos*
Allow: /model/*/photos

User-agent: Twitterbot
Disallow:

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of pornhub.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/pornhub.com