nypost.com: what it tells AI crawlers

On 28 October 2026, nypost.com's robots.txt fully blocked ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel

top 10k (Tranco)

robots.txt

ok (HTTP 200) · https://nypost.com/robots.txt
unchanged since 2026-10-26 · sha256 868d2f594a02a1f7… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-28

Other signals

robots.txt as recorded sha256 74830e2f0c0ca0da…

# If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
# Please see https://developer.wordpress.com/docs/firehose/ for more details.

Sitemap: https://nypost.com/news-sitemap.xml

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /5850/
User-agent: *
Disallow: /search/
Disallow: /ss-post/
Disallow: /tv/channel-finder-modal/
Disallow: /*?s=
Disallow: /*&s=
Disallow: /*?s&
Disallow: /*&s&
Disallow: /*?q=
Disallow: /*&q=
Disallow: /*?q&
Disallow: /*&q&
Disallow: /*?x=
Disallow: /*?_ga=
Disallow: /*&_ga=
Disallow: /*?action=
Disallow: /*&action=
Disallow: /*?adid=
Disallow: /*&adid=
Disallow: /*?bitrate=
Disallow: /*&bitrate=
Disallow: /*?channel=
Disallow: /*&channel=
Disallow: /*?clipid=
Disallow: /*&clipid=
Disallow: /*?CMP=
Disallow: /*&CMP=
Disallow: /*?contentCollection=
Disallow: /*&contentCollection=
Disallow: /*?contentId=
Disallow: /*&contentId=
Disallow: /*?ei=
Disallow: /*&ei=
Disallow: /*?FEEDNAME=
Disallow: /*&FEEDNAME=
Disallow: /*?format=
Disallow: /*&format=
Disallow: /*?height=
Disallow: /*&height=
Disallow: /*?iframe=
Disallow: /*&iframe=
Disallow: /*?link=
Disallow: /*&link=
Disallow: /*?mediaId=
Disallow: /*&mediaId=
Disallow: /*?module=
Disallow: /*&module=
Disallow: /*?monthnum=
Disallow: /*&monthnum=
Disallow: /*?pgtype=
Disallow: /*&pgtype=
Disallow: /*?preview=
Disallow: /*&preview=
Disallow: /*?priority=
Disallow: /*&priority=
Disallow: /*?ref=
Disallow: /*&ref=
Disallow: /*?referrer=
Disallow: /*&referrer=
Disallow: /*?sa=
Disallow: /*&sa=
Disallow: /*?section=
Disallow: /*&section=
Disallow: /*?share=
Disallow: /*&share=
Disallow: /*?TB_iframe=
Disallow: /*&TB_iframe=
Disallow: /*?TBL_Prod100Page=
Disallow: /*&TBL_Prod100Page=
Disallow: /*?theme_preview=
Disallow: /*&theme_preview=
Disallow: /*?usg=
Disallow: /*&usg=
Disallow: /*?ved=
Disallow: /*&ved=
Disallow: /*?version=
Disallow: /*&version=
Disallow: /*?VID=
Disallow: /*&VID=
Disallow: /*?vxBitrate=
Disallow: /*&vxBitrate=
Disallow: /*?vxChannel=
Disallow: /*&vxChannel=
Disallow: /*?vxClipId=
Disallow: /*&vxClipId=
Disallow: /*?vxSiteId=
Disallow: /*&vxSiteId=
Disallow: /*?width=
Disallow: /*&width=
Disallow: /*?year=
Disallow: /*&year=
Disallow: /*.xml?uuid=
Disallow: /*?post_type=feedback&p=
Disallow: /*?wptouch_preview_theme=
Disallow: /*.htm$
Disallow: /*>
Disallow: /*/p/
Disallow: /p/
Disallow: /?p=
Disallow: /?attachment_id=
# Sitemap archive

Disallow: /wp-login.php
Disallow: /activate/ # har har
Disallow: /cgi-bin/ # MT refugees
Disallow: /mshots/v1/
Disallow: /next/
Disallow: /public.api/

User-agent: IRLbot
Crawl-delay: 3600

# This file was generated on Fri, 20 Apr 2018 21:02:54 +0000
# Additional sitemaps
Sitemap: https://nypost.com/sitemap-nypost-section.xml
Sitemap: https://nypost.com/sitemap-nypost-post_tag.xml
Sitemap: https://nypost.com/sitemap-nypost-authors.xml
Sitemap: https://nypost.com/sitemap-nypost-pages.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of nypost.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).

API: https://infrarabbit.com/crawlers/api/v1/policy/nypost.com