Points to an RSL licence (robots.txt License line or Link header)
not observed
Answers with an x402 payment offer (lower bound: only the homepage is probed)
not observed
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)
not observed
Sends noai / noimageai directives
not observed
Refused our fetcher (403 / challenge)
no
robots.txt as recorded sha256 2737b80bbf47a6bc…
# If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
# Please see https://developer.wordpress.com/docs/firehose/ for more details.
Sitemap: https://wordpress.com/sitemap.xml
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /activate/ # har har
Disallow: /cgi-bin/ # MT refugees
Disallow: /mshots/v1/
Disallow: /next/
Disallow: /public.api/
User-agent: IRLbot
Crawl-delay: 3600
# This file was generated on Wed, 31 Aug 2016 23:18:58 +0000
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
Every daily observation of wp.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).