washingtonpost.com: what it tells AI crawlers

On 27 October 2026, washingtonpost.com's robots.txt fully blocked ClaudeBot, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.washingtonpost.com/robots.txt
unchanged since 2026-10-26 · sha256 918970c4ae59aa1f… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 0ab3b1648a100fe6…

User-agent: *
Disallow: /*_print.html
Disallow: /*_email.html
Disallow: /*_singlePage.html
Disallow: /*_allComments.html
Disallow: /*_jsn.json
Disallow: /*_jsonpStatic.js
Disallow: /*_nitf.xml
Disallow: /*_newsml.html
Disallow: /*_qa.html
Disallow: /*_meta.xml
Disallow: /*_jsnp.js
Disallow: /*_json.json
Disallow: /*_search.html
Disallow: /*_jsonp.js
Disallow: /*_jsnpStatic.js
Disallow: /*_rss.xml
Disallow: /*_mobile.mobile
Disallow: /*_mobile.xml
Disallow: /*_allCommentsClassicBlog.html
Disallow: /*_seo.html
Disallow: /*_nimbusJson.json
Disallow: /*_nimbusJsonp.js
Disallow: /*_nimbusJsonpStatic.js
Disallow: /*_modal.html
Disallow: /todays_paper/
Disallow: /rw/WashingtonPost/Content/Epaper/
Disallow: /ac2/
Disallow: /blogs/slow-ride/
Disallow: /local/blogsandcolumns/slow-ride-story-tanked
Disallow: /local/blogsandcolumns/slow-ride-story-achenblog
Disallow: /local/blogsandcolumns/slow-ride-stream-tanked
Disallow: /local/blogsandcolumns/slow-ride-front
Disallow: /utils/
Disallow: /jobs/JS_JobSearchResult
Disallow: /jobs/UpdateJobEmployerCounterServlet
Disallow: /jobs/JS_Login
Disallow: /jobs/EU_UpdateJobEmployerCounter
Disallow: /blogs/nationals-journal-beta/
Disallow: /blogs/test/
Disallow: /posttv-beta/
Disallow: /posttv/sponsored-video/
Disallow: /posttv/c/trendex/
Disallow: /posttv/c/video_search/
Disallow: /posttv/posttv/trendex
Disallow: /posttv/c/embed/
Disallow: /rweb/
Disallow: /wp-stat/vrroom/
Disallow: /classic-apps/
Disallow: /news/test/
Disallow: /tablet/
Disallow: /news/tablet/
Disallow: /sf/test/
Disallow: /news/test-liveblog/
Disallow: /pb/
Allow: /pb/resources/
Allow: /pb/gr/
Allow: /pb/resource/
Disallow: /homepage-video-test
Disallow: /testpage-forhomepage
Disallow: /knowmore
Disallow: /test/
Disallow: /brand-studio/
Disallow: /sslsingle
Disallow: /sf/brand-connect/
Disallow: /amphtml/news/test/
Disallow: /amphtml/blogs/test/
Disallow: /amphtml/classic-apps/
Disallow: /amphtml/utils/
Disallow: /newsletter/
Disallow: /sf/brand-connect/wp/
Disallow: /wp-dyn/
Disallow: /bandito/
Disallow: /Fragment/SysConfig/
Disallow: /my-post/
Disallow: /recipes/search/
Disallow: /talk/
Disallow: /wp-stat/ad/
Disallow: /*?*outputType=comment
Disallow: /pwapi-proxy/pwproxy/*
Disallow: /pwapiv2/
Disallow: /*?*outputType=accessibility
Disallow: /*?*resType=
Disallow: /*?*nid=
Disallow: /*?*category=
Disallow: /*?*node=
Disallow: /*?*noredirect=
Disallow: /*?*wprss=
Disallow: /*?*destination=
Disallow: /*?*wpisrc=
Disallow: /*?*resType=
Disallow: /*?*postshare=
Disallow: /*?*fbclid=
Disallow: /*?*wpmm=
Disallow: /wp-adv/
Disallow: /subscribe/
Disallow: /newssearch/
Disallow: /wp-admin/
Disallow: /gdpr-consent/
Disallow: /*?*outputType=tracking

User-agent: Twitterbot
Allow: /posttv-beta/
Disallow: /amphtml/*

Sitemap: https://www.washingtonpost.com/arcio/sitemap/master/index/
Sitemap: https://www.washingtonpost.com/arcio/news-sitemap/

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of washingtonpost.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/washingtonpost.com