washingtonpost.com: what it tells AI crawlers

On 27 October 2026, washingtonpost.com's robots.txt fully blocked ClaudeBot, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.washingtonpost.com/robots.txt
unchanged since 2026-10-26 · sha256 918970c4ae59aa1f… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 8b8cc86b07e0f414…

User-agent: *
Disallow: /*_print.html
Disallow: /*_email.html
Disallow: /*_singlePage.html
Disallow: /*_allComments.html
Disallow: /*_jsn.json
Disallow: /*_jsonpStatic.js
Disallow: /*_nitf.xml
Disallow: /*_newsml.html
Disallow: /*_qa.html
Disallow: /*_meta.xml
Disallow: /*_jsnp.js
Disallow: /*_json.json
Disallow: /*_search.html
Disallow: /*_jsonp.js
Disallow: /*_jsnpStatic.js
Disallow: /*_rss.xml
Disallow: /*_mobile.mobile
Disallow: /*_mobile.xml
Disallow: /*_allCommentsClassicBlog.html
Disallow: /*_seo.html
Disallow: /*_nimbusJson.json
Disallow: /*_nimbusJsonp.js
Disallow: /*_nimbusJsonpStatic.js
Disallow: /*_modal.html
Disallow: /todays_paper/
Disallow: /rw/WashingtonPost/Content/Epaper/
Disallow: /ac2/
Disallow: /blogs/slow-ride/
Disallow: /local/blogsandcolumns/slow-ride-story-tanked
Disallow: /local/blogsandcolumns/slow-ride-story-achenblog
Disallow: /local/blogsandcolumns/slow-ride-stream-tanked
Disallow: /local/blogsandcolumns/slow-ride-front
Disallow: /utils/
Disallow: /jobs/JS_JobSearchResult
Disallow: /jobs/UpdateJobEmployerCounterServlet
Disallow: /jobs/JS_Login
Disallow: /jobs/EU_UpdateJobEmployerCounter
Disallow: /blogs/nationals-journal-beta/
Disallow: /blogs/test/
Disallow: /posttv-beta/
Disallow: /posttv/sponsored-video/
Disallow: /posttv/c/trendex/
Disallow: /posttv/c/video_search/
Disallow: /posttv/posttv/trendex
Disallow: /posttv/c/embed/
Disallow: /rweb/
Disallow: /wp-stat/vrroom/
Disallow: /classic-apps/
Disallow: /news/test/
Disallow: /tablet/
Disallow: /news/tablet/
Disallow: /sf/test/
Disallow: /news/test-liveblog/
Disallow: /pb/
Allow: /pb/resources/
Allow: /pb/gr/
Allow: /pb/resource/
Disallow: /homepage-video-test
Disallow: /testpage-forhomepage
Disallow: /knowmore
Disallow: /test/
Disallow: /sslsingle
Disallow: /amphtml/news/test/
Disallow: /amphtml/blogs/test/
Disallow: /amphtml/classic-apps/
Disallow: /amphtml/utils/
Disallow: /newsletter/
Disallow: /wp-dyn/
Disallow: /wp-srv/
Disallow: /bandito/
Disallow: /Fragment/SysConfig/
Disallow: /recipes/search/
Disallow: /talk/
Disallow: /wp-stat/ad/
Disallow: /*?*outputType=comment
Disallow: /pwapi-proxy/pwproxy/*
Disallow: /pwapiv2/
Disallow: /*?*outputType=accessibility
Disallow: /wp-adv/
Disallow: /newssearch/
Disallow: /wp-admin/
Disallow: /gdpr-consent/
Disallow: /*?*outputType=tracking
Disallow: /tetro/
Disallow: /comments/
Disallow: /comments
Disallow: /search
Disallow: /s/*
Disallow: /embed/
Disallow: /native/
Disallow: /subscribe/braintree/
Disallow: /subscribe/enterpriseportal/
Disallow: /subscribe/foryouapi/
Disallow: /subscribe/lagoon/
Disallow: /subscribe/offers/service/
Disallow: /subscribe/onsiteapi/
Disallow: /subscribe/paywall/
Disallow: /subscribe/person/
Disallow: /subscribe/preferenceapi/
Disallow: /subscribe/subscriptionapi/
Disallow: /subscribe/user/
Disallow: /subscribe/signin/
Disallow: /subscribe/signup/
Disallow: /wpost/proxy
Disallow: /ehf/
Disallow: /ehf/*
Disallow: /subscribe/logging/*
Disallow: /blogs/*
Disallow: /gog/*
Disallow: /arcio/fact-checker/
Disallow: /recipes/*?q=
Disallow: /ripple/

User-agent: Twitterbot
Allow: /posttv-beta/
Disallow: /amphtml/*

User-agent: SemrushBot
Disallow: /licensing-syndication
Disallow: /licensing-syndication/*

User-agent: Amazonbot
Disallow: /
Allow: /wp-apps/imrs.php


# Disallow Rules

User-agent: AhrefsBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: archive.org_bot
Disallow: /

User-agent: ArchiveBot
Disallow: /

User-agent: Arquivo-web-crawler
Disallow: /

User-agent: AwarioRssBot
Disallow: /

User-agent: AwarioSmartBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: cohere-training-data-crawler
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: europarchive.org
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: heritrix
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: ia_archiver-web.archive.org
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: LinkArchiver
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: meta-externalfetcher
Disallow: /

User-agent: Nicecrawler
Disallow: /

User-agent: omgili
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Primalbot
Disallow: /

User-agent: web-archive-net.com.bot
Disallow: /

User-agent: YandexBot
Disallow: /

Sitemap: https://www.washingtonpost.com/sitemaps/sitemap.xml.gz
Sitemap: https://www.washingtonpost.com/sitemaps/news-sitemap.xml.gz
Sitemap: https://www.washingtonpost.com/sitemaps/author-sitemap.xml.gz
Sitemap: https://www.washingtonpost.com/sitemaps/section-sitemap.xml.gz
Sitemap: https://www.washingtonpost.com/elections/results/sitemap.xml
Sitemap: https://www.washingtonpost.com/arcio/sitemap/video/index/

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of washingtonpost.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/washingtonpost.com