tandfonline.com: what it tells AI crawlers

On 27 October 2026, tandfonline.com's robots.txt fully blocked GPTBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.tandfonline.com/robots.txt
unchanged since 2026-10-26 · sha256 70adb5768df9af8c… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 0667b26be1b8b2e0…

Sitemap: https://www.tandfonline.com/sitemap-index-1.xml
User-agent: *
Disallow: /action
Disallow: /help
Disallow: /search
Disallow: /feedback
Disallow: /rss
Disallow: /page/account-confirmation-thanks
Disallow: /media
Disallow: /medical-research
Disallow: /servlet/linkout
Disallow: /na101/
Disallow: /na101v1/
Disallow: /na102/
Disallow: /doi/mlt/
Disallow: /topic
Disallow: /author/
Disallow: /keyword/
Disallow: /action/showPublications*ConceptID=*
Disallow: /action/showPublications*alphabetRange=*
Allow: /action/showPublications
Allow: /action/showXml
Allow: /action/showCoverImage
Allow: /sda/journalIssueList/
Allow: /action/showOpenGraphCoverImage
Allow: /doi/cover-img
Allow: /action/journalInformation*aimsScope*
Allow: /action/journalInformation*editorialBoard*
Allow: /action/journalInformation*journalMetrics*
Allow: /topic/articlecollection
Allow: /topic/sections

Crawl-delay: 1

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of tandfonline.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/tandfonline.com