tumblr.com: what it tells AI crawlers

On 27 October 2026, tumblr.com's robots.txt fully blocked ClaudeBot, Google-Extended and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.tumblr.com/robots.txt
unchanged since 2026-10-26 · sha256 2892c51fb7e06e76… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 e56030cc03fd1eec…

# Google Search Engine Robot
User-agent: Googlebot
Disallow: /radar
Disallow: /audio_file
Disallow: /video_file
Disallow: /dashboard
Disallow: /x
Disallow: /svc/account
Disallow: /dashboard/notes
Disallow: /customize
Disallow: /impixu
Disallow: /liked
Disallow: /search/*
Disallow: /tagged/*+*
Disallow: /tagged/*?before=*
Disallow: /tagged/*?language=*
Disallow: /open
Disallow: /link_to
Disallow: /sticky-ad-iframe.html
Disallow: /privacy/consent

# Yahoo! Search Engine Robot
User-agent: Slurp
Disallow: /radar
Disallow: /audio_file
Disallow: /video_file
Disallow: /dashboard
Disallow: /x
Disallow: /svc/account
Disallow: /dashboard/notes
Disallow: /customize
Disallow: /impixu
Disallow: /liked
Disallow: /search/*
Disallow: /tagged/*+*
Disallow: /tagged/*?before=*
Disallow: /tagged/*?language=*
Disallow: /open
Disallow: /link_to
Disallow: /sticky-ad-iframe.html
Disallow: /privacy/consent

# Yandex Search Engine Robot
User-agent: Yandex
Disallow: /radar
Disallow: /audio_file
Disallow: /video_file
Disallow: /dashboard
Disallow: /x
Disallow: /svc/account
Disallow: /dashboard/notes
Disallow: /customize
Disallow: /impixu
Disallow: /liked
Disallow: /search/*
Disallow: /tagged/*+*
Disallow: /tagged/*?before=*
Disallow: /tagged/*?language=*
Disallow: /open
Disallow: /link_to
Disallow: /sticky-ad-iframe.html
Disallow: /privacy/consent

# Microsoft Search Engine Robot
User-agent: msnbot
Disallow: /radar
Disallow: /audio_file
Disallow: /video_file
Disallow: /dashboard
Disallow: /x
Disallow: /svc/account
Disallow: /dashboard/notes
Disallow: /customize
Disallow: /impixu
Disallow: /liked
Disallow: /search/*
Disallow: /tagged/*+*
Disallow: /tagged/*?before=*
Disallow: /tagged/*?language=*
Disallow: /open
Disallow: /link_to
Disallow: /sticky-ad-iframe.html
Disallow: /privacy/consent

# Every bot that might possibly read and respect this file.
User-agent: *
Disallow: /radar
Disallow: /audio_file
Disallow: /video_file
Disallow: /dashboard
Disallow: /x
Disallow: /svc/account
Disallow: /dashboard/notes
Disallow: /customize
Disallow: /impixu
Disallow: /liked
Disallow: /search/*
Disallow: /tagged/*+*
Disallow: /tagged/*?before=*
Disallow: /tagged/*?language=*
Disallow: /open
Disallow: /link_to
Disallow: /sticky-ad-iframe.html
Disallow: /privacy/consent
Crawl-delay: 1

# Common Crawl's crawler
User-agent: CCBot
Disallow: /

# SentiBot's crawler
User-agent: sentibot
Disallow: /

# Google Bard's crawler
User-agent: Google-Extended
Disallow: /

# Facebook's crawler
User-agent: FacebookBot
Disallow: /

# webz.io's crawler
User-agent: omgili
Disallow: /

# webz.io's crawler
User-agent: omgilibot
Disallow: /

# Amazon's crawler
User-agent: Amazonbot
Disallow: /

# ClaudeBot's crawler
User-agent: ClaudeBot
Disallow: /

# anthropic-ai's crawler
User-agent: anthropic-ai
Disallow: /

# ImageSift's AI crawler
User-agent: ImagesiftBot
Disallow: /

# Apple's AI crawler
User-agent: Applebot-Extended
Disallow: /

# TurnitinBot crawler
User-agent: TurnitinBot
Disallow: /

# Meta AI crawler
User-agent: meta-externalagent
Disallow: /

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of tumblr.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/tumblr.com