weibo.com: what it tells AI crawlers

On 26 October 2026, weibo.com's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot; it also signalled ai-train=no.

Last verified 2026-10-26 21:13 UTC (daily run of 2026-10-26). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://weibo.com/robots.txt
unchanged since 2026-10-26 · sha256 15b0db632716dd85… · view as recorded

The * group

blocks applies to every agent without its own group

Per AI bot, 2026-10-26

Other signals

Content signals: search=yes, ai-input=yes, ai-train=no for baiduspider, googlebot, sogou, bingbot, smspider, haosouspider, yisouspider; search=yes, ai-input=yes, ai-train=no for google-adstxt; search=yes, ai-input=yes, ai-train=no for perplexity-user, deepseek, deepseekbot, bytespider, deepseek, oai-searchbot, chatgpt-user, kimi, doubao, meta-externalagent, chatglm; search=yes, ai-input=yes, ai-train=no for *

robots.txt as recorded sha256 15b0db632716dd85…

Sitemap: https://weibo.com/sitemap.xml

User-Agent: Baiduspider
User-Agent: 360Spider
User-Agent: Googlebot
User-Agent: Sogou web spider
User-Agent: bingbot
User-Agent: smspider
User-Agent: HaosouSpider
User-Agent: YisouSpider
Content-Signal: search=yes, ai-input=yes, ai-train=no
Crawl-delay: 1
Disallow:

User-Agent: Google-adstxt
Content-Signal: search=yes, ai-input=yes, ai-train=no
Crawl-delay: 1
Allow: /app-ads.txt
Disallow: /

User-Agent: Perplexity-User
User-Agent: DeepSeek Chat
User-Agent: DeepSeekBot
User-Agent: Bytespider
User-Agent: DeepSeek
User-Agent: OAI-SearchBot
User-Agent: ChatGPT-User
User-Agent: Kimi
User-Agent: Doubao
User-Agent: meta-externalagent
User-Agent: ChatGLM
Content-Signal: search=yes, ai-input=yes, ai-train=no
Crawl-delay: 1
Allow: /sitemap.xml
Allow: /sitemap/
Allow: /2/detail/
Allow: /ttarticle/p/show
Disallow: /

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Crawl-delay: 1
Disallow: /
Allow: /ads.txt

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of weibo.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-26: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973294)).

API: https://infrarabbit.com/crawlers/api/v1/policy/weibo.com