worldbank.org: what it tells AI crawlers

On 27 October 2026, worldbank.org's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.worldbank.org/robots.txt
unchanged since 2026-10-26 · sha256 5c36a22fe5b1a380… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 5c36a22fe5b1a380…

User-agent: *
Allow: /

# --- Adobe Edge Optimize crawler ---
User-agent: AdobeEdgeOptimize/1.0
Allow: /

# --- System / platform paths ---
Disallow: /apps/
Disallow: /conf/
Disallow: /misc/
Disallow: /wbg/services/
Disallow: /wbg/aem/
Disallow: /fragments/
Disallow: /fragment/
Disallow: /downloadstats/
Disallow: /content/search
Disallow: /*/search
Disallow: /*/webarchives/archive
Disallow: /*.print

# --- One-off retired pages ---
Disallow: /*/programs/umbrellafacilityforgenderequality/brief/ufge-partnership-council-meeting

# --- Retired templates: blocked entirely, all years, all locales ---
Disallow: /*/news/infographic/
Disallow: /*/news/loans-credits/
Disallow: /*/news/opinion/
Disallow: /*/data/interactive/

# --- Migrated templates: pre-2020 content retired (19xx, 200x, 201x) ---
Disallow: /*/news/factsheet/19
Disallow: /*/news/factsheet/200
Disallow: /*/news/factsheet/201
Disallow: /*/news/feature/19
Disallow: /*/news/feature/200
Disallow: /*/news/feature/201
Disallow: /*/news/immersive-story/19
Disallow: /*/news/immersive-story/200
Disallow: /*/news/immersive-story/201
Disallow: /*/news/press-release/19
Disallow: /*/news/press-release/200
Disallow: /*/news/press-release/201
Disallow: /*/news/speech/19
Disallow: /*/news/speech/200
Disallow: /*/news/speech/201
Disallow: /*/news/statement/19
Disallow: /*/news/statement/200
Disallow: /*/news/statement/201
Disallow: /*/results/19
Disallow: /*/results/200
Disallow: /*/results/201

Sitemap: https://www.worldbank.org/sitemap.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of worldbank.org is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/worldbank.org