rakuten.com: what it tells AI crawlers

On 27 October 2026, rakuten.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.rakuten.com/robots.txt
unchanged since 2026-10-26 · sha256 8bccbc078da0f012… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 8bccbc078da0f012…

User-agent: *
Disallow: *disallow=1*
Disallow: /.well-known/
Disallow: /*-xfas
Disallow: /*-xfcat
Disallow: /product/
Disallow: /prod-cats/
Disallow: /editUserInfos.do
Disallow: /utilities/
Disallow: /landing_merchant.jsp
Disallow: /2010.htm
Disallow: /compare/
Disallow: /searchproducts.htm
Disallow: /searchproducts_all.htm
Disallow: /searchcomparison.htm
Disallow: /search/special.htm
Disallow: /search/all.htm
Disallow: /search/stores.htm
Disallow: /myus/
Disallow: /account-info.htm
Disallow: /whitelist/
Disallow: /*?referrerid
Disallow: /*&campaign_entity_id
Disallow: /*&sourceName
Disallow: /*&eeid
Disallow: /*&sourceId
Disallow: /*&src
Disallow: /*&utm_source
Disallow: /*&utm_medium
Disallow: /*&utm_campaign
Disallow: /*&AutoLoginID
Disallow: /*&mpl_id
Disallow: /*&referrid
Disallow: /*&referrerdid
Disallow: /*&referrer
Disallow: /unifiedSearch/
Disallow: /refer-a-friend/
Disallow: /frontendEvents.do
Disallow: /button/
Disallow: /giveaway_rules/
Disallow: /wp-json/
Disallow: /feed/merchant-feed.xml
Disallow: /feed/offer-feed.xml
Disallow: /customer/feed/offer-feed.xml
Disallow: /customer/feed/merchant-feed.xml
Disallow: /logonUserServicesStartupAction.do
Disallow: /myaccount/
Disallow: /pending-cash-back.htm
Disallow: /sem/
Disallow: /ebatesCaptcha.do
Disallow: /mobile/
Disallow: /cgi-bin/
Disallow: /blog-preview/
Disallow: /blog/wp-includes/
Disallow: /blog/wp-login.php
Disallow: /blog/wp-admin/*
Disallow: /blog/wp-register.php
Disallow: /blog/search/*
Disallow: /email/
Disallow: /account/
Disallow: /ajax/
Disallow: /auth/getLogonForm.do
Disallow: /zh/index.htm
Disallow: /kr/index.htm
Disallow: /sg/index.htm
Disallow: /sem2015/
Disallow: /ad-scans/
Disallow: /_stores/
Disallow: /hotels/search-results.htm
Disallow: /hotels/verify.htm
Disallow: /hotels/reservation-management.htm
Disallow: /hotels/reservation-payment.htm
Disallow: /hotels/reservation-confirmation.htm
Disallow: /kr/
Disallow: /blog1/
Disallow: *bvstate=
Disallow: *maxp=
Disallow: *minp=
Disallow: *rtg=
Disallow: *po_c=
Disallow: *bo=
Disallow: *queryId=
Disallow: *querySubId=
Disallow: /*.vm
Disallow: /sorting/
Disallow: /m/
Disallow: /shop/search/
Disallow: /shop/*/search/
Disallow: /shop/member/
Disallow: /shop/order/
Disallow: /shop/messages/
Disallow: /shop/wishlist/
Disallow: /shop/mycoupons/
Disallow: /shop/browsing_history/
Disallow: /shop/preference-center-rmail/
Disallow: /amexblue/

User-agent: naverbot
Allow: /kr/
User-agent: yeti
Allow: /kr/
User-agent: grapeshot
Allow: /shop/search/

Sitemap: https://www.rakuten.com/sitemap-index.xml
Sitemap: https://www.rakuten.com/products/product_sitemap.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of rakuten.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/rakuten.com