weather.com: what it tells AI crawlers

On 27 October 2026, weather.com's robots.txt fully blocked CCBot; it also published llms.txt.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://weather.com/robots.txt
unchanged since 2026-10-26 · sha256 489d33a498d23ca3… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

llms.txt: "Weather.com", 9,101 bytes, 78 links.

robots.txt as recorded sha256 489d33a498d23ca3…

#
# /robots.txt
#

#
# Last updated by nick.hester 10/06/2026
#

#
# AI / AEO bot policy (CW-9994)
# We explicitly allow AI bots that fetch pages on behalf of a user's direct
# request (answer engines like ChatGPT-User, Brightbot) so our content can
# be cited in AI-generated answers. Bulk AI-training/indexing crawlers
# remain blocked below under "Block Bots" (e.g. YouBot, Google-CloudVertexBot).
#

# Explicit Allows

User-agent: ChatGPT-User
Allow: /

User-agent: AmazonAdBot
Allow: /

User-agent: Brightbot 1.0
Allow: /

User-agent: CriteoBot/0.1
Allow: /

# Disallowed for PhantomJS

User-agent: *
Crawl-delay: 10
# Directories
Disallow: /includes/
Disallow: /life/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
Disallow: /api/payload
Allow: /api/payload/v1/sitemaps
Disallow: /taxonomy/
# Files
Disallow: /appspromo
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /INSTALL.sqlite.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
# Paths (clean URLs)
Disallow: /migration/
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
# Paths (no clean URLs)
Disallow: /*?apiKey=
Disallow: /*&apiKey=
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=filter/tips/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Disallow: /?q=user/logout/

Disallow: /sponsored
Disallow: /ugc
Disallow: /sponsored-content

Disallow: /api/v1/p/redux-dal
# International weather language URL paths (404s)
Disallow: /tiempo/
Disallow: /clima/
Disallow: /temps/
Disallow: /vreme/
Disallow: /idojaras/
Disallow: /pogoda/
Disallow: /wetter/
Disallow: /vejret/
Disallow: /pocasi/
Disallow: /cuaca/
Disallow: /weer/
# Duplicate forecast URLs that canonicalize to the param-less page (CW-12397)
Disallow: /*/hourbyhour/day-
Disallow: /*?t=
Disallow: /*&t=
Disallow: /*?metric=
Disallow: /*&metric=
# Scoped to /localalerts so legacy /l/{id}?phenomena= 301s stay crawlable (CW-10292)
Disallow: /*/localalerts?phenomena=
Disallow: /*/localalerts?*&phenomena=

#
# Block Bots
#

User-agent: Amazonbot
Disallow: /

User-agent: AwarioRssBot
User-agent: AwarioSmartBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: FriendlyCrawler
Disallow: /

User-agent: Google-CloudVertexBot
Disallow: /

User-agent: GoogleOther
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: magpie-crawler
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: news-please
Disallow: /

User-agent: omgili
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: peer39_crawler
User-agent: peer39_crawler/1.0
Disallow: /

User-agent: Quora-Bot
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: YouBot
Disallow: /

#
# Sitemaps
#

Sitemap: https://weather.com/en-US/sitemaps/news-0-1300.xml
Sitemap: https://weather.com/en-US/sitemaps/bios-sitemap.xml
Sitemap: https://weather.com/en-US/sitemaps/collections-manual.xml
Sitemap: https://weather.com/api/payload/v1/sitemaps/en-US/news.xml
Sitemap: https://weather.com/de-DE/sitemaps/sitemap.xml
Sitemap: https://weather.com/es-US/sitemaps/sitemap.xml
Sitemap: https://weather.com/api/payload/v1/sitemaps/en-US/sitemap.xml
Sitemap: https://weather.com/api/location/sitemaps/en-US/sitemap.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of weather.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/weather.com