weather.com: what it tells AI crawlers
On 27 October 2026, weather.com's robots.txt fully blocked CCBot; it also published llms.txt.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://weather.com/robots.txt
unchanged since 2026-10-26 · sha256489d33a498d23ca3…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-27
| User agent | Verdict | How |
|---|---|---|
| GPTBot | allows | through the * group |
| OAI-SearchBot | allows | through the * group |
| ChatGPT-User | allows | named in its own group |
| ClaudeBot | allows | through the * group |
| Claude-SearchBot | allows | through the * group |
| Claude-User | allows | through the * group |
| PerplexityBot | allows | through the * group |
| Perplexity-User | allows | through the * group |
| Googlebot | allows | through the * group |
| Google-Extended | allows | through the * group |
| Applebot-Extended | allows | through the * group |
| CCBot | blocks | named in its own group |
| Bytespider | blocks | named in its own group |
| Meta-ExternalAgent | allows | through the * group |
| Amazonbot | blocks | named in its own group |
| Brightbot | allows | named in its own group |
| Diffbot | blocks | named in its own group |
| FacebookBot | blocks | named in its own group |
| FriendlyCrawler | blocks | named in its own group |
| Google-CloudVertexBot | blocks | named in its own group |
| GoogleOther | blocks | named in its own group |
| ImagesiftBot | blocks | named in its own group |
| omgili | blocks | named in its own group |
| omgilibot | blocks | named in its own group |
| Scrapy | blocks | named in its own group |
| YouBot | blocks | named in its own group |
Other signals
| Publishes /llms.txt | yes |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | no |
| Declares Content Signals (robots.txt or header) | no |
| Says ai-train=no in a Content Signal | no |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
llms.txt: "Weather.com", 9,101 bytes, 78 links.
robots.txt as recorded sha256 8ea0bd10a20b2173…
# # /robots.txt # # # Last updated by shaun.wallace 01/21/2025 # # Explicit Allows User-agent: ChatGPT-User Allow: / # Disallowed for PhantomJS User-agent: * # Crawl-delay: 10 User-agent: CriteoBot/0.1 Disallow: Crawl-delay: .02 # Directories Disallow: /includes/ Disallow: /life/ Disallow: /misc/ Disallow: /modules/ Disallow: /profiles/ Disallow: /scripts/ Disallow: /themes/ # Files Disallow: /appspromo Disallow: /CHANGELOG.txt Disallow: /cron.php Disallow: /INSTALL.mysql.txt Disallow: /INSTALL.pgsql.txt Disallow: /INSTALL.sqlite.txt Disallow: /install.php Disallow: /INSTALL.txt Disallow: /LICENSE.txt Disallow: /MAINTAINERS.txt Disallow: /update.php Disallow: /UPGRADE.txt Disallow: /xmlrpc.php # Paths (clean URLs) Disallow: /migration/ Disallow: /admin/ Disallow: /comment/reply/ Disallow: /filter/tips/ Disallow: /node/add/ Disallow: /search/ Disallow: /user/register/ Disallow: /user/password/ Disallow: /user/login/ Disallow: /user/logout/ Disallow: /g00/ Disallow: /g01/ Disallow: /g02/ Disallow: /g03/ Disallow: /g04/ Disallow: /g05/ Disallow: /g06/ Disallow: /g07/ Disallow: /g08/ Disallow: /g09/ Disallow: /g10/ Disallow: /g11/ Disallow: /g12/ Disallow: /g13/ Disallow: /g14/ Disallow: /g15/ Disallow: /g16/ Disallow: /g17/ Disallow: /g18/ Disallow: /g19/ Disallow: /g20/ Disallow: /g21/ Disallow: /g22/ Disallow: /g23/ Disallow: /g24/ Disallow: /g25/ Disallow: /g26/ Disallow: /g27/ Disallow: /g28/ Disallow: /g29/ Disallow: /g30/ Disallow: /g31/ Disallow: /g32/ Disallow: /g33/ Disallow: /g34/ Disallow: /g35/ Disallow: /g36/ Disallow: /g37/ Disallow: /g38/ Disallow: /g39/ Disallow: /g40/ Disallow: /g41/ Disallow: /g42/ Disallow: /g43/ Disallow: /g44/ Disallow: /g45/ Disallow: /g46/ Disallow: /g47/ Disallow: /g48/ Disallow: /g49/ Disallow: /g50/ Disallow: /g51/ Disallow: /g52/ Disallow: /g53/ Disallow: /g54/ Disallow: /g55/ Disallow: /g56/ Disallow: /g57/ Disallow: /g58/ Disallow: /g59/ Disallow: /g60/ Disallow: /g61/ Disallow: /g62/ Disallow: /g63/ Disallow: /g64/ Disallow: /g65/ Disallow: /g66/ Disallow: /g67/ Disallow: /g68/ Disallow: /g69/ Disallow: /g70/ Disallow: /g71/ Disallow: /g72/ Disallow: /g73/ Disallow: /g74/ Disallow: /g75/ Disallow: /g76/ Disallow: /g77/ Disallow: /g78/ Disallow: /g79/ Disallow: /g80/ Disallow: /g81/ Disallow: /g82/ Disallow: /g83/ Disallow: /g84/ Disallow: /g85/ Disallow: /g86/ Disallow: /g87/ Disallow: /g88/ Disallow: /g89/ Disallow: /g90/ Disallow: /g91/ Disallow: /g92/ Disallow: /g93/ Disallow: /g94/ Disallow: /g95/ Disallow: /g96/ Disallow: /g97/ Disallow: /g98/ Disallow: /g99/ # Paths (no clean URLs) Disallow: /*?apiKey= Disallow: /*&apiKey= Disallow: /?q=admin/ Disallow: /?q=comment/reply/ Disallow: /?q=filter/tips/ Disallow: /?q=node/add/ Disallow: /?q=search/ Disallow: /?q=user/password/ Disallow: /?q=user/register/ Disallow: /?q=user/login/ Disallow: /?q=user/logout/ Disallow: /sponsored Disallow: /ugc Disallow: /sponsored-content # # Block Bots # User-agent: Amazonbot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: AwarioRssBot User-agent: AwarioSmartBot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: Diffbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Meta-ExternalAgent User-agent: meta-externalagent Disallow: / User-agent: NewsNow Disallow: / User-agent: news-please Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: peer39_crawler User-agent: peer39_crawler/1.0 Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Quora-Bot Disallow: / User-agent: Scrapy Disallow: / User-agent: TurnitinBot Disallow: / # # Sitemaps # Sitemap: https://weather.com/en-US/sitemaps/sitemap.xml Sitemap: https://weather.com/pt-PT/sitemaps/sitemap.xml Sitemap: https://weather.com/de-DE/sitemaps/sitemap.xml Sitemap: https://weather.com/fr-FR/sitemaps/sitemap.xml Sitemap: https://weather.com/es-US/sitemaps/sitemap.xml Sitemap: https://weather.com/es-ES/sitemaps/sitemap.xml Sitemap: https://weather.com/en-IN/sitemaps/sitemap.xml Sitemap: https://weather.com/en-GB/sitemaps/sitemap.xml Sitemap: https://weather.com/en-CA/sitemaps/sitemap.xml
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-27 | ok 200 | 489d33a498d2 |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2025-09-15 CC-MAIN-2025-38 | weather.com | ok view | blocks | allows | unknown | blocks | allows | unknown | blocks |
| 2024-09-16 CC-MAIN-2024-38 | weather.com | ok view | blocks | blocks | unknown | blocks | allows | unknown | blocks |
| 2023-10-02 CC-MAIN-2023-40 | weather.com | ok view | blocks | allows | unknown | allows | allows | unknown | allows |
| 2022-10-03 CC-MAIN-2022-40 | weather.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | weather.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | weather.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | weather.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | weather.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | weather.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-01-23 CC-MAIN-2017-04 | weather.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-10-03 CC-MAIN-2016-40 | weather.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-09-05 CC-MAIN-2016-36 | weather.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
Evidence
Every daily observation of weather.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).
API: https://infrarabbit.com/crawlers/api/v1/policy/weather.com