aol.com: what it tells AI crawlers

On 27 October 2026, aol.com's robots.txt fully blocked GPTBot, ClaudeBot, Google-Extended and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.aol.com:443/robots.txt
unchanged since 2026-10-26 · sha256 25f13d669345b3ff… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 25f13d669345b3ff…

User-agent: *
Allow: /author/
Disallow: /forward
Disallow: /traffic
Disallow: /mm_track
Disallow: /dl_track
Disallow: /_uac/adpage.html
Disallow: /api/
Disallow: /amp-proxy/
Disallow: /auth
Disallow: /login
Disallow: /logout
Disallow: /config
Disallow: /_/authenticate/
Disallow: /video/play/
Disallow: /m/
Disallow: /article/m/
Disallow: /metrics/pageView
Disallow: /game/playUpdate
User-agent: ADmantX
Disallow: /
User-agent: AlphaBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: AwarioRssBot
Disallow: /
User-agent: AwarioSmartBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: Buzzbot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User
Allow: /
Disallow: /forward
Disallow: /traffic
Disallow: /mm_track
Disallow: /dl_track
Disallow: /_uac/adpage.html
Disallow: /api/
Disallow: /amp-proxy/
Disallow: /auth
Disallow: /login
Disallow: /logout
Disallow: /config
Disallow: /_/authenticate/
Disallow: /video/play/
Disallow: /m/
Disallow: /article/m/
Disallow: /metrics/pageView
Disallow: /game/playUpdate
User-agent: claritybot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: FriendlyCrawler
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: huggingface
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: img2dataset
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: Meltwater
Disallow: /
User-agent: Neevabot
Disallow: /
User-agent: news-please
Disallow: /
User-agent: NewsNow
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: omgili
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: panscient.com
Disallow: /
User-agent: Perplexity-ai
Disallow: /
User-agent: PerplexityBot
Allow: /
Disallow: /forward
Disallow: /traffic
Disallow: /mm_track
Disallow: /dl_track
Disallow: /_uac/adpage.html
Disallow: /api/
Disallow: /amp-proxy/
Disallow: /auth
Disallow: /login
Disallow: /logout
Disallow: /config
Disallow: /_/authenticate/
Disallow: /video/play/
Disallow: /m/
Disallow: /article/m/
Disallow: /metrics/pageView
Disallow: /game/playUpdate
User-agent: PetalBot
Disallow: /
User-agent: PiplBot
Disallow: /
User-agent: scoop.it
Disallow: /
User-agent: Scrapy
Disallow: /
User-agent: Seekr
Disallow: /
User-agent: SentiBot
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: YouBot
Disallow: /
User-agent: ZumBot
Disallow: /
Sitemap: https://www.aol.com/products/sitemap.xml
Sitemap: https://www.aol.com/editorial/experience-manager/us/channel_xml/sitemap.xml
Sitemap: https://www.aol.com/sitemaps/us/sitemap-index.xml.gz
Sitemap: https://www.aol.com/sitemaps/us/news-sitemap-index.xml.gz

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of aol.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/aol.com