meta.com: what it tells AI crawlers

On 27 October 2026, meta.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.meta.com/robots.txt
unchanged since 2026-10-26 · sha256 5be2af84fc90337b… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 5fb64b46ad88de50…

# Notice: Collection of data on Facebook through automated means is
# prohibited unless you have express written permission from Facebook
# and may only be conducted for the limited purpose contained in said
# permission.
# See: http://www.facebook.com/apps/site_scraping_tos_terms.php

User-agent: facebookexternalhit
Allow: *

User-agent: *
Disallow: /*cursor=
Disallow: /*fb_comment_id=
Disallow: /ajax/
Disallow: /tealium/
Disallow: /intern/
Disallow: /internal/
Disallow: /login/
Disallow: /oidc/callback/
Disallow: /*.php
Disallow: *account/
Disallow: *help/search/
Disallow: *help/support/
Disallow: *help_app/
Disallow: *rma/*
Disallow: /a/fl?
Disallow: /affiliates/
Disallow: /bv/upload/
Disallow: /campaign/
Disallow: /common/referer_frame.php
Disallow: /connect/invite/
Disallow: /meta-employee-store/
Disallow: /order/
Disallow: /return/
Disallow: /s/
Disallow: /survey/
Disallow: /my/
Disallow: /rx/


Sitemap: https://www.meta.com/sitemap/www_meta_com_sitemap.xml.gz
Sitemap: https://www.meta.com/sitemap/www_meta_com_help_sitemap.xml.gz
Sitemap: https://www.meta.com/sitemap/www_meta_com_blog_sitemap.xml.gz
Sitemap: https://www.meta.com/sitemap/www_meta_com_experiences_sitemap.xml.gz
Sitemap: https://www.meta.com/sitemap/www_meta_com_about_sitemap.xml.gz

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of meta.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/meta.com