Methodology

What is fetched, how it is read, what is counted, and how each day is sealed. Every rule here is implemented in code and covered by tests; anyone can check a day with verify.py.

Panel

The top sites of the Tranco list, a research ranking that averages several popularity lists over 30 days. The panel is frozen on the first run of each calendar month, so a month's daily series measures the same sites. v1 observes the top 10,000 every day; shares are reported for the top 1,000 and the top 10,000. Ranks are used to cut the panel and are not republished. Current panel: list W3ZV9 of 2026-10-25.

Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczyński and Wouter Joosen. "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation." NDSS 2019. https://tranco-list.eu/

What is fetched, per site, per day

/robots.txt over https on the registrable domain, falling back to www. and then http when the apex does not answer, and to www. when the apex redirects /robots.txt to an ordinary page. Redirects are followed by hand, at most five hops, wherever they lead (RFC 9309); the final URL is recorded.
Then, only where that robots.txt allows our own token InfraRabbitBot: /llms.txt, /ai.txt, /.well-known/tdmrep.json and the homepage (headers and the first 64 KB, for TDM-Reservation and Content-Signal headers, RSL licence Link headers, tdm-reservation and noai meta tags, HTTP 402 and x402 offers). These follow redirects only within the same site (registrable domain, per the Public Suffix List); a redirect to another site is recorded as offsite and counts as absent.
When robots.txt is unavailable (5xx, unreachable, a redirect loop, 402) nothing else is requested that day and the other files are recorded skipped (RFC 9309 §2.3.1.4). When robots.txt refuses us (401/403/429, a challenge page) only the homepage is tried once, to confirm.
Each file must be what it claims to be: llms.txt starts with a Markdown H1 (# ), ai.txt carries robots-style directives, tdmrep.json is JSON naming tdm-reservation, and none of them is an HTML page. A body identical to another file of the same site that day, or matching what the site returns for a made-up path (one extra request, only when a file looked present), is a soft 404.

Politeness: about three requests per second in total, at most one per second per host, an Accept header per file, a 15 s timeout, two retries with backoff on timeouts, 5xx and 429, conditional requests (ETag, Last-Modified) for files that did not change. See our fetcher.

States

Each file gets one state per day: ok; absent (404, 410 and other "not here" answers); soft404 (a 2xx HTML page or a duplicate, not the file); invalid (a text file that is not in the file's format, counted as not published); offsite; fetch-blocked (401/403/429 or a challenge page: the site refused our fetcher, typically because it blocks datacenter IPs; this says nothing about its AI policy); payment-required and x402 (HTTP 402, with or without an x402 payment offer); server-error; unreachable; tls-error; redirect-loop; disallowed (not fetched: the site's robots.txt disallows us); skipped. Only an ok or absent robots.txt gives a verdict; every other state makes the verdicts unknown.

Verdicts

robots.txt is read with Google's rules (google/robotstxt, the reference implementation behind RFC 9309): a group is a run of user-agent lines and the rules after it, and only an allow or disallow line ends the run; the user-agent token is the leading letters, hyphens and underscores of the value, matched without regard to case; all groups naming the agent are merged and the * group applies only when none does; the longest matching pattern wins and allow wins a tie. The parser is tested against 288 conformance cases (every assertion of Google's own test suite, the RFC's examples and adversarial bodies) and against 40 real files, with Google's code as the oracle.

The verdict for an agent comes from a fixed set of probe paths: the homepage / and eleven typical content paths (/a, /about, /blog/, /blog/2026/10/some-post, /news/article-123.html, /products/item, /en/, /p/12345, /docs/getting-started, /images/photo.jpg, /category/topic/):

blocks: every content path is disallowed (the homepage alone may stay open, as with Disallow: / plus Allow: /$);
allows: every probe path is allowed. Generic rules such as Disallow: /admin/ or Disallow: /search leave a site "allows";
partly: anything in between, for example Disallow: /$ (only the homepage) or Disallow: /blog/.

Whether / itself is disallowed is recorded separately. Where RFC 9309 tells a crawler to assume "disallow everything" (a 5xx robots.txt), the ledger records unknown: that is a fetch outcome, not the site's stated policy.

Each verdict records whether the agent was named or caught by the * group. Agents come from ai.robots.txt, refreshed daily; the fifteen headline agents are fixed and each carries its role (training, search or user fetch) as its operator documents it.

Denominators

A share is k of n, where n is the panel sites whose relevant file could be observed that day. Sites where it could not be observed (unreadable robots.txt, a file not fetched, a homepage that refused us) are counted as U and shown next to every figure, with the bounds k/n and (k+U)/n, never guessed. HTTP 402 and x402 figures are lower bounds: only the homepage is probed.

Also recorded from robots.txt and headers: Cloudflare Content Signals (Content-Signal: search=yes, ai-train=no, in robots.txt or as a header) and RSL licence references (License: <url> in robots.txt, or a Link header).

Storage and changes

Raw bodies are stored once per distinct SHA-256. Observations are run-length: a row covers the days on which a file's result was identical, and is extended only when the previous run also saw it, so a missed day is a gap, never filled in. A change is recorded when a day's result differs from the previous run's; bot verdicts are compared with the last readable robots.txt, so a day of 403s does not read as two policy changes. The first observation of a site is its baseline.

Proofs

Each day's observations are sorted and hashed into a Merkle root; the statement (day, leaf count, root) is timestamped with OpenTimestamps. See Proofs.

Reconstructed history (2016 onwards)

Common Crawl publishes the robots.txt files its crawler (CCBot) fetched in each monthly crawl. The ledger reads Common Crawl's columnar index for those captures, fetches only the panel's records with byte-range requests, and evaluates them with the same parser: one September or October crawl per year since 2016 first, then one per quarter. This series is labelled reconstructed, stored separately and drawn dashed. Its trend lines use only the sites whose robots.txt was readable in every crawl plotted, and each agent's line starts at that agent's public launch date (see below). Differences from the daily series: it is what CCBot was served (some sites serve crawler-specific files or refuse CCBot), capture times vary within a crawl, and it applies today's panel to past years (survivorship).

Website cohort and bounds

Panel lists include hosts that are not websites (CDN, DNS and API domains). A site is in the website cohort W on a day when its host answered at all; U of those could not be read (refused us, a challenge or HTML page, a server error). Every headline share is printed as k / (W − U) with the bounds k/W (no unreadable site blocks) and (k+U)/W (all of them do). Trend lines use one fixed cohort, the panel of the first top-10k day, so a line never moves because the list moved.

Masking of the reconstruction

The reconstructed series of an agent starts at that agent's first public date (for example GPTBot 7 Aug 2023, Google-Extended 28 Sep 2023, Applebot-Extended 10 Jun 2024, Meta-ExternalAgent 17 Jul 2024, OAI-SearchBot 25 Jul 2024, Claude-SearchBot Mar 2025). Before it, a site could not have meant to address the agent, so no point is drawn, even where a * rule would block it in a parser. The * group's own series ("closes the whole site to every unnamed crawler") is drawn separately for the whole period.

Limits

robots.txt is a request, not an enforcement; sites also block at the CDN (for example Cloudflare's AI bot blocking), which robots.txt does not show. A fetch-blocked state is a hint of that, not proof.
Only the registrable domain is observed, not every subdomain.
Google-Extended and Applebot-Extended are opt-out tokens without their own crawlers; their verdicts are policy, not traffic.

Changes to this method

Every figure, CSV row and API answer carries the method version it was computed with.

3.0, 24 October 2026: First public method: Google's robots.txt semantics (parser v3, tested against Google's own suite), verdicts from a fixed probe set, website cohort with bounds, leaf v2 of raw facts sealed daily with OpenTimestamps, Common Crawl reconstruction masked before each agent's public launch date.