Data downloads
CSV files, rebuilt after every daily run. Licence: CC BY 4.0; credit "InfraRabbit AI Crawl Ledger" with a link.
| File | What it holds |
|---|---|
| shares.csv | Daily share of each panel blocking, partly blocking and naming each AI user agent (first-party observations). |
| shares-reconstructed.csv | The same measures reconstructed from Common Crawl, one crawl per quarter since 2016. Separate on purpose. |
| signals.csv | Daily adoption of llms.txt, ai.txt, TDMRep, TDM headers, Content Signals, RSL, HTTP 402 / x402, noai directives, and how many sites refused our fetcher. |
| changes.csv | Every recorded change: bot verdicts, robots.txt edits, files added or removed, header changes, fetch states. |
| current.csv | Each panel site on the latest day: verdict per headline AI bot and every signal. Rank bucket only, not the Tranco rank. |
| proofs.csv | Per day: Merkle root, number of leaves, statement hash, OpenTimestamps status. |
/crawlers/data/manifest/<day>.tsv | The sorted leaves of a day's Merkle tree, to rebuild and check its root. |
Panel and sources
Panel: 10,000 sites from Tranco list W3ZV9 (2026-10-25), frozen for 2026-10. Please cite Tranco when you use panel shares: Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczyński and Wouter Joosen. "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation." NDSS 2019. https://tranco-list.eu/
Agent list: ai.robots.txt (MIT). Reconstructed series: Common Crawl robots.txt captures, used under the Common Crawl terms of use.
Bulk history and evidence certificates
Full raw history (every robots.txt, llms.txt and TDMRep version with its first and last day seen) is available under a bulk licence, and a per-site evidence certificate (PDF with the recorded file, its hash, the Merkle audit path and the OpenTimestamps proof) is planned.