REST API
No key, JSON, CORS open, rate-limited per IP. Data CC BY 4.0.
| Endpoint | Returns |
|---|---|
GET /crawlers/api/v1/policy/<domain>?date=YYYY-MM-DD | What a site told AI crawlers on a day (default: latest): verdict per bot, signals, robots.txt hash, Merkle root of the day. |
GET /crawlers/api/v1/share/<ua>?from=&to=&panel=top10k | Share of the panel blocking an agent per day, with the reconstructed series alongside. |
GET /crawlers/api/v1/changes?since=&ua=&kind=&domain=&limit= | Recorded changes, newest first. |
GET /crawlers/api/v1/bots | The agent registry. |
GET /crawlers/api/v1/domain/<domain>/history | Every robots.txt version, file history, changes and reconstructed captures of a site. |
GET /crawlers/api/v1/proof/<day>[?domain=&resource=robots] | A day's Merkle root and timestamp; with a domain, the leaf and its audit path. |
GET /crawlers/api/v1/status | Last run, panel, reconstructed crawls. |
Example
curl -s https://infrarabbit.com/crawlers/api/v1/policy/nytimes.com