About our fetcher

InfraRabbitBot fetches a handful of public, machine-readable files from the top websites once a day, to record what each site tells AI crawlers.

User-Agent

Mozilla/5.0 (compatible; InfraRabbitBot/1.0; +https://infrarabbit.com/crawlers/fetcher)

What it fetches

/robots.txt; and, only where robots.txt allows InfraRabbitBot: /llms.txt, /ai.txt, /.well-known/tdmrep.json, and the homepage's headers plus its first 64 KB.

How often

Once a day, about six requests per site (robots.txt, llms.txt, ai.txt, .well-known/tdmrep.json, the homepage, and one random path when a file is found, to tell real files from catch-all pages), at most one per second per host.

What it does not do

It does not follow links, crawl pages, run JavaScript, fill forms, or use the content for training anything.

Opt out

User-agent: InfraRabbitBot / Disallow: / in your robots.txt stops every fetch except robots.txt itself (which the protocol requires us to read).

Asking to be left out

A site's owner can have the site excluded from the ledger: it is then not fetched, recorded or reported, and it is left out of every share (the number of excluded sites is published). Requests are applied automatically once we can see they come from the site, by one of:

1. DNS: a TXT record infrarabbit-exclude=… on the domain (any non-empty value, for example infrarabbit-exclude=yes; it is checked on the domain named in the request);
2. robots.txt: User-agent: InfraRabbitBot / Disallow: / (this already stops every fetch but robots.txt itself);
3. Email from an address at the domain.

A request for a domain covers the whole site, every subdomain included; a request for one subdomain covers that host. Complaints forwarded by our hosting provider are applied the same way.

Observations sealed before the request stay in the daily proofs (they cannot be changed without breaking the proof), but they are no longer shown or counted.