github.com: what it tells AI crawlers
On 26 October 2026, github.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot; it also published llms.txt.
Last verified 2026-10-26 21:13 UTC (daily run of 2026-10-26). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://github.com/robots.txt
unchanged since 2026-10-26 · sha256b622e2dd9b60e854…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-26
| User agent | Verdict | How |
|---|---|---|
| GPTBot | allows | named in its own group |
| OAI-SearchBot | allows | named in its own group |
| ChatGPT-User | allows | through the * group |
| ClaudeBot | allows | named in its own group |
| Claude-SearchBot | allows | through the * group |
| Claude-User | allows | through the * group |
| PerplexityBot | allows | named in its own group |
| Perplexity-User | allows | through the * group |
| Googlebot | allows | through the * group |
| Google-Extended | allows | through the * group |
| Applebot-Extended | allows | through the * group |
| CCBot | allows | through the * group |
| Bytespider | blocks | named in its own group |
| Meta-ExternalAgent | allows | through the * group |
| Amazonbot | allows | through the * group |
| anthropic-ai | allows | named in its own group |
Other signals
| Publishes /llms.txt | yes |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | no |
| Declares Content Signals (robots.txt or header) | no |
| Says ai-train=no in a Content Signal | no |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
llms.txt: "GitHub", 28,658 bytes, 117 links.
robots.txt as recorded sha256 b622e2dd9b60e854…
# If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots # We also provide an extensive API: https://docs.github.com Sitemap: https://github.com/sitemap_marketing.xml User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: PerplexityBot Crawl-delay: 1 Allow: /*?tab=achievements&achievement=* Allow: /marketplace/*?tab=readme$ Allow: /marketplace/*?tab=transparency$ Disallow: /account-login Disallow: /copilot/ Disallow: /copilot/c/ Disallow: /ekansa/Open-Context-Data Disallow: /ekansa/opencontext-* Disallow: /Explodingstuff/ Disallow: /gist/ Disallow: /search/advanced Disallow: /search$ Disallow: /*/*/pulse_committer_data.json$ Disallow: /*/*/pulse_committer_data.json?* Disallow: /*/archive/ Disallow: /*/blame/ Disallow: /*/cache/ Disallow: /*/raw/ Disallow: /*/tree/ Disallow: */tarball/ Disallow: */zipball/ Disallow: /.git/ Disallow: */.git/ Disallow: /*.git$ Disallow: /*.atom$ Disallow: /*report-abuse?report=* Disallow: /*/*/commits/ Disallow: /marketplace/*/*?tab=readme$ Disallow: /marketplace/*/*?tab=transparency$ Disallow: /marketplace/*?*?tab=readme$ Disallow: /marketplace/*?*?tab=transparency$ Disallow: /marketplace/actions?tab=readme$ Disallow: /marketplace/actions?tab=transparency$ Disallow: /marketplace/free-trials?tab=readme$ Disallow: /marketplace/free-trials?tab=transparency$ Disallow: /marketplace/manage?tab=readme$ Disallow: /marketplace/manage?tab=transparency$ Disallow: /marketplace/models?tab=readme$ Disallow: /marketplace/models?tab=transparency$ Disallow: /marketplace/new?tab=readme$ Disallow: /marketplace/new?tab=transparency$ Disallow: /marketplace/publishers?tab=readme$ Disallow: /marketplace/publishers?tab=transparency$ Disallow: /marketplace/recently-added?tab=readme$ Disallow: /marketplace/recently-added?tab=transparency$ Disallow: /biztools$ Disallow: /biztools?* Disallow: /biztools/* Disallow: /businesses$ Disallow: /businesses?* Disallow: /businesses/* Disallow: /enterprises$ Disallow: /enterprises?* Disallow: /enterprises/* Disallow: /organizations$ Disallow: /organizations?* Disallow: /organizations/* Disallow: /settings$ Disallow: /settings?* Disallow: /settings/* Disallow: /stafftools$ Disallow: /stafftools?* Disallow: /stafftools/* Disallow: /users/*/billing$ Disallow: /users/*/billing?* Disallow: /users/*/billing/* Disallow: /writing$ Disallow: /writing?* Disallow: /writing/* Disallow: /*/*/branches$ Disallow: /*/*/branches?* Disallow: /*/*/branches/* Disallow: /*/*/comments$ Disallow: /*/*/comments?* Disallow: /*/*/comments/* Disallow: /*/*/compare$ Disallow: /*/*/compare?* Disallow: /*/*/compare/* Disallow: /*/*/contributors$ Disallow: /*/*/contributors?* Disallow: /*/*/contributors/* Disallow: /*/*/contributors_list$ Disallow: /*/*/contributors_list?* Disallow: /*/*/contributors_list/* Disallow: /*/*/contributors-list$ Disallow: /*/*/contributors-list?* Disallow: /*/*/contributors-list/* Disallow: /*/*/download$ Disallow: /*/*/download?* Disallow: /*/*/download/* Disallow: /*/*/forks$ Disallow: /*/*/forks?* Disallow: /*/*/forks/* Disallow: /*/*/graphs$ Disallow: /*/*/graphs?* Disallow: /*/*/graphs/* Disallow: /*/*/issues/new$ Disallow: /*/*/issues/new?* Disallow: /*/*/issues/new/* Disallow: /*/*/issues/search$ Disallow: /*/*/issues/search?* Disallow: /*/*/issues/search/* Disallow: /*/*/milestones/new$ Disallow: /*/*/milestones/new?* Disallow: /*/*/milestones/new/* Disallow: /*/*/network$ Disallow: /*/*/network?* Disallow: /*/*/network/* Disallow: /*/*/projects$ Disallow: /*/*/projects?* Disallow: /*/*/projects/* Disallow: /*/*/pulse$ Disallow: /*/*/pulse?* Disallow: /*/*/pulse/* Disallow: /*/*/pulse-new$ Disallow: /*/*/pulse-new?* Disallow: /*/*/pulse-new/* Disallow: /*/*/pulse_committer_data$ Disallow: /*/*/pulse_committer_data?* Disallow: /*/*/pulse_committer_data/* Disallow: /*/*/pulse_diffstat_summary$ Disallow: /*/*/pulse_diffstat_summary?* Disallow: /*/*/pulse_diffstat_summary/* Disallow: /*/*/revisions$ Disallow: /*/*/revisions?* Disallow: /*/*/revisions/* Disallow: /*/*/stargazers$ Disallow: /*/*/stargazers?* Disallow: /*/*/stargazers/* Disallow: /*/*/tags$ Disallow: /*/*/tags?* Disallow: /*/*/tags/* Disallow: /*/*/watchers$ Disallow: /*/*/watchers?* Disallow: /*/*/watchers/* Disallow: /*?source=* Disallow: /*&source=* Disallow: /*?ref_cta=* Disallow: /*&ref_cta=* Disallow: /*?plan=* Disallow: /*&plan=* Disallow: /*?return_to=* Disallow: /*&return_to=* Disallow: /*?ref_loc=* Disallow: /*&ref_loc=* Disallow: /*?setup_organization=* Disallow: /*&setup_organization=* Disallow: /*?source_repo=* Disallow: /*&source_repo=* Disallow: /*?ref_page=* Disallow: /*&ref_page=* Disallow: /*?referrer=* Disallow: /*&referrer=* Disallow: /*?report=* Disallow: /*&report=* Disallow: /*?author=* Disallow: /*&author=* Disallow: /*?since=* Disallow: /*&since=* Disallow: /*?until=* Disallow: /*&until=* Disallow: /*?tab=* Disallow: /*&tab=* Disallow: /*?q=* Disallow: /*&q=* User-agent: Bytespider Disallow: / User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile Disallow: /account-login Disallow: /copilot/ Disallow: /copilot/c/ Disallow: /ekansa/Open-Context-Data Disallow: /ekansa/opencontext-* Disallow: /Explodingstuff/ Disallow: /gist/ Disallow: /search/advanced Disallow: /search$ Disallow: /*/*/pulse_committer_data.json$ Disallow: /*/*/pulse_committer_data.json?* Disallow: /*/archive/ Disallow: /*/blame/ Disallow: /*/cache/ Disallow: /*/raw/ Disallow: /*/tree/ Disallow: */tarball/ Disallow: */zipball/ Disallow: /.git/ Disallow: */.git/ Disallow: /*.git$ Disallow: /*.atom$ Disallow: /*report-abuse?report=* Disallow: /*/*/commits/ Disallow: /marketplace/*/*?tab=readme$ Disallow: /marketplace/*/*?tab=transparency$ Disallow: /marketplace/*?*?tab=readme$ Disallow: /marketplace/*?*?tab=transparency$ Disallow: /marketplace/actions?tab=readme$ Disallow: /marketplace/actions?tab=transparency$ Disallow: /marketplace/free-trials?tab=readme$ Disallow: /marketplace/free-trials?tab=transparency$ Disallow: /marketplace/manage?tab=readme$ Disallow: /marketplace/manage?tab=transparency$ Disallow: /marketplace/models?tab=readme$ Disallow: /marketplace/models?tab=transparency$ Disallow: /marketplace/new?tab=readme$ Disallow: /marketplace/new?tab=transparency$ Disallow: /marketplace/publishers?tab=readme$ Disallow: /marketplace/publishers?tab=transparency$ Disallow: /marketplace/recently-added?tab=readme$ Disallow: /marketplace/recently-added?tab=transparency$ Disallow: /biztools$ Disallow: /biztools?* Disallow: /biztools/* Disallow: /businesses$ Disallow: /businesses?* Disallow: /businesses/* Disallow: /enterprises$ Disallow: /enterprises?* Disallow: /enterprises/* Disallow: /organizations$ Disallow: /organizations?* Disallow: /organizations/* Disallow: /settings$ Disallow: /settings?* Disallow: /settings/* Disallow: /stafftools$ Disallow: /stafftools?* Disallow: /stafftools/* Disallow: /users/*/billing$ Disallow: /users/*/billing?* Disallow: /users/*/billing/* Disallow: /writing$ Disallow: /writing?* Disallow: /writing/* Disallow: /*/*/branches$ Disallow: /*/*/branches?* Disallow: /*/*/branches/* Disallow: /*/*/comments$ Disallow: /*/*/comments?* Disallow: /*/*/comments/* Disallow: /*/*/compare$ Disallow: /*/*/compare?* Disallow: /*/*/compare/* Disallow: /*/*/contributors$ Disallow: /*/*/contributors?* Disallow: /*/*/contributors/* Disallow: /*/*/contributors_list$ Disallow: /*/*/contributors_list?* Disallow: /*/*/contributors_list/* Disallow: /*/*/contributors-list$ Disallow: /*/*/contributors-list?* Disallow: /*/*/contributors-list/* Disallow: /*/*/download$ Disallow: /*/*/download?* Disallow: /*/*/download/* Disallow: /*/*/forks$ Disallow: /*/*/forks?* Disallow: /*/*/forks/* Disallow: /*/*/graphs$ Disallow: /*/*/graphs?* Disallow: /*/*/graphs/* Disallow: /*/*/issues/new$ Disallow: /*/*/issues/new?* Disallow: /*/*/issues/new/* Disallow: /*/*/issues/search$ Disallow: /*/*/issues/search?* Disallow: /*/*/issues/search/* Disallow: /*/*/milestones/new$ Disallow: /*/*/milestones/new?* Disallow: /*/*/milestones/new/* Disallow: /*/*/network$ Disallow: /*/*/network?* Disallow: /*/*/network/* Disallow: /*/*/projects$ Disallow: /*/*/projects?* Disallow: /*/*/projects/* Disallow: /*/*/pulse$ Disallow: /*/*/pulse?* Disallow: /*/*/pulse/* Disallow: /*/*/pulse-new$ Disallow: /*/*/pulse-new?* Disallow: /*/*/pulse-new/* Disallow: /*/*/pulse_committer_data$ Disallow: /*/*/pulse_committer_data?* Disallow: /*/*/pulse_committer_data/* Disallow: /*/*/pulse_diffstat_summary$ Disallow: /*/*/pulse_diffstat_summary?* Disallow: /*/*/pulse_diffstat_summary/* Disallow: /*/*/revisions$ Disallow: /*/*/revisions?* Disallow: /*/*/revisions/* Disallow: /*/*/stargazers$ Disallow: /*/*/stargazers?* Disallow: /*/*/stargazers/* Disallow: /*/*/tags$ Disallow: /*/*/tags?* Disallow: /*/*/tags/* Disallow: /*/*/watchers$ Disallow: /*/*/watchers?* Disallow: /*/*/watchers/* Disallow: /*?source=* Disallow: /*&source=* Disallow: /*?ref_cta=* Disallow: /*&ref_cta=* Disallow: /*?plan=* Disallow: /*&plan=* Disallow: /*?return_to=* Disallow: /*&return_to=* Disallow: /*?ref_loc=* Disallow: /*&ref_loc=* Disallow: /*?setup_organization=* Disallow: /*&setup_organization=* Disallow: /*?source_repo=* Disallow: /*&source_repo=* Disallow: /*?ref_page=* Disallow: /*&ref_page=* Disallow: /*?referrer=* Disallow: /*&referrer=* Disallow: /*?report=* Disallow: /*&report=* Disallow: /*?author=* Disallow: /*&author=* Disallow: /*?since=* Disallow: /*&since=* Disallow: /*?until=* Disallow: /*&until=* Disallow: /*?tab=* Disallow: /*&tab=* Disallow: /*?q=* Disallow: /*&q=* User-agent: bingbot User-agent: adidxbot User-agent: BingPreview Disallow: /ekansa/Open-Context-Data Disallow: /ekansa/opencontext-* Disallow: /account-login Disallow: */tarball/ Disallow: */zipball/ Disallow: /Explodingstuff/ Disallow: /copilot/ Disallow: /copilot/c/ User-agent: baidu crawl-delay: 1 User-agent: * Allow: /*?tab=achievements&achievement=* Allow: /marketplace/*?tab=readme$ Allow: /marketplace/*?tab=transparency$ Disallow: /account-login Disallow: /copilot/ Disallow: /copilot/c/ Disallow: /ekansa/Open-Context-Data Disallow: /ekansa/opencontext-* Disallow: /Explodingstuff/ Disallow: /gist/ Disallow: /search/advanced Disallow: /search$ Disallow: /*/*/pulse_committer_data.json$ Disallow: /*/*/pulse_committer_data.json?* Disallow: /*/archive/ Disallow: /*/blame/ Disallow: /*/cache/ Disallow: /*/raw/ Disallow: /*/tree/ Disallow: */tarball/ Disallow: */zipball/ Disallow: /.git/ Disallow: */.git/ Disallow: /*.git$ Disallow: /*.atom$ Disallow: /*report-abuse?report=* Disallow: /*/*/commits/ Disallow: /marketplace/*/*?tab=readme$ Disallow: /marketplace/*/*?tab=transparency$ Disallow: /marketplace/*?*?tab=readme$ Disallow: /marketplace/*?*?tab=transparency$ Disallow: /marketplace/actions?tab=readme$ Disallow: /marketplace/actions?tab=transparency$ Disallow: /marketplace/free-trials?tab=readme$ Disallow: /marketplace/free-trials?tab=transparency$ Disallow: /marketplace/manage?tab=readme$ Disallow: /marketplace/manage?tab=transparency$ Disallow: /marketplace/models?tab=readme$ Disallow: /marketplace/models?tab=transparency$ Disallow: /marketplace/new?tab=readme$ Disallow: /marketplace/new?tab=transparency$ Disallow: /marketplace/publishers?tab=readme$ Disallow: /marketplace/publishers?tab=transparency$ Disallow: /marketplace/recently-added?tab=readme$ Disallow: /marketplace/recently-added?tab=transparency$ Disallow: /biztools$ Disallow: /biztools?* Disallow: /biztools/* Disallow: /businesses$ Disallow: /businesses?* Disallow: /businesses/* Disallow: /enterprises$ Disallow: /enterprises?* Disallow: /enterprises/* Disallow: /organizations$ Disallow: /organizations?* Disallow: /organizations/* Disallow: /settings$ Disallow: /settings?* Disallow: /settings/* Disallow: /stafftools$ Disallow: /stafftools?* Disallow: /stafftools/* Disallow: /users/*/billing$ Disallow: /users/*/billing?* Disallow: /users/*/billing/* Disallow: /writing$ Disallow: /writing?* Disallow: /writing/* Disallow: /*/*/branches$ Disallow: /*/*/branches?* Disallow: /*/*/branches/* Disallow: /*/*/comments$ Disallow: /*/*/comments?* Disallow: /*/*/comments/* Disallow: /*/*/compare$ Disallow: /*/*/compare?* Disallow: /*/*/compare/* Disallow: /*/*/contributors$ Disallow: /*/*/contributors?* Disallow: /*/*/contributors/* Disallow: /*/*/contributors_list$ Disallow: /*/*/contributors_list?* Disallow: /*/*/contributors_list/* Disallow: /*/*/contributors-list$ Disallow: /*/*/contributors-list?* Disallow: /*/*/contributors-list/* Disallow: /*/*/download$ Disallow: /*/*/download?* Disallow: /*/*/download/* Disallow: /*/*/forks$ Disallow: /*/*/forks?* Disallow: /*/*/forks/* Disallow: /*/*/graphs$ Disallow: /*/*/graphs?* Disallow: /*/*/graphs/* Disallow: /*/*/issues/new$ Disallow: /*/*/issues/new?* Disallow: /*/*/issues/new/* Disallow: /*/*/issues/search$ Disallow: /*/*/issues/search?* Disallow: /*/*/issues/search/* Disallow: /*/*/milestones/new$ Disallow: /*/*/milestones/new?* Disallow: /*/*/milestones/new/* Disallow: /*/*/network$ Disallow: /*/*/network?* Disallow: /*/*/network/* Disallow: /*/*/projects$ Disallow: /*/*/projects?* Disallow: /*/*/projects/* Disallow: /*/*/pulse$ Disallow: /*/*/pulse?* Disallow: /*/*/pulse/* Disallow: /*/*/pulse-new$ Disallow: /*/*/pulse-new?* Disallow: /*/*/pulse-new/* Disallow: /*/*/pulse_committer_data$ Disallow: /*/*/pulse_committer_data?* Disallow: /*/*/pulse_committer_data/* Disallow: /*/*/pulse_diffstat_summary$ Disallow: /*/*/pulse_diffstat_summary?* Disallow: /*/*/pulse_diffstat_summary/* Disallow: /*/*/revisions$ Disallow: /*/*/revisions?* Disallow: /*/*/revisions/* Disallow: /*/*/stargazers$ Disallow: /*/*/stargazers?* Disallow: /*/*/stargazers/* Disallow: /*/*/tags$ Disallow: /*/*/tags?* Disallow: /*/*/tags/* Disallow: /*/*/watchers$ Disallow: /*/*/watchers?* Disallow: /*/*/watchers/* Disallow: /*?source=* Disallow: /*&source=* Disallow: /*?ref_cta=* Disallow: /*&ref_cta=* Disallow: /*?plan=* Disallow: /*&plan=* Disallow: /*?return_to=* Disallow: /*&return_to=* Disallow: /*?ref_loc=* Disallow: /*&ref_loc=* Disallow: /*?setup_organization=* Disallow: /*&setup_organization=* Disallow: /*?source_repo=* Disallow: /*&source_repo=* Disallow: /*?ref_page=* Disallow: /*&ref_page=* Disallow: /*?referrer=* Disallow: /*&referrer=* Disallow: /*?report=* Disallow: /*&report=* Disallow: /*?author=* Disallow: /*&author=* Disallow: /*?since=* Disallow: /*&since=* Disallow: /*?until=* Disallow: /*&until=* Disallow: /*?tab=* Disallow: /*&tab=* Disallow: /*?q=* Disallow: /*&q=*
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-26 | 2026-10-27 | ok 200 | b622e2dd9b60 |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | github.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2024-09-16 CC-MAIN-2024-38 | github.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2022-10-03 CC-MAIN-2022-40 | github.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | github.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | github.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | github.com | ok view | blocks | blocks | unknown | blocks | blocks | unknown | blocks |
| 2018-09-24 CC-MAIN-2018-39 | github.com | ok view | blocks | blocks | unknown | blocks | blocks | unknown | blocks |
| 2017-09-25 CC-MAIN-2017-39 | github.com | ok view | blocks | blocks | unknown | blocks | blocks | unknown | blocks |
| 2016-09-05 CC-MAIN-2016-36 | github.com | ok view | blocks | blocks | unknown | blocks | blocks | unknown | blocks |
Evidence
Every daily observation of github.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-26: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973294)).
API: https://infrarabbit.com/crawlers/api/v1/policy/github.com