nasa.gov: what it tells AI crawlers

On 28 October 2026, nasa.gov's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.nasa.gov/robots.txt
unchanged since 2026-10-26 · sha256 44d72c5ad1a94bb4… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-28

Other signals

robots.txt as recorded sha256 9cfdcf5a10eb7fd3…

# Robots.txt file from http://www.nasa.gov
#
# All robots will spider the domain

User-agent: *
Disallow: /
Allow: /tribal_affairs/comments/
Allow: /aeroresearch/programs/aavp/hicam/leadership-and-contact/
Allow: /content/questions-about-nasa-s-equity-action-plan/
Allow: /office/procurement/covid19-contractor-eo-questions/
Allow: /content/anonymous-cui-challenge/
Allow: /centers/nssc/forms/sbir-search-form/
Allow: /centers/nssc/forms/grant-status-form/
Allow: /centers/nssc/forms/sbir-sttr-extension-request/
Allow: /centers/nssc/forms/grant-pi-transfer-request/
Allow: /centers/nssc/forms/grantcooperative-agreement-no-cost-extension-request/
Allow: /centers/nssc/forms/etdy-questionnaire/
Allow: /centers/nssc/forms/change-of-station/
Allow: /centers/nssc/forms/web-inquiry/
Allow: /centers/nssc/forms/relocation-amendment-form/
Allow: /centers/nssc/forms/relocation-form/
Allow: /centers/nssc/forms/accounts-receivable/
Allow: /aeroresearch/programs/tacp/cas/contacts/
Allow: /partnerships/form.html
Allow: /offices/meo/software/meo_form.html
Allow: /offices/oeod/cf/OEOD-Customer-Service-Survey/
Allow: /centers/glenn/about/contact_us.html
Allow: /centers/goddard/about/email-goddard.html
Allow: /offices/nesc/contacts/contact_us.html
Allow: /content/submit-a-question-for-nasa/
Allow: /help/contact/index.html
Allow: /flightlog/
Allow: /mail.php
Allow: /content/global-reach/
Allow: /send-your-name-with-artemis/
Allow: /envia-tu-nombre-con-artemis/
Allow: /send-your-name-with-viper/
Allow: /envia-tu-nombre-con-viper/
Allow: /skymap/
Allow: /spaceforus/
Allow: /centers/kennedy/about/foia/abacus_index.html
Allow: /centers/kennedy/about/foia/capps_index.html
Allow: /centers/kennedy/about/foia/cards.html
Allow: /centers/kennedy/about/foia/delaware-north_index.html
Allow: /centers/kennedy/about/foia/guide.html
Allow: /centers/kennedy/about/foia/indyne_index.html
Allow: /centers/kennedy/about/foia/jbosc_index.html
Allow: /centers/kennedy/about/foia/ksc_debris.html
Allow: /centers/kennedy/about/foia/ksc_FRR.html
Allow: /centers/kennedy/about/foia/ksc_OMI.html
Allow: /centers/kennedy/about/foia/ksc_related-docs.html
Allow: /centers/kennedy/about/foia/ksc_sts107_emails.html
Allow: /centers/kennedy/about/foia/ksc_TPS.html
Allow: /centers/kennedy/about/foia/lssc_index.html
Allow: /centers/kennedy/about/foia/TOSC_index.html
Allow: /centers/kennedy/about/foia/ustdc_index.html
Allow: /robots.txt
Allow: /content/request-submitted-successfully/
Allow: /content/request-failed/
Allow: /help/contact/contact_us_success.html
Allow: /help/contact/contact_us_failed.html
Allow: /contractor-eo-questions-success/
Allow: /contractor-eo-questions-failure/
Allow: /nssc-form-success/
Allow: /nssc-form-failure/
Allow: /offices/oeod/cf/success/
Allow: /offices/oeod/customer_feedback/failed.html
Allow: /goddard-form-success/
Allow: /goddard-form-failure/
Allow: /offices/nesc/contacts/contact_us_success.html
Allow: /offices/nesc/contacts/contact_us_failed.html
Allow: /centers/kennedy/pdf/
Allow: /audience/formedia/features/Breaking_News_Collection_archive_2.html
Allow: /audience/formedia/archives/2012-all-archives.html
Allow: /audience/formedia/archives/2011-all-archives.html
Allow: /audience/formedia/archives/2010-all-archives.html
Allow: /audience/formedia/archives/2010-all-archives.html
Allow: /audience/formedia/archives/2009-all-archives.html
Allow: /audience/formedia/archives/2008-all-archives.html
Allow: /audience/formedia/archives/MP_Archive_02.html
Allow: /audience/formedia/archives/MP_Archive_01.html
Allow: /audience/formedia/archives/MP_Archive_00.html
Allow: /audience/formedia/archives/MP_Archive_99.html
Allow: /audience/formedia/archives/MP_Archive_98.html
Allow: /audience/formedia/archives/MP_Archive_97.html
Allow: /audience/formedia/archives/MP_Archive_96.html
Allow: /audience/formedia/archives/MP_Archive_95.html
Allow: /audience/formedia/archives/MP_Archive_94.html
Allow: /audience/formedia/archives/MP_Archive_93.html
Allow: /audience/formedia/archives/MP_Archive_92.html
Allow: /audience/formedia/archives/MP_Archive_91.html
Allow: /audience/formedia/archives/MP_Archive_90.html
Allow: /audience/formedia/archives/2007-all-archives.html
Allow: /audience/formedia/archives/2006-all-archives.html
Allow: /audience/formedia/archives/2005-all-archives.html
Allow: /audience/formedia/archives/2004-all-archives.html
Allow: /audience/formedia/archives/2003-all-archives.html
Allow: /www3-sitemap.xml
Allow: /captcha-failure/
Allow: /specials/
Allow: /office/procurement/nail-industry-interest/

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of nasa.gov is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).

API: https://infrarabbit.com/crawlers/api/v1/policy/nasa.gov