sourceforge.net: what it tells AI crawlers

On 27 October 2026, sourceforge.net's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot; it also published llms.txt.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://sourceforge.net/robots.txt
unchanged since 2026-10-26 · sha256 afd526e8de7f2ca5… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

llms.txt: "SourceForge", 1,000,000 bytes, 128 links.

robots.txt as recorded sha256 afd526e8de7f2ca5…

User-agent: Mediapartners-Google
Allow: /

User-agent: AmazonAdBot
Allow: /

User-agent: *
Allow: /p/*/ci/master/tree/$
Allow: /p/*/ci/main/tree/$
Allow: /p/*/ci/default/tree/$
Allow: /p/*/HEAD/tree/$
Allow: /p/*/HEAD/tree/trunk/$
Allow: /adobe/*/HEAD/tree/trunk/$
Allow: /software/vendors/optimize-software-for-ai-search/
Disallow: /p/*/code/
Disallow: /p/*/git/
Disallow: /p/*/svn/
Disallow: /p/*/hg/
Disallow: /p/*/search
Disallow: /p/*/code-0/
Disallow: /p/*/tree/
Disallow: /adobe/*/tree/
Disallow: /p/*/search_feed/
Disallow: /p/*/search_help/
Disallow: /p/*/wiki/*/edit
Disallow: /p/*/wiki/browse_tags/
Disallow: /p/*/discussion/create_topic/
Disallow: /p/*/discussion/stats
Disallow: /*/ci/
Disallow: /*/commit_browser
Disallow: /*/commit_browser_data
Disallow: /*?barediff=
Disallow: /*?diff=
Disallow: /*/tarball
Disallow: /code-snapshots/
Disallow: /*/bugs/new/
Disallow: /*/support-requests/new/
Disallow: /*/patches/new/
Disallow: /*/feature-requests/new/
Disallow: /*/tickets/new/
Disallow: /*/edit/
Disallow: /*/attachment/
Disallow: /*/_discuss/
Disallow: /projects/*/files/*/stats/timeline
Disallow: /projects/*/files/stats/timeline
Disallow: /projects/*/files/*/stats/map
Disallow: /projects/*/files/stats/map
Disallow: /projects/*/files/*/stats/os
Disallow: /projects/*/files/stats/os
Disallow: /projects/*/files/stats/json
Disallow: /projects/*/files/*/stats/json
Disallow: /projects/*/files/latest/download
Disallow: /projects/*/latest.json
Disallow: /projects/*/report_inappropriate
Disallow: /projects/*/reviews/new
Disallow: /projects/*/rss
Disallow: /projects/*/lists/*/unsubscribe
Disallow: /projects/*/moderate_review
Disallow: /projects/*/rate_review
Disallow: /projects/*/best_release.html
Disallow: /projects/*/postdownload
Disallow: /projects/*/get_updates
Disallow: /app/*/files/
Disallow: /app/*/latest.json
Disallow: /app/*/report_inappropriate
Disallow: /app/*/reviews/
Disallow: /app/*/rss
Disallow: /app/*/moderate_review
Disallow: /app/*/rate_review
Disallow: /app/*/best_release.html
Disallow: /app/*/postdownload
Disallow: /app/*/get_updates
Disallow: /*_escaped_fragment_
Disallow: /auth/
Disallow: /auth/do_login
Disallow: /software/link?
Disallow: /software/link$
Disallow: /software/visit?
Disallow: /software/visit$
Disallow: /software/transfer_site
Disallow: /software/compare/*-vs-*-vs-*-vs-
Disallow: /software/compare/*/add-software
Disallow: /software/product/*/reviews/new
Disallow: /software/product/*/reviews/add
Disallow: /software/product/*/claim
Disallow: /software/vendors/new
Disallow: /software/vendors/inquire
Disallow: /software/vendors/ai-llm-optimization
Disallow: /software/*?q=
Disallow: /software/*&q=
Disallow: /software/*?categories=
Disallow: /software/*&categories=
Disallow: /software/*?company_sizes=
Disallow: /software/*&company_sizes=
Disallow: /software/*?deployment=
Disallow: /software/*&deployment=
Disallow: /software/*?free_options=
Disallow: /software/*&free_options=
Disallow: /software/*?integrates_with=
Disallow: /software/*&integrates_with=
Disallow: /software/*?org_types=
Disallow: /software/*&org_types=
Disallow: /software/*?api=
Disallow: /software/*&api=
Disallow: /software/*?regions=
Disallow: /software/*&regions=
Disallow: /software/*?support=
Disallow: /software/*&support=
Disallow: /software/*?training=
Disallow: /software/*&training=
Disallow: /software/*?feature_
Disallow: /software/*&feature_
Disallow: /rest/
Allow: /rest/*/activity
Disallow: /user/registration
Disallow: /user/verified
Disallow: */bin_counts
Disallow: */milestone_counts
Disallow: */stats_data
Disallow: *//users$
Disallow: */users$
Disallow: /u/*/link/
Disallow: /p/*/link/
Disallow: /p/*/donate
Disallow: */feed.rss
Disallow: */feed.atom
Disallow: /p/*/news/feed
Disallow: /p/*/news/*/history
Disallow: /arris/*/feed$
Disallow: */activity/feed
Disallow: */activity/pjax
Disallow: /p/*/admin/
Disallow: /directory/release_feed
Disallow: *?css-reload=
Disallow: *&css-reload=
Disallow: /settings/mirror_choices
Disallow: /directory/os:windows/?q=
Disallow: /directory/tp
Disallow: /software/tp
Disallow: /sd-jobs
Disallow: /about/mirror-agreement
Disallow: /t7
Disallow: /p/*/wiki/*/history
Disallow: /p/*/*/milestone/
Disallow: /p/*/activity/
Disallow: /u/*/wiki/*
Disallow: /u/*/activity/
Allow: /u/*/profile/$
Disallow: /u/*/*/
Disallow: /*/wp-login.php
Disallow: /blog/tag/
Disallow: /blog/category/
Disallow: /articles/tag/
Disallow: /articles/category/open-source/
Disallow: /articles/category/developer-tools/
Disallow: /articles/category/artificial-intelligence/
Disallow: /articles/category/database/
Disallow: /articles/category/mobile/
Disallow: /articles/category/payroll/
Disallow: /articles/category/telephony/
Disallow: /articles/category/networking/
Disallow: /articles/category/cloud/
Disallow: /articles/category/general/
Disallow: /articles/category/software/
Disallow: /p/allura/chat/
Disallow: /p/allura/tickets/
Disallow: /about/maxnav
Disallow: /hi
Allow: /llms.txt

# /directory param rules: permit ?q= and ?page= on their own, but no other params
# longest match takes precedence.  listed in length order
Disallow: /directory/*?
Allow: /directory/*?q=
Disallow: /directory/*?q=*&
Allow: /directory/*?page=
Disallow: /directory/*?page=*&

# mailman flags
Disallow: *?style=flat
Disallow: *&style=flat
Disallow: *?style=threaded
Disallow: *&style=threaded
Disallow: *?viewmonth=
Disallow: *&viewmonth=
Disallow: *?viewmont=
Disallow: *&viewmont=
Disallow: *?viewday=
Disallow: *&viewday=

Sitemap: https://sourceforge.net/sitemap.xml
Sitemap: https://sourceforge.net/allura_sitemap/sitemap.xml
Sitemap: https://sourceforge.net/directory_sitemap.xml
Sitemap: https://sourceforge.net/software_sitemap.xml
Sitemap: https://sourceforge.net/blog/sitemap_index.xml
Sitemap: https://sourceforge.net/articles/sitemap_index.xml
Sitemap: https://sourceforge.net/freeware_sitemap.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of sourceforge.net is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/sourceforge.net