ubuntu.com: what it tells AI crawlers

On 27 October 2026, ubuntu.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot; it also published llms.txt.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://ubuntu.com/robots.txt
unchanged since 2026-10-26 · sha256 aab936ae074e1cf7… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

llms.txt: "Ubuntu", 12,879 bytes, 65 links.

robots.txt as recorded sha256 aab936ae074e1cf7…

# ===========================================
# ubuntu.com robots.txt
# Strategy: 
#   - Block private/transactional/search paths for all.
#   - Group major AI Retrieval & Crawling bots to optimize crawl budget.
#   - Allow high-value product/doc paths; block noise.
# ===========================================

# ===========================================
# DEFAULT RULES — all crawlers
# ===========================================
User-Agent: *
Disallow: /search
Disallow: /search*
Disallow: /*/search*
Allow: /certified/search
Disallow: /account
Disallow: /account/*
Disallow: /login
Disallow: /logout
Disallow: /pro/dashboard
Disallow: /pro/users
Disallow: /pro/account-users
Disallow: /pro/subscribe
Disallow: /pro/activate
Disallow: /pro/attach
Disallow: /pro/offer
Disallow: /pro/offers
Disallow: /pro/renewals/
Disallow: /pro/contracts/
Disallow: /pro/trial/
Disallow: /pro/set-auto-renewal
Disallow: /pro/user-subscriptions
Disallow: /pro/distributor/users
Disallow: /pro/distributor/invoice
Disallow: /pro/distributor/thank-you
Disallow: /account.json
Disallow: /mirrors.json
Disallow: /pro/subscriptions.json
Disallow: /pro/offers.json
Disallow: /pro/channel-offers.json
Disallow: /thank-you
Disallow: /*/thank-you
Disallow: /blog/draft-blogs
Disallow: /blog/draft-blogs/*
Disallow: /tests/
Disallow: /tests/*
Disallow: /sentry-debug
Disallow: /mobile
Disallow: /mobile/*
Disallow: /phone
Disallow: /phone/*
Disallow: /tablet
Disallow: /tablet/*
Disallow: /tv
Disallow: /tv/*
Disallow: /devices
Disallow: /devices/*
Disallow: /credentials/exam*
Crawl-delay: 1

# ===========================================
# AI OPTIMIZED RULES
# Includes: OpenAI (Browsing & Crawling), Perplexity, Anthropic, and Apple
# ===========================================
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: anthropic-ai
User-agent: Claude-SearchBot
User-agent: Google-Extended
User-agent: meta-externalagent
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot-Extended
User-agent: cohere-ai
User-agent: Bytespider

# Nudge toward Markdown endpoints
Allow: /*?format=md

# High-value Content Priority
Allow: /server
Allow: /desktop
Allow: /cloud
Allow: /openstack
Allow: /kubernetes
Allow: /ceph
Allow: /containers
Allow: /core
Allow: /ai
Allow: /pro
Allow: /landscape
Allow: /security
Allow: /internet-of-things
Allow: /embedded
Allow: /hpc
Allow: /real-time
Allow: /confidential-computing
Allow: /enterprise-store
Allow: /kernel
Allow: /toolchains
Allow: /robotics
Allow: /certified
Allow: /about
Allow: /community
Allow: /download
Allow: /pricing
Allow: /training
Allow: /credentials
Allow: /support
Allow: /managed
Allow: /managed-infrastructure
Allow: /aws
Allow: /azure
Allow: /gcp
Allow: /dell
Allow: /ibm
Allow: /nvidia
Allow: /hpe
Allow: /supermicro
Allow: /blender
Allow: /blog
Allow: /tutorials
Allow: /appliance
Allow: /cpu-compatibility
Allow: /what-is-enterprise-linux

# Block "Noise" (Forms, fragments, and archives that exhaust context windows)
Disallow: /search
Disallow: /search*
Disallow: /*/search*
Allow: /certified/search
Disallow: /account
Disallow: /account/*
Disallow: /account.json
Disallow: /mirrors.json
Disallow: /login
Disallow: /logout
Disallow: /contact-us
Disallow: /contact-us/*
Disallow: /*/contact-us
Disallow: /thank-you
Disallow: /*/thank-you
Disallow: /engage
Disallow: /engage/*
Disallow: /takeovers
Disallow: /takeovers.json
Disallow: /templates/
Disallow: /frame
Disallow: /marketo/submit
Disallow: /blog/feed
Disallow: /blog/archives
Disallow: /blog/tag/
Disallow: /blog/author/
Disallow: /blog/topic/
Disallow: /blog/group/
Disallow: /blog/latest-news
Disallow: /blog/events-and-webinars
Disallow: /blog/draft-blogs
Disallow: /blog/draft-blogs/*
Disallow: /engage/resources.json
Disallow: /engage/metadata.json
Disallow: /pro/dashboard
Disallow: /pro/users
Disallow: /pro/account-users
Disallow: /pro/subscribe
Disallow: /pro/activate
Disallow: /pro/attach
Disallow: /pro/offer
Disallow: /pro/offers
Disallow: /pro/renewals/
Disallow: /pro/contracts/
Disallow: /pro/trial/
Disallow: /pro/set-auto-renewal
Disallow: /pro/user-subscriptions
Disallow: /pro/subscriptions.json
Disallow: /pro/offers.json
Disallow: /pro/channel-offers.json
Disallow: /pro/distributor/users
Disallow: /pro/distributor/invoice
Disallow: /pro/distributor/thank-you
Disallow: /subscription-centre
Disallow: /rfp
Disallow: /tests/
Disallow: /tests/*
Disallow: /_image-testing/
Disallow: /sentry-debug
Disallow: /credentials/exam*
Disallow: /mobile
Disallow: /mobile/*
Disallow: /phone
Disallow: /phone/*
Disallow: /tablet
Disallow: /tablet/*
Disallow: /tv
Disallow: /tv/*
Disallow: /devices
Disallow: /devices/*
Disallow: /kubeconeurope2020
Disallow: /masters-conference

# Performance
Crawl-delay: 2

# ===========================================
# SITEMAPS
# ===========================================
Sitemap: https://ubuntu.com/sitemap.xml

# High-Value Content Trees (Prioritized for AI Crawlers)
Sitemap: https://ubuntu.com/sitemap_tree.xml
Sitemap: https://ubuntu.com/blog/sitemap.xml
Sitemap: https://ubuntu.com/tutorials/sitemap.xml
Sitemap: https://ubuntu.com/server/docs/doc-sitemap.xml
Sitemap: https://ubuntu.com/openstack/docs/sitemap.xml
Sitemap: https://ubuntu.com/ceph/docs/sitemap.xml
Sitemap: https://ubuntu.com/security/notices/sitemap.xml
Sitemap: https://ubuntu.com/security/cves/sitemap.xml
Sitemap: https://ubuntu.com/security/vulnerabilities/sitemap.xml
Sitemap: https://ubuntu.com/community/docs/sitemap.xml

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of ubuntu.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/ubuntu.com