airbnb.com: what it tells AI crawlers

On 27 October 2026, airbnb.com's robots.txt fully blocked GPTBot and ClaudeBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 1k (Tranco)
robots.txt: ok (HTTP 200) · https://www.airbnb.com/robots.txt
unchanged since 2026-10-26 · sha256 e274a840ea29686b… · view as recorded
The * group: allows applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	blocks	named in its own group
OAI-SearchBot	allows	named in its own group
ChatGPT-User	allows	named in its own group
ClaudeBot	blocks	named in its own group
Claude-SearchBot	allows	through the * group
Claude-User	allows	through the * group
PerplexityBot	allows	named in its own group
Perplexity-User	allows	through the * group
Googlebot	allows	named in its own group
Google-Extended	allows	through the * group
Applebot-Extended	blocks	named in its own group
CCBot	allows	through the * group
Bytespider	allows	through the * group
Meta-ExternalAgent	allows	named in its own group
Amazonbot	allows	through the * group
anthropic-ai	allows	named in its own group
cohere-ai	allows	named in its own group
ImagesiftBot	blocks	named in its own group
MistralAI-Training	blocks	named in its own group
Webzio-Extended	blocks	named in its own group

Other signals

Publishes /llms.txt	no
Publishes /ai.txt	no
Publishes /.well-known/tdmrep.json	no
Reserves TDM rights (TDMRep file, header or meta)	no
Declares Content Signals (robots.txt or header)	no
Says ai-train=no in a Content Signal	no
Points to an RSL licence (robots.txt License line or Link header)	no
Answers with an x402 payment offer (lower bound: only the homepage is probed)	no
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	no
Sends noai / noimageai directives	no
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 0625fc5582cf29f2…

#                      ///////
#                     //     //
#                    //       //
#                   //         //                           ///             ///                      ///
#                  //           //                                          ///                      ///
#                 //     ///     //               //// ///  ///  /// ////   /// ////     /// ////    /// ////
#                //   ///   ///   //            //////////  ///  ////////// ///////////  //////////  ///////////
#               //   //       //   //          ///     ///  ///  ///        ///      /// ///     /// ///      ///
#              //    //       //    //        ///      ///  ///  ///        ///      /// ///     /// ///      ///
#             //      //     //      //        ///     ///  ///  ///        ///     ///  ///     /// ///     ///
#            //        //   //        //        //////////  ///  ///        //////////   ///     /// //////////
#            //         /////         //
#            //         /////         //
#             //      ///   ///      //
#               //////         //////
#
#
#    We thought you'd never make it!
#    We hope you feel right at home in this file...unless you're a disallowed subfolder.
#    And since you're here, read up on our culture and team: https://www.airbnb.com/careers/departments/engineering
#    There's even a bring your robot to work day.


User-agent: Googlebot
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
Disallow: /account
Disallow: /alumni
Disallow: /associates/click
Disallow: /api/v1/trebuchet
Disallow: /calendar/
Disallow: /disaster/lookup
Disallow: /email/unsubscribe
Disallow: /fix-it
Disallow: /fixit
Disallow: /forgot_password
Disallow: /groups
Disallow: /help/search
Disallow: /help/feedback
Disallow: /home/dashboard
Disallow: /inbox
Disallow: /logout
Disallow: /manage-listing
Disallow: /messaging/ajax_already_messaged/
Disallow: /my_listings
Disallow: /skeleton$
Disallow: /skeleton/
Disallow: /payments/book
Disallow: /signup_modal
Disallow: /signed_out_modal.json
Disallow: /.well-known/assetlinks.json
Disallow: /device_id_bev_map
Disallow: /help/search
Disallow: /trips/upcoming
Disallow: /trips/v1/
Disallow: /update-your-browser
Disallow: /reservation

        Disallow: /rooms/*/safety
        Disallow: /rooms/*/reviews
        Disallow: /rooms/*/photos
        Disallow: /rooms/*/location
        Disallow: /rooms/*/house-rules
        Disallow: /rooms/*/enhanced-cleaning
        Disallow: /rooms/*/amenities
        Disallow: /users/show
        Disallow: /users/*/listings
        Disallow: /contact_host
        Disallow: /book
        Disallow: /stories
        Disallow: /embeddable
        Disallow: /guidebooks
        



User-agent: Bingbot
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
Disallow: /account
Disallow: /alumni
Disallow: /associates/click
Disallow: /api/v1/trebuchet
Disallow: /calendar/
Disallow: /disaster/lookup
Disallow: /email/unsubscribe
Disallow: /fix-it
Disallow: /fixit
Disallow: /forgot_password
Disallow: /groups
Disallow: /help/search
Disallow: /help/feedback
Disallow: /home/dashboard
Disallow: /inbox
Disallow: /logout
Disallow: /manage-listing
Disallow: /messaging/ajax_already_messaged/
Disallow: /my_listings
Disallow: /skeleton$
Disallow: /skeleton/
Disallow: /payments/book
Disallow: /signup_modal
Disallow: /signed_out_modal.json
Disallow: /.well-known/assetlinks.json
Disallow: /device_id_bev_map
Disallow: /help/search
Disallow: /trips/upcoming
Disallow: /trips/v1/
Disallow: /update-your-browser
Disallow: /reservation

        Disallow: /rooms/*/safety
        Disallow: /rooms/*/reviews
        Disallow: /rooms/*/photos
        Disallow: /rooms/*/location
        Disallow: /rooms/*/house-rules
        Disallow: /rooms/*/enhanced-cleaning
        Disallow: /rooms/*/amenities
        Disallow: /users/show
        Disallow: /users/*/listings
        Disallow: /contact_host
        Disallow: /book
        Disallow: /stories
        Disallow: /embeddable
        Disallow: /guidebooks
        



User-agent: YandexBot
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
Disallow: /account
Disallow: /alumni
Disallow: /associates/click
Disallow: /api/v1/trebuchet
Disallow: /calendar/
Disallow: /disaster/lookup
Disallow: /email/unsubscribe
Disallow: /fix-it
Disallow: /fixit
Disallow: /forgot_password
Disallow: /groups
Disallow: /help/search
Disallow: /help/feedback
Disallow: /home/dashboard
Disallow: /inbox
Disallow: /logout
Disallow: /manage-listing
Disallow: /messaging/ajax_already_messaged/
Disallow: /my_listings
Disallow: /skeleton$
Disallow: /skeleton/
Disallow: /payments/book
Disallow: /signup_modal
Disallow: /signed_out_modal.json
Disallow: /.well-known/assetlinks.json
Disallow: /device_id_bev_map
Disallow: /help/search
Disallow: /trips/upcoming
Disallow: /trips/v1/
Disallow: /update-your-browser
Disallow: /reservation

        Disallow: /rooms/*/safety
        Disallow: /rooms/*/reviews
        Disallow: /rooms/*/photos
        Disallow: /rooms/*/location
        Disallow: /rooms/*/house-rules
        Disallow: /rooms/*/enhanced-cleaning
        Disallow: /rooms/*/amenities
        Disallow: /users/show
        Disallow: /users/*/listings
        Disallow: /contact_host
        Disallow: /book
        Disallow: /stories
        Disallow: /embeddable
        Disallow: /guidebooks
        



User-agent: Baiduspider
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
Disallow: /account
Disallow: /alumni
Disallow: /associates/click
Disallow: /api/v1/trebuchet
Disallow: /calendar/
Disallow: /disaster/lookup
Disallow: /email/unsubscribe
Disallow: /fix-it
Disallow: /fixit
Disallow: /forgot_password
Disallow: /groups
Disallow: /help/search
Disallow: /help/feedback
Disallow: /home/dashboard
Disallow: /inbox
Disallow: /logout
Disallow: /manage-listing
Disallow: /messaging/ajax_already_messaged/
Disallow: /my_listings
Disallow: /skeleton$
Disallow: /skeleton/
Disallow: /payments/book
Disallow: /signup_modal
Disallow: /signed_out_modal.json
Disallow: /.well-known/assetlinks.json
Disallow: /device_id_bev_map
Disallow: /help/search
Disallow: /trips/upcoming
Disallow: /trips/v1/
Disallow: /update-your-browser
Disallow: /reservation

        Disallow: /rooms/*/safety
        Disallow: /rooms/*/reviews
        Disallow: /rooms/*/photos
        Disallow: /rooms/*/location
        Disallow: /rooms/*/house-rules
        Disallow: /rooms/*/enhanced-cleaning
        Disallow: /rooms/*/amenities
        Disallow: /users/show
        Disallow: /users/*/listings
        Disallow: /contact_host
        Disallow: /book
        Disallow: /stories
        Disallow: /embeddable
        Disallow: /guidebooks
        



User-Agent: *
Allow: /calendar/ical/
Allow: /.well-known/amphtml/apikey.pub
Disallow: /account
Disallow: /alumni
Disallow: /associates/click
Disallow: /api/v1/trebuchet
Disallow: /calendar/
Disallow: /disaster/lookup
Disallow: /email/unsubscribe
Disallow: /fix-it
Disallow: /fixit
Disallow: /forgot_password
Disallow: /groups
Disallow: /help/search
Disallow: /help/feedback
Disallow: /home/dashboard
Disallow: /inbox
Disallow: /logout
Disallow: /manage-listing
Disallow: /messaging/ajax_already_messaged/
Disallow: /my_listings
Disallow: /skeleton$
Disallow: /skeleton/
Disallow: /payments/book
Disallow: /signup_modal
Disallow: /signed_out_modal.json
Disallow: /.well-known/assetlinks.json
Disallow: /device_id_bev_map
Disallow: /help/search
Disallow: /trips/upcoming
Disallow: /trips/v1/
Disallow: /update-your-browser
Disallow: /reservation

        Disallow: /rooms/*/safety
        Disallow: /rooms/*/reviews
        Disallow: /rooms/*/photos
        Disallow: /rooms/*/location
        Disallow: /rooms/*/house-rules
        Disallow: /rooms/*/enhanced-cleaning
        Disallow: /rooms/*/amenities
        Disallow: /users/show
        Disallow: /users/*/listings
        Disallow: /contact_host
        Disallow: /book
        Disallow: /stories
        Disallow: /embeddable
        Disallow: /guidebooks
        



Sitemap: https://www.airbnb.com/sitemap-master-index.xml.gz

robots.txt versions

From	Through	State	Version
2026-10-26	2026-10-28	ok 200	`e274a840ea29`

Timeline of changes

Date	What changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2025-09-15 CC-MAIN-2025-38	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2024-09-16 CC-MAIN-2024-38	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2023-10-02 CC-MAIN-2023-40	www.airbnb.com	ok view	blocks	allows	unknown	allows	allows	unknown	allows
2022-10-03 CC-MAIN-2022-40	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2021-09-27 CC-MAIN-2021-39	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2019-09-23 CC-MAIN-2019-39	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-09-24 CC-MAIN-2018-39	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-04-23 CC-MAIN-2018-17	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-01-29 CC-MAIN-2018-05	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-10-23 CC-MAIN-2017-43	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-07-24 CC-MAIN-2017-30	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-04-24 CC-MAIN-2017-17	airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-01-23 CC-MAIN-2017-04	airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-10-03 CC-MAIN-2016-40	www.airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-09-05 CC-MAIN-2016-36	airbnb.com	ok view	allows	allows	unknown	allows	allows	unknown	allows

Evidence

Every daily observation of airbnb.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/airbnb.com