property24.com: what it tells AI crawlers

On 27 October 2026, property24.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 10k (Tranco)

robots.txt

ok (HTTP 200) · https://www.property24.com/robots.txt
unchanged since 2026-10-27 · sha256 d38305a507db60ef… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 1f0182e9a8fd2b4c…

User-agent: *
Disallow: /Agent/AgentHandler.ashx
Disallow: /agent/agenthandler.ashx
Disallow: /AgentZone/*
Disallow: /agentzone/*
Disallow: /Services
Disallow: /services
Disallow: /Error
Disallow: /error
Disallow: /Developers
Disallow: /developers
Disallow: /Test
Disallow: /test
Disallow: /MyProperty24
Disallow: /myproperty24
Disallow: /Login
Disallow: /login
Disallow: /Signup
Disallow: /signup
Disallow: /Reset-password
Disallow: /reset-password
Disallow: /OpenLogin
Disallow: /openlogin
Disallow: /reference-search
Disallow: /Admin
Disallow: /admin
Disallow: /Handlers
Disallow: /handlers
Disallow: /Shared/AdWords
Disallow: /shared/adwords
Disallow: /User/AlertRemoval
Disallow: /user/alertremoval
Disallow: /Autocomplete
Disallow: /autocomplete
Disallow: /General/LogContactLead
Disallow: /general/logcontactlead
Disallow: /General/SendContactMessage
Disallow: /general/sendcontactmessage
Disallow: /General/SendEmail
Disallow: /general/sendemail
Disallow: /General/SendEmail
Disallow: /general/sendemail
Disallow: /General/Open
Disallow: /general/open
Disallow: /general/open
Disallow: /general/open
Disallow: /Home/SubscribeNewsletter
Disallow: /home/subscribenewsletter
Disallow: /Listing/DownloadBrochure
Disallow: /listing/downloadbrochure
Disallow: /Listing/ListingOffenceAdd
Disallow: /listing/listingoffenceadd
Disallow: /Listing/RecentSalesView
Disallow: /listing/recentsalesview
Disallow: /Listing/RecentRentalsView
Disallow: /listing/recentrentalsview
Disallow: /ListingReadOnly/GetImage
Disallow: /listingreadonly/getimage
Disallow: /Listing/ReportListing
Disallow: /listing/reportlisting
Disallow: /Listing/SaveUserNote
Disallow: /listing/saveusernote
Disallow: /ListingReadOnly/SendAgentAMessage
Disallow: /listingreadonly/sendagentamessage
Disallow: /ListingReadOnly/ShowContactEmails
Disallow: /listingreadonly/showcontactemails
Disallow: /ListingReadOnly/ShowContactNumbers
Disallow: /listingreadonly/showcontactnumbers
Disallow: /Listing/TrendsView
Disallow: /listing/trendsview
Disallow: /ListingReadOnly/PointsOfInterestView
Disallow: /listingreadonly/pointsofinterestview
Disallow: /ListingReadOnly/PointsOfInterest
Disallow: /listingreadonly/pointsofinterest
Disallow: /ListingReadOnly/PointsOfInterestForListing
Disallow: /listingreadonly/pointsofinterestforlisting
Disallow: /for-sale/advanced-search
Disallow: /houses-for-sale/advanced-search
Disallow: /apartments-for-sale/advanced-search
Disallow: /townhouses-for-sale/advanced-search
Disallow: /vacant-land-for-sale/advanced-search
Disallow: /farms-for-sale/advanced-search
Disallow: /commercial-property-for-sale/advanced-search
Disallow: /industrial-property-for-sale/advanced-search
Disallow: /to-rent/advanced-search
Disallow: /houses-to-rent/advanced-search
Disallow: /apartments-to-rent/advanced-search
Disallow: /townhouses-to-rent/advanced-search
Disallow: /vacant-land-to-rent/advanced-search
Disallow: /farms-to-rent/advanced-search
Disallow: /commercial-property-to-rent/advanced-search
Disallow: /industrial-property-to-rent/advanced-search
Disallow: /New-developments/advanced-search
Disallow: /new-developments/advanced-search
Disallow: /Results/GroupedListings
Disallow: /results/groupedlistings
Disallow: /Results/SearchBarDevelopmentsResult
Disallow: /results/searchbardevelopmentsresult
Disallow: /user/GoogleOneTapVerification
Disallow: /user/googleonetapverification
Disallow: /everlytic/template
Disallow: /everlytic/template
Disallow: /everlytic/template/hotproperties
Disallow: /everlytic/template/hotproperties
Disallow: /mapSearch/getAreaSearchMapModel
Disallow: /mapsearch/getareasearchmapmodel
Disallow: /mapSearch/getClickedAreas
Disallow: /mapsearch/getclickedareas
Disallow: /oauth2login
Disallow: /oauth2login
Disallow: /oauth2callback
Disallow: /oauth2callback
Disallow: /user/sendDeleteUserAccountEmail
Disallow: /user/senddeleteuseraccountemail
Disallow: /User/DeleteUser
Disallow: /user/deleteuser
Disallow: /Financial/ExternalBondCalculator.aspx
Disallow: /financial/externalbondcalculator.aspx
User-agent: trovitBot
Disallow: /
Sitemap: https://www.property24.com/sitemap/Index

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of property24.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/property24.com