olhardigital.com.br: what it tells AI crawlers

On 27 October 2026, olhardigital.com.br's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 10k (Tranco)
robots.txt: ok (HTTP 200) · https://olhardigital.com.br/robots.txt
unchanged since 2026-10-27 · sha256 ec97c20304f7970a… · view as recorded
The * group: partial applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	partial	through the * group
OAI-SearchBot	partial	through the * group
ChatGPT-User	partial	through the * group
ClaudeBot	partial	through the * group
Claude-SearchBot	partial	through the * group
Claude-User	partial	through the * group
PerplexityBot	partial	through the * group
Perplexity-User	partial	through the * group
Googlebot	partial	through the * group
Google-Extended	partial	through the * group
Applebot-Extended	partial	through the * group
CCBot	partial	through the * group
Bytespider	partial	through the * group
Meta-ExternalAgent	partial	through the * group
Amazonbot	partial	through the * group

Other signals

Publishes /llms.txt	no
Publishes /ai.txt	no
Publishes /.well-known/tdmrep.json	no
Reserves TDM rights (TDMRep file, header or meta)	no
Declares Content Signals (robots.txt or header)	no
Says ai-train=no in a Content Signal	no
Points to an RSL licence (robots.txt License line or Link header)	no
Answers with an x402 payment offer (lower bound: only the homepage is probed)	no
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	no
Sends noai / noimageai directives	no
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 31687586ace3eff3…

User-agent: *
Sitemap: https://olhardigital.com.br/sitemap/sitemap.xml
Sitemap: https://olhardigital.com.br/sitemap-news.xml

Sitemap: https://olhardigital.com.br/curiosidades/sitemap.xml
Sitemap: https://olhardigital.com.br/curiosidades/sitemap-news.xml

Disallow: /wp-admin/
Disallow: /search/
Disallow: /entrar/
Disallow: /assinantes/api/
Disallow: /af/
Disallow: /sq/
Disallow: /am/
Disallow: /ar/
Disallow: /hy/
Disallow: /az/
Disallow: /eu/
Disallow: /be/
Disallow: /bn/
Disallow: /bs/
Disallow: /bg/
Disallow: /ca/
Disallow: /ceb/
Disallow: /ny/
Disallow: /zh-CN/
Disallow: /zh-TW/
Disallow: /co/
Disallow: /hr/
Disallow: /cs/
Disallow: /da/
Disallow: /nl/
Disallow: /en/
Disallow: /eo/
Disallow: /et/
Disallow: /tl/
Disallow: /fi/
Disallow: /fr/
Disallow: /fy/
Disallow: /gl/
Disallow: /ka/
Disallow: /de/
Disallow: /el/
Disallow: /gu/
Disallow: /ht/
Disallow: /ha/
Disallow: /haw/
Disallow: /iw/
Disallow: /hi/
Disallow: /hmn/
Disallow: /hu/
Disallow: /is/
Disallow: /ig/
Disallow: /id/
Disallow: /ga/
Disallow: /it/
Disallow: /ja/
Disallow: /jw/
Disallow: /kn/
Disallow: /kk/
Disallow: /km/
Disallow: /ko/
Disallow: /ku/
Disallow: /ky/
Disallow: /lo/
Disallow: /la/
Disallow: /lv/
Disallow: /lt/
Disallow: /lb/
Disallow: /mk/
Disallow: /mg/
Disallow: /ms/
Disallow: /ml/
Disallow: /mt/
Disallow: /mi/
Disallow: /mr/
Disallow: /mn/
Disallow: /my/
Disallow: /ne/
Disallow: /no/
Disallow: /ps/
Disallow: /fa/
Disallow: /pl/
Disallow: /pa/
Disallow: /ro/
Disallow: /ru/
Disallow: /sm/
Disallow: /gd/
Disallow: /sr/
Disallow: /st/
Disallow: /sn/
Disallow: /sd/
Disallow: /si/
Disallow: /sk/
Disallow: /sl/
Disallow: /so/
Disallow: /es/
Disallow: /su/
Disallow: /sw/
Disallow: /sv/
Disallow: /tg/
Disallow: /ta/
Disallow: /te/
Disallow: /th/
Disallow: /tr/
Disallow: /uk/
Disallow: /ur/
Disallow: /uz/
Disallow: /vi/
Disallow: /cy/
Disallow: /xh/
Disallow: /yi/
Disallow: /yo/
Disallow: /zu/


User-agent: Amazonbot
Disallow: /

User-agent: magpie-crawler
Disallow: /

User-agent: CCBot
Disallow: /

User-Agent: omgili
Disallow: /
 
User-Agent: omgilibot
Disallow: /
 
User-agent: Claude-Web
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /
 
User-agent: cohere-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-Agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: Google-CloudVertexBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: YandexAdditional
Disallow: /

User-agent: YandexAdditionalBot
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: Amzn-SearchBot
Disallow: /

User-agent: Amzn-User
Disallow: /

User-agent: ProRataInc
Disallow: /

robots.txt versions

From	Through	State	Version
2026-10-27	2026-10-27	ok 200	`ec97c20304f7`	diff with previous
2026-10-26	2026-10-26	ok 200	`31687586ace3`

Timeline of changes

Date	What changed
2026-10-27	robots.txt changed
2026-10-27	now partly blocks Amazonbot (was block)
2026-10-27	now partly blocks Amzn-SearchBot (was block)
2026-10-27	now partly blocks Amzn-User (was block)
2026-10-27	now partly blocks anthropic-ai (was block)
2026-10-27	now partly blocks Applebot-Extended (was block)
2026-10-27	now partly blocks Bytespider (was block)
2026-10-27	now partly blocks CCBot (was block)
2026-10-27	now partly blocks ChatGPT-User (was block)
2026-10-27	now partly blocks Claude-Web (was block)
2026-10-27	now partly blocks ClaudeBot (was block)
2026-10-27	now partly blocks cohere-ai (was block)
2026-10-27	now partly blocks Google-CloudVertexBot (was block)
2026-10-27	now partly blocks Google-Extended (was block)
2026-10-27	now partly blocks GPTBot (was block)
2026-10-27	now partly blocks Meta-ExternalAgent (was block)
2026-10-27	now partly blocks OAI-SearchBot (was block)
2026-10-27	now partly blocks omgili (was block)
2026-10-27	now partly blocks omgilibot (was block)
2026-10-27	now partly blocks Perplexity-User (was block)
2026-10-27	now partly blocks PerplexityBot (was block)
2026-10-27	now partly blocks PetalBot (was block)
2026-10-27	now partly blocks Scrapy (was block)
2026-10-27	now partly blocks YandexAdditional (was block)
2026-10-27	now partly blocks YandexAdditionalBot (was block)

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	olhardigital.com.br	ok view	blocks	blocks	unknown	blocks	partial	unknown	blocks
2025-09-15 CC-MAIN-2025-38	olhardigital.com.br	ok view	blocks	partial	unknown	partial	partial	unknown	partial
2024-09-16 CC-MAIN-2024-38	olhardigital.com.br	ok view	blocks	partial	unknown	partial	partial	unknown	partial
2021-09-27 CC-MAIN-2021-39	olhardigital.com.br	ok view	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	olhardigital.com.br	ok view	allows	allows	unknown	allows	allows	unknown	allows
2019-09-23 CC-MAIN-2019-39	olhardigital.com.br	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-09-24 CC-MAIN-2018-39	olhardigital.com.br	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-04-23 CC-MAIN-2018-17	olhardigital.com.br	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-01-29 CC-MAIN-2018-05	olhardigital.com.br	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-10-23 CC-MAIN-2017-43	olhardigital.com.br	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	olhardigital.com.br	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-07-24 CC-MAIN-2017-30	olhardigital.com.br	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-04-24 CC-MAIN-2017-17	olhardigital.com.br	redirect-only	unknown	unknown	unknown	unknown	unknown	unknown	unknown
2017-01-23 CC-MAIN-2017-04	www.olhardigital.com.br	redirect-only	unknown	unknown	unknown	unknown	unknown	unknown	unknown

Evidence

Every daily observation of olhardigital.com.br is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/olhardigital.com.br