tjk.org: what it tells AI crawlers

On 27 October 2026, tjk.org's robots.txt fully blocked GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 10k (Tranco)
robots.txt: ok (HTTP 200) · https://www.tjk.org/robots.txt
unchanged since 2026-10-26 · sha256 ea5527fc41297921… · view as recorded
The * group: blocks applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	blocks	named in its own group
OAI-SearchBot	blocks	named in its own group
ChatGPT-User	blocks	named in its own group
ClaudeBot	blocks	named in its own group
Claude-SearchBot	blocks	named in its own group
Claude-User	blocks	named in its own group
PerplexityBot	blocks	named in its own group
Perplexity-User	blocks	named in its own group
Googlebot	allows	named in its own group
Google-Extended	blocks	named in its own group
Applebot-Extended	blocks	named in its own group
CCBot	blocks	named in its own group
Bytespider	blocks	named in its own group
Meta-ExternalAgent	blocks	named in its own group
Amazonbot	blocks	named in its own group
Amzn-SearchBot	blocks	named in its own group
anthropic-ai	blocks	named in its own group
Claude-Web	blocks	named in its own group
cohere-ai	blocks	named in its own group
DeepSeekBot	blocks	named in its own group
Diffbot	blocks	named in its own group
DuckAssistBot	blocks	named in its own group
FacebookBot	blocks	named in its own group
facebookexternalhit	allows	named in its own group
Gemini-Deep-Research	blocks	named in its own group
Google-CloudVertexBot	blocks	named in its own group
Google-NotebookLM	blocks	named in its own group
GrokBot	blocks	named in its own group
iaskspider	blocks	named in its own group
ImagesiftBot	blocks	named in its own group
meta-externalfetcher	blocks	named in its own group
meta-webindexer	blocks	named in its own group
MistralAI-User	blocks	named in its own group
omgili	blocks	named in its own group
omgilibot	blocks	named in its own group
PetalBot	blocks	named in its own group
Scrapy	blocks	named in its own group
Webzio-Extended	blocks	named in its own group
xAI-Grok	blocks	named in its own group
YouBot	blocks	named in its own group

Other signals

Publishes /llms.txt	not observed
Publishes /ai.txt	not observed
Publishes /.well-known/tdmrep.json	not observed
Reserves TDM rights (TDMRep file, header or meta)	not observed
Declares Content Signals (robots.txt or header)	not observed
Says ai-train=no in a Content Signal	not observed
Points to an RSL licence (robots.txt License line or Link header)	not observed
Answers with an x402 payment offer (lower bound: only the homepage is probed)	not observed
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	not observed
Sends noai / noimageai directives	not observed
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 ea5527fc41297921…

# ============================================
# Genel kural - kendine özel blok bulunmayan
# tüm botlar için varsayılan: engelli
# ============================================
User-agent: *
Disallow: /

# ============================================
# Google arama - izinli
# ============================================
User-agent: Googlebot
Allow: /

# ============================================
# Sosyal medya önizleme botları - izinli
# (zararsız, sadece link paylaşım kartı için)
# ============================================
User-agent: facebookexternalhit
Allow: /

User-agent: Facebot
Allow: /

User-agent: Twitterbot
Allow: /

User-agent: LinkedInBot
Allow: /

# ============================================
# OpenAI / ChatGPT
# ============================================
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

# ============================================
# Anthropic / Claude
# ============================================
User-agent: anthropic-ai
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: claude-web
Disallow: /

# ============================================
# Google AI (Googlebot'tan farklı - eğitim/özet amaçlı)
# ============================================
User-agent: Google-Extended
Disallow: /

User-agent: Gemini-Deep-Research
Disallow: /

User-agent: Google-CloudVertexBot
Disallow: /

User-agent: Google-NotebookLM
Disallow: /

# ============================================
# Meta AI
# ============================================
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: Meta-WebIndexer
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: MetaBot
Disallow: /

User-agent: MetaAI
Disallow: /

# ============================================
# Apple AI
# ============================================
User-agent: Applebot-Extended
Disallow: /

# ============================================
# Amazon
# ============================================
User-agent: Amazonbot
Disallow: /

User-agent: Amzn-SearchBot
Disallow: /

# ============================================
# Perplexity AI
# ============================================
User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

# ============================================
# xAI / Grok
# ============================================
User-agent: GrokBot
Disallow: /

User-agent: xAI-Grok
Disallow: /

User-agent: Grok-DeepSearch
Disallow: /

# ============================================
# Diğer AI botları
# ============================================
User-agent: Bytespider
Disallow: /

User-agent: DuckAssistBot
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: MistralAI-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Omgili
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: webzio-extended
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: proximic
Disallow: /

# ============================================
# SEO / backlink analiz botları
# ============================================
User-agent: SemrushBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: SEOkicks
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: serpstatbot
Disallow: /

# ============================================
# Arşivleme botları
# ============================================
User-agent: ia_archiver
Disallow: /

# ============================================
# Genel scraping araçları (varsayılan UA)
# ============================================
User-agent: Scrapy
Disallow: /

User-agent: python-requests
Disallow: /

User-agent: curl
Disallow: /

User-agent: Wget
Disallow: /

# ============================================
# Diğer arama motorları (yalnızca Google izinli)
# ============================================
User-agent: Bingbot
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: DuckDuckBot
Disallow: /

User-agent: iaskspider

Disallow: /



User-agent: img2dataset

Disallow: /

robots.txt versions

From	Through	State	Version
2026-10-26	2026-10-27	ok 200	`ea5527fc4129`

Timeline of changes

Date	What changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	www.tjk.org	ok view	blocks	blocks	unknown	blocks	blocks	unknown	blocks
2025-09-15 CC-MAIN-2025-38	www.tjk.org	absent	allows	allows	unknown	allows	allows	unknown	allows
2024-09-16 CC-MAIN-2024-38	www.tjk.org	absent	allows	allows	unknown	allows	allows	unknown	allows
2023-10-02 CC-MAIN-2023-40	www.tjk.org	absent	allows	allows	unknown	allows	allows	unknown	allows
2022-10-03 CC-MAIN-2022-40	www.tjk.org	absent	allows	allows	unknown	allows	allows	unknown	allows
2021-09-27 CC-MAIN-2021-39	www.tjk.org	absent	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	www.tjk.org	server-error	unknown	unknown	unknown	unknown	unknown	unknown	unknown
2019-09-23 CC-MAIN-2019-39	www.tjk.org	server-error	unknown	unknown	unknown	unknown	unknown	unknown	unknown
2018-09-24 CC-MAIN-2018-39	www.tjk.org	server-error	unknown	unknown	unknown	unknown	unknown	unknown	unknown
2017-09-25 CC-MAIN-2017-39	www.tjk.org	server-error	unknown	unknown	unknown	unknown	unknown	unknown	unknown
2016-10-03 CC-MAIN-2016-40	tjk.org	server-error	unknown	unknown	unknown	unknown	unknown	unknown	unknown
2016-09-05 CC-MAIN-2016-36	tjk.org	server-error	unknown	unknown	unknown	unknown	unknown	unknown	unknown

Evidence

Every daily observation of tjk.org is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/tjk.org