meb.gov.tr: what it tells AI crawlers

On 28 October 2026, meb.gov.tr's robots.txt fully blocked GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel
top 10k (Tranco)
robots.txt
ok (HTTP 200) · https://meb.gov.tr/robots.txt
unchanged since 2026-10-26 · sha256 b8659a4d6e1317ed… · view as recorded

Per AI bot, 2026-10-28

User agentVerdictHow
GPTBotblocksnamed in its own group
OAI-SearchBotallowsno applicable rules
ChatGPT-Userblocksnamed in its own group
ClaudeBotblocksnamed in its own group
Claude-SearchBotallowsno applicable rules
Claude-Userallowsno applicable rules
PerplexityBotblocksnamed in its own group
Perplexity-Userallowsno applicable rules
Googlebotallowsno applicable rules
Google-Extendedblocksnamed in its own group
Applebot-Extendedblocksnamed in its own group
CCBotblocksnamed in its own group
Bytespiderblocksnamed in its own group
Meta-ExternalAgentallowsno applicable rules
Amazonbotblocksnamed in its own group
anthropic-aiblocksnamed in its own group
Applebotblocksnamed in its own group
Claude-Webblocksnamed in its own group
Diffbotblocksnamed in its own group
ExaBotblocksnamed in its own group
FacebookBotblocksnamed in its own group
ImagesiftBotblocksnamed in its own group
omgiliblocksnamed in its own group
omgilibotblocksnamed in its own group
PetalBotblocksnamed in its own group
YouBotblocksnamed in its own group

Other signals

Publishes /llms.txtno
Publishes /ai.txtno
Publishes /.well-known/tdmrep.jsonno
Reserves TDM rights (TDMRep file, header or meta)no
Declares Content Signals (robots.txt or header)no
Says ai-train=no in a Content Signalno
Points to an RSL licence (robots.txt License line or Link header)no
Answers with an x402 payment offer (lower bound: only the homepage is probed)no
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)no
Sends noai / noimageai directivesno
Refused our fetcher (403 / challenge)no

robots.txt as recorded sha256 b8659a4d6e1317ed…

User-agent: ia_archiver
Disallow: /
User-agent: archive.org_bot
Disallow: /
User-agent: WaybackMachine
Disallow: /
User-agent: archive.ph
Disallow: /
User-agent: archive.today
Disallow: /
User-agent: special_archiver
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Amazonbot
Disallow: /

User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: seoscanners.net
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: ZoominfoBot
Disallow: /
User-agent: MauiBot
Disallow: /
User-agent: Barkrowler
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: AspiegelBot
Disallow: /      
User-agent: PetalBot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: DotBot/1.0
Disallow: /      
User-agent: ZoominfoBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-Agent: FacebookBot
Disallow: /
User-Agent: Applebot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: YouBot
Disallow: /

robots.txt versions

FromThroughStateVersion
2026-10-262026-10-28ok 200b8659a4d6e13

Timeline of changes

DateWhat changed
No changes since the first observation (2026-10-26).

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

CrawlHostStateGPTBotOAI-SearchBotChatGPT-UserClaudeBotClaude-SearchBotClaude-UserPerplexityBot
2026-09-21 CC-MAIN-2026-39meb.gov.trok viewblocksallowsunknownblocksallowsunknownblocks
2025-09-15 CC-MAIN-2025-38meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2024-09-16 CC-MAIN-2024-38meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2023-10-02 CC-MAIN-2023-40meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2022-10-03 CC-MAIN-2022-40meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2021-09-27 CC-MAIN-2021-39meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2020-09-28 CC-MAIN-2020-40meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2019-09-23 CC-MAIN-2019-39meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2018-09-24 CC-MAIN-2018-39meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2018-01-29 CC-MAIN-2018-05meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2017-10-23 CC-MAIN-2017-43meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2017-09-25 CC-MAIN-2017-39meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2017-07-24 CC-MAIN-2017-30meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2017-04-24 CC-MAIN-2017-17meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2017-01-23 CC-MAIN-2017-04meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2016-10-03 CC-MAIN-2016-40meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows
2016-09-05 CC-MAIN-2016-36meb.gov.trok viewallowsallowsunknownallowsallowsunknownallows

Evidence

Every daily observation of meb.gov.tr is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (pending (4 calendars)).

API: https://infrarabbit.com/crawlers/api/v1/policy/meb.gov.tr