fmkorea.com: what it tells AI crawlers

On 27 October 2026, fmkorea.com's robots.txt fully blocked GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel

top 10k (Tranco)

robots.txt

ok (HTTP 200) · https://www.fmkorea.com/robots.txt
unchanged since 2026-10-26 · sha256 e787da32aedebb00… · view as recorded

The * group

blocks applies to every agent without its own group

Per AI bot, 2026-10-27

Other signals

robots.txt as recorded sha256 e787da32aedebb00…

##############################
# 0) Disallowed bots
# Allow only main page
##############################
User-agent: AhrefsBot
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: BLEXBot
User-agent: BoardReader
User-agent: Bytespider
User-agent: CCBot
User-agent: ClaudeBot
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: DeepSeek
User-agent: DeepSeekBot
User-agent: Diffbot
User-agent: dotbot
User-Agent: FacebookBot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ia_archiver
User-agent: ICCrawler
User-agent: ImagesiftBot
User-agent: Linguee Bot
User-agent: Meta-ExternalAgent
User-Agent: MJ12bot
User-agent: Omgili
User-agent: Omgilibot
User-agent: PerplexityBot
User-agent: PetalBot
User-Agent: SemrushBot
User-agent: Sogou inst spider
User-agent: Sogou web spider
User-Agent: The Knowledge AI
User-Agent: trendictionbot
User-agent: TurnitinBot
User-agent: Tweetmemebot
User-agent: YouBot
User-agent: webzio
User-agent: Webzio-Extended
Disallow: /
Allow: /$

##############################
# 1A) Bots with unrestricted access to all pages
# Allow ALL pages (overrides generic rules)
##############################
User-agent: AmazonAdBot
User-agent: facebookexternalhit
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: OAI-SearchBot
User-agent: Perplexity-User
User-agent: Mediapartners-Google
User-agent: Twitterbot
Allow: /
Disallow:

##############################
# 1) Allowed bots (others)
# Allow most pages, with some path/query blocks
##############################
User-agent: Googlebot
User-agent: Yeti
User-agent: Daum
User-agent: ZumBot
User-agent: bingbot
User-agent: Applebot
User-agent: DuckDuckBot
Allow: /

# Blocks for these bots:
Disallow: /*listStyle=
Disallow: /*act=IS$
Disallow: /*act=IS&
Disallow: /*act=dispBoardCategory
Disallow: /*act=procFileDownload
Disallow: /*search_keyword=
Disallow: /*search_target=
Disallow: /*module_srl=
Disallow: /*act=dispMemberBookmark
Disallow: /*_filter=
Disallow: /*m=0&
Disallow: /*m=0$
Disallow: /*m=1&
Disallow: /*m=1$
Disallow: /*m=6&
Disallow: /*m=6$
Disallow: /_loader

##############################
# 2) Other bots
# (Default: block everything, then allow specific paths)
##############################
User-agent: *
Disallow: /

# Open these for all other bots:
Allow: /$
Allow: /best
Allow: /best2
Allow: /humor

# Blocks apply to everyone in this group:
Disallow: /*listStyle=
Disallow: /*act=IS$
Disallow: /*act=IS&
Disallow: /*act=dispBoardCategory
Disallow: /*act=procFileDownload
Disallow: /*search_keyword=
Disallow: /*search_target=
Disallow: /*module_srl=
Disallow: /*act=dispMemberBookmark
Disallow: /*_filter=
Disallow: /*m=0&
Disallow: /*m=0$
Disallow: /*m=1&
Disallow: /*m=1$
Disallow: /*m=6&
Disallow: /*m=6$
Disallow: /_loader

#DaumWebMasterTool:126e8987b7185a10b189be1660c2008a3d8c29e0b7c63563c57ac5af1e368a89:aAqAmLBe1oQFjbn0AvTMAw==

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of fmkorea.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/fmkorea.com