vnexpress.net: what it tells AI crawlers

On 27 October 2026, vnexpress.net's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.

Panel: top 10k (Tranco)
robots.txt: ok (HTTP 200) · https://vnexpress.net/robots.txt
unchanged since 2026-10-27 · sha256 cb80f8bc213a10b9… · view as recorded
The * group: allows applies to every agent without its own group

Per AI bot, 2026-10-27

User agent	Verdict	How
GPTBot	allows	through the * group
OAI-SearchBot	allows	named in its own group
ChatGPT-User	allows	named in its own group
ClaudeBot	allows	through the * group
Claude-SearchBot	allows	named in its own group
Claude-User	allows	named in its own group
PerplexityBot	allows	named in its own group
Perplexity-User	allows	named in its own group
Googlebot	allows	through the * group
Google-Extended	allows	through the * group
Applebot-Extended	allows	through the * group
CCBot	allows	through the * group
Bytespider	allows	through the * group
Meta-ExternalAgent	allows	through the * group
Amazonbot	blocks	named in its own group
Diffbot	blocks	named in its own group
DuckAssistBot	allows	named in its own group
EchoboxBot	blocks	named in its own group
facebookexternalhit	allows	named in its own group
FriendlyCrawler	blocks	named in its own group
QuillBot	blocks	named in its own group
Scrapy	blocks	named in its own group
Timpibot	blocks	named in its own group
YouBot	blocks	named in its own group

Other signals

Publishes /llms.txt	no
Publishes /ai.txt	no
Publishes /.well-known/tdmrep.json	no
Reserves TDM rights (TDMRep file, header or meta)	no
Declares Content Signals (robots.txt or header)	no
Says ai-train=no in a Content Signal	no
Points to an RSL licence (robots.txt License line or Link header)	no
Answers with an x402 payment offer (lower bound: only the homepage is probed)	no
Answers HTTP 402 Payment Required (lower bound: only the homepage is probed)	no
Sends noai / noimageai directives	no
Refused our fetcher (403 / challenge)	no

robots.txt as recorded sha256 cb80f8bc213a10b9…

User-agent: *
Allow: /
Allow: /ads.txt

Sitemap: https://vnexpress.net/sitemap.xml
Sitemap: https://vnexpress.net/google-news-sitemap.xml
Sitemap: https://vnexpress.net/images-2026-sitemap.xml



User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: DuckAssistBot
Allow: /

User-agent: Bingbot
Allow: /
User-agent: Baiduspider
Allow: /
User-agent: Slurp
Allow: /

User-agent: AliyunSecBot
Disallow: /
User-agent: AmazonBot
Disallow: /
User-agent: archive.org_bot
Disallow: /
User-agent: AudigentAdBot
Disallow: /
User-agent: AwarioRssBot
User-agent: AwarioSmartBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: EchoboxBot
Disallow: /
User-agent: FriendlyCrawler
Disallow: /
User-agent: Jetslide
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: NewsNow
Disallow: /
User-agent: peer39_crawler
User-agent: peer39_crawler/1.0
Disallow: /
User-agent: Poseidon Research Crawler
Disallow: /
User-agent: quillbot.com
Disallow: /
User-agent: Quora-Bot
Disallow: /
User-agent: Scrapy
Disallow: /
User-agent: SeznamHomepageCrawler
Disallow: /
User-agent: TaraGroup Intelligent Bot
Disallow: /
User-agent: Timpibot
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: ViennaTinyBot
Disallow: /
User-agent: YouBot
Disallow: /

User-agent: facebookexternalhit
Allow: /

robots.txt versions

From	Through	State	Version
2026-10-27	2026-10-27	ok 200	`cb80f8bc213a`	diff with previous
2026-10-26	2026-10-26	ok 200	`87cb383e5f97`

Timeline of changes

Date	What changed
2026-10-27	robots.txt changed
2026-10-27	now allows anthropic-ai (was block)
2026-10-27	now allows Applebot-Extended (was block)
2026-10-27	now allows Bytespider (was block)
2026-10-27	now allows CCBot (was block)
2026-10-27	now allows Claude-Web (was block)
2026-10-27	now allows ClaudeBot (was block)
2026-10-27	now allows cohere-ai (was block)
2026-10-27	now allows Google-CloudVertexBot (was block)
2026-10-27	now allows Google-Extended (was block)
2026-10-27	now allows GPTBot (was block)
2026-10-27	now allows ImagesiftBot (was block)
2026-10-27	now allows MyCentralAIScraperBot (was block)
2026-10-27	now allows omgili (was block)
2026-10-27	now allows omgilibot (was block)

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Crawl	Host	State	GPTBot	OAI-SearchBot	ChatGPT-User	ClaudeBot	Claude-SearchBot	Claude-User	PerplexityBot
2026-09-21 CC-MAIN-2026-39	vnexpress.net	absent	allows	allows	unknown	allows	allows	unknown	allows
2025-09-15 CC-MAIN-2025-38	vnexpress.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2024-09-16 CC-MAIN-2024-38	vnexpress.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2023-10-02 CC-MAIN-2023-40	vnexpress.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2022-10-03 CC-MAIN-2022-40	vnexpress.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2021-09-27 CC-MAIN-2021-39	vnexpress.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2020-09-28 CC-MAIN-2020-40	vnexpress.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2019-09-23 CC-MAIN-2019-39	vnexpress.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2018-09-24 CC-MAIN-2018-39	vnexpress.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2017-09-25 CC-MAIN-2017-39	vnexpress.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-10-03 CC-MAIN-2016-40	vnexpress.net	ok view	allows	allows	unknown	allows	allows	unknown	allows
2016-09-05 CC-MAIN-2016-36	vnexpress.net	ok view	allows	allows	unknown	allows	allows	unknown	allows

Evidence

Every daily observation of vnexpress.net is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).

API: https://infrarabbit.com/crawlers/api/v1/policy/vnexpress.net