livejournal.com: what it tells AI crawlers

On 28 October 2026, livejournal.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.

Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.

Panel

top 1k (Tranco)

robots.txt

ok (HTTP 200) · https://www.livejournal.com/robots.txt
unchanged since 2026-10-26 · sha256 2c2788645a1587a0… · view as recorded

The * group

allows applies to every agent without its own group

Per AI bot, 2026-10-28

Other signals

robots.txt as recorded sha256 2c2788645a1587a0…

User-agent: Yandex
Allow: /
Disallow: /allpics.bml
Disallow: /update.bml
Disallow: /identity
Disallow: /login.bml
Disallow: /manage
Disallow: /poll
Disallow: /profile
Disallow: /schools
Disallow: /todo
Disallow: /tools
Disallow: /update.bml
Disallow: /userinfo.bml
Disallow: /users
Allow: /ratings/$
Disallow: /ratings
Disallow: /syn
Disallow: /latest
Disallow: /ljtimes
Disallow: /talkread
Disallow: /inbox
Disallow: /misc
Disallow: /legal
Disallow: /checklistposts
Disallow: /away
Disallow: /rsearch
Disallow: /gsearch
Disallow: /register.bml
Disallow: /delcomment.bml
Disallow: /talkscreen.bml
Disallow: /give_tokens.bml
Clean-param: event&add&amp&area&askname&auth&authas&auto_forwhat
Clean-param: bookmark_off&cat&cc&change&cid&comm&country&ctc&date&del_filter&dest_name&dir
Clean-param: ELEMENT_ID&errmsg&expand&f116&f16&faqid&fb&filter&for&form&format&fp&fromuser&fs&g
Clean-param: gift&hash&id&iglocation&int_like&it&item&itemid&items&journal&keywords&lang&loc_ci&loc_cn
Clean-param: loc_st&m&metal&mode&moodtheme&name&nc&nodraft&nojs&nopics&opt_pagesize&opt_sort
Clean-param: ownerid&p&page&PAGEN_1&PAGEN_2&PAGEN_4&postID&ppid&prop_taglist&q&qid&ref&replyto&repost
Clean-param: repost_type&return&returnto&s&s_loc&s2id&sc&sessid&set_filter&show&sid&sign&sim&skip&sort&spam
Clean-param: st&start_search&state&stone&style&styleid&subject&subs_id&t
Clean-param: tab&tags&talkid&theme1&theme2&theme3&theme4&thread&title&to&type
Clean-param: u&uid&url&usejournal&uselang&user&userid&usescheme&view&xtor&y
Host: https://www.livejournal.com

User-Agent: spbot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-Agent: *
Allow: /
Disallow: /allpics.bml
Disallow: /update.bml
Disallow: /identity
Disallow: /login.bml
Disallow: /manage
Disallow: /poll
Disallow: /profile
Disallow: /schools
Disallow: /todo
Disallow: /tools
Disallow: /update.bml
Disallow: /userinfo.bml
Disallow: /users
Allow: /ratings/$
Disallow: /ratings
Disallow: /syn
Disallow: /latest
Disallow: /ljtimes
Disallow: /talkread
Disallow: /inbox
Disallow: /misc
Disallow: /legal
Disallow: /checklistposts
Disallow: /away
Disallow: /rsearch
Disallow: /gsearch
Disallow: /register.bml
Disallow: /delcomment.bml
Disallow: /talkscreen.bml
Disallow: /give_tokens.bml

Sitemap: https://www.livejournal.com/sitemap.xml

#
## Blocked journals aren't listed here because robots.txt files
## can't be above 50k or so, depending on the spider.
## 
## Instead, blocked journals have HTML inserted in them which
## should prevent behaved spiders from indexing it.
##
## Note that http://username.livejournal.com journals have an
## autogenerated robots.txt, since it can be small.
##
#
#

robots.txt versions

Timeline of changes

Reconstructed history (Common Crawl)

From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.

Evidence

Every daily observation of livejournal.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973640)).

API: https://infrarabbit.com/crawlers/api/v1/policy/livejournal.com