cambridge.org: what it tells AI crawlers
On 28 October 2026, cambridge.org's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot; it also reserved TDM rights.
Last verified 2026-10-29 03:59 UTC (daily run of 2026-10-28). Proof for that day.
- Panel
- top 1k (Tranco)
- robots.txt
- ok (HTTP 200) · https://www.cambridge.org/robots.txt
unchanged since 2026-10-28 · sha256dddc775cf11f6d53…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-28
| User agent | Verdict | How |
|---|---|---|
| GPTBot | allows | through the * group |
| OAI-SearchBot | allows | through the * group |
| ChatGPT-User | blocks | named in its own group |
| ClaudeBot | allows | through the * group |
| Claude-SearchBot | allows | through the * group |
| Claude-User | allows | through the * group |
| PerplexityBot | allows | through the * group |
| Perplexity-User | allows | through the * group |
| Googlebot | allows | through the * group |
| Google-Extended | allows | through the * group |
| Applebot-Extended | allows | through the * group |
| CCBot | allows | through the * group |
| Bytespider | allows | through the * group |
| Meta-ExternalAgent | allows | through the * group |
| Amazonbot | allows | through the * group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | yes |
| Reserves TDM rights (TDMRep file, header or meta) | yes |
| Declares Content Signals (robots.txt or header) | not observed |
| Says ai-train=no in a Content Signal | not observed |
| Points to an RSL licence (robots.txt License line or Link header) | not observed |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | not observed |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | not observed |
| Sends noai / noimageai directives | not observed |
| Refused our fetcher (403 / challenge) | no |
TDMRep rules: / reservation=1 policy https://www.cambridge.org/tdm/tdmrep-policy.json
robots.txt as recorded sha256 37e3f471dd91f491…
User-agent: * Allow: /concrete/*.css Allow: /concrete/*.js Allow: /concrete/*.jpg Allow: /concrete/*.svg Allow: /concrete/*.png Allow: /concrete/*.gif Allow: /packages/*.css Allow: /packages/*.js Allow: /packages/*.jpg Allow: /packages/*.svg Allow: /packages/*.png Allow: /packages/*.gif Allow: /blocks/*.css Allow: /blocks/*.js Allow: /blocks/*.jpg Allow: /blocks/*.svg Allow: /blocks/*.png Allow: /blocks/*.gif Allow: /tools/packages/cambridge_themes/getSocialImage Allow: /tools/packages/cambridge_themes/miniCart* Disallow: /blocks Disallow: /concrete Disallow: /config Disallow: /controllers Disallow: /css Disallow: /elements Disallow: /helpers Disallow: /jobs Disallow: /js Disallow: /languages Disallow: /libraries Disallow: /mail Disallow: /models Disallow: /packages Disallow: /single_pages Disallow: /themes Disallow: /tools Disallow: /updates Disallow: /branding Disallow: /wm-ecommerce-web/academic/addtocart Disallow: /wm-ecommerce-web/english/addtocart Disallow: /wm-ecommerce-web/academic/cart Disallow: /wm-ecommerce-web/english/cart Disallow: /wm-ecommerce-web/education/cart Disallow: */c5catalogue/ Disallow: /resources Disallow: /core/dashboard Disallow: /core/my-core Disallow: /core/services/* Allow: /core/services/aop-cambridge-core/content/view/ Allow: /core/services/aop-cambridge-core/altmetric Allow: /core/services/aop-file-manager/file Allow: /aus/interchange Allow: /files/*.jpg Allow: /files/*.jpeg Allow: /files/*.png Allow: /files/*.gif Disallow: /files Disallow: /series/ Disallow: /titles/ Disallow: /uk Allow: /uk/bookshop Allow: /uk/education Allow: /uk/jobs Allow: /uk/linguistics Allow: /uk/chemistry Allow: /uk/stm Allow: /uk/resources User-agent: Twitterbot Disallow: * Disallow: /download_file/ Disallow: /manifests Disallow: /infographic Allow: /ad/education/sitemap_index.xml Allow: /ae/education/sitemap_index.xml Allow: /af/education/sitemap_index.xml Allow: /ag/education/sitemap_index.xml Allow: /ai/education/sitemap_index.xml Allow: /al/education/sitemap_index.xml Allow: /am/education/sitemap_index.xml Allow: /an/education/sitemap_index.xml Allow: /ao/education/sitemap_index.xml Allow: /aq/education/sitemap_index.xml Allow: /ar/education/sitemap_index.xml Allow: /as/education/sitemap_index.xml Allow: /at/education/sitemap_index.xml Allow: /au/education/sitemap_index.xml Allow: /aw/education/sitemap_index.xml Allow: /ax/education/sitemap_index.xml Allow: /az/education/sitemap_index.xml Allow: /ba/education/sitemap_index.xml Allow: /bb/education/sitemap_index.xml Allow: /bd/education/sitemap_index.xml Allow: /be/education/sitemap_index.xml Allow: /bf/education/sitemap_index.xml Allow: /bg/education/sitemap_index.xml Allow: /bh/education/sitemap_index.xml Allow: /bi/education/sitemap_index.xml Allow: /bj/education/sitemap_index.xml Allow: /bl/education/sitemap_index.xml Allow: /bm/education/sitemap_index.xml Allow: /bn/education/sitemap_index.xml Allow: /bo/education/sitemap_index.xml Allow: /bq/education/sitemap_index.xml Allow: /br/education/sitemap_index.xml Allow: /bs/education/sitemap_index.xml Allow: /bt/education/sitemap_index.xml Allow: /bv/education/sitemap_index.xml Allow: /bw/education/sitemap_index.xml Allow: /by/education/sitemap_index.xml Allow: /bz/education/sitemap_index.xml Allow: /ca/education/sitemap_index.xml Allow: /cc/education/sitemap_index.xml Allow: /cd/education/sitemap_index.xml Allow: /cf/education/sitemap_index.xml Allow: /cg/education/sitemap_index.xml Allow: /ch/education/sitemap_index.xml Allow: /ci/education/sitemap_index.xml Allow: /ck/education/sitemap_index.xml Allow: /cl/education/sitemap_index.xml Allow: /cm/education/sitemap_index.xml Allow: /cn/education/sitemap_index.xml Allow: /co/education/sitemap_index.xml Allow: /cr/education/sitemap_index.xml Allow: /cs/education/sitemap_index.xml Allow: /ct/education/sitemap_index.xml Allow: /cu/education/sitemap_index.xml Allow: /cv/education/sitemap_index.xml Allow: /cx/education/sitemap_index.xml Allow: /cy/education/sitemap_index.xml Allow: /cz/education/sitemap_index.xml Allow: /dd/education/sitemap_index.xml Allow: /de/education/sitemap_index.xml Allow: /dj/education/sitemap_index.xml Allow: /dk/education/sitemap_index.xml Allow: /dm/education/sitemap_index.xml Allow: /do/education/sitemap_index.xml Allow: /dz/education/sitemap_index.xml Allow: /ec/education/sitemap_index.xml Allow: /ee/education/sitemap_index.xml Allow: /eg/education/sitemap_index.xml Allow: /eh/education/sitemap_index.xml Allow: /er/education/sitemap_index.xml Allow: /es/education/sitemap_index.xml Allow: /et/education/sitemap_index.xml Allow: /fi/education/sitemap_index.xml Allow: /fj/education/sitemap_index.xml Allow: /fk/education/sitemap_index.xml Allow: /fm/education/sitemap_index.xml Allow: /fo/education/sitemap_index.xml Allow: /fq/education/sitemap_index.xml Allow: /fr/education/sitemap_index.xml Allow: /fx/education/sitemap_index.xml Allow: /ga/education/sitemap_index.xml Allow: /gb/education/sitemap_index.xml Allow: /gd/education/sitemap_index.xml Allow: /ge/education/sitemap_index.xml Allow: /gf/education/sitemap_index.xml Allow: /gg/education/sitemap_index.xml Allow: /gh/education/sitemap_index.xml Allow: /gi/education/sitemap_index.xml Allow: /gl/education/sitemap_index.xml Allow: /gm/education/sitemap_index.xml Allow: /gn/education/sitemap_index.xml Allow: /gp/education/sitemap_index.xml Allow: /gq/education/sitemap_index.xml Allow: /gr/education/sitemap_index.xml Allow: /gs/education/sitemap_index.xml Allow: /gt/education/sitemap_index.xml Allow: /gu/education/sitemap_index.xml Allow: /gw/education/sitemap_index.xml Allow: /gy/education/sitemap_index.xml Allow: /hk/education/sitemap_index.xml Allow: /hm/education/sitemap_index.xml Allow: /hn/education/sitemap_index.xml Allow: /hr/education/sitemap_index.xml Allow: /ht/education/sitemap_index.xml Allow: /hu/education/sitemap_index.xml Allow: /id/education/sitemap_index.xml Allow: /ie/education/sitemap_index.xml Allow: /il/education/sitemap_index.xml Allow: /im/education/sitemap_index.xml Allow: /in/education/sitemap_index.xml Allow: /io/education/sitemap_index.xml Allow: /iq/education/sitemap_index.xml Allow: /is/education/sitemap_index.xml Allow: /it/education/sitemap_index.xml Allow: /je/education/sitemap_index.xml Allow: /jm/education/sitemap_index.xml Allow: /jo/education/sitemap_index.xml Allow: /jp/education/sitemap_index.xml Allow: /jt/education/sitemap_index.xml Allow: /ke/education/sitemap_index.xml Allow: /kg/education/sitemap_index.xml Allow: /kh/education/sitemap_index.xml Allow: /ki/education/sitemap_index.xml Allow: /km/education/sitemap_index.xml Allow: /kn/education/sitemap_index.xml Allow: /kr/education/sitemap_index.xml Allow: /kw/education/sitemap_index.xml Allow: /ky/education/sitemap_index.xml Allow: /kz/education/sitemap_index.xml Allow: /la/education/sitemap_index.xml Allow: /lb/education/sitemap_index.xml Allow: /lc/education/sitemap_index.xml Allow: /li/education/sitemap_index.xml Allow: /lk/education/sitemap_index.xml Allow: /lr/education/sitemap_index.xml Allow: /ls/education/sitemap_index.xml Allow: /lt/education/sitemap_index.xml Allow: /lu/education/sitemap_index.xml Allow: /lv/education/sitemap_index.xml Allow: /ly/education/sitemap_index.xml Allow: /ma/education/sitemap_index.xml Allow: /mc/education/sitemap_index.xml Allow: /md/education/sitemap_index.xml Allow: /me/education/sitemap_index.xml Allow: /mf/education/sitemap_index.xml Allow: /mg/education/sitemap_index.xml Allow: /mh/education/sitemap_index.xml Allow: /mi/education/sitemap_index.xml Allow: /mk/education/sitemap_index.xml Allow: /ml/education/sitemap_index.xml Allow: /mm/education/sitemap_index.xml Allow: /mn/education/sitemap_index.xml Allow: /mo/education/sitemap_index.xml Allow: /mp/education/sitemap_index.xml Allow: /mq/education/sitemap_index.xml Allow: /mr/education/sitemap_index.xml Allow: /ms/education/sitemap_index.xml Allow: /mt/education/sitemap_index.xml Allow: /mu/education/sitemap_index.xml Allow: /mv/education/sitemap_index.xml Allow: /mw/education/sitemap_index.xml Allow: /mx/education/sitemap_index.xml Allow: /my/education/sitemap_index.xml Allow: /mz/education/sitemap_index.xml Allow: /na/education/sitemap_index.xml Allow: /nc/education/sitemap_index.xml Allow: /ne/education/sitemap_index.xml Allow: /nf/education/sitemap_index.xml Allow: /ng/education/sitemap_index.xml Allow: /ni/education/sitemap_index.xml Allow: /nl/education/sitemap_index.xml Allow: /no/education/sitemap_index.xml Allow: /np/education/sitemap_index.xml Allow: /nq/education/sitemap_index.xml Allow: /nr/education/sitemap_index.xml Allow: /nt/education/sitemap_index.xml Allow: /nu/education/sitemap_index.xml Allow: /nz/education/sitemap_index.xml Allow: /om/education/sitemap_index.xml Allow: /pa/education/sitemap_index.xml Allow: /pc/education/sitemap_index.xml Allow: /pe/education/sitemap_index.xml Allow: /pf/education/sitemap_index.xml Allow: /pg/education/sitemap_index.xml Allow: /ph/education/sitemap_index.xml Allow: /pk/education/sitemap_index.xml Allow: /pl/education/sitemap_index.xml Allow: /pm/education/sitemap_index.xml Allow: /pn/education/sitemap_index.xml Allow: /pr/education/sitemap_index.xml Allow: /ps/education/sitemap_index.xml Allow: /pt/education/sitemap_index.xml Allow: /pu/education/sitemap_index.xml Allow: /pw/education/sitemap_index.xml Allow: /py/education/sitemap_index.xml Allow: /pz/education/sitemap_index.xml Allow: /qa/education/sitemap_index.xml Allow: /re/education/sitemap_index.xml Allow: /ro/education/sitemap_index.xml Allow: /rs/education/sitemap_index.xml Allow: /ru/education/sitemap_index.xml Allow: /rw/education/sitemap_index.xml Allow: /sa/education/sitemap_index.xml Allow: /sb/education/sitemap_index.xml Allow: /sc/education/sitemap_index.xml Allow: /se/education/sitemap_index.xml Allow: /sg/education/sitemap_index.xml Allow: /sh/education/sitemap_index.xml Allow: /si/education/sitemap_index.xml Allow: /sj/education/sitemap_index.xml Allow: /sk/education/sitemap_index.xml Allow: /sl/education/sitemap_index.xml Allow: /sm/education/sitemap_index.xml Allow: /sn/education/sitemap_index.xml Allow: /so/education/sitemap_index.xml Allow: /sr/education/sitemap_index.xml Allow: /st/education/sitemap_index.xml Allow: /su/education/sitemap_index.xml Allow: /sv/education/sitemap_index.xml Allow: /sz/education/sitemap_index.xml Allow: /tc/education/sitemap_index.xml Allow: /td/education/sitemap_index.xml Allow: /tf/education/sitemap_index.xml Allow: /tg/education/sitemap_index.xml Allow: /th/education/sitemap_index.xml Allow: /tj/education/sitemap_index.xml Allow: /tk/education/sitemap_index.xml Allow: /tl/education/sitemap_index.xml Allow: /tm/education/sitemap_index.xml Allow: /tn/education/sitemap_index.xml Allow: /to/education/sitemap_index.xml Allow: /tr/education/sitemap_index.xml Allow: /tt/education/sitemap_index.xml Allow: /tv/education/sitemap_index.xml Allow: /tw/education/sitemap_index.xml Allow: /tz/education/sitemap_index.xml Allow: /ua/education/sitemap_index.xml Allow: /ug/education/sitemap_index.xml Allow: /um/education/sitemap_index.xml Allow: /us/education/sitemap_index.xml Allow: /uy/education/sitemap_index.xml Allow: /uz/education/sitemap_index.xml Allow: /va/education/sitemap_index.xml Allow: /vc/education/sitemap_index.xml Allow: /vd/education/sitemap_index.xml Allow: /ve/education/sitemap_index.xml Allow: /vg/education/sitemap_index.xml Allow: /vi/education/sitemap_index.xml Allow: /vn/education/sitemap_index.xml Allow: /vu/education/sitemap_index.xml Allow: /wf/education/sitemap_index.xml Allow: /wk/education/sitemap_index.xml Allow: /ws/education/sitemap_index.xml Allow: /yd/education/sitemap_index.xml Allow: /ye/education/sitemap_index.xml Allow: /yt/education/sitemap_index.xml Allow: /za/education/sitemap_index.xml Allow: /zm/education/sitemap_index.xml Allow: /zw/education/sitemap_index.xml Allow: /zz/education/sitemap_index.xml
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-28 | 2026-10-28 | ok 200 | dddc775cf11f | diff with previous |
| 2026-10-26 | 2026-10-27 | ok 200 | dddc775cf11f |
Timeline of changes
| Date | What changed |
|---|---|
| No changes since the first observation (2026-10-26). | |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | cambridge.org | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2025-09-15 CC-MAIN-2025-38 | cambridge.org | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2024-09-16 CC-MAIN-2024-38 | cambridge.org | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2023-10-02 CC-MAIN-2023-40 | cambridge.org | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2022-10-03 CC-MAIN-2022-40 | cambridge.org | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | www.cambridge.org | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | www.cambridge.org | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2019-09-23 CC-MAIN-2019-39 | www.cambridge.org | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | www.cambridge.org | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-01-29 CC-MAIN-2018-05 | www.cambridge.org | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-10-23 CC-MAIN-2017-43 | www.cambridge.org | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | www.cambridge.org | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-07-24 CC-MAIN-2017-30 | www2.cambridge.org | ok view | blocks | blocks | unknown | blocks | blocks | unknown | blocks |
| 2017-04-24 CC-MAIN-2017-17 | www2.cambridge.org | ok view | blocks | blocks | unknown | blocks | blocks | unknown | blocks |
| 2017-01-23 CC-MAIN-2017-04 | www2.cambridge.org | ok view | blocks | blocks | unknown | blocks | blocks | unknown | blocks |
| 2016-10-03 CC-MAIN-2016-40 | www2.cambridge.org | ok view | blocks | blocks | unknown | blocks | blocks | unknown | blocks |
| 2016-09-05 CC-MAIN-2016-36 | www2.cambridge.org | ok view | blocks | blocks | unknown | blocks | blocks | unknown | blocks |
Evidence
Every daily observation of cambridge.org is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-28: audit path (JSON), day root and OpenTimestamps proof (pending (4 calendars)).
API: https://infrarabbit.com/crawlers/api/v1/policy/cambridge.org