proofpoint.com: what it tells AI crawlers
On 27 October 2026, proofpoint.com's robots.txt allowed all of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot and CCBot.
Last verified 2026-10-28 03:59 UTC (daily run of 2026-10-27). Proof for that day.
- Panel
- top 10k (Tranco)
- robots.txt
- ok (HTTP 200) · https://www.proofpoint.com/robots.txt
unchanged since 2026-10-27 · sha25697f7e6cfe0381a57…· view as recorded - The * group
- allows applies to every agent without its own group
Per AI bot, 2026-10-27
| User agent | Verdict | How |
|---|---|---|
| GPTBot | allows | through the * group |
| OAI-SearchBot | allows | through the * group |
| ChatGPT-User | allows | through the * group |
| ClaudeBot | allows | through the * group |
| Claude-SearchBot | allows | through the * group |
| Claude-User | allows | through the * group |
| PerplexityBot | allows | through the * group |
| Perplexity-User | allows | through the * group |
| Googlebot | allows | through the * group |
| Google-Extended | allows | through the * group |
| Applebot-Extended | allows | through the * group |
| CCBot | allows | through the * group |
| Bytespider | allows | through the * group |
| Meta-ExternalAgent | blocks | named in its own group |
| Amazonbot | allows | through the * group |
| Diffbot | blocks | named in its own group |
| omgili | blocks | named in its own group |
| Webzio-Extended | blocks | named in its own group |
Other signals
| Publishes /llms.txt | no |
| Publishes /ai.txt | no |
| Publishes /.well-known/tdmrep.json | no |
| Reserves TDM rights (TDMRep file, header or meta) | no |
| Declares Content Signals (robots.txt or header) | no |
| Says ai-train=no in a Content Signal | no |
| Points to an RSL licence (robots.txt License line or Link header) | no |
| Answers with an x402 payment offer (lower bound: only the homepage is probed) | no |
| Answers HTTP 402 Payment Required (lower bound: only the homepage is probed) | no |
| Sends noai / noimageai directives | no |
| Refused our fetcher (403 / challenge) | no |
robots.txt as recorded sha256 8fe8aa22a1d5116e…
# # robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources. # # This file will be ignored unless it is at the root of your host: # Used: http://example.com/robots.txt # Ignored: http://example.com/site/robots.txt # # For more information about the robots.txt standard, see: # http://www.robotstxt.org/robotstxt.html User-agent: * Crawl-delay: 10 # CSS, JS, Images Allow: /core/*.css$ Allow: /core/*.css? Allow: /core/*.js$ Allow: /core/*.js? Allow: /core/*.gif Allow: /core/*.jpg Allow: /core/*.jpeg Allow: /core/*.png Allow: /core/*.svg Allow: /profiles/*.css$ Allow: /profiles/*.css? Allow: /profiles/*.js$ Allow: /profiles/*.js? Allow: /profiles/*.gif Allow: /profiles/*.jpg Allow: /profiles/*.jpeg Allow: /profiles/*.png Allow: /profiles/*.svg Allow: /misc/*.css$ Allow: /misc/*.css? Allow: /misc/*.js$ Allow: /misc/*.js? Allow: /misc/*.gif Allow: /misc/*.jpg Allow: /misc/*.jpeg Allow: /misc/*.png Allow: /modules/*.css$ Allow: /modules/*.css? Allow: /modules/*.js$ Allow: /modules/*.js? Allow: /modules/*.gif Allow: /modules/*.jpg Allow: /modules/*.jpeg Allow: /modules/*.png Allow: /profiles/*.css$ Allow: /profiles/*.css? Allow: /profiles/*.js$ Allow: /profiles/*.js? Allow: /profiles/*.gif Allow: /profiles/*.jpg Allow: /profiles/*.jpeg Allow: /profiles/*.png Allow: /themes/*.css$ Allow: /themes/*.css? Allow: /themes/*.js$ Allow: /themes/*.js? Allow: /themes/*.gif Allow: /themes/*.jpg Allow: /themes/*.jpeg Allow: /themes/*.png # Directories Disallow: /core/ Disallow: /profiles/ # Files Disallow: /README.txt Disallow: /web.config Disallow: /CHANGELOG.txt Disallow: /cron.php Disallow: /INSTALL.mysql.txt Disallow: /INSTALL.pgsql.txt Disallow: /INSTALL.sqlite.txt Disallow: /install.php Disallow: /INSTALL.txt Disallow: /LICENSE.txt Disallow: /MAINTAINERS.txt Disallow: /update.php Disallow: /UPGRADE.txt Disallow: /xmlrpc.php Disallow: /sites/default/files/*gtd-*.pdf Disallow: /sites/default/files/*lgy-*.pdf Disallow: /thank-you-* Disallow: /sites/default/files/pfpt-us-tr-state-of-the-phish-2019.pdf Disallow: /sites/default/files/mail-routing-agent-datasheet-v2.pdf # Paths (clean URLs) Disallow: /admin/ Disallow: /comment/reply/ Disallow: /filter/tips Disallow: /node/add/ Disallow: /search/ Disallow: /user/register/ Disallow: /user/password/ Disallow: /user/login/ Disallow: /user/logout/ Disallow: /admin/ Disallow: /comment/reply/ Disallow: /node/add/ Disallow: /search/ Disallow: /user/register* Disallow: /user/password* Disallow: /user/login* Disallow: /user/logout* Disallow: /events/* Disallow: /upcoming-events/* Disallow: /lp/* Disallow: /conquer Disallow: /merck Disallow: /johnson-controls Disallow: /baxter Disallow: /sutter-health Disallow: /chevron Disallow: /meeting-request # Paths (no clean URLs) Disallow: /index.php/admin/ Disallow: /index.php/comment/reply/ Disallow: /index.php/filter/tips Disallow: /index.php/node/add/ Disallow: /index.php/search/ Disallow: /index.php/user/password/ Disallow: /index.php/user/register/ Disallow: /index.php/user/login/ Disallow: /index.php/user/logout/ Disallow: /?q=admin/ Disallow: /?q=comment/reply/ Disallow: /?q=filter/tips/ Disallow: /?q=node/add/ Disallow: /?q=search/ Disallow: /?q=user/password/ Disallow: /?q=user/register/ Disallow: /?q=user/login/ Disallow: /?q=user/logout/ Disallow: /?q=events/* Disallow: /?q=upcoming-events/* Disallow: /sites/default/files/e-books/pfpt-jp-eb-getting-started-with-DMARC-ebook.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-setting-up-itm.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-itm-overview.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-role-of-itm.pdf Disallow: /sites/default/files/solution-briefs/pfpt-uk-sb-casb-adaptive-access-controls.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-itm-cybersecurity.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-itm-overview.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-setting-up-itm.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-itm-overview.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-role-of-itm.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-demystifying-bec-eac.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-deploying-people-centric-security-3.pdf Disallow: /sites/default/files/e-books/pfpt-fr-dr-demystifying-bec-eac.pdf Disallow: /sites/default/files/e-books/pfpt-fr-da-deploying-people-centric-security-3.pdf Disallow: /sites/default/files/solution-briefs/pfpt-de-sb-CASB-adaptive-access-controls.pdf Disallow: /sites/default/files/e-books/pfpt-de-da-deploying-people-centric-security-3.pdf Disallow: /sites/default/files/solution-briefs/RemoteWorkAlliance_Webinar.pdf Disallow: /sites/default/files/white-papers/pfpt-us-eb-the-hidden-cost-of-mso365-security.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-dr-whack-a-mail.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-no-compromises.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-putting-people-at-center-of-your-email-security-strategy.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-calculated-risks.pdf Disallow: /sites/default/files/solution-briefs/pfpt-ja-sb-CASB-adaptive-access-controls.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-demystifying-bec-eac.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-no-compromises.pdf Disallow: /sites/default/files/e-books/pfpt-jp-da-deploying-people-centric-security-3.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-whack-a-mail.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-unmasking-bec-and-eac.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-role-of-itm.pdf Disallow: /sites/default/files/e-books/pfpt-es-eb-calculated-risks.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-top-10-reasons-customers-choose-pfpt-to-secure-ms365.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-de-eb-top-10-reasons-customers-choose-pfpt-to-secure-ms365.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-de-eb-putting-people-at-center-of-your-email-security-strategy.pdf Disallow: /sites/default/files/solution-briefs/pfpt-jp-eb-unmasking-bec-and-eac.pdf Disallow: /sites/default/files/e-books/pfpt-fr-eb-calculated-risks.pdf Disallow: /sites/default/files/e-books/pfpt-es-eb-top-10-reasons-customers-choose-pfpt-to-secure-ms365.pdf Disallow: /sites/default/files/e-books/pfpt-fr-eb-top-10-reasons-customers-choose-pfpt-to-secure-ms365.pdf Disallow: /sites/default/files/e-books/pfpt-it-eb-top-10-reasons-customers-choose-pfpt-to-secure-ms365.pdf Disallow: /sites/default/files/misc/pfpt-us-eb-the-evolution-of-the-casb.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-the-evolution-of-the-casb.pdf Disallow: /sites/default/files/infographics/Pfpt-Phishing%20Infographic%20Poster-17x24-jp.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-calculated-risks.pdf Disallow: /sites/default/files/e-books/pfpt-es-da-deploying-people-centric-security-3.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-de-eb-calculated-risks.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-it-eb-calculated-risks.pdf Disallow: /sites/default/files/white-papers/gtyd-pfpt-fr-wp-driving-real-behavior-change.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-es-eb-demystifying-bec-eac.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-getting-started-with-DMARC-ebook_2020.pdf Disallow: /sites/default/files/solution-briefs/gtd-pfpt-es-sb-CASB-adaptive-access-controls.pdf Disallow: /sites/default/files/securityawarenesskit/Cybersecurity-Awareness-Kit-BEC.zip Disallow: /sites/default/files/e-books/pfpt-us-eb-rethinking-insider-risk.pdf Disallow: /sites/default/files/data-sheets/pfpt-jp-eval_2020-evaluation-jp.pdf Disallow: /sites/default/files/e-books/pfpt-us-tr-healthcare-report.pdf Disallow: /sites/default/files/white-papers/pfpt-de-eb-the-hidden-cost-of-mso365-security.pdf Disallow: /sites/default/files/white-papers/pfpt-uk-eb-the-hidden-cost-of-mso365-security.pdf Disallow: /sites/default/files/white-papers/pfpt-au-eb-the-hidden-cost-of-mso365-security.pdf Disallow: /sites/default/files/white-papers/pfpt-fr-eb-the-hidden-cost-of-mso365-security.pdf Disallow: /sites/default/files/white-papers/pfpt-es-eb-the-hidden-cost-of-mso365-security.pdf Disallow: /sites/default/files/white-papers/pfpt-it-eb-the-hidden-cost-of-mso365-security.pdf Disallow: /sites/default/files/white-papers/pfpt-jp-eb-the-hidden-cost-of-mso365-security.pdf Disallow: /sites/default/files/misc/gtd-pfpt-jp-a4-eb-top-10-reasons-customers-choose-pfpt-to-secure-ms365.pdf Disallow: /sites/default/files/analyst-reports/pfpt-us-ar-esg-economic-validation-sep-2020.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-it-eb-stopping-bec-and-eac.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-jp-eb-stopping-bec-and-eac.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-fr-eb-stopping-bec-and-eac.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-es-eb-stopping-bec-and-eac.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-it-eb-putting-people-at-center-of-your-email-security-strategy.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-jp-eb-putting-people-at-center-of-your-email-security-strategy.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-fr-eb-putting-people-at-center-of-your-email-security-strategy.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-es-eb-putting-people-at-center-of-your-email-security-strategy.pdf Disallow: /sites/default/files/analyst-reports/pfpt-es-ar-esg-economic-validation-sep-2020.pdf Disallow: /sites/default/files/analyst-reports/pfpt-fr-ar-esg-economic-validation-sep-2020.pdf Disallow: /sites/default/files/analyst-reports/pfpt-it-ar-esg-economic-validation-sep-2020.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-fr-eb-role-of-itm.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-fr-eb-itm-overview.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-de-eb-itm-cybersecurity.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-fr-eb-itm-cybersecurity.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-de-eb-setting-up-itm.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-fr-eb-setting-up-itm.pdf Disallow: /sites/default/files/analyst-reports/pfpt-uk-ar-esg-economic-validation-sep-2020.pdf Disallow: /sites/default/files/solution-briefs/pfpt-jp-a4-sb-solutions-bundles_201111a.pdf Disallow: /sites/default/files/analyst-reports/pfpt-jp-ar-esg-economic-validation-sep-2020.pdf Disallow: /sites/default/files/white-papers/pfpt-us-wp-cyber-risk-alliance-research-report.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-uk-eb-top-10-reasons-customers-choose-pfpt-to-secure-ms365.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-au-eb-top-10-reasons-customers-choose-pfpt-to-secure-ms365.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-it-eb-role-of-itm.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-it-eb-itm-cybersecurity.pdf Disallow: /sites/default/files/analyst-reports/pfpt-de-ar-esg-economic-validation-sep-2020.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-it-eb-setting-up-itm.pdf Disallow: /sites/default/files/white-papers/pfpt-us-wp-radicati-secure-email-gateway-market-quadrant-2020.pdf Disallow: /sites/default/files/white-papers/gtd-pfpt-it-wp-driving-real-behavior-change.pdf Disallow: /sites/default/files/white-papers/gtd-pfpt-es-wp-driving-real-behavior-change.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-itm-overview-v7.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-itm-overview-v7.pdf Disallow: /sites/default/files/e-books/pfpt-au-eb-itm-overview-v7.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-fr-eb-itm-overview-v5.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-it-eb-itm-overview.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-itm-overview-v7.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-rethinking-insider-risk.pdf Disallow: /sites/default/files/e-books/pfpt-au-eb-rethinking-insider-risk.pdf Disallow: /sites/default/files/solution-briefs/gtd-pfpt-fr-sb-CASB-adaptive-access-controls.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-it-eb-demystifying-bec-eac.pdf Disallow: /sites/default/files/misc/gtd-pfpt-uk-eb-role-of-itm.pdf.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-2021-ransomware-survival-guide.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-2021-ransomware-survival-guide.pdf Disallow: /sites/default/files/e-books/pfpt-a4-da-ransomware-survival-guide-executive-summary.pdf Disallow: /sites/default/files/e-books/pfpt-es-eb-rethinking-insider-risk.pdf Disallow: /sites/default/files/e-books/PROO3754_fr_RethinkingIT_GIG_eBook.pdf Disallow: /sites/default/files/e-books/pfpt-fr-eb-rethinking-insider-risk.pdf Disallow: /sites/default/files/e-books/PROO3773_FinServ_eBook.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-FinServ_eBook.pdf Disallow: /sites/default/files/threat-reports/pfpt-us-tr-healthcare.pdf Disallow: /sites/default/files/e-books/pfpt-it-eb-rethinking-insider-risk.pdf Disallow: /sites/default/files/threat-reports/pfpt-es-tr-healthcare-report.pdf Disallow: /sites/default/files/threat-reports/pfpt-uk-tr-healthcare-report.pdf Disallow: /sites/default/files/threat-reports/pfpt-fr-tr-healthcare-report.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-driving-real-behavior-change.pdf Disallow: /sites/default/files/white-papers/pfpt-us-wp-driving-real-behavior-change.pdf Disallow: /sites/default/files/misc/pfpt-2020-type2-soc2-report-clickwrap.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-rethinking-insider-risk.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-rethinking-insider-risk.pdf Disallow: /sites/default/files/threat-reports/pfpt-jp-tr-healthcare-report.pdf Disallow: /sites/default/files/threat-reports/pfpt-it-tr-healthcare-report.pdf Disallow: /sites/default/files/2020-05/pfpt-us-wp-how-proofpoint-helps-organizations-meet-NIST-cybersecurity-guidelines.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-biggest-and-boldest-insider-threats-2019-2020.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-compliance-in-the-age-of-digital-collaboration.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-compliance-in-the-age-of-digital-collaborations.pdf Disallow: /sites/default/files/2020-06/pfpt-us-eb-the-definitive-email-security-guide.pdf Disallow: /sites/default/files/ciso_ebook_sponsored_by_proofpoint.pdf Disallow: /sites/default/files/threat-reports/pfpt-us-tr-state-of-the-phish-2021.pdf Disallow: /sites/default/files/threat-reports/pfpt-de-tr-healthcare-report.pdf Disallow: /sites/default/files/threat-reports/pfpt-us-tr-financial-services-and-insurance.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-managing-insider-threats-in-the-technology-sector.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-managing-insider-threats-in-the-technology-sector-.pdf Disallow: /sites/default/files/threat-reports/gtd-pfpt-uk-a4-r-state-of-the-phish-2021.pdf Disallow: /sites/default/files/misc/gtd-pfpt-de-a4-r-state-of-the-phish-2021.pdf Disallow: /sites/default/files/misc/gtd-pfpt-fr-a4-r-state-of-the-phish-2021.pdf Disallow: /sites/default/files/misc/Proofpoint-jp-a4-eb-2021-ransomware-survival-guide.pdf Disallow: /sites/default/files/white-papers/pfpt-us-wp-securing-digital-healthcare.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-ransomware-survival-guide-executive-summary.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-complete-guide-to-migrating-your-enterprise-archiving-system.pdf Disallow: /sites/default/files/e-books/0503-001-01-01-pfpt-A4-e-book-youve-got-BEC-JP-30APR2020.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-es-eb-role-of-itm.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-es-eb-itm-cybersecurity.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-es-eb-setting-up-itm.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-es-eb-itm-overview.pdf Disallow: /sites/default/files/white-papers/gtd-pfpt-es-a4-r-state-of-the-phish-2021.pdf.pdf Disallow: /sites/default/files/white-papers/gtd-pfpt-it-a4-r-state-of-the-phish-2021.pdf.pdf Disallow: /sites/default/files/threat-reports/gtd-pfpt-es-a4-r-state-of-the-phish-2021.pdf Disallow: /sites/default/files/threat-reports/gtd-pfpt-it-a4-r-state-of-the-phish-2021.pdf Disallow: /sites/default/files/threat-reports/220252.37-pfpt-jp-a4-r-state-of-the-phish-2021.pdf Disallow: /sites/default/files/white-papers/pfpt-us-wp-business-impact-brief.pdf Disallow: /sites/default/files/white-papers/gtd-pfpt-us-wp-people-centric-cybersecurity-federal-perspective.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-complete-guide-to-migrating-your-enterprise-archiving-system.pdf Disallow: /sites/default/files/e-books/pfpt-au-eb-complete-guide-to-migrating-your-enterprise-archiving-system.pdf Disallow: /sites/default/files/solution-briefs/pfpt-jp-wp-a4-people-centric-approach-insider-threat-management-3.pdf Disallow: /sites/default/files/analyst-reports/pfpt-us-ar-cost-of-cloud-compromise-and-shadow-IT.pdf Disallow: /sites/default/files/e-books/pfpt-uk-a4-eb-10-reasons-orgs-choose-pfpt-to-secure-email-and-cloud.pdf Disallow: /sites/default/files/misc/pfpt-es-a4-eb-10-reasons-orgs-choose-pfpt-to-secure-email-and-cloud.pdf Disallow: /sites/default/files/customer-stories/pfpt-jp-cs-macdonald_20210426a.pdf Disallow: /sites/default/files/customer-stories/pfpt-jp-cs-mcdonald_20210426a.pdf Disallow: /sites/default/files/misc/Password%20Day%20Awareness%20Kit.zip Disallow: /sites/default/files/e-books/pfpt-uk-eb-managing-insider-threats-in-the-technology-sector.pdf Disallow: /sites/default/files/e-books/pfpt-au-eb-managing-insider-threats-in-the-technology-sector.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-compliance-in-the-age-of-digital-collaboration.pdf Disallow: /sites/default/files/e-books/pfpt-au-eb-compliance-in-the-age-of-digital-collaboration.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-reimagining-email-security.pdf Disallow: /sites/default/files/white-papers/pfpt-us-wp-beyond-the-cloud.pdf Disallow: /sites/default/files/white-papers/pfpt-us-wp-voice-of-the-CISO-report.pdf Disallow: /sites/default/files/e-books/pfpt-it-eb-reimagining-email-security.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-reimagining-email-security.pdf Disallow: /sites/default/files/e-books/pfpt-au-eb-reimagining-email-security.pdf Disallow: /sites/default/files/e-books/pfpt-fr-eb-reimagining-email-security.pdf Disallow: /sites/default/files/e-books/pfpt-es-eb-reimagining-email-security.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-reimagining-email-security.pdf Disallow: /sites/default/files/e-books/gtd-pfpt-us-wp-getting-started-with-casb-v33.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-reimagining-email-security-v3.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-frost-sullivan-guarding-the-virtual-doorway.pdf Disallow: /sites/default/files/awareness-materials/Email%20Fraud%20and%20Security%20Awareness%20Kit%202021.zip Disallow: /sites/default/files/threat-reports/pfpt-us-tr-human-factor-report.pdf Disallow: /sites/default/files/pfpt-uk-tr-human-factor-report.pdf Disallow: /sites/default/files/pfpt-au-tr-human-factor-report.pdf Disallow: /sites/default/files/pfpt-de-tr-human-factor-report.pdf Disallow: /sites/default/files/pfpt-es-tr-human-factor-report.pdf Disallow: /sites/default/files/pfpt-fr-tr-human-factor-report.pdf Disallow: /sites/default/files/pfpt-it-tr-human-factor-report_0.pdf Disallow: /sites/default/files/pfpt-it-tr-human-factor-report_1.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-unmasking-bec-and-eac.pdf Disallow: /sites/default/files/white-papers/pfpt-jp-wp-voice-of-the-CISO-report.pdf Disallow: /sites/default/files/white-papers/pfpt-us-wp-beyond-vpn-securing-your-remote-workforce.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-rethinking-dlp.pdf Disallow: /sites/default/files/white-papers/pfpt-jp-wp-voice-of-the-CISO-report_v2.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-the-anatomy-of-an-insider-threat-investigation.pdf Disallow: /sites/default/files/e-books/pfpt-fr-eb-complete-guide-to-migrating-your-enterprise-archiving-system.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-complete-guide-to-migrating-your-enterprise-archiving-system.pdf Disallow: /sites/default/files/customer-stories/pfpt-jp-cs-segasammy_20210622.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-the-bec-handbook.pdf Disallow: /sites/default/files/analyst-reports/pfpt-us-ar-frost-radar-global-email-security-market-report.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-rethinking-dlp.pdf Disallow: /sites/default/files/e-books/pfpt-au-eb-rethinking-dlp.pdf Disallow: /sites/default/files/e-books/pfpt-it-eb-rethinking-dlp.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-rethinking-dlp.pdf Disallow: /sites/default/files/e-books/pfpt-es-eb-rethinking-dlp.pdf Disallow: /sites/default/files/e-books/pfpt-fr-eb-rethinking-dlp.pdf Disallow: /sites/default/files/threat-reports/pfpt-us-tr-the-cost-of-insider-threats-ponemon-report.pdf Disallow: /sites/default/files/analyst-reports/pfpt-uk-ar-frost-radar-global-email-security-market-report.pdf Disallow: /sites/default/files/analyst-reports/pfpt-au-ar-frost-radar-global-email-security-market-report.pdf Disallow: /sites/default/files/analyst-reports/pfpt-uk-ar-cost-of-cloud-compromise-and-shadow-IT.pdf Disallow: /sites/default/files/analyst-reports/pfpt-au-ar-cost-of-cloud-compromise-and-shadow-IT.pdf Disallow: /sites/default/files/event-materials/210713_CYBERGYMxProofpoint.zip Disallow: /sites/default/files/white-papers/pfpt-uk-wp-beyond-the-cloud.pdf Disallow: /sites/default/files/white-papers/pfpt-au-wp-beyond-the-cloud.pdf Disallow: /sites/default/files/event-materials/210713Webinar_Crowdstrike-Netskope-Okta-Proofpoint.zip Disallow: /sites/default/files/threat-reports/pfpt-it-tr-human-factor-report.pdf Disallow: /sites/default/files/analyst-reports/pfpt-us-ar-ponemon-2021-cost-of-phishing-study.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-rethinking-dlp.pdf Disallow: /sites/default/files/white-papers/pfpt-uk-wp-securing-digital-healthcare.pdf Disallow: /sites/default/files/white-papers/pfpt-au-wp-securing-digital-healthcare.pdf Disallow: /sites/default/files/white-papers/pfpt-es-wp-securing-digital-healthcare.pdf Disallow: /sites/default/files/white-papers/pfpt-de-wp-securing-digital-healthcare.pdf Disallow: /sites/default/files/white-papers/pfpt-it-wp-securing-digital-healthcare.pdf Disallow: /sites/default/files/white-papers/pfpt-fr-wp-securing-digital-healthcare.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-three-information-protection-use-cases-in-the-cloud-era.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-ige-modern-blueprint-to-information-protection.pdf Disallow: /sites/default/files/analyst-reports/gtd-pfpt-us-ar-frost-radar-security-awareness-training-market-report-2021.pdf.pdf Disallow: /sites/default/files/analyst-reports/gtd-pfpt-us-ar-frost-radar-security-awareness-training-market-report-2021.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-the-bec-handbook.pdf Disallow: /sites/default/files/e-books/pfpt-au-eb-the-bec-handbook.pdf Disallow: /sites/default/files/e-books/pfpt-fr-eb-the-bec-handbook.pdf Disallow: /sites/default/files/e-books/pfpt-it-eb-the-bec-handbook.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-the-bec-handbook.pdf Disallow: /sites/default/files/e-books/pfpt-es-eb-the-bec-handbook.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-data-breach-is-coming-from-inside-the-house.pdf Disallow: /sites/default/files/e-books/pfpt-uk-eb-the-anatomy-of-an-insider-threat-investigation.pdf Disallow: /sites/default/files/e-books/pfpt-au-eb-the-anatomy-of-an-insider-threat-investigation.pdf Disallow: /sites/default/files/e-books/pfpt-fr-eb-the-anatomy-of-an-insider-threat-investigation.pdf Disallow: /sites/default/files/e-books/pfpt-es-eb-the-anatomy-of-an-insider-threat-investigation.pdf Disallow: /sites/default/files/e-books/pfpt-it-eb-the-anatomy-of-an-insider-threat-investigation.pdf Disallow: /sites/default/files/e-books/pfpt-de-eb-the-anatomy-of-an-insider-threat-investigation.pdf Disallow: /sites/default/files/white-papers/pfpt-de-wp-voice-of-the-CISO-report.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-the-bec-handbook.pdf Disallow: /sites/default/files/threat-reports/pfpt-japanese-tr-human-factor-report.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-biggest-and-boldest-insider-threats-2020-2021.pdf Disallow: /sites/default/files/white-papers/pfpt-fr-wp-voice-of-the-CISO-report.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-esg-ediscovery-market-trends-and-challenges.pdf Disallow: /sites/default/files/e-books/pfpt-jp-eb-the-anatomy-of-an-insider-threat-investigation.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-measuring-security-awareness-impact.pdf Disallow: /sites/default/files/white-papers/pfpt-us-wp-building-a-business-case-for-itm.pdf Disallow: /sites/default/files/e-books/pfpt-us-eb-2021-user-risk-report.pdf Sitemap: https://www.proofpoint.com/sitemap.xml
robots.txt versions
| From | Through | State | Version | |
|---|---|---|---|---|
| 2026-10-27 | 2026-10-27 | ok 200 | 97f7e6cfe038 | diff with previous |
| 2026-10-26 | 2026-10-26 | ok 200 | 5ffcda252eb0 |
Timeline of changes
| Date | What changed |
|---|---|
| 2026-10-27 | robots.txt changed |
Reconstructed history (Common Crawl)
From Common Crawl's robots.txt captures: what CCBot was served, one crawl per quarter. A separate series from the daily observations above; a 403 here means Common Crawl was refused, not this ledger.
| Crawl | Host | State | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | PerplexityBot |
|---|---|---|---|---|---|---|---|---|---|
| 2026-09-21 CC-MAIN-2026-39 | www.proofpoint.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2025-09-15 CC-MAIN-2025-38 | www.proofpoint.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2024-09-16 CC-MAIN-2024-38 | www.proofpoint.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2023-10-02 CC-MAIN-2023-40 | www.proofpoint.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2022-10-03 CC-MAIN-2022-40 | www.proofpoint.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2021-09-27 CC-MAIN-2021-39 | www.proofpoint.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2020-09-28 CC-MAIN-2020-40 | proofpoint.com | redirect-only | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| 2019-09-23 CC-MAIN-2019-39 | www.proofpoint.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2018-09-24 CC-MAIN-2018-39 | www.proofpoint.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-09-25 CC-MAIN-2017-39 | www.proofpoint.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2017-01-23 CC-MAIN-2017-04 | proofpoint.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-10-03 CC-MAIN-2016-40 | www.proofpoint.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
| 2016-09-05 CC-MAIN-2016-36 | www.proofpoint.com | ok view | allows | allows | unknown | allows | allows | unknown | allows |
Evidence
Every daily observation of proofpoint.com is a leaf in that day's Merkle tree. Proof of the robots.txt line for 2026-10-27: audit path (JSON), day root and OpenTimestamps proof (complete (bitcoin block 973493)).
API: https://infrarabbit.com/crawlers/api/v1/policy/proofpoint.com