If your website uses an SSL certificate that was issued on March 15 or 16, 2023 by one of the following certificate authorities:
- Amazon Trust Services (operated by DigiCert)
- NAVER Cloud
- Viking Cloud
then your site might not load in Chrome, Safari, and Edge, and you should contact your certificate authority to get a new, working certificate ASAP. You can use our Certificate Transparency Policy Analyzer to see for sure if your site is affected. This blog post will explain why over 150,000 certificates issued last week are being rejected by browsers, and how SSLMate analyzed the incident.
Certificate Transparency: A Brief Overview
Certificate Transparency (CT) is a system for publishing SSL certificates in public logs. Certificate Transparency monitors like Cert Spotter download the logs to help you track certificate expiration and detect unauthorized certificates.
At a high level, Certificate Transparency works as follows:
- Before issuing a certificate, the certificate authority (CA) creates a "precertificate" containing the details of the certificate it intends to issue.
- The CA submits the precertificate to multiple Certificate Transparency logs.
- Each log returns a receipt, called a Signed Certificate Timestamp (SCT), which confirms submission of the precertificate.
- The CA embeds the SCTs in the certificate which it gives to the site operator.
- When a browser loads a website, it makes sure the website's certificate
has SCTs from a sufficient number of recognized logs. If it doesn't,
the browser throws up an error page (
ERR_CERTIFICATE_TRANSPARENCY_REQUIREDin Chrome) and refuses to load the website.
Note that only the precertificate needs to be submitted to logs; the certificate itself may or may not end up in logs. Some CAs submit them automatically, and crawlers such as the Googlebot submit certificates that they observe when crawling the Internet.
What Went Wrong
The incident began last Wednesday, March 15 at 15:13 UTC, when Sectigo, a CT log operator, began a migration of their "Sabre" log to newer software. There was about an hour and a half of planned downtime. At 16:30 UTC the log came back online. The migration was mostly successful, except for one rather significant problem: the new log software had been configured with the private key of a test log called "Dodo". The problem wasn't noticed and corrected until the next day, March 16, at 12:28 UTC. During this 20 hour window, all of the artifacts produced by Sabre, including the SCTs returned to CAs, were signed with Dodo's private key.
In theory, this should not have been a problem - CAs could have used Sabre's public key to validate the SCT signatures, rejected the ones with invalid signatures, and gotten SCTs from other logs.
Unfortunately, history shows that certificate authorities are prone to sloppiness and shortcut-taking. There was no doubt that at least some CAs would blindly take SCTs and embed them in certificates without validating the signature. Browsers, of course, do validate SCT signatures, and reject SCTs signed by Dodo, since it is a test log.
The only question was how many CAs were affected. To answer this question, SSLMate examined the approximately 164,000 precertificates which were logged to Sabre during the incident. For every precertificate, we looked for its corresponding certificate (in any log, not just Sabre), and evaluated the certificate's embedded SCTs for compliance with browser Certificate Transparency policies. Among the 35 distinct CAs logging to Sabre during the incident, we found:
- 12 CAs which had at least one non-compliant certificate, indicating that the CA does not validate SCT signatures: Amazon Trust Services (operated by DigiCert), Buypass, CFCA, certSIGN, Certum, DigiCert, Globaltrust, Microsoft, NAVER Cloud, Netlock, SwissSign, Viking Cloud
- 6 CAs with only complaint certificates, indicating that they correctly rejected SCTs from Dodo: Certainly, Chunghwa Telecom, Cybertrust Japan, IdenTrust, Let's Encrypt, Sectigo
- 17 CAs for which we could not find any certificates, making it indeterminate whether or not they validate SCT signatures: Atos Trustcenter, Certigna, D-Trust, Deutsche Telekom Security, Firmaprofesional, Google Trust Services, Government of Spain - ACCV, Government of Spain - FNMT, HARICA, Izenpe, Microsec, OISTE, QuoVadis, SSL.com, Shanghai Electronic Certification Authority, Taiwan-CA, Telia
Below are the total number of precertificates that were logged to Sabre during the incident by CAs which do not validate SCT signatures. Presumably all of the corresponding certificates contain Dodo SCTs that will be rejected by browsers:
|Amazon Trust Services (operated by DigiCert)||42,363|
Cloudflare for the Win
As a content distribution network, Cloudflare automatically obtains certificates for its customers' websites. Despite getting some of its certificates from DigiCert, Cloudflare-hosted websites were not affected by the incident. That's because Cloudflare evaluates every certificate for Certificate Transparency policy compliance, and if a certificate lacks a sufficient number of SCTs, Cloudflare submits the certificate to additional logs and serves the resulting SCTs via the TLS handshake.
We found several examples of Cloudflare-hosted websites that were serving a certificate with insufficient SCTs yet still worked fine thanks to the additional SCTs in the TLS handshake. Here's an example of the SCTs served by one such website:
|Embedded||Google Argon 2024||Authentic||Usable||Usable||2023-03-16T00:32:03.504Z|
|Embedded||Google Xenon 2024||Authentic||Usable||Usable||2023-03-16T00:32:03.59Z|
|TLS||Let's Encrypt Oak 2024h1||Authentic||Usable||Usable||2023-03-16T00:35:28.898Z|
|TLS||DigiCert Yeti 2024||Authentic||Usable||Usable||2023-03-16T00:35:29.071Z|
This is great engineering by Cloudflare and helps improve the robustness of the WebPKI in the face of certificate authority blunders or catastrophic log failures. Unfortunately, Chrome is considering removing support for SCTs delivered via the TLS handshake. We believe this would be a mistake and hope they reconsider.
This isn't the first time that certificate authorities have botched Certificate Transparency compliance and issued nonfunctional certificates: in 2021, several CAs missed the memo that Apple had increased the minimum number of SCTs.
It's unfortunate that certificate authorities cannot be relied upon to validate the information they place in their certificates. This means that site owners need to vigilant about the mistakes made by CAs: both mistakes that lead to attackers getting certificates for your domains, and mistakes that cause your certificates to not work in browsers. You can learn more about how certificate monitoring by SSLMate can help.