Certificate Transparency: A Brief Overview

Certificate Transparency (CT) is a system for publishing SSL certificates in public logs. Certificate Transparency monitors like Cert Spotter download the logs to help you track certificate expiration and detect unauthorized certificates.

At a high level, Certificate Transparency works as follows:

1. Before issuing a certificate, the certificate authority (CA) creates a "precertificate" containing the details of the certificate it intends to issue.
2. The CA submits the precertificate to multiple Certificate Transparency logs.
3. Each log returns a receipt, called a Signed Certificate Timestamp (SCT), which confirms submission of the precertificate.
4. The CA embeds the SCTs in the certificate which it gives to the site operator.
5. When a browser loads a website, it makes sure the website's certificate has SCTs from a sufficient number of recognized logs. If it doesn't, the browser throws up an error page (ERR_CERTIFICATE_TRANSPARENCY_REQUIRED in Chrome) and refuses to load the website.

Note that only the precertificate needs to be submitted to logs; the certificate itself may or may not end up in logs. Some CAs submit them automatically, and crawlers such as the Googlebot submit certificates that they observe when crawling the Internet.

What Went Wrong

The incident began last Wednesday, March 15 at 15:13 UTC, when Sectigo, a CT log operator, began a migration of their "Sabre" log to newer software. There was about an hour and a half of planned downtime. At 16:30 UTC the log came back online. The migration was mostly successful, except for one rather significant problem: the new log software had been configured with the private key of a test log called "Dodo". The problem wasn't noticed and corrected until the next day, March 16, at 12:28 UTC. During this 20 hour window, all of the artifacts produced by Sabre, including the SCTs returned to CAs, were signed with Dodo's private key.

In theory, this should not have been a problem - CAs could have used Sabre's public key to validate the SCT signatures, rejected the ones with invalid signatures, and gotten SCTs from other logs.

Unfortunately, history shows that certificate authorities are prone to sloppiness and shortcut-taking. There was no doubt that at least some CAs would blindly take SCTs and embed them in certificates without validating the signature. Browsers, of course, do validate SCT signatures, and reject SCTs signed by Dodo, since it is a test log.

The only question was how many CAs were affected. To answer this question, SSLMate examined the approximately 164,000 precertificates which were logged to Sabre during the incident. For every precertificate, we looked for its corresponding certificate (in any log, not just Sabre), and evaluated the certificate's embedded SCTs for compliance with browser Certificate Transparency policies. Among the 35 distinct CAs logging to Sabre during the incident, we found:

• 12 CAs which had at least one non-compliant certificate, indicating that the CA does not validate SCT signatures: Amazon Trust Services (operated by DigiCert), Buypass, CFCA, certSIGN, Certum, DigiCert, Globaltrust, Microsoft, NAVER Cloud, Netlock, SwissSign, Viking Cloud
• 6 CAs with only complaint certificates, indicating that they correctly rejected SCTs from Dodo: Certainly, Chunghwa Telecom, Cybertrust Japan, IdenTrust, Let's Encrypt, Sectigo
• 17 CAs for which we could not find any certificates, making it indeterminate whether or not they validate SCT signatures: Atos Trustcenter, Certigna, D-Trust, Deutsche Telekom Security, Firmaprofesional, Google Trust Services, Government of Spain - ACCV, Government of Spain - FNMT, HARICA, Izenpe, Microsec, OISTE, QuoVadis, SSL.com, Shanghai Electronic Certification Authority, Taiwan-CA, Telia

Below are the total number of precertificates that were logged to Sabre during the incident by CAs which do not validate SCT signatures. Presumably all of the corresponding certificates contain Dodo SCTs that will be rejected by browsers:

Amazon Trust Services (operated by DigiCert)	42,363
Buypass	1
CFCA	19
certSIGN	2
Certum	878
DigiCert	27,367
Globaltrust	1
Microsoft	83,024
NAVER Cloud	1
Netlock	5
SwissSign	4
Viking Cloud	29
Total	153,694

Cloudflare for the Win

As a content distribution network, Cloudflare automatically obtains certificates for its customers' websites. Despite getting some of its certificates from DigiCert, Cloudflare-hosted websites were not affected by the incident. That's because Cloudflare evaluates every certificate for Certificate Transparency policy compliance, and if a certificate lacks a sufficient number of SCTs, Cloudflare submits the certificate to additional logs and serves the resulting SCTs via the TLS handshake.

We found several examples of Cloudflare-hosted websites that were serving a certificate with insufficient SCTs yet still worked fine thanks to the additional SCTs in the TLS handshake. Here's an example of the SCTs served by one such website:

Mechanism	Log	Signature	Log Status		Timestamp
			Apple	Chrome
Embedded	Google Argon 2024	Authentic	Usable	Usable	2023-03-16T00:32:03.504Z
Embedded	Sectigo Dodo	Authentic	Unrecognized	Unrecognized	2023-03-16T00:32:03.854Z
Embedded	Google Xenon 2024	Authentic	Usable	Usable	2023-03-16T00:32:03.59Z
TLS	Let's Encrypt Oak 2024h1	Authentic	Usable	Usable	2023-03-16T00:35:28.898Z
TLS	DigiCert Yeti 2024	Authentic	Usable	Usable	2023-03-16T00:35:29.071Z

This is great engineering by Cloudflare and helps improve the robustness of the WebPKI in the face of certificate authority blunders or catastrophic log failures. Unfortunately, Chrome is considering removing support for SCTs delivered via the TLS handshake. We believe this would be a mistake and hope they reconsider.

Final Thoughts

This isn't the first time that certificate authorities have botched Certificate Transparency compliance and issued nonfunctional certificates: in 2021, several CAs missed the memo that Apple had increased the minimum number of SCTs.

It's unfortunate that certificate authorities cannot be relied upon to validate the information they place in their certificates. This means that site owners need to vigilant about the mistakes made by CAs: both mistakes that lead to attackers getting certificates for your domains, and mistakes that cause your certificates to not work in browsers. You can learn more about how certificate monitoring by SSLMate can help.