Skip to content

SSLMate Blog

Why Your Website Is Failing With ERR_CERTIFICATE_TRANSPARENCY_REQUIRED (March 2023 Edition)

If your website uses an SSL certificate that was issued on March 15 or 16, 2023 by one of the following certificate authorities:

  • Amazon Trust Services (operated by DigiCert)
  • Buypass
  • CFCA
  • certSIGN
  • Certum
  • DigiCert
  • Globaltrust
  • Microsoft
  • NAVER Cloud
  • Netlock
  • SwissSign
  • Viking Cloud

then your site might not load in Chrome, Safari, and Edge, and you should contact your certificate authority to get a new, working certificate ASAP. You can use our Certificate Transparency Policy Analyzer to see for sure if your site is affected. This blog post will explain why over 150,000 certificates issued last week are being rejected by browsers, and how SSLMate analyzed the incident.

Certificate Transparency: A Brief Overview

Certificate Transparency (CT) is a system for publishing SSL certificates in public logs. Certificate Transparency monitors like Cert Spotter download the logs to help you track certificate expiration and detect unauthorized certificates.

At a high level, Certificate Transparency works as follows:

  1. Before issuing a certificate, the certificate authority (CA) creates a "precertificate" containing the details of the certificate it intends to issue.
  2. The CA submits the precertificate to multiple Certificate Transparency logs.
  3. Each log returns a receipt, called a Signed Certificate Timestamp (SCT), which confirms submission of the precertificate.
  4. The CA embeds the SCTs in the certificate which it gives to the site operator.
  5. When a browser loads a website, it makes sure the website's certificate has SCTs from a sufficient number of recognized logs. If it doesn't, the browser throws up an error page (ERR_CERTIFICATE_TRANSPARENCY_REQUIRED in Chrome) and refuses to load the website.

Note that only the precertificate needs to be submitted to logs; the certificate itself may or may not end up in logs. Some CAs submit them automatically, and crawlers such as the Googlebot submit certificates that they observe when crawling the Internet.

What Went Wrong

The incident began last Wednesday, March 15 at 15:13 UTC, when Sectigo, a CT log operator, began a migration of their "Sabre" log to newer software. There was about an hour and a half of planned downtime. At 16:30 UTC the log came back online. The migration was mostly successful, except for one rather significant problem: the new log software had been configured with the private key of a test log called "Dodo". The problem wasn't noticed and corrected until the next day, March 16, at 12:28 UTC. During this 20 hour window, all of the artifacts produced by Sabre, including the SCTs returned to CAs, were signed with Dodo's private key.

In theory, this should not have been a problem - CAs could have used Sabre's public key to validate the SCT signatures, rejected the ones with invalid signatures, and gotten SCTs from other logs.

Unfortunately, history shows that certificate authorities are prone to sloppiness and shortcut-taking. There was no doubt that at least some CAs would blindly take SCTs and embed them in certificates without validating the signature. Browsers, of course, do validate SCT signatures, and reject SCTs signed by Dodo, since it is a test log.

The only question was how many CAs were affected. To answer this question, SSLMate examined the approximately 164,000 precertificates which were logged to Sabre during the incident. For every precertificate, we looked for its corresponding certificate (in any log, not just Sabre), and evaluated the certificate's embedded SCTs for compliance with browser Certificate Transparency policies. Among the 35 distinct CAs logging to Sabre during the incident, we found:

  • 12 CAs which had at least one non-compliant certificate, indicating that the CA does not validate SCT signatures: Amazon Trust Services (operated by DigiCert), Buypass, CFCA, certSIGN, Certum, DigiCert, Globaltrust, Microsoft, NAVER Cloud, Netlock, SwissSign, Viking Cloud
  • 6 CAs with only complaint certificates, indicating that they correctly rejected SCTs from Dodo: Certainly, Chunghwa Telecom, Cybertrust Japan, IdenTrust, Let's Encrypt, Sectigo
  • 17 CAs for which we could not find any certificates, making it indeterminate whether or not they validate SCT signatures: Atos Trustcenter, Certigna, D-Trust, Deutsche Telekom Security, Firmaprofesional, Google Trust Services, Government of Spain - ACCV, Government of Spain - FNMT, HARICA, Izenpe, Microsec, OISTE, QuoVadis, SSL.com, Shanghai Electronic Certification Authority, Taiwan-CA, Telia

Below are the total number of precertificates that were logged to Sabre during the incident by CAs which do not validate SCT signatures. Presumably all of the corresponding certificates contain Dodo SCTs that will be rejected by browsers:

Amazon Trust Services (operated by DigiCert) 42,363
Buypass 1
CFCA 19
certSIGN 2
Certum 878
DigiCert 27,367
Globaltrust 1
Microsoft 83,024
NAVER Cloud 1
Netlock 5
SwissSign 4
Viking Cloud 29
Total 153,694

Cloudflare for the Win

As a content distribution network, Cloudflare automatically obtains certificates for its customers' websites. Despite getting some of its certificates from DigiCert, Cloudflare-hosted websites were not affected by the incident. That's because Cloudflare evaluates every certificate for Certificate Transparency policy compliance, and if a certificate lacks a sufficient number of SCTs, Cloudflare submits the certificate to additional logs and serves the resulting SCTs via the TLS handshake.

We found several examples of Cloudflare-hosted websites that were serving a certificate with insufficient SCTs yet still worked fine thanks to the additional SCTs in the TLS handshake. Here's an example of the SCTs served by one such website:

Mechanism Log Signature Log Status Timestamp
Apple Chrome
Embedded Google Argon 2024 Authentic Usable Usable 2023-03-16T00:32:03.504Z
Embedded Sectigo Dodo Authentic Unrecognized Unrecognized 2023-03-16T00:32:03.854Z
Embedded Google Xenon 2024 Authentic Usable Usable 2023-03-16T00:32:03.59Z
TLS Let's Encrypt Oak 2024h1 Authentic Usable Usable 2023-03-16T00:35:28.898Z
TLS DigiCert Yeti 2024 Authentic Usable Usable 2023-03-16T00:35:29.071Z

This is great engineering by Cloudflare and helps improve the robustness of the WebPKI in the face of certificate authority blunders or catastrophic log failures. Unfortunately, Chrome is considering removing support for SCTs delivered via the TLS handshake. We believe this would be a mistake and hope they reconsider.

Final Thoughts

This isn't the first time that certificate authorities have botched Certificate Transparency compliance and issued nonfunctional certificates: in 2021, several CAs missed the memo that Apple had increased the minimum number of SCTs.

It's unfortunate that certificate authorities cannot be relied upon to validate the information they place in their certificates. This means that site owners need to vigilant about the mistakes made by CAs: both mistakes that lead to attackers getting certificates for your domains, and mistakes that cause your certificates to not work in browsers. You can learn more about how certificate monitoring by SSLMate can help.

Tens of Thousands of GoDaddy Certificates Will Break in macOS 11.4 and iOS 14.6

macOS 11.4 and iOS 14.6 impose some new requirements on publicly-trusted SSL certificates which were issued on or after April 21 2021. In general, as a website owner you don't need to worry about this change, as it's your certificate authority's job to stay abreast of policy changes and provide you certificates which are compliant with web browser policies (all certificates issued through SSLMate are compliant).

Unfortunately, some certificate authorities, namely GoDaddy, GlobalSign, Certigna, and WidePoint, messed up and issued tens of thousands of non-compliant certificates between April 21 and April 27 that will not work in macOS 11.4 and iOS 14.6 (Safari will say "This Connection is Not Private"). If you got a certificate from one of these CAs during this time frame, you might need to replace your certificate. You can use our Certificate Transparency Policy Analyzer to check if your website is affected. If you're a Cert Spotter customer, Cert Spotter checks for this problem automatically and will alert you if you need to take action.

Background

Apple and Chrome require certificate authorities to publish information about their certificates in public Certificate Transparency logs so that misbehavior by certificate authorities can be detected. For example, Cert Spotter monitors Certificate Transparency logs and alerts you when a certificate is issued for your website that you didn't authorize.

Before issuing a certificate, the certificate authority publicly announces its intent to issue the certificate by publishing a precertificate in multiple Certificate Transparency Logs. Precertificates are similar to certificates and contain the same information that will be included in the certificate, such as the domain names that the certificate will be valid for.

Each Certificate Transparency log returns a receipt, called a Signed Certificate Timestamp or SCT, promising to publish the precertificate. The certificate authority embeds the SCTs in the certificate. When Chrome or Safari load a website, they examine the SCTs, and if the certificate lacks a sufficient number of SCTs, they reject the certificate and refuse to load the website.

The number of required SCTs depends on the certificate lifetime. For certificates issued before April 21, 2021, both Chrome and Apple used the following policy:

Lifetime # of SCTs From Distinct Logs
< 15 months 2
15 - 27 months 3
27 - 39 months 4
> 39 months 5

But for certificates issued on or after April 21, Apple now requires:

Lifetime # of SCTs From Distinct Logs
180 days or less 2
181 to 398 days 3

This change was communicated to certificate authorities ahead-of-time, and most of them got the memo. Unfortunately, certificate authorities often have a hard time complying with the rules.

Estimating the Amount of Non-Compliance

To estimate how many non-compliant certificates have been issued, Rob Stradling of crt.sh searched for precertificates which were issued on or after April 21, are valid for more than 180 days, and which were logged to fewer than three logs. This provides a lower bound on the number of non-compliant certificates, since if a precertificate is in fewer than 3 logs, its corresponding certificate can't possibly have SCTs from 3 or more logs. However, it's only a lower bound: a precertificate might be in 3 or more logs, but the corresponding certificate doesn't necessarily embed SCTs from all of them.

Here's a summary of the data collected by Rob:

Certificate Authority # Non-Compliant Precertificates
GoDaddy 76,376
GlobalSign 183
Certigna 15
WidePoint 5

To get an alternative lower bound, SSLMate searched for certificates (not precertificates) issued between April 21 and May 3 which are valid for more than 180 days, embed at least one SCT, and which are not compliant with Apple's Certificate Transparency policy. (We skipped certificates containing zero SCTs, as such certificates were probably intentionally not logged, which is permissible as long as you don't need your certificate to work in Safari or Chrome.) Evaluating the certificate, rather than the precertificate, definitively tells us whether or not the certificate is compliant. However, most certificate authorities only log precertificates, and their certificates will only be logged if they're discovered and submitted by a third party, such as the Googlebot. Therefore, evaluating certificates doesn't provide us a full picture of a certificate authority's compliance.

Here are our results, broken down by issuance date so that we can determine when each CA fixed the problem:

Certificate Authority Issuance Date # Non-Compliant Certificates
GlobalSign 2021-04-21 39
GlobalSign 2021-04-22 72
GlobalSign 2021-04-23 31
GlobalSign 2021-04-24 14
GlobalSign 2021-04-25 10
GlobalSign 2021-04-26 338
GoDaddy 2021-04-21 67
GoDaddy 2021-04-22 72
GoDaddy 2021-04-23 50
GoDaddy 2021-04-24 47
GoDaddy 2021-04-25 32
GoDaddy 2021-04-26 74
GoDaddy 2021-04-27 39

Our data increases the lower bound for GlobalSign to 504 non-compliant certificates.

How Did This Happen?

Looking at the non-compliant certificates, some interesting patterns emerge.

Every single one of GlobalSign's non-compliant certificates has a validity period of 182 days, 14 hours, 54 minutes, and 36 seconds. This happens to be just under the number of days in the six month period between April and October. It appears that GlobalSign thought that Apple's new policy affected certificates with validity periods greater than six months rather than greater than 180 days.

As for GoDaddy, the validity periods of the non-compliant certificates range from 181 to 254 days. None of the certificates has a validity period of a full year, suggesting that only reissued certificates (which have the same expiration dates as the original certificate) were affected. It's possible that GoDaddy was consulting the issuance date of the original certificate, rather than the reissued certificate, when deciding how many SCTs the reissued certificate needed to have.

Monitor and Automate MTA-STS with Cert Spotter

By default, email is insecure: any email sent to your domains is at risk of being intercepted and read by network attackers. Even if your domains support STARTTLS, an active attacker can downgrade the connection to plaintext. MTA-STS changes that. It's a new standard that lets you publish a policy requiring that inbound email to your domains be encrypted in transit using authenticated STARTTLS. Major providers like Gmail already support MTA-STS. If your domains receive email, whether via a third party provider or your own mail servers, you should consider deploying MTA-STS.

However, MTA-STS is tricky to deploy correctly, and a mistake can prevent email from getting through. MTA-STS imposes specific requirements on your domain's mail servers; if the mail servers fall out of compliance, due to problems such as an expired or incorrectly-installed certificate, you'll lose email. If your MTA-STS policy falls out of sync with your domain's DNS, you'll lose email. The antidotes are monitoring to detect mistakes, and automation to prevent mistakes in the first place. Cert Spotter now does both: monitor your domains for MTA-STS problems, and automate the publication of correct MTA-STS policies.

Monitoring

Cert Spotter continuously monitors your domains' MTA-STS policies and alerts you if a problem would prevent the delivery of mail. Cert Spotter automatically detects MTA-STS policies for any of your monitored endpoints; you don't need to enable MTA-STS monitoring separately.

Among the problems detected by Cert Spotter are:

  • A mail server is not eligible for MTA-STS (doesn't support STARTTLS with TLSv1.2 or higher, or has an expired, mismatched, untrusted, or otherwise invalid certificate).
  • The MTA-STS policy can't be retrieved or is invalid.
  • The MX servers listed in the policy don't match your domain's MX records.
  • The MTA-STS policy has changed without rotating the ID in the _mta-sts DNS record.
  • The MTA-STS policy has been removed without following the proper procedure to flush policies cached by sending mail servers.

Additionally, Cert Spotter provides a dashboard which shows MTA-STS eligibility and deployment status in one place. You can use this dashboard to track MTA-STS deployment across your entire domain portfolio:

Screenshot of MTA-STS dashboard

Automatic Policy Publication

Deploying an MTA-STS policy initially requires:

  • Publishing a text file over HTTPS at mta-sts.YOURDOMAIN, which requires a valid certificate for mta-sts.YOURDOMAIN.
  • Publishing a DNS TXT record at _mta-sts.YOURDOMAIN containing the ID of the policy.

Maintaining an MTA-STS policy on an ongoing basis requires:

  • Renewing the HTTPS certificate for mta-sts.YOURDOMAIN.
  • Changing the policy any time you change your domain's MX records.
  • Changing the TXT record's ID any time you change the policy.

Cert Spotter can optionally automate the above steps. All you need to do is create DNS CNAME records pointing mta-sts.YOURDOMAIN and _mta-sts.YOURDOMAIN to domain names hosted by SSLMate. We'll automatically publish a policy matching your MX records. For transparency, we'll alert you any time your MX records change, so you can take corrective action if the change was unauthorized. To set up automated MTA-STS, visit the endpoint page for your domain to retrieve the necessary DNS records and configure policy settings like the enforcement mode and the maximum age:

Screenshot of automatic MTA-STS settings

Final Thoughts

Email is a critical communication tool, and despite its flaws, it's here to stay. It deserves the same level of encryption and authentication that we take for granted with web browsing. MTA-STS is a major step in that direction, and Cert Spotter is here to help you easily and fearlessly deploy MTA-STS on your domains. Sign up for Cert Spotter today

You Can Now Use CAA to Auto-Authorize Certificates in Cert Spotter, Without Compromising Transparency

One of the most requested features for the new Cert Spotter has been for Cert Spotter to consult CAA records to decide if a certificate is authorized. This makes a lot of sense - many domain operators have already published CAA records, and it's redundant to duplicate the information in your Cert Spotter settings. Also, CAA records can apply different policies to different domain names, which is more flexible than the list of authorized CAs in your Cert Spotter settings, which applies to every domain you monitor.

However, implementing this feature without degrading the security of Cert Spotter was trickier than it sounds. DNS is, by and large, not authenticated, which means that when Cert Spotter goes to look up CAA records, a network-layer attacker could return bogus CAA records which trick Cert Spotter into thinking a certificate is authorized.

DNSSEC, if we set aside its low adoption rate and deployment challenges, would at least make CAA lookups authenticated, but they still wouldn't be transparent. A misbehaving parent zone, such as a TLD registry, could return validly-signed but malicious CAA records and no one would be the wiser. That's a problem for Cert Spotter, which takes great care not to blindly trust third parties. Cert Spotter verifies the Merkle Trees of all Certificate Transparency logs, and gossips information about them with other auditors, including Google's Certificate Transparency monitor and a network of "honeybees" located all over the Internet. This gives Cert Spotter a high degree of confidence that the logs which it relies upon are not hiding certificates. DNSSEC doesn't provide the same level of confidence that DNS servers are not misbehaving.

For this reason, using CAA was never a consideration during the development of Cert Spotter. But our customers have spoken, so we set out to find a way to mitigate the risks of CAA as much as possible, and settled on the following solution:

  1. You can optionally enable the use of CAA records for certificate authorization in your settings.

  2. At any time you can examine the list of CAA records that Cert Spotter is relying upon to make authorization decisions.

  3. Cert Spotter will email you any time it detects that a CAA record has been added, modified, or removed.

While this solution won't stop Cert Spotter from being fooled by a malicious CAA record, you will at least know about it and be able to take action in response - much like Certificate Transparency doesn't stop malicious certificates, but makes sure you know about them.

Cert Spotter's use of CAA records is unique among Certificate Transparency monitors and is just one example of how SSLMate is innovating to bring you certificate monitoring that reduces alert fatigue by making it easy to automatically authorize (and elide notifications for) legitimate certificates - without reducing security. We have some other features in the pipeline to offer even more flexibility in auto-authorizing certificates. Get in touch if you are interested in being added to our beta tester list.

Do you operate a website? Do you like security and uptime? Cert Spotter monitors your website's security certificates to give you early warning about security and uptime problems like malicious certificates, DNS tampering, and certificate expiration - so you can act before your customers notice a problem, not after.

Introducing the New Cert Spotter - with Expiration Monitoring, Alert Fatigue Reduction, and More

Today, we're launching a brand new version of Cert Spotter which protects your websites from costly outages caused by expired certificates, reduces alert fatigue by eliding notifications about legitimate certificates, and saves you time managing certificates with a new web dashboard.

We're also making some changes to Cert Spotter's pricing to reflect the new value that Cert Spotter provides. Please read on to learn about our changes.

Avoid Outages From Expired Certificates

Cert Spotter now monitors every one of your sub-domains found in Certificate Transparency logs for an up-to-date certificate. If any sub-domain has a certificate expiring in less than 30 days, we'll alert you, so you can prevent outages from forgotten certificates and broken automation.

We check for an up-to-date certificate by fetching the certificate used for HTTPS on port 443. If we can't connect to port 443, because the host is firewalled or otherwise inaccessible, we consult Certificate Transparency for the latest certificate for the hostname.

Our use of Certificate Transparency makes Cert Spotter more likely to find expiring certificates than traditional certificate expiration monitors, which only work with publicly-accessible endpoints and require you to manually configure individual sub-domains.

Reduce Alert Fatigue So You'll Always React When You Need To

Certificate Transparency alerts should be rare and taken seriously, not frequent and ignored like car alarms. Cert Spotter now provides two ways you can tell us about legitimate certificates so that you are only alerted about unauthorized certificates.

If you trust certain certificate authorities, you can choose not to be notified about certificates which they issue. Select your authorized certificate authorities

Alternatively, if your certificate issuance is automated, you can use a new API to automatically notify Cert Spotter about your legitimate certificates. We won't notify you about these certificates when we discover them. More information about the API

Easier Certificate Management With Central Web Dashboard

You no longer need to track your certificates in an error-prone spreadsheet or shared calendar. The new endpoint dashboard shows information about all of your domains and sub-domains in one place. For each endpoint, Cert Spotter tells you the expiration of the endpoint's most recent certificate, the issuer(s) of the endpoint's certificates, and whether the most recent certificate has been installed properly on the endpoint. A separate dashboard shows all endpoints with certificates expiring in the next 30 days so you know where to focus your attention.

New Pricing

First, since the new Cert Spotter does work for each of your sub-domains by monitoring them for expiration, pricing is now based on total number of endpoints, rather than domains. For example, monitoring 5 domains with 2 sub-domains each now costs the same as monitoring a single domain with 10 sub-domains. If you are currently a customer, this change won't affect you: we've grandfathered your account so you will immediately gain access to all of Cert Spotter's new features at the same price you're currently paying. (If you want, you can convert to per-endpoint pricing here.)

Second, we're discontinuing the free tier, and introducing a new set of plans, starting at $15 / month.

Discontinuing the free tier wasn't an easy decision to make, but it was a necessary one. Since Cert Spotter launched in 2016, the number of certificates has grown from 50 million to over 2 billion, making it significantly more costly to monitor Certificate Transparency. The free tier wasn't sustainable if we wanted to continue operating Cert Spotter while improving it to provide you with the best infrastructure monitoring possible.

Although SSLMate has been approached by investors, we've chosen to remain an entirely customer-funded company. That means we're guided by what is best for you, not what is best for venture capitalists. We're building a service for the long haul, not one that will be discontinued after an acquisition, or neglected when the next shiny object takes priority. Additionally, we're guided by what is good for the Internet, which is why we participate in the standards and policy forums for Certificate Transparency. Your subscription to Cert Spotter supports our efforts and helps ensure that Internet standards and policy are not decided solely by huge corporations.

We understand that some people might not want to pay for the new Cert Spotter, and for them we recommend the Facebook or Cloudflare monitors, which provide functionality similar to the old Cert Spotter. However, we think you'll want to use the new Cert Spotter. The price of Cert Spotter is far below the cost of a single outage from an expired certificate, which Cert Spotter helps prevent. And since Cert Spotter suppresses alerts for legitimate certificates, you're more likely to notice an alert for a rogue certificate, making you more secure. What's more, we reply to your emails, so if you ever have a question about a certificate, or need help getting an unwanted certificate revoked, expert help is just an email away.

If you currently use the free tier, we have subscribed you to an extended 90 day free trial of the new Cert Spotter. When the trial is over, you'll receive 20% off your subscription, forever, as long as you choose a plan before the trial ends on February 12, 2020.

If you have any questions about our changes or would like help using Cert Spotter's new features, shoot me an email at andrew@sslmate.com. I thank you for your business.

Andrew Ayer
Founder, SSLMate

Show Older Posts