SSLMate Blog

Chronicling the revolution of SSL certificate management.

May 5, 2021

Tens of Thousands of GoDaddy Certificates Will Break in macOS 11.4 and iOS 14.6

macOS 11.4 and iOS 14.6 impose some new requirements on publicly-trusted SSL certificates which were issued on or after April 21 2021. In general, as a website owner you don't need to worry about this change, as it's your certificate authority's job to stay abreast of policy changes and provide you certificates which are compliant with web browser policies (all certificates issued through SSLMate are compliant).

Unfortunately, some certificate authorities, namely GoDaddy, GlobalSign, Certigna, and WidePoint, messed up and issued tens of thousands of non-compliant certificates between April 21 and April 27 that will not work in macOS 11.4 and iOS 14.6. If you got a certificate from one of these CAs during this time frame, you might need to replace your certificate. You can use our Certificate Transparency Policy Analyzer to check if your website is affected. If you're a Cert Spotter customer, Cert Spotter checks for this problem automatically and will alert you if you need to take action.

Background

Apple and Chrome require certificate authorities to publish information about their certificates in public Certificate Transparency logs so that misbehavior by certificate authorities can be detected. For example, Cert Spotter monitors Certificate Transparency logs and alerts you when a certificate is issued for your website that you didn't authorize.

Before issuing a certificate, the certificate authority publicly announces its intent to issue the certificate by publishing a precertificate in multiple Certificate Transparency Logs. Precertificates are similar to certificates and contain the same information that will be included in the certificate, such as the domain names that the certificate will be valid for.

Each Certificate Transparency log returns a receipt, called a Signed Certificate Timestamp or SCT, promising to publish the precertificate. The certificate authority embeds the SCTs in the certificate. When Chrome or Safari load a website, they examine the SCTs, and if the certificate lacks a sufficient number of SCTs, they reject the certificate and refuse to load the website.

The number of required SCTs depends on the certificate lifetime. For certificates issued before April 21, 2021, both Chrome and Apple used the following policy:

Lifetime# of SCTs From Distinct Logs
< 15 months2
15 - 27 months3
27 - 39 months4
> 39 months5

But for certificates issued on or after April 21, Apple now requires:

Lifetime# of SCTs From Distinct Logs
180 days or less2
181 to 398 days3

This change was communicated to certificate authorities ahead-of-time, and most of them got the memo. Unfortunately, certificate authorities often have a hard time complying with the rules.

Estimating the Amount of Non-Compliance

To estimate how many non-compliant certificates have been issued, Rob Stradling of crt.sh searched for precertificates which were issued on or after April 21, are valid for more than 180 days, and which were logged to fewer than three logs. This provides a lower bound on the number of non-compliant certificates, since if a precertificate is in fewer than 3 logs, its corresponding certificate can't possibly have SCTs from 3 or more logs. However, it's only a lower bound: a precertificate might be in 3 or more logs, but the corresponding certificate doesn't necessarily embed SCTs from all of them.

Here's a summary of the data collected by Rob:

Certificate Authority# Non-Compliant Precertificates
GoDaddy76,376
GlobalSign183
Certigna15
WidePoint5

To get an alternative lower bound, SSLMate searched for certificates (not precertificates) issued between April 21 and May 3 which are valid for more than 180 days, embed at least one SCT, and which are not compliant with Apple's Certificate Transparency policy. (We skipped certificates containing zero SCTs, as such certificates were probably intentionally not logged, which is permissible as long as you don't need your certificate to work in Safari or Chrome.) Evaluating the certificate, rather than the precertificate, definitively tells us whether or not the certificate is compliant. However, most certificate authorities only log precertificates, and their certificates will only be logged if they're discovered and submitted by a third party, such as the Googlebot. Therefore, evaluating certificates doesn't provide us a full picture of a certificate authority's compliance.

Here are our results, broken down by issuance date so that we can determine when each CA fixed the problem:

Certificate AuthorityIssuance Date# Non-Compliant Certificates
GlobalSign2021-04-21 39
GlobalSign2021-04-22 72
GlobalSign2021-04-23 31
GlobalSign2021-04-24 14
GlobalSign2021-04-25 10
GlobalSign2021-04-26338
GoDaddy 2021-04-21 67
GoDaddy 2021-04-22 72
GoDaddy 2021-04-23 50
GoDaddy 2021-04-24 47
GoDaddy 2021-04-25 32
GoDaddy 2021-04-26 74
GoDaddy 2021-04-27 39

Our data increases the lower bound for GlobalSign to 504 non-compliant certificates.

How Did This Happen?

Looking at the non-compliant certificates, some interesting patterns emerge.

Every single one of GlobalSign's non-compliant certificates has a validity period of 182 days, 14 hours, 54 minutes, and 36 seconds. This happens to be just under the number of days in the six month period between April and October. It appears that GlobalSign thought that Apple's new policy affected certificates with validity periods greater than six months rather than greater than 180 days.

As for GoDaddy, the validity periods of the non-compliant certificates range from 181 to 254 days. None of the certificates has a validity period of a full year, suggesting that only reissued certificates (which have the same expiration dates as the original certificate) were affected. It's possible that GoDaddy was consulting the issuance date of the original certificate, rather than the reissued certificate, when deciding how many SCTs the reissued certificate needed to have.

November 12, 2020

Monitor and Automate MTA-STS with Cert Spotter

By default, email is insecure: any email sent to your domains is at risk of being intercepted and read by network attackers. Even if your domains support STARTTLS, an active attacker can downgrade the connection to plaintext. MTA-STS changes that. It's a new standard that lets you publish a policy requiring that inbound email to your domains be encrypted in transit using authenticated STARTTLS. Major providers like Gmail already support MTA-STS. If your domains receive email, whether via a third party provider or your own mail servers, you should consider deploying MTA-STS.

However, MTA-STS is tricky to deploy correctly, and a mistake can prevent email from getting through. MTA-STS imposes specific requirements on your domain's mail servers; if the mail servers fall out of compliance, due to problems such as an expired or incorrectly-installed certificate, you'll lose email. If your MTA-STS policy falls out of sync with your domain's DNS, you'll lose email. The antidotes are monitoring to detect mistakes, and automation to prevent mistakes in the first place. Cert Spotter now does both: monitor your domains for MTA-STS problems, and automate the publication of correct MTA-STS policies.

Monitoring

Cert Spotter continuously monitors your domains' MTA-STS policies and alerts you if a problem would prevent the delivery of mail. Cert Spotter automatically detects MTA-STS policies for any of your monitored endpoints; you don't need to enable MTA-STS monitoring separately.

Among the problems detected by Cert Spotter are:

  • A mail server is not eligible for MTA-STS (doesn't support STARTTLS with TLSv1.2 or higher, or has an expired, mismatched, untrusted, or otherwise invalid certificate).
  • The MTA-STS policy can't be retrieved or is invalid.
  • The MX servers listed in the policy don't match your domain's MX records.
  • The MTA-STS policy has changed without rotating the ID in the _mta-sts DNS record.
  • The MTA-STS policy has been removed without following the proper procedure to flush policies cached by sending mail servers.

Additionally, Cert Spotter provides a dashboard which shows MTA-STS eligibility and deployment status in one place. You can use this dashboard to track MTA-STS deployment across your entire domain portfolio:

Screenshot of MTA-STS dashboard

Automatic Policy Publication

Deploying an MTA-STS policy initially requires:

  • Publishing a text file over HTTPS at mta-sts.YOURDOMAIN, which requires a valid certificate for mta-sts.YOURDOMAIN.
  • Publishing a DNS TXT record at _mta-sts.YOURDOMAIN containing the ID of the policy.

Maintaining an MTA-STS policy on an ongoing basis requires:

  • Renewing the HTTPS certificate for mta-sts.YOURDOMAIN.
  • Changing the policy any time you change your domain's MX records.
  • Changing the TXT record's ID any time you change the policy.

Cert Spotter can optionally automate the above steps. All you need to do is create DNS CNAME records pointing mta-sts.YOURDOMAIN and _mta-sts.YOURDOMAIN to domain names hosted by SSLMate. We'll automatically publish a policy matching your MX records. For transparency, we'll alert you any time your MX records change, so you can take corrective action if the change was unauthorized. To set up automated MTA-STS, visit the endpoint page for your domain to retrieve the necessary DNS records and configure policy settings like the enforcement mode and the maximum age:

Screenshot of automatic MTA-STS settings

Final Thoughts

Email is a critical communication tool, and despite its flaws, it's here to stay. It deserves the same level of encryption and authentication that we take for granted with web browsing. MTA-STS is a major step in that direction, and Cert Spotter is here to help you easily and fearlessly deploy MTA-STS on your domains. Sign up for Cert Spotter today

February 10, 2020

You Can Now Use CAA to Auto-Authorize Certificates in Cert Spotter, Without Compromising Transparency

One of the most requested features for the new Cert Spotter has been for Cert Spotter to consult CAA records to decide if a certificate is authorized. This makes a lot of sense - many domain operators have already published CAA records, and it's redundant to duplicate the information in your Cert Spotter settings. Also, CAA records can apply different policies to different domain names, which is more flexible than the list of authorized CAs in your Cert Spotter settings, which applies to every domain you monitor.

However, implementing this feature without degrading the security of Cert Spotter was trickier than it sounds. DNS is, by and large, not authenticated, which means that when Cert Spotter goes to look up CAA records, a network-layer attacker could return bogus CAA records which trick Cert Spotter into thinking a certificate is authorized.

DNSSEC, if we set aside its low adoption rate and deployment challenges, would at least make CAA lookups authenticated, but they still wouldn't be transparent. A misbehaving parent zone, such as a TLD registry, could return validly-signed but malicious CAA records and no one would be the wiser. That's a problem for Cert Spotter, which takes great care not to blindly trust third parties. Cert Spotter verifies the Merkle Trees of all Certificate Transparency logs, and gossips information about them with other auditors, including Google's Certificate Transparency monitor and a network of "honeybees" located all over the Internet. This gives Cert Spotter a high degree of confidence that the logs which it relies upon are not hiding certificates. DNSSEC doesn't provide the same level of confidence that DNS servers are not misbehaving.

For this reason, using CAA was never a consideration during the development of Cert Spotter. But our customers have spoken, so we set out to find a way to mitigate the risks of CAA as much as possible, and settled on the following solution:

  1. You can optionally enable the use of CAA records for certificate authorization in your settings.

  2. At any time you can examine the list of CAA records that Cert Spotter is relying upon to make authorization decisions.

  3. Cert Spotter will email you any time it detects that a CAA record has been added, modified, or removed.

While this solution won't stop Cert Spotter from being fooled by a malicious CAA record, you will at least know about it and be able to take action in response - much like Certificate Transparency doesn't stop malicious certificates, but makes sure you know about them.

Cert Spotter's use of CAA records is unique among Certificate Transparency monitors and is just one example of how SSLMate is innovating to bring you certificate monitoring that reduces alert fatigue by making it easy to automatically authorize (and elide notifications for) legitimate certificates - without reducing security. We have some other features in the pipeline to offer even more flexibility in auto-authorizing certificates. Get in touch if you are interested in being added to our beta tester list.

Do you operate a website? Do you like security and uptime? Cert Spotter monitors your website's security certificates to give you early warning about security and uptime problems like malicious certificates, DNS tampering, and certificate expiration - so you can act before your customers notice a problem, not after.

November 14, 2019

Introducing the New Cert Spotter - with Expiration Monitoring, Alert Fatigue Reduction, and More

Today, we're launching a brand new version of Cert Spotter which protects your websites from costly outages caused by expired certificates, reduces alert fatigue by eliding notifications about legitimate certificates, and saves you time managing certificates with a new web dashboard.

We're also making some changes to Cert Spotter's pricing to reflect the new value that Cert Spotter provides. Please read on to learn about our changes.

Avoid Outages From Expired Certificates

Cert Spotter now monitors every one of your sub-domains found in Certificate Transparency logs for an up-to-date certificate. If any sub-domain has a certificate expiring in less than 30 days, we'll alert you, so you can prevent outages from forgotten certificates and broken automation.

We check for an up-to-date certificate by fetching the certificate used for HTTPS on port 443. If we can't connect to port 443, because the host is firewalled or otherwise inaccessible, we consult Certificate Transparency for the latest certificate for the hostname.

Our use of Certificate Transparency makes Cert Spotter more likely to find expiring certificates than traditional certificate expiration monitors, which only work with publicly-accessible endpoints and require you to manually configure individual sub-domains.

Reduce Alert Fatigue So You'll Always React When You Need To

Certificate Transparency alerts should be rare and taken seriously, not frequent and ignored like car alarms. Cert Spotter now provides two ways you can tell us about legitimate certificates so that you are only alerted about unauthorized certificates.

If you trust certain certificate authorities, you can choose not to be notified about certificates which they issue. Select your authorized certificate authorities

Alternatively, if your certificate issuance is automated, you can use a new API to automatically notify Cert Spotter about your legitimate certificates. We won't notify you about these certificates when we discover them. More information about the API

Easier Certificate Management With Central Web Dashboard

You no longer need to track your certificates in an error-prone spreadsheet or shared calendar. The new endpoint dashboard shows information about all of your domains and sub-domains in one place. For each endpoint, Cert Spotter tells you the expiration of the endpoint's most recent certificate, the issuer(s) of the endpoint's certificates, and whether the most recent certificate has been installed properly on the endpoint. A separate dashboard shows all endpoints with certificates expiring in the next 30 days so you know where to focus your attention.

New Pricing

First, since the new Cert Spotter does work for each of your sub-domains by monitoring them for expiration, pricing is now based on total number of endpoints, rather than domains. For example, monitoring 5 domains with 2 sub-domains each now costs the same as monitoring a single domain with 10 sub-domains. If you are currently a customer, this change won't affect you: we've grandfathered your account so you will immediately gain access to all of Cert Spotter's new features at the same price you're currently paying. (If you want, you can convert to per-endpoint pricing here.)

Second, we're discontinuing the free tier, and introducing a new set of plans, starting at $15 / month.

Discontinuing the free tier wasn't an easy decision to make, but it was a necessary one. Since Cert Spotter launched in 2016, the number of certificates has grown from 50 million to over 2 billion, making it significantly more costly to monitor Certificate Transparency. The free tier wasn't sustainable if we wanted to continue operating Cert Spotter while improving it to provide you with the best infrastructure monitoring possible.

Although SSLMate has been approached by investors, we've chosen to remain an entirely customer-funded company. That means we're guided by what is best for you, not what is best for venture capitalists. We're building a service for the long haul, not one that will be discontinued after an acquisition, or neglected when the next shiny object takes priority. Additionally, we're guided by what is good for the Internet, which is why we participate in the standards and policy forums for Certificate Transparency. Your subscription to Cert Spotter supports our efforts and helps ensure that Internet standards and policy are not decided solely by huge corporations.

We understand that some people might not want to pay for the new Cert Spotter, and for them we recommend the Facebook or Cloudflare monitors, which provide functionality similar to the old Cert Spotter. However, we think you'll want to use the new Cert Spotter. The price of Cert Spotter is far below the cost of a single outage from an expired certificate, which Cert Spotter helps prevent. And since Cert Spotter suppresses alerts for legitimate certificates, you're more likely to notice an alert for a rogue certificate, making you more secure. What's more, we reply to your emails, so if you ever have a question about a certificate, or need help getting an unwanted certificate revoked, expert help is just an email away.

If you currently use the free tier, we have subscribed you to an extended 90 day free trial of the new Cert Spotter. When the trial is over, you'll receive 20% off your subscription, forever, as long as you choose a plan before the trial ends on February 12, 2020.

If you have any questions about our changes or would like help using Cert Spotter's new features, shoot me an email at andrew@sslmate.com. I thank you for your business.

Andrew Ayer
Founder, SSLMate

November 5, 2019

Elizabeth Warren's Website Uses a Foreign TLD. Hopefully She's Monitoring Certificate Transparency!

Recently, Elizabeth Warren's presidential campaign began using the domain name ewar.ren on Twitter. Links to this domain appear in the Senator's Twitter bio and in her Tweets.

Screenshot of Elizabeth Warren's Twitter bio, showing ewren.ren URL

It's a cute, short domain, but what's the deal with this .ren TLD? According to the ICANNWiki:

.ren is a Brand TLD delegated in ICANN's New gTLD Program. Beijing Qianxiang Wangjing Technology Development Co., Ltd. manages the TLD and is its Registry. The proposed application succeeded and was delegated to the Root Zone on 27 March, 2014. The proposed application succeeded and was delegated to the Root Zone on 27 March 2014.

Unfortunately, Senator Warren's use of .ren exposes her website to possible tampering by the Chinese government, which is of great concern after the 2016 election showed that political campaigns have to consider foreign governments as potential adversaries. The Warren campaign should probably switch to a more mainstream TLD operated in the United States, though by using a Certificate Transparency monitor such as Cert Spotter, they could at least detect if an attack occurs.

The Attack

Before someone can visit the ewar.ren website, their browser has to look up the IP address for ewar.ren using the Domain Name System (DNS). Their browser sends a DNS query to the user's resolver, which is a service typically operated by the user's Internet service provider, but could be another resolver such as Google's 8.8.8.8 or Cloudflare's 1.1.1.1. The resolver performs a series of DNS lookups until it finds the name server which is authoritative for ewar.ren.

Since DNS is a hierarchical system, the resolver first queries the Internet's root name servers. The root name servers refer the resolver to the name servers for the .ren TLD. When the resolver asks the .ren name servers, .ren returns a referral to the ewar.ren name servers. When the resolver asks the ewar.ren name servers, they reply with the IP address, which the resolver returns to the user's web browser. Finally, the browser connects to that IP address to retrieve the web page.

Picture showing a DNS query for ewar.ren

Instead of referring the user's resolver to the ewar.ren name servers, the .ren name servers could return a referral to rogue name servers that return an IP address for a rogue web server. The rogue web server could serve up disinformation intended to discredit Senator Warren.

There are a couple ways the Chinese government could execute this attack. The heavy-handed approach would be to compel Beijing Qianxiang Wangjing Technology Development Co., the operator of the .ren TLD, to update their DNS zone file to replace the real name servers with the rogue ones.

A far more devious approach would use the Great Firewall of China. Since the name servers for .ren are located in China, DNS queries for .ren that originate outside China have to pass through the Great Firewall. The Great Firewall could intercept the DNS queries and reply with referrals to the rogue name servers. We know from the Great Cannon attack against GitHub in 2015 that the Great Firewall has the ability to intercept requests originating from outside China.

Picture showing a DNS query for ewar.ren that is intercepted by the Great Firewall of China

Using the Great Firewall wouldn't require the involvement of the .ren TLD owners, and the attackers could be quite surgical in who they target, by letting some DNS requests through to the real name servers while intercepting others. At the very minimum, they could base their decision on the IP address of the user's resolver. If the resolver sends the client subnet of the user, as Google's 8.8.8.8 resolver does, the attackers could make their decision based on a prefix of the user's own IP address, which reveals the user's ISP and possibly their company and rough geographic location. For example, the attackers might be able to conceal their attack from known DNS monitoring services, the FBI, and the U.S. Congress. If the attackers are judicious in who they target, the attack could go undetected for quite some time.

Detecting the Attack with Certificate Transparency

Fortunately, if the user's web browser uses HTTPS to connect to ewar.ren, there is hope for detecting the attack. That's because the attackers must perform one additional step: obtain a valid SSL certificate for ewar.ren from a publicly-trusted certificate authority. This is not difficult for the attackers - since they can manipulate the DNS for ewar.ren, they can successfully convince certificate authorities that they control ewar.ren and are therefore authorized to receive a certificate for it.

However, this part of the attack can be detected. That's because both Chrome and Safari require that certificates be logged to public Certificate Transparency logs. So, as long as Senator Warren's campaign is monitoring Certificate Transparency logs for certificates for ewar.ren, they'll be notified when the attackers launch their attack. (Note the attackers could try to target only browsers that don't require Certificate Transparency yet, but that is a minority of Web traffic and they would need to be very careful not to accidentally target Chrome or Safari, which would cause a certificate error and increase the likelihood of detection.)

Since there are almost 50 logs to monitor, and millions of certificates to sift through every day, the easiest way to monitor Certificate Transparency logs is to use a Certificate Transparency monitor such as Cert Spotter. It's as simple as entering in your domain name and getting notified by email whenever a certificate is issued for your domain. If you get a notification about a certificate you didn't request, you know that something is wrong. Cert Spotter also has an easy API that lets you search certificates by domain name.

Even if Senator Warren switches to a TLD in the United States, it would still be a good idea for her campaign to monitor Certificate Transparency logs. That's because Certificate Transparency can help you detect compromises of your own DNS infrastructure, such as the recent Sea Turtle attacks which used stolen credentials to hijack the DNS of numerous organizations. In fact, the Department of Homeland Security now requires all federal agencies to monitor Certificate Transparency logs. (Certificate Transparency is also useful for non-security reasons, like cataloging your certificates to track their expiration dates.)

One last note - for Certificate Transparency to reliably detect DNS tampering, you need to make sure that visitors only access your website over HTTPS. Requests made over unencrypted HTTP don't need a certificate, so the attackers could intercept those requests without being detected by Certificate Transparency. To ensure visitors only use HTTPS, always use links that start with https://, and get your domain preloaded in the HSTS preload list so browsers will automatically upgrade HTTP links for your domain to HTTPS. (Currently, ewar.ren is not preloaded, and several of the Senator's Tweets include HTTP links to ewar.ren. The Warren campaign should fix both problems in addition to monitoring Certificate Transparency.)

Sidebar: DNSSEC: Not Even Once

It's impossible to talk about DNS security without someone claiming that DNSSEC, a standard for securing DNS that is still barely deployed despite existing for over 20 years, would solve the problem. The only difference DNSSEC would make here is that the attackers would need to know the DNSSEC keys for .ren so they could sign their rogue responses with a valid DNSSEC signature. Presumably, the Chinese government could compel the .ren TLD owners to hand over their DNSSEC keys. If they did, no one would know. It's this lack of transparency that makes DNSSEC vastly inferior to the Web PKI for securing Internet communication.

All posts

Get Started with SSLMate Today

Click to sign up