SSLMate Blog

Chronicling the revolution of SSL certificate management.

September 7, 2016

Why Chrome 53 is Rejecting Chase Bank's Symantec Certificate

Update (Thursday, November 10, 2016): Due to a bug in Chrome, versions of Chrome 53 that are more than 10 weeks old now display this error message for all websites using Symantec certificates that were issued on or after June 1, 2016 (including from Symantec-owned brands like Thawte and GeoTrust). As of Friday, November 11, 2016, Chrome users can simply restart Chrome to avoid the bug. Chromium users must upgrade. Website operators can switch to a certificate from a different certificate authority. (SSLMate issues Comodo certificates, which are not affected.) The rest of this post still applies to new versions of Chrome.

Update (Saturday, November 26, 2016): Out-of-date versions of Android 5 and higher are affected by this bug when viewing websites from within apps. The solution is to upgrade Android System WebView from the Play Store.

If you're using Chrome 53, which was released last week, you might find that some websites which worked under Chrome 52 now fail with "Your connection is not private" with an error code of NET::ERR_CERTIFICATE_TRANSPARENCY_REQUIRED. For example, choosemyreward.chase.com shows the following error as of publication time:

Chrome error page saying that the connection to choosemyreward.chase.com is not private

The short explanation is that Chase's system administrators made a mistake when they requested their SSL certificate from their certificate authority, Symantec, but as we shall see, Symantec shares responsibility too.

The History of Certificate Transparency

The security of HTTPS relies on organizations called certificate authorities, who issue certificates that help ensure your connections to websites are secure and private. HTTPS is only secure if certificate authorities do their job properly. If a certificate authority messes up and issues an unauthorized certificate, an attacker can use it to intercept HTTPS connections.

Unfortunately, it's difficult to ensure that certificate authorities do their job properly, and certificate authorities have repeatedly violated the public's trust by issuing unauthorized certificates, including ones which have been used in real attacks against HTTPS connections.

In response, Google created Certificate Transparency. Under Certificate Transparency, all certificates are submitted to publicly-auditable logs by either the certificate authority or a third-party observer such as the Googlebot. Domain owners can monitor these logs using a service like Cert Spotter and take action if they see an unauthorized certificate for one of their domains.

Web browsers will eventually reject certificates that aren't logged using Certificate Transparency. However, Google is proceeding slowly towards mandatory logging so that they and others can gain operational experience first. The first milestone towards mandatory logging came in January 2015 when Chrome started requiring Certificate Transparency for Extended Validation certificates.

The second milestone came last October, when Google caught Symantec, a large certificate authority, issuing unauthorized "test" certificates for google.com and 75 other domains. Since issuing certificates for a domain without its owner's approval is such a serious violation of trust, Google announced that Chrome would require Certificate Transparency for all certificates issued by Symantec on or after June 1, 2016. This change rolled out last week in Chrome 53.

Symantec and Certificate Transparency

Symantec is, for the most part, complying with Google's logging requirement, and by default any certificate they issue will be properly logged and will work in Chrome 53. However, Certificate Transparency has a downside: it requires the complete contents of every certificate, including the hostnames, to be logged to a public log. For a public website, this is no big deal, but some organizations prefer to keep the hostnames of their internal servers private. Even the hostnames of public websites might need to be kept private until a certain date to avoid leaking information such as new product announcements or corporate acquisitions.

To address the privacy concerns, the IETF working group responsible for Certificate Transparency developed a redaction mechanism which would allow certificate authorities to redact components of the hostname beneath the registered domain. For example, a certificate for secretserver.secretdivision.example.com could be logged as ?.secretdivision.example.com, ?.?.example.com, but not ?.?.?.com. Redaction allows domain owners to keep their hostnames private, while still allowing them to detect that a certificate has been issued for some hostname under their domain.

Unfortunately for Symantec, there were some obstacles in the way of offering redaction to their privacy-sensitive customers.

First, redaction is only defined for the next version of Certificate Transparency, which is still a draft and has not been implemented by Chrome or any public log server.

Second, the Chrome team has raised several concerns with redaction, and stated that Chrome will not support redaction unless their concerns are addressed. Partly because of Chrome's concerns, the IETF working group removed redaction from the next version of Certificate Transparency and placed it in a separate document which has not yet been officially adopted by the working group.

Despite the fact that redaction, practically speaking, does not exist, Symantec forged ahead and grafted redaction onto the original version of Certificate Transparency. The result is a Franken-certificate that works fine in browsers that don't support Certificate Transparency, but fails to validate in Chrome.

Pointless Redaction

Symantec defaults to logging certificates in a compliant, unredacted form, but they provide their customers the option to log certificates in redacted form instead. Customers who choose this option get Franken-certificates that cause the above warning in Chrome 53.

Despite the incompatibility with Chrome and the utter pointlessness of redacting the certificates of public websites, both Chase Bank and United Airlines have chosen to redact such certificates. United fixed their websites before Chrome 53 became stable by replacing their certificates with fully-logged ones, but as of publication time, choosemyreward.chase.com is still serving a Franken-certificate that's rejected by Chrome 53.

Data collected from Certificate Transparency logs reveal quite a few other websites that are probably public yet use redaction, including websites at Amazon, Fedex, Goldman Sachs, Mitsubishi, and Siemens.

Why would someone choose redacted logging for a public website? Symantec's documentation might be to blame. Their documentation describes the two options as follows:

  • Full domain names: Publicly logs root domain names and subdomains in the certificate. Recommended for all public websites.
  • Only root domain names: Publicly logs only root domain names in the certificate. Intended only for private internal domains.

Although they say that logging root domain names is "intended only for private internal domains" while recommending full domain name logging for public websites, they don't mention the downside until later in the document:

All certificates with root domain logging may display browser warnings when users connect to the website.

Saying that a warning "may" be displayed doesn't seem adequate when a warning absolutely will be displayed, by the world's most popular web browser to boot!

Symantec needs to do a better job informing their customers of the downsides of choosing redaction. Too many websites have chosen redaction incorrectly, and I expect this to continue unless Symantec improves their messaging. Meanwhile, Chrome users will encounter avoidable browser errors when visiting these websites, which is a horrible experience for Symantec's customer's customers, and risks desensitizing people to security warnings.

Finally, if you represent an organization that wants to use redaction appropriately (that is, to hide the hostnames of a non-public server), please send an email to the IETF working group mailing list. The working group has had a difficult time designing redaction, and addressing Chrome's concerns will require hearing from the people who want to use redaction. The fate of redaction depends on your input!

If you're worried about certificate authorities like Symantec issuing unauthorized "test" certificates for your domains, you should check out Cert Spotter, a tool to monitor Certificate Transparency logs for unauthorized certificates. Available as open source or a hosted service.

September 2, 2016

What Will Happen to WoSign? A Look at Past Certificate Authority Sanctions

Certificate authority (in)security is in the news again, with the recent revelation of several security vulnerabilities and other serious oversights at the WoSign certificate authority. The Mozilla community is currently debating what sanctions should be taken against WoSign. While we wait for Mozilla and other browser vendors to make their decisions, I thought it would be interesting to explore what sanctions other certificate authorities have faced in the past.

Note that this post will focus on sanctions imposed by Mozilla and Google, since they discuss certificate authorities more openly than other browser vendors.

2011: Comodo avoids consequences for reseller compromise

Although security researchers had discovered flaws in certificate authorities prior to 2011, the first serious compromise occurred in 2011 when a Comodo reseller was breached and used to issue unauthorized, publicly-trusted certificates for high profile domains such as google.com, mozilla.org, and skype.com. Although Comodo themselves were not breached, at the time Comodo trusted their resellers to perform domain control validation, meaning a compromised reseller could cause the issuance of publicly-trusted certificates for arbitrary domains.

Since Comodo weren't breached, they were able to detect and contain the compromise quickly, which likely helped them avoid sanctions. They remain widely trusted to this day, but no longer allow resellers to perform domain control validation.

2011: Death penalty for DigiNotar

Not long after the Comodo reseller compromise, a much more serious compromise occurred that remains the most serious certificate authority incident to date, and which resulted in the most severe sanctions possible. DigiNotar, a Dutch certificate authority, was totally compromised. Over 500 unauthorized certificates were detected and a rogue wildcard certificate for google.com used for mass interception of Internet traffic in Iran. Since DigiNotar was so thoroughly compromised, the full extent of the breach was never known.

Due to the seriousness of the breach, all major browsers (Mozilla, Google, Microsoft, Apple) revoked trust in DigiNotar. A few months later, DigiNotar declared bankruptcy and shut down.

2012: Amnesty for MITM certificates

In 2012, Trustwave disclosed that they had willfully issued an intermediate CA certificate to a company for the express purpose of signing unauthorized, publicly-trusted certificates to intercept and inspect encrypted traffic.

Although the certificates issued by this intermediate CA were clearly not authorized by domain owners, there was some concern that this practice was neither clearly prohibited nor uncommon, and that it would be unfair to punish the first CA to admit to it. After a public discussion, Mozilla opted for an amnesty policy instead of sanctioning Trustwave. Certificate authorities were explicitly informed that this practice was forbidden and given two months to comply or face sanctions.

2013: No green bar for TURKTRUST

In 2013, Google learned that TURKTRUST had "mistakenly" issued two intermediate CA certificates to subscribers instead of regular certificates, and that one of them had been used to sign a rogue certificate for google.com. TURKTRUST provided a plausible explanation for their mistake and there was no indication that they had been compromised. Google decided to revoke TURKTRUST's ability to issue extended validation certificates, and Mozilla temporarily suspended the inclusion of a new TURKTRUST root certificate. No further action was taken, making this a fairly mild sanction.

2013 - 2014: Regional death penalty for ANSSI and India CCA

In 2013, ANSSI, a French certificate authority, was caught doing what was expressly prohibited after the Trustwave revelation: they issued an intermediate CA that was used to intercept and inspect encrypted traffic. Google and Mozilla responded by "name constraining" ANSSI so that it could only issue certificates for domain names ending in .fr or one of the other top-level domains used by French territories. Although this sanction fell short of a full death penalty, it killed off ANSSI for most of the world.

In 2014, an intermediate certificate authority signed by India CCA was compromised and used to issue rogue certificates for Google and Yahoo. After investigating the compromise, India CCA found only four rogue certificates. However, Google was aware of additional rogue certificates that India CCA did not detect and concluded that India CCA was unaware of the full scope of the breach. Consequentially, they decided to name constrain India CCA to a small set of Indian domains.

2015: Suspension of CNNIC

In 2015, Google learned that CNNIC, like ANSSI and Trustwave, had issued an intermediate certificate authority that was used to intercept encrypted traffic. CNNIC did this in violation of their own Certificate Practice Statement, which said that they would not issue intermediate certificate authorities to external organizations. In response, both Google and Mozilla revoked trust in new CNNIC certificates.

Since existing certificates remained trusted, this was a less severe sanction than DigiNotar faced. To implement this sanction, Chrome and Firefox embedded a whitelist containing the hash of every existing CNNIC certificate. Since there were only a couple thousand CNNIC certificates, this was practical.

Google welcomed CNNIC to reapply for inclusion, but stipulated that all CNNIC certificates would have to be logged to Certificate Transparency logs. Unlike DigiNotar, CNNIC remains operational.

2015: Certificate Transparency required for Symantec

In 2015, Google discovered that Symantec had issued over 100 publicly-trusted "test" certificates for 76 different domains, including google.com, without the authorization of domain owners. In response, Google announced that all Symantec certificates issued on or after June 1, 2016 would have to be logged to Certificate Transparency logs. Chrome 53, released this week, enforces this requirement and rejects any unlogged Symantec certificate with an issuance (a.k.a. notBefore) date on or after June 1, 2016.

2016: What will WoSign face?

As we can see from the above, every certificate authority incident was different and therefore merited a different response. Would any of the above sanctions work with WoSign?

  • Death penalty, like DigiNotar: Although WoSign has not been compromised like DigiNotar was, they have displayed a shocking amount of sloppiness for an organization entrusted to sign certificates for any domain on the Internet, which arguably warrants distrust. However, there are over 100,000 unexpired WoSign certificates in existence. Distrusting WoSign would cause any site using one of these certificates to break, which risks desensitizing people to security warnings. Does the security benefit from distrusting WoSign outweigh this risk?

  • Revoking extended validation status, like TURKTRUST: Is this a strong enough sanction for WoSign's failures?

  • Name constraining, like ANNSI: Since WoSign has issued certificates worldwide, there is no set of name constraints that could be meaningfully applied to WoSign.

  • Suspension, like CNNIC: Could new WoSign certificates be distrusted, but existing certificates grandfathered? Unfortunately, since there are over 100,000 unexpired WoSign certificates, it would be impractical to include a whitelist in browsers as was done with CNNIC. And since WoSign has been caught backdating certificates, it would be imprudent to rely on the notBefore date to decide if an existing certificate should be grandfathered.

  • Require Certificate Transparency, like Symantec: WoSign has already promised to log all certificates issued after July 5, 2016 to Certificate Transparency logs and has asked browsers to block certificates issued after this date that are not logged. But how can a browser really know if a certificate was issued after July 5, 2016 or not? WoSign can't be trusted to set the notBefore date faithfully, and there are too many existing certificates to embed a whitelist in the browser.

If you're worried about certificate authorities like WoSign issuing certificates for your domains, you should check out Cert Spotter, a service to monitor Certificate Transparency logs for unauthorized certificates. Available as open source or a hosted service.

July 28, 2016

Introducing Cert Spotter: Easy Certificate Transparency Monitoring from SSLMate

SSLMate is excited to announce the launch of Cert Spotter, an easy-to-use Certificate Transparency log monitor that helps you detect certificates for your domains. Cert Spotter helps you track down undocumented certificates so you can avoid outages from unexpected expirations, and improves your security by alerting you to unauthorized certificates.

Click here to sign up and start monitoring up to 5 domains for free, or get the open source version and set it up on your own server.

The Cert Spotter Story

The Web's public key infrastructure has a trust problem. Its security is based on the assumption that certificate authorities are trustworthy and competent and will only issue a certificate for a domain with the proper authorization of the domain owner. Unfortunately, there are a lot of certificate authorities trusted by your browser, and there's a history of certificate authorities making mistakes. Even if you choose a good certificate authority for your website, it doesn't stop an attacker from exploiting a bad certificate authority and obtaining fraudulent certificates to attack your website.

Certificate Transparency provides a compelling solution. With Certificate Transparency, certificates are submitted to public, append-only logs which domain owners can monitor. If a domain owner sees an unauthorized certificate for one of their domains, they can tell the offending certificate authority to revoke the certificate, and notify the Internet community so that corrective action can be taken against the certificate authority. Although Certificate Transparency doesn't directly stop unauthorized certificates, it reduces their likelihood. Since unauthorized certificates can be detected, certificate authority vulnerabilities can be fixed before being exploited again. Certificate Transparency encourages certificate authorities to improve their security, and discourages attackers who would prefer to keep their attacks secret.

Certificate Transparency only works if domain owners are monitoring logs. Cert Spotter makes this easy. Sign up for an SSLMate account, add the domains you want to monitor, and SSLMate will send you an email when a certificate is detected for one of your domains. You can review all of your discovered certificates from a simple dashboard. To avoid distracting you with false alarms, we don't bother you about certificates which we know you acquired through SSLMate.

Cert Spotter is free for up to 5 domains (get in touch if you'd like to monitor more), and we're planning to introduce premium features such as integration with monitoring systems in the near future. Click here to sign up and start monitoring.

We think the security of the Web PKI will only improve if every domain owner is monitoring Certificate Transparency logs. That's why we're open sourcing the core of our log monitor. Unlike existing open source log monitors, Cert Spotter does not require setting up a database. You just list the domains you want to monitor in a file and set up a cron job. Check it out on GitHub to learn more.

Certificate Transparency is in its infancy, but is already showing great promise. There are over 13 million unique unexpired certificates in Certificate Transparency logs. Chrome requires Certificate Transparency for Extended Validation certificates and all certificates issued by Symantec. Let's Encrypt, StartCom, and WoSign voluntarily log all certificates that they issue. The Google crawler logs every certificate it encounters, so any certificate on a public website is likely to be logged.

We expect more certificate authorities to begin voluntarily logging certificates in the coming years, and others to be opted-in to mandatory certificate transparency in response to security failures. Ultimately, all certificates will be logged in Certificate Transparency and web browsers will only accept a certificate if it's accompanied by a verifiable promise that it's logged.

If you have a website, now is the time to get started with Certificate Transparency. Sign up now to start monitoring up to 5 domains for free. If you're already an SSLMate customer, visit your Cert Spotter page to set up your watchlist. If you want to learn more, read about how Cert Spotter helps you, and about how Cert Spotter works.

July 2, 2015

SSLMate 1.4.0: Multi-Domain Certificates

SSLMate 1.4.0 was released today, with support for buying, importing, and managing multi-domain certificates. Multi-domain certificates allow a single certificate to work with up to 100 different hostnames by specifying the hostnames in the certificate's Subject Alternative Name field.

Multi-domain certificates differ from wildcard certificates in that wildcard certificates secure any name directly below a particular domain, whereas multi-domain certificates protect specific single names that may or may not be part of a related domain. Multi-domain certificates are useful if you have several different hostnames from different domains that are hosted on the same server and you desire the convenience of using a single certificate, or you need to support legacy clients that don't support SNI (SNI is a mechanism that allows multiple certificates to co-exist on the same IP address).

One nice feature of multi-domain certificates is that they're very flexible: you can modify the list of hostnames protected by a multi-domain certificate at any time after the certificate is purchased. Replacing an existing name is free, and when you add a new name you pay only for the new name, not for a whole new certificate. Adding or removing a name requires the installation of a new certificate file, which can be completely automated with sslmate download.

Getting a multi-domain certificate is easy: just specify more than one hostname on the command line to sslmate buy. For more details, such as how to add and remove names, check out the documentation.

As with all certificates offered by SSLMate, multi-domain certificates feature the full range of automation and approval options, including DNS-based approval, and if you're using the SSLMate API, HTTP-based approval. The flexibility and automation offered by SSLMate's multi-domain certificates makes them ideal for SaaS services where a dedicated hostname under the customer's domain is provisioned for every customer.

If you've installed SSLMate through APT or Yum, upgrading to 1.4.0 is as simple as running apt-get update && apt-get upgrade or yum update. If you're using Homebrew, an updated formula should be available in the near future. Otherwise, head over to our install page or our GitHub repository to download and install the new version.

June 18, 2015

SSLMate 1.3.0: Certificate Formats for Everyone

SSLMate 1.3.0 was released today, featuring automatic generation of a variety of certificate formats, including PKCS#12 (aka PFX) and Java Keystore.

Traditionally, SSLMate created key and certificate files only in PEM format (specifically, the PEM encoding of the ASN.1 DER serialization of the private key or the X.509 certificate). This is the default file format of the OpenSSL command line tools and is accepted by the most common server applications on Unix platforms. This left other users, particularly of Java applications such as Tomcat, needing to perform manual conversions of certificate files using obscure commands - exactly the problem that SSLMate tries to solve.

So, SSLMate can now create the following types of files:

  • combined – a concatenation of the private key, certificate, and intermediate certificate chain, in PEM encoding. This is intended for programs like haproxy that require you specify the private key and certificate in the same file.

  • p12 – a PKCS#12 (also known as PFX) file containing the private key, certificate, and intermediate certificate chain. The password for the file is sslmate.

  • jks – a Java Keystore file containing the private key, certificate, and intermediate certificate chain. The password for the file is sslmate. (Note: the keytool program from the Java Runtime Environment must be installed to use this format.)

  • root – the root certificate, in PEM encoding.

  • chain+root – the intermediate certificate chain, including the root certificate, in PEM encoding. This is required by nginx for OCSP stapling.

To enable a particular format, put the following in your SSLMate config file, replacing name with one of the bolded names from the above list (e.g. combined):

cert_format.name yes

Once a format is enabled, a file of that format will be created by any call to sslmate download, sslmate buy, sslmate renew, or sslmate reissue. So you can configure your application to directly use the files created by SSLMate, and when a renewed certificate is downloaded, your application can automatically use the new certificate.

Now that SSLMate supports these formats, expect to see additional applications supported by sslmate mkconfig in the near future.

File Permissions

SSLMate now preserves the filesystem permissions of key and certificate files when updating them. By default, files containing the private key have restrictive permissions (0600) and other files have world-readable permissions (0644). This is usually sufficient, since most server applications read the private key as root before dropping to a less privileged user. However, some applications read the private key after dropping to an unprivileged user, making the default 0600 permissions too restrictive. You can now set whatever filesystem permissions you like on your key files: user ownership, group ownership, traditional permissions, and (on Linux only) even filesystem ACLs, and SSLMate won't revert them back to their restrictive defaults.

Of course, you should make sure the permissions of private keys are no more permissive than necessary. Linux filesystem ACLs are useful for this. For example, if your application runs as user appuser, you can set a read-only ACL for that user by running setfacl -m user:appuser:r /etc/sslmate/example.com.key.

Getting SSLMate 1.3.0

If you've installed SSLMate through APT or Yum, upgrading to 1.3.0 is as simple as running apt-get update && apt-get upgrade or yum update. If you're using Homebrew, an updated formula should be available in the near future. Otherwise, head over to our install page or our GitHub repository to download and install the new version.

All posts

Get Started with SSLMate Today

Buy a new certificate, or import your existing certs for free.

Click to sign up