Fixing the Breakage from the AddTrust External CA Root Expiration

Blog

May 30, 2020

Fixing the Breakage from the AddTrust External CA Root Expiration

A lot of stuff on the Internet is currently broken on account of a Sectigo root certificate expiring at 10:48:38 UTC today. Generally speaking, this is affecting older, non-browser clients (notably OpenSSL 1.0.x) which talk to TLS servers which serve a Sectigo certificate chain ending in the expired certificate. See also this Twitter thread by Ryan Sleevi.

This post is going to explain what you should do to avoid problems, from the perspectives of both server operators (tldr: test your server with What's My Chain Cert? and do what it says) and client operators (tldr: upgrade your TLS libraries if possible, otherwise remove AddTrust External CA Root from your trust store).

Quick primer on certificate chains

When you connect to a TLS server, the server sends the client a certificate that proves its identity. The client needs to build a chain of certificates from the server certificate to a root certificate that the client trusts. To help the client build this chain, the server sends back one or more intermediate certificates after its own certificate.

For example, my website sends the following two certificates:

Subject	Issuer	Expiration
www.agwa.name	Sectigo RSA Domain Validation Secure Server CA	2021-04-03
Sectigo RSA Domain Validation Secure Server CA	USERTrust RSA Certification Authority	2030-12-31

The first certificate is mine and is issued by Sectigo RSA Domain Validation Secure Server CA. The second certificate is Sectigo RSA Domain Validation Secure Server CA and is issued by USERTrust RSA Certification Authority, which is a root certificate. These two certificates form a complete chain to a trusted root.

However, USERTrust RSA Certification Authority is a relatively new root. It was created in 2010, and it took many years for it to become trusted by all clients. As recently as last year I heard reports of clients not trusting this root.

For this reason, some servers send back a chain with an additional intermediate certificate:

Subject	Issuer	Expiration
www.agwa.name	Sectigo RSA Domain Validation Secure Server CA	2021-04-03
Sectigo RSA Domain Validation Secure Server CA	USERTrust RSA Certification Authority	2030-12-31
USERTrust RSA Certification Authority	AddTrust External CA Root	2020-05-30

This sequence of certificates form a chain to another root called AddTrust External CA Root which was created in 2000 and is trusted by many client platforms. Or rather, it was trusted before it expired today.

Fortunately, modern clients with well-written certificate validators (this includes all mainstream web browsers) won't have a problem with the expiration. Since they trust the USERTrust RSA Certification Authority root, they will build a chain to that root and ignore the fact that the server sent an expired intermediate certificate.

Other clients, notably anything using OpenSSL 1.0.x or GnuTLS, will have a problem. Even if these clients trust the USERTrust RSA Certification Authority root, and could build a chain to it if they wanted, they'll end up building a chain to AddTrust External CA Root instead, causing the certificate validation to fail with an expired certificate error.

Fixing this problem as a server operator

Basically, you need to remove the intermediate certificate issued by AddTrust External CA Root from your certificate chain.

If you get your certificates from SSLMate, you don't need to worry. I saw this coming over a year ago, and configured SSLMate to start providing a chain without AddTrust External CA Root. As certificates renewed, SSLMate customers received the new chain, and since SSLMate has long capped certificate lifetimes at one year, the older chain was cycled out before the intermediate expired.

But if your server is using Sectigo certificates from another source, you might need to worry. You can quickly test if your server is affected using What's My Chain Cert?. If your server is OK, it will say "correct chain". If it's sending the expired intermediate, it will say "trusted chain containing an expired certificate" and provide you with a link to download a correct, non-expired chain.

Fixing this problem as a client operator

In a perfect world, all of your libraries would be up-to-date and you wouldn't be using clownish TLS implementations like GnuTLS. But the world isn't perfect. OpenSSL 1.0.x is still common, and curl used it as recently as Debian Stretch. And APT, the package manager used by Debian and Ubuntu, links with GnuTLS.

Fortunately, OpenSSL 1.0.x and GnuTLS (at least on Debian) only choke on the expired intermediate if the AddTrust External CA Root root is in the local trust store. If it isn't, they will build a chain to USERTrust RSA Certification Authority instead. On Debian (and probably Ubuntu but I haven't tested), you can easily remove this root from the trust store as follows:

Edit /etc/ca-certificates.conf and put a bang/exclamation mark (!) before mozilla/AddTrust_External_Root.crt
Run update-ca-certificates

For Fedora and RHEL, see this Tweet by Christian Heimes.

Older (View Archive)

Short Take: Why Trust-On-First-Use Doesn't Work (Even for SSH)

Newer

Security Review of CFSSL Signer Code

Comments

Anonymous on 2020-05-30 at 20:04:

I am not certain that the following statement is always true:

Fortunately, OpenSSL 1.0.x and GnuTLS only choke on the expired intermediate if the AddTrust External CA Root root is in the local trust store.

On my RHEL and Fedora systems, removing AddTrust External CA Root from my trust stores by following Christian Heimes' twitter link in your post, indeed resolved the openssl s_client -connect test which no longer shows an error but wget which is compiled with GnuTLS still returns the "is not trusted" error, when testing connection to servers that did not remove the expired intermediate certificate issued by AddTrust External CA Root from their certificate chain.

Am I missing something?

Reader Ryan H on 2020-05-30 at 20:23:

You need to remove the USERTrust cert from your cert bundle served up by your web server on the server side to fix this Issue with wget

Andrew Ayer on 2020-05-30 at 20:49:

Yes, deleting the expired cert on the server (which to be clear is issued by AddTrust External CA Root not USERTrust) will fix all clients, but there should be a client-side fix as well.

Reader Hasan Qadir on 2020-06-02 at 15:12:

How can we do it? Can you please tell me the command for ubuntu 16.04?

Andrew Ayer on 2020-05-30 at 20:47:

Thanks for reporting this! I can't speak for RHEL/Fedora, but on Debian, removing AddTrust External CA Root does indeed fix wget (which is linked with GnuTLS). I've added a caveat to the blog post about this.

Reader Christian Heimes on 2020-05-30 at 20:50:

AFAIK GnuTLS uses PKCS#11 to fetch the trust anchors directly from libnssckbi.so. The shared library contains the hard-coded trust anchors from Mozilla's NSS.

My workaround does not solve the problem for GnuTLS, see https://bugzilla.redhat.com/show_bug.cgi?id=1842174

Reader Christian Heimes on 2020-05-30 at 20:55:

It's actually wget -> p11-kit -> libnssckbi.so.

Andrew Ayer on 2020-05-30 at 20:57:

Thanks for commenting Christian! (And for filing these bugs!)

Reader F30 on 2020-05-30 at 23:09:

Are you aware that there is at least one other current Root CA which used to be cross-signed by "AddTrust External CA Root" - "COMODO RSA Certification Authority"? (https://crt.sh/?id=1720081, old cross-signed cert is https://crt.sh/?id=1044348)

Additionally there is another variant of the "USERTrust RSA Certification Authority" cert (https://crt.sh/?id=1282303295) issued by the "AAA Certificate Services" Root CA (https://crt.sh/?id=331986). That one should still be valid until 2028.

However, cURL on macOS 10.14 (built against LibreSSL 2.6.5) also fails to connect to servers sending a security chain containing the latter. This is interesting, since "AddTrust External CA Root" never occurs in that chain. One example for it is https://electroncash.org/.

Andrew Ayer on 2020-05-30 at 23:24:

Are you aware that there is at least one other current Root CA which used to be cross-signed by "AddTrust External CA Root" - "COMODO RSA Certification Authority"? (https://crt.sh/?id=1720081 , old cross-signed cert is https://crt.sh/?id=1044348 )

Yes. https://whatsmychaincert.com will catch this case.

Additionally there is another variant of the "USERTrust RSA Certification Authority" cert (https://crt.sh/?id=1282303295 ) issued by the "AAA Certificate Services" Root CA (https://crt.sh/?id=331986 ). That one should still be valid until 2028.

Including this certificate in your chain is a good option if you need to support really old clients that don't have "USERTrust RSA Certification Authority" in their trust stores. However, there's a caveat: since "AAA Certificate Services" isn't enabled for EV by Apple, Mozilla, or Chrome, your expensive EV certificate might not get EV treatment. That said, EV certificates are silly ;-)

However, cURL on macOS 10.14 (built against LibreSSL 2.6.5) also fails to connect to servers sending a security chain containing the latter. This is interesting, since "AddTrust External CA Root" never occurs in that chain. One example for it is https://electroncash.org/

The chain that I see ends in "AddTrust External CA Root" not "AAA Certificate Services".

Reader F30 on 2020-05-30 at 23:56:

The chain that I see ends in "AddTrust External CA Root" not "AAA Certificate Services".

My bad. This was caused by LibreSSL 2.6.5 not doing SNI by default and me looking at the wrong certificate chain.

Reader Lovely Sharma on 2020-06-01 at 04:30:

When I am using "openssl s_client -connect domain_name" it shows expired certificate in chain i.e AddTrust External. I have commented this cert in ca-bundle.crt and my curl command is running for https://url after commenting the cert in ca-bundle.crt but as I said s_client shows this expired certficate in chain, do we need a fix for that, if yes? then what possible could be the fix. I am using AWS Linux instance.

Anonymous on 2020-05-30 at 23:26:

Thanks - This helped me! :)

Andrew Ayer on 2020-05-30 at 23:27:

You're welcome! I'm glad the post helped you!

Reader Renan on 2020-05-31 at 00:19:

Hi Andrew,

Thanks a lot for the information, me and my company we are facing this Add trust expiracy problem in some in some centOS machines. When we try to curl the machines we get the ssl certificate expiracy error. Do you know what are the equivalent steps to fix in a client running centOS ?

Great post & thanks anyways, Renan

Andrew Ayer on 2020-05-31 at 00:34:

Hi Renan - I assume that the RHEL instructions provided by Christian Heimes will work on CentOS also.

Reader Renan on 2020-05-31 at 01:01:

I tryed his approach, but my OS doenst have this 'trust' package, and I didn't find a way to install it, so I kind tryied for other approaches. I'm a very begginer centOS user.

Any idea how to install this package?

Reader Christian Heimes on 2020-05-31 at 08:37:

The trust command is part of the p11-kit-trust package. You can also generate the file on another CentOS, RHEL, or Fedora system, copy it to /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit on your CentOS machines, and run "update-ca-trust extract" on each machine.

I've attached the file to https://bugzilla.redhat.com/show_bug.cgi?id=1842174, too.

Reader Quinn on 2020-05-31 at 23:56:

Hi Cristian,

Thanks for sharing the addtrust-external-root.p11-kit file. I was unable to use it directly on CentOS 6 and 7 machines; both get this error when running sudo update-ca-trust extract:

  p11-kit: addtrust-external-root.p11-kit: nss-server-distrust-after: invalid or unsupported attribute
  p11-kit: addtrust-external-root.p11-kit: nss-server-distrust-after: invalid or unsupported attribute
  p11-kit: addtrust-external-root.p11-kit: nss-server-distrust-after: invalid or unsupported attribute
  …

After commenting-out the lines starting with nss-* I was able to run sudo update-ca-trust extract successfully, and I'm able to connect to the test site.

I'm testing this issue with this command:

  php -r 'echo file_get_contents("https://addtrust-chain.demo.sslmate.com/");'

Thanks again. :)

Reader heyzea1 on 2020-05-31 at 13:38:

Great tnx! Worked like a charm

Reader Sara on 2020-05-31 at 17:07:

I am trying to access my account in car.org to generate contracts and I get the error message. All of this is a new language for me....how do I fix it for a basic non-techy person or do I need to hire someone?

Andrew Ayer on 2020-05-31 at 21:06:

Sara - I'm sorry to hear about your situation. Problems like these are very frustrating. Unfortunately, the problem can't be fixed on your end. The problem lies on car.org's end, or with one of their providers. If you can get in touch with the operators of car.org, this blog post should be able to help them fix the problem.

Reader Brendan Mannix on 2020-05-31 at 23:44:

Thank you Andrew and Christian. You really helped me and my team out this weekend.

Reader Jack on 2020-06-01 at 00:45:

Hello!

We're running into an issue on our RHEL systems relating to LDAPS. Our AD admin swears he removed the cert but I think it may still be lingering somewhere. After implementing the suggested workaround then running openssl s_client -showcerts -CApath /etc/ssl/certs -verify 10 -connect <ad.server.com> it appears to throw the error "error:num=2 unable to get issuer certificate". We've blocked USERtrust and addtrust external. Each trust we block, the error seems to move to the next cert in the chain.

Are there any suggestions for how to work around this issue?

Thank you!

Andrew Ayer on 2020-06-01 at 12:23:

Jack - you definitely should not block USERTrust. That is an unexpired root and if you block it nothing issued by Sectigo will work anymore.

Reader Jack on 2020-06-01 at 13:44:

Thank you!

Blocking AddTrust External only, leaving USERTrust, still results in "Verify return code: 2 (unable to get issuer certificate)"

I've updated the rpm certificate package for the system as well as manually added the new SHA2 cert from Sectigo to /etc/pki/ca-trust/anchors/ and still no dice. RH support is being a bit....not responsive...

I don't want to hijack the comment section unless others are having this issue, I'm happy to take this to a support thread if anyone can recommend one.

Reader Guy on 2020-06-01 at 14:46:

Having the same issue as you Jack, I do not have the AddTrust in my server's chain and I have blacklisted the AddTrust root, and OpenSSL 1.0.2k is throwing the "unable to get issuer certificate" error

Andrew Ayer on 2020-06-01 at 16:28:

It's possible that the clients are so ancient that they don't trust the USERTrust RSA Certification Authority root. If this is the problem, you can try appending this certificate to the end of your server's chain: https://api.certspotter.com/v1/certs/68B9C761219A5B1F0131784474665DB61BBDB109E00F05CA9F74244EE5F5F52B.pem

This will create a chain to the older AAA Certificate Services root, which expires in 2028.

Reader Jack on 2020-06-01 at 17:44:

Comment #22 on the bug report resolved the issue for us. The conclusion we've come to is the blacklist does not remove the offending cert from the ca-bundle.crt file, that has to be done manually. If we blacklist AddTrust [editor's note: removed incorrect information], then modify /etc/pki/tls/certs/ca-bundle.crt (you'll need to chmod +w to mod it) and remove the AddTrust [editor's note: removed incorrect information] certs.

You will need to have the blacklist in place otherwise if you run update-ca-trust the offending certs will be added back into the bundle.

You'd think having the certs in blacklist would remove them from the cert bundle...... 0_0

Reader Dallin on 2020-06-01 at 11:16:

Thanks for posting this. I've not been able to get this to work with curl in PHP, though. Can anyone provide any pointers on working around this in that environment?

Reader Nidheesh on 2020-06-01 at 11:17:

Thanks this helped to resolve my issue :)

Reader LoveDealer on 2020-06-01 at 19:38:

Hi Andrew

Thanks for your blog post. We are still experiencing the issue after applying your change. we are on Debian 8, we have commented out the line mozilla/AddTrust_External_Root.crt and run the command update-ca-certificates

What else could be wrong ?

Andrew Ayer on 2020-06-01 at 19:47:

First, make sure that you put a ! in front of the mozilla/AddTrust_External_Root.crt line - not a comment character (#).

Second, make sure your ca-certificates package is up-to-date, by running:

apt-get update && apt-get install ca-certificates

Reader Stefan Richter on 2020-06-01 at 20:11:

Many thanks for this. I'm technically slightly out of my depth here but could you tell me why one of my customers running the latest version of Chrome and Safari on Mac OS X had issues with an affected site while I could load the same resources fine, also on latest Chrome and Mac OS X?

Andrew Ayer on 2020-06-01 at 20:19:

What version of macOS? If it's 10.11 or earlier, then it doesn't trust the new USERTrust RSA Certification Authority root. Apart from having your customer upgrade, there are two options:

1. Have your customer use Firefox, which uses its own trust store which includes USERTrust RSA Certification Authority.

2. Configure your server to send an additional intermediate certificate which is signed by the AAA Certificate Services root. Add the following certificate to the end of your existing certificate chain: https://api.certspotter.com/v1/certs/68B9C761219A5B1F0131784474665DB61BBDB109E00F05CA9F74244EE5F5F52B.pem

Reader Nicholas Yap on 2020-06-02 at 01:48:

Thank you! This saved all my servers yesterday. Worked like a charm for my wildcard cert. Woot!

Reader Jan Rise on 2020-06-02 at 06:43:

Thank you! Used the workaround for a CentOS (the one from Christians Tweet) and Ubuntu machine while waiting for the servers to be updated. Saved the day for a couple of web services.

Anonymous on 2020-06-02 at 12:33:

This was the fix for CentOS 6.10

Manually remove expired AddTrust cert from ca-bundle.legacy.crt
Add expired AddTrust cert to blacklist dir (Cert ends with hgQ=)
Goto Comodo website -> Knowledgebase : > Root & Intermediate(s)
and download this SHA-2 Cert 
  [Intermediate #1 (SHA-2)] COMODO RSA Certification Authority
Put in anchors directory (Cert ends with 8fxV)

run "update-ca-trust extract"

Reader Fkscorpion on 2020-06-03 at 09:37:

Thank you very much for this, but I think you've confused two certs:

The cert ending with 8fxV is the

"[Intermediate CA #2] COMODO RSA Organization Validation Secure Server CA (SHA-2)"

Whereas "[Intermediate #1 (SHA-2)] COMODO RSA Certification Authority" ends with OG8=

Which one did you mean? :)

Reader Tristan on 2020-06-03 at 11:54:

Thank you to original Anonymous commenter and to you for pointing out that problem with the original instructions!

I've just tried both, and using the certificate that ends OG8= causes the openssl test to fail for me. Using the one ending 8fxV instead restores openssl test to working for me.

However, the problem I'm having on Centos 6 is that the openssl tests appear to work, but connections from PHP and PHP-FPM still do not.

I'm a bit lost as to what to try next so this post and thread has been fantastic so far - hoping someone out there can point me towards getting PHP to work too?

Anonymous on 2020-06-03 at 17:04:

My understanding is that PHP uses certs from this directory: /etc/ssl/certs/ On my CentOS 6.10 this directory still has certs ending in hgQ=. I would remove the certs in this directory and add the updated one ending in 8fxV. Check php.ini and see if it is referencing a cert file.

Anonymous on 2020-06-03 at 16:03:

Correct use - [Intermediate CA #2] COMODO RSA Organization Validation Secure Server CA (SHA-2) named comodorsaorganizationvalidationsecureserverca.crt (ends with 8xfV)

Reader Hui Chin on 2020-06-03 at 03:22:

Thank you! this worked like charm!!!!!

Reader Carlos on 2020-06-03 at 17:30:

Hello, I'm working with Amazon EC2 Linux and I'm trying to update just the "AddTrust External Root" in the ssl certificate so I can use CURL, anyone know how can I update it from mi EC2 with a command? thank you

Anonymous on 2020-06-04 at 00:03:

I got my MacBook Air in 2014 and never updated it until now, my mistake I know. I am not able to access any websites, such as university websites that use the ADDtrust CA root certificate. I already deleted the expired, and all other expired certificates from my keychain. I am not sure how to get the new and updated certificate onto my computer. Please let me know if you can help me with this.

Andrew Ayer on 2020-06-04 at 13:44:

Use Firefox, which has a newer set of trusted roots: https://www.mozilla.org/en-US/firefox/download/thanks/

Anonymous on 2020-06-04 at 17:47:

There is a command to update the certificate in Linux? Or anyone know how can I update automatically?

Andrew Ayer on 2020-06-04 at 18:34:

The blog post describes how to fix Linux systems. Could you clarify what you're asking about?

Anonymous on 2020-06-04 at 20:17:

Sure, I know that by deleting the AddTrustExternalRoot in the ca-bundle.crt file the issue can be solved, but I would like to found a way to delete that expired chain by using the linux terminal, with a kind of command by automate the deleting of that chain

Andrew Ayer on 2020-06-04 at 20:54:

It depends on the distro. On Debian/Ubuntu, you would use these two commands:

sed -i -e 's|^mozilla/AddTrust_External_Root.crt$|!&|' /etc/ca-certificates.conf
update-ca-certificates

Anonymous on 2020-06-05 at 14:42:

Thank you very much!! :)

Reader Nicolas on 2020-06-05 at 07:35:

Thanks for your article it helped us a lot !!!!

Reader Garrett Michael Hayes on 2020-06-05 at 18:58:

Is there a way to run the same sort of test as "WhatsMyChainCert.com" against an internal server - i.e. one not exposed to the internet?

I'm helping a user troubleshoot an issue, but don't manage the server in question. I'd like to assist the user in checking if this is the cause of their observed problem.

Reader Michael on 2020-06-07 at 08:06:

we're using AWS Linux and apache 2.2.1 version. From browser no issue, but we got error on curl calls. we so far tried the followings - - try to regenerate the Correct Chain from this link (https://whatsmychaincert.com) and replace with our SSLCertificateChainFile on etc/httpd/conf.d/ssl.conf, not working. - download bundle https://www.ssl.com/blogs/addtrust-external-ca-root-expired-may-30-2020/ (bcoz we bought SSL from there) and replace the SSLCertificateChainFile on ssl.conf also not working. can you help us give advice?

Reader Moritz Kobel on 2021-09-30 at 20:55:

Thank you very much! I had a similar problem due to the expiring Let’s Encrypt DST Root CA X3 certificate. I described a similar workaround on https://www.kobelnet.ch/2021/09/30/old-lets-encrypt-root-certificate-expiration-workaround

Reader Annie on 2021-12-07 at 11:54:

Hi Andrew, due to the expiring Let’s Encrypt DST Root CA X3 certificate, I cannot see some of the websites by internet in the computer. Could you please teaching me how to solve this problem by steps. Thanks for your kindness!

Best Regards, Annie

Andrew Ayer

Sections

Blog