Andrew Ayer

You can protect the users of your Go modules from supply chain attacks, such as a compromise of your GitHub account, by monitoring Go's checksum database (sumdb). Since the go command won't install a module unless its checksum is published in the sumdb, monitoring the sumdb lets you discover unauthorized versions of your modules. Since the sumdb is a transparency log, you can even detect if Google themselves go rogue and publish a malicious version of your module. Although you can only detect, not prevent, attacks, Go's Minimal Version Selection makes it possible to respond before most of your users have installed the malicious version. Dependency cooldowns, which are coming to Go, will make it even easier to respond in time.

How To Monitor

Source Spotter, which is operated by my company SSLMate as a free service to the Go community, provides Atom feeds listing all versions of your modules found in the sumdb. For example, this Atom feed returns all versions of modules under the src.agwa.name/ prefix:

https://feeds.api.sourcespotter.com/modules/versions.atom?module=src.agwa.name%2F

You can also get Prometheus-compatible metrics:

https://metrics.api.sourcespotter.com/modules?module=src.agwa.name%2F

You can scrape the metrics endpoint using Prometheus and alert in your usual way, subscribe to the Atom feed using your favorite feed reader, or use one of the free services that converts Atom feeds to emails.

Avoiding Noise

Discovery is only half the story. The more important half is deciding whether to alert on a discovered module. Approximately 100% of the records published in the sumdb are legitimate. If you're alerted every time a new version of your module is legitimately published, it will be very hard to notice the one time it's an attack.

I wasn't sure at first how Source Spotter could facilitate no-noise monitoring. Initially, I thought Source Spotter would need access to your module's Git repository so it could cross-check sumdb records against the repo's contents. But that seemed complicated, and it would fail to detect a compromise of your repository host. I also really wanted a solution that wouldn't require users to create accounts.

I finally found a lightweight solution that I really like. I'll show you how you use it, and then explain how it works.

First, you install a small command line tool called sourcespotter-authorize and generate a public/private key pair:

go install software.sslmate.com/src/sourcespotter/cmd/sourcespotter-authorize@latest

sourcespotter-authorize -keygen

Run sourcespotter-authorize again with the -feed-for or -metrics-for flags to output Atom and Prometheus URLs for the module prefix you want to monitor (src.agwa.name/ in this example):

sourcespotter-authorize -feed-for src.agwa.name/

https://feeds.api.sourcespotter.com/modules/versions.atom?module=src.agwa.name%2F&mldsa=efbcb2bcb2d4decdf1ad9cab3224b2dd0087b6c6877abe00448a00d31716dff1

sourcespotter-authorize -metrics-for src.agwa.name/

https://metrics.api.sourcespotter.com/modules?module=src.agwa.name%2F&mldsa=efbcb2bcb2d4decdf1ad9cab3224b2dd0087b6c6877abe00448a00d31716dff1

These are the same URLs shown earlier, but with a new mldsa= parameter in the query string which is the hash of the public key you generated in the previous step.

Initially, these URLs return the same contents as the URLs without the mldsa= parameter. That's because you haven't marked any module versions as authorized yet.

To mark a module version as authorized, change into the Git repository for the module and run sourcespotter-authorize with a Git tag:

cd ~/src/snid

sourcespotter-authorize v0.4.0

Now, v0.4.0 of this module is omitted from the feed and metrics URLs.

The command accepts multiple tags as arguments, so you can authorize all the tags in a repo like this:

sourcespotter-authorize $(git tag)

Moving forward, you should run sourcespotter-authorize any time you tag a new version. I've written a tiny shell script called gotag that runs git tag followed by sourcespotter-authorize:

#!/bin/sh -e git tag "$1" sourcespotter-authorize "$1"

As long as you authorize every tag you create, the feed and metrics URLs will report zero unauthorized module versions, eliminating false positive alerts. A supply chain attacker can't hide their malicious versions from your feeds unless they also compromise sourcespotter-authorize's private key.

How It Works

sourcespotter-authorize -keygen generates an ML-DSA-44 private key (stored under $XDG_CONFIG_HOME/sourcespotter-authorize). The SHA-256 hash of the corresponding public key goes in the mldsa query string parameter.

When you authorize a tag, sourcespotter-authorize uses the golang.org/x/mod/zip and golang.org/x/mod/sumdb/dirhash packages to compute the checksums of the go.mod and module zip files for the tag. It formats the module path, version, and hashes for each tag as a go.sum file (no need to invent a new format here), and signs the file with your private ML-DSA key. Finally, it uploads a JSON object to the Source Spotter server containing your public key, the go.sum file, and the signature.

The Source Spotter server verifies the signature using the public key. If valid, it records the module path, version, and hash as being authorized by that public key, and excludes that module version from the feeds and metrics endpoints for the key.

The protocol is simple and documented if you want to implement your own client.

Why Not Just Sign Your Releases?

If you're generating a private key and signing stuff anyway, why not just sign your releases and distribute the signatures? After all, this is the traditional approach to stopping unauthorized software releases.

The problem with the traditional approach is that consumers of your software have to verify the signatures, which means they need to know what your public key is. Key distribution is a hard, hard problem, and most software ecosystems do a bad job at it. I would guess that in practice, few people actually verify signatures of software releases.

In contrast, the go command automatically verifies that every module it installs is listed in the checksum database, without the user needing to do anything. Although sourcespotter-authorize needs a private key, you only need to distribute the public key to Source Spotter (or whatever other monitor you might choose to use), rather than every single consumer of your software. That's a vastly simpler problem, especially when (not if) you need to rotate your keys.

Everyone who signs their software releases will need to rotate their keys soon, as quantum computers are expected to break RSA and elliptic curves within the next few years. An earlier version of sourcespotter-authorize used elliptic curve keys. After it gained ML-DSA support, upgrading my key was super easy - generate a new key, transfer my list of authorized versions to it, and finally update the URL in my feed reader:

sourcespotter-authorize -export > tmpfile

rm ~/.config/sourcespotter-authorize/private_key

sourcespotter-authorize -keygen

sourcespotter-authorize -import < tmpfile

sourcespotter-authorize -feed-for src.agwa.name/

This was so much easier than communicating a new key to every consumer of my software would have been, and didn't require a lengthy transition period during which I was signing with two keys.

You can learn more about Source Spotter's module monitoring, get the source code on GitHub, or read my blog post about how Source Spotter also verifies Go's reproducible builds.

Try the following Git clone with libcurl 8.7.1 (which happens to be the version shipped in macOS 14.6 and newer) and it fails or hangs:

git clone https://software.sslmate.com/src/macosgitbug.git

Disable HTTP/2 and it works:

git clone -c http.version=HTTP/1.1 https://software.sslmate.com/src/macosgitbug.git

The bug is in libcurl 8.7.1's handling of the FAILONERROR option. FAILONERROR tells libcurl to treat unsuccessful HTTP status codes, such as 404, as a request failure. When HTTP/2 is used, the bug causes other in-flight requests on the same HTTP/2 connection to also fail, or even to hang. The bug was fixed over two years ago in curl 8.8.0, but Apple continues to ship a buggy version, even in last week's macOS 27 release.

When retrieving a repository over the "dumb" transfer protocol, Git makes certain HTTP requests with the FAILONERROR option set, notably requests to objects/info/alternates and objects/info/http-alternates, which list alternate locations where the repository's content can be found. Most repositories don't have alternate locations, so these files don't exist, and the URLs return 404 errors. When the buggy version of libcurl is used, this 404 error causes Git's other HTTP requests to also fail, and Git is unable to clone the repository.

The bug affects not just direct uses of Git, but also go get with GOPROXY=direct or a module listed in GOPRIVATE, which invoke Git under the hood.

Working around the bug on the client side is easy: just force Git to use HTTP/1.1:

git config --global http.version HTTP/1.1

Even better, install Git through MacPorts, since Apple has clearly dropped the ball. (Homebrew won't help - unlike MacPorts, they use the system libcurl.)

But most clients won't know to apply this workaround, and if you host Git repositories with the dumb protocol, you probably want macOS users to be able to clone your repositories! Fortunately, there's a really easy server-side workaround: create objects/info/alternates and objects/info/http-alternates as empty files, so they don't return a 404 error anymore. Git treats the empty files the same as it would treat a 404 error, and the libcurl bug isn't triggered.

touch /path/to/repo.git/objects/info/alternates /path/to/repo.git/objects/info/http-alternates

The bug isn't triggered when the repository supports the "smart" protocol, which is why macOS can clone repositories from GitHub and other popular forges despite them using HTTP/2. But I do not want to use the smart protocol for my repositories: although it has many advantages over the dumb protocol, it requires heavy server-side computation and even a modest load can knock a server over. In contrast, the dumb protocol can be served entirely from static files, which makes a huge difference for withstanding the horde of AI scrapers currently terrorizing the Web. I hope that we will see innovations to the dumb protocol that bring it some of the advantages of the smart protocol while still being served from static files.

Thanks to Romain of the Traefik project for noticing that SSLMate's repos couldn't be cloned on macOS, Sebastiaan van Stijn for asking that this problem be reported upstream instead of silently hacked around in Traefik's go.mod file, and Kangmin Kim for pointing me to the libcurl bug as the root cause. Claude Code proposed the empty file workaround so I didn't have to waste (too much) time on this. Zero thanks to Apple for shipping a two-year-old show-stopping bug in libcurl.

HTTP reverse proxying is a minefield. Just the other week, a researcher disclosed a desync vulnerability in Discord's media proxy that allowed spying on private attachments. This is not unusual; these vulnerabilities just keep coming.

The problem is the widespread use of HTTP as the protocol between reverse proxies and backends, even though it's unfit for the job. But we don't have to use HTTP here. There's a 30-year-old protocol for proxy-to-backend communication that avoids HTTP's pitfalls. It's called FastCGI, and its specification was released 30 years ago today.

FastCGI is a Wire Protocol, not a Process Model

It's true that some web servers can automatically spawn FastCGI processes to handle requests for files with the .fcgi extension, much like they would for .cgi files. But you don't have to use FastCGI this way - you can also use the FastCGI protocol just like HTTP, with requests sent over a TCP or UNIX socket to a long-running daemon that handles them as if they were HTTP requests.

For example, in Go all you have to do is import the net/http/fcgi standard library package and replace http.Serve with fcgi.Serve:

Everything else about your app stays the same - even your handler, which continues to use the standard http.ResponseWriter and http.Request types.

Popular proxies like Apache, Caddy, nginx, and HAProxy support FastCGI backends, and the configuration is simple:

Show more config examples

Apache HTTP

ProxyPass / http://localhost:8080/

Apache FastCGI

ProxyPass / fcgi://localhost:8080/

Caddy HTTP

reverse_proxy localhost:8080 { transport http { } }

Caddy FastCGI

reverse_proxy localhost:8080 { transport fastcgi { } }

HAProxy HTTP

backend app_backend server s1 localhost:8080

HAProxy FastCGI

fcgi-app fcgi_app docroot / backend app_backend use-fcgi-app fcgi_app server s1 localhost:8080 proto fcgi

Why HTTP Sucks for Reverse Proxies: Desync Attacks / Request Smuggling

HTTP/1.1 has the tragic property of looking simple on the surface (it's just text!) but actually being a nightmare to parse robustly. There are so many different ways to format the same HTTP message, and there are too many edge cases and ambiguities for implementations to handle consistently. As a result, no two HTTP/1.1 implementations are exactly the same, and the same message can be parsed differently by different parsers.

The most serious problem is that there is no explicit framing of HTTP messages - the message itself describes where it ends, and there are multiple ways for a message to do that, all with their own edge cases. Implementations can disagree about where a message ends, and consequently, where the next message begins. This is the foundation of HTTP desync attacks, also known as request smuggling, wherein a reverse proxy and a backend disagree about the boundaries between HTTP messages, causing all sorts of nightmare security issues, such as the Discord vulnerability I linked above.

A lot of people seem to think you can just patch the parser divergences, but this is a losing strategy. James Kettle just keeps finding new ones. After finding another batch last year, he declared "HTTP/1.1 must die".

HTTP/2, when consistently used between the proxy and backend, fixes desync by putting clear boundaries around messages, but FastCGI has been doing that since 1996 with a simpler protocol. For context, nginx has supported FastCGI backends since its first release, but only got support for HTTP/2 backends in late 2025. Apache's support for HTTP/2 backends is still "experimental".

Why HTTP Sucks for Reverse Proxies: Untrusted Headers

If desync attacks were the only problem, you could just use HTTP/2 and call it a day. Unfortunately, there's another problem: HTTP has no robust way for the proxy to convey trusted information about the request, such as the real client IP address, authenticated username (if the proxy handles authentication), or client certificate details (if mTLS is used).

The only option is to stick this information in HTTP headers, alongside the headers proxied from the client, without a clear structural distinction between trusted headers from the proxy and untrusted headers from a potential attacker. For example, the X-Real-IP header is often used to convey the client's real IP address. In theory, if your proxy correctly deletes all instances of the X-Real-IP header (not just the first, and including case variations like x-REaL-ip) before adding its own, you're safe.

In practice, this is a minefield and there are an awful lot of ways your backend can end up trusting attacker-controlled data. Your proxy really needs to delete not just X-Real-IP, but any header that's used for this sort of thing, just in case some part of your stack relies on it without your knowledge. For example, the Chi middleware determines the client's real IP address by looking at the True-Client-IP header first. Only if True-Client-IP doesn't exist does it use X-Real-IP. So even if your proxy does the right thing with X-Real-IP, you can still be pwned by an attacker sending a True-Client-IP header.

FastCGI completely avoids this class of problem by providing domain separation between headers from the client and information added by the proxy. Though trusted data from the proxy and HTTP request headers are transmitted to the backend in the same key/value parameter list, HTTP header names are prefixed with the string "HTTP_", making it structurally impossible for clients to send a header that would be interpreted as trusted data.

FastCGI defines some standard parameters such as REMOTE_ADDR to convey the real client IP address. Go's net/http/fcgi package automatically uses this parameter to populate the RemoteAddr field of http.Request, rendering middleware unnecessary. It Just Works. Proxies can also use non-standard parameters to report whether HTTPS was used, what TLS ciphersuite was negotiated, and what client certificate was presented, if any. Go automatically sets the Request's TLS field to a non-nil (but empty) value if the request used HTTPS, which is very handy for enforcing the use of HTTPS. The fcgi.ProcessEnv function can be used to access the full set of trusted parameters sent by the proxy.

Closing Thoughts

If FastCGI is the better protocol, why isn't it more popular? Maybe it's the name - while capitalizing on CGI's popularity made sense in 1996, CGI feels dated in 2026. There's also an enduring lack of awareness of the security problems with HTTP reverse proxying. Watchfire described desync attacks in 2005, and gave a prescient warning of their intractability, but the attacks were inexplicably ignored for over a decade. In an alternate timeline, Watchfire's research was taken seriously and people went looking for other protocols for reverse proxies.

FastCGI is very usable today, and has been in production use at SSLMate for over 10 years. That said, using a vintage technology has some downsides. It was never updated to support WebSockets. The tooling is not as good. For example, curl has no way to make requests to a FastCGI server. It supports FTP, Gopher, and even SMTP (however that works), but not FastCGI. When I benchmarked Go's FastCGI server behind a variety of reverse proxies, some workloads had worse throughput compared to HTTP/1.1 or HTTP/2. I don't think that's inherent to the protocol, but a reflection that FastCGI code paths have not been optimized as much as HTTP.

Despite these shortcomings, I still think FastCGI is worth using. I don't use WebSockets, and it's fast enough for my use case (and maybe yours too). If it ever became the bottleneck, I'd rather buy more hardware than deal with the nightmare of HTTP reverse proxying.

Happy 30th birthday, FastCGI!

Imagine you're connecting to an IP address over TLS. The server presents a valid, properly-issued IP address certificate from a publicly-trusted certificate authority. But you're not really talking to that IP address, but to a man-in-the-middle (MitM) who was assigned the IP address over a year ago. How could this be? Unfortunately, publicly-trusted IP address certificates just don't provide the same level of security as certificates for domain names. Unless you're operating a DNS-over-TLS or DNS-over-HTTPS resolver, you should not use IP address certificates. You'll be much more secure with domain name certificates.

(When I say IP address certificate, I mean a TLS certificate with an IP address subjectAltName, which allows a certificate to authenticate a connection to an IP address, rather than a domain name as is typical on the Web. Public certificate authorities have been allowed to issue IP address certificates for a long time, but I fear they will become more popular since Let's Encrypt added support for them last month.)

The basic security property provided by a certificate is that the certificate authority has validated that the certificate subscriber (the person who applies for the certificate and knows its private key) is authorized to represent the domain name or IP address in the certificate. This ensures that the other end of a TLS connection is truly the domain or IP address that you want to connect to, not a MitM impostor.

But the validation is not done every time a TLS connection is established; rather, it was done at some point in the past. Thus, the certificate subscriber may no longer be authorized to represent the domain or IP address.

How old might the validation be? As of February 2026, certificate authorities are allowed to issue certificates that are valid for up to 398 days. (Let's Encrypt limits their IP address certs to 6 days, but you have to worry about attackers using other CAs, not just the CA you use.) So the validation may be 398 days old. But it gets worse. When issuing a certificate, CAs are allowed to rely on a validation that was done up to 398 days prior to issuance. So when you establish a TLS connection, you may be relying on a validation that was performed a whopping 796 days ago. You could be talking not to the current assignee of the domain or IP address, but to anyone who was assigned the domain or IP address at any point in the last 2+ years.

This is a problem with both domains and IP addresses, but it's way worse with IP addresses. While it's still very possible to register a domain that no one has ever registered before, you don't have this luxury with IPv4 addresses. There are no unassigned IPv4 addresses left; when you get an IPv4 address, it has already been assigned to someone else. In some cases, particularly in cloud environments, the address could have had as many tenants as a room at a Motel 6.

For some use cases, the risk is unavoidable. For example, DNS-over-TLS and DNS-over-HTTPS resolvers have to be contacted by IP address, to avoid a circular dependency on DNS resolution. Domain name certificates are not an option. Fortunately, by 2029, the maximum certificate lifetime will be 47 days, and the maximum validation reuse period will be 10 days, so the risk window will be only 57 days. That's not too bad for a service that's likely to exist for many years with the same IP address.

But even with a 57 day window, other use cases should just use domain name certificates. One use case for IP address certificates is particularly misguided: securing short-lived cloud workloads. Let's Encrypt describes this use case as:

Securing ephemeral connections within cloud hosting infrastructure, like connections between one back-end cloud server and another, or ephemeral connections to administer new or short-lived back-end servers via HTTPS - as long as those servers have at least one public IP address available.

On Bluesky, Ryan Hurst goes into more detail:

Modern infrastructure no longer has stable hostnames, static IPs, or long-lived trust anchors. Workloads spin up before DNS exists, live briefly, and disappear. Trust has to keep up.

Short-lived and IP certificates make it possible to use TLS before a DNS name exists, reduce friction for DNS over HTTPS adoption, secure ephemeral devices and services by default, and shift trust from long-lived credentials to automated renewal.

As I understand it, the use case is running servers which accept TLS connections and exist for a very short period, like minutes. The most secure implementation would be to assign the server a unique, never-before-used hostname under your domain, publish a DNS record pointing the hostname to the server's IP address, get a domain name certificate, and have clients connect to the hostname. Thanks to the unique hostname, clients can be sure they're connecting to the right server and not to an impostor who was assigned the IP address at some point in the past.

But Ryan suggests that it might not be possible to wait for a DNS record to be published before the server needs to be usable. Fortunately, it's still possible to avoid IP address certificates. Instead, you can publish, in advance, a DNS record under your domain for every IP address you might need to use. For example, ip-192-0-2-1.yourcompany.example would resolve to 192.0.2.1. ISPs commonly publish DNS records like this for their entire IP address space, so it's possible to do at scale. Have your clients connect to your short-lived server using this DNS record, rather than an IP address. To prevent would-be attackers from completing domain validation under your domain using a non-DNS validation method, you should also publish a CAA record for your domain that uses ACME account binding to restrict issuance to your ACME account. Now clients can be sure they're connecting to a server controlled by your organization, rather than anyone who happened to be assigned the same cloud IP address in the last 2 years.

You can also monitor Certificate Transparency logs to look for unauthorized certificates for your domain. Monitoring Certificate Transparency is not practical when using IP address certificates with short-lived cloud workloads, since the set of IP addresses that you need to monitor will be constantly changing as you create and destroy your servers, and it will be hard to tell whether an unknown certificate is malicious or just an innocent issuance to someone who got assigned the same IP address.

I encourage you to think long and hard before using IP address certificates, especially for short-lived cloud servers. The high risk of stale validation, combined with the inability to effectively monitor Certificate Transparency, bring IP address certificates uncomfortably close to opportunistic encryption (TLS without certificate validation), even with post-2029 shortened lifetimes. Fortunately, you can embed IP addresses in pre-published DNS records if you can't publish DNS records on the fly. Doing a little extra DNS work is a small price to pay for avoiding the inherent risks of IP address certificates.

Twice a year, the Certificate Transparency ecosystem undergoes a transition as certificate authorities start to submit certificates to new semiannual log partitions. And recently, the ecosystem has started transitioning to the new static-ct-api specification. Unfortunately, despite efforts to make these transitions extremely easy for certificate authorities, in the past week I have detected 16 certificate authorities who have bungled these transitions, issuing certificates that are rejected by some or all mainstream web browsers with an error message like "This Connection Is Not Private" or ERR_CERTIFICATE_TRANSPARENCY_REQUIRED.

If you're not familiar, Certificate Transparency (CT) is a system for publishing SSL certificates in public logs. Certificate Transparency monitors like Cert Spotter download the logs to help you track certificate expiration and detect unauthorized certificates for your domains.

At a high level, Certificate Transparency works like this:

• Before issuing a certificate, the certificate authority (CA) creates a "precertificate" containing the details of the certificate it intends to issue.
• The CA submits the precertificate to multiple Certificate Transparency logs.
• Each log returns a receipt, called a Signed Certificate Timestamp (SCT), which confirms submission of the precertificate.
• The CA embeds the SCTs in the certificate which it gives to the site operator.
• When a browser loads a website, it makes sure the website's certificate has SCTs from a sufficient number of recognized logs. If it doesn't, the browser throws up an error page and refuses to load the website.

Billions of SSL certificates are issued and logged to CT every year. To prevent logs from growing indefinitely, logs only accept (pre)certificates which expire within a certain range, typically six months long. Every log will eventually contain only expired certificates, allowing it to be shut down. Meanwhile, new logs are created to contain certificates expiring further in the future.

How do CAs know what logs to submit precertificates to? It's easy: Apple and Chrome each publish a JSON file containing a list of logs. (Firefox and Edge use Chrome's list.) Apple's is at https://valid.apple.com/ct/log_list/current_log_list.json and Chrome's is at https://www.gstatic.com/ct/log_list/v3/log_list.json. Each log object contains the log's name, URL, public key, range of expiration dates accepted by the log, and crucially, the log's state.

{ "description": "Sectigo 'Elephant2027h1'", "log_id": "YEyar3p/d18B1Ab8kg3ImesLHH34yVIb+voXdzuXi8k=", "key": "MFkwEwYHKoZIzj0CAQYIKoZI...AScw2woA==", "url": "https://elephant2027h1.ct.sectigo.com/", "mmd": 86400, "state": { "usable": { "timestamp": "2025-07-22T01:33:20Z" } }, "temporal_interval": { "start_inclusive": "2027-01-01T00:00:00Z", "end_exclusive": "2027-07-01T00:00:00Z" } }

The state is very simple: if it's "usable", then CAs should use it. If it's something else, CAs should not use it.

The full process of logging is a bit more complicated, because CAs have to include SCTs from a sufficiently-diverse set of logs, but when it comes to finding the initial set of logs to consider, it's hard to imagine how it could be any easier for CAs. They just need to download the Apple and Chrome lists and find the logs whose state is Usable in both lists and whose expiration range covers the expiration date of the certificate.

Despite this, a number of CAs appear to either disregard the state or only consider Chrome's log list. Historically, this has not caused problems because new logs have become Usable in both Chrome and Apple before they were needed for new certificates. Since the maximum certificate lifetime is 398 days, logs for certificates expiring in the first half of 2027 (2027h1) needed to be Usable by November 29, 2025. Unfortunately, not all 2027h1 logs were Usable by this date.

First, Google's 2027h1 logs (Argon 2027h1 and Xenon 2027h1) were added to Chrome 40 days later than they should have been. Normally, new logs are added to Chrome after 30 days of successful monitoring, but this process is still very manual and human error led to Chrome setting a 70 day timer instead of a 30 day timer. Consequentially, these logs are still in the Qualified state in Chrome. Although Qualified logs are recognized by up-to-date installations of Chrome (and Firefox and Edge), there may be out-of-date installations which do not recognize them, making it a very bad idea for CAs to use Qualified logs if they care about compatibility. Chrome, Firefox, and Edge automatically disable Certificate Transparency enforcement once they become 70 days out-of-date, so Argon and Xenon 2027h1 will become Usable on December 27, 2025, which is 70 days after they became Qualified. (Argon and Xenon 2027h1 are already Usable in Apple's list.)

Second, DigiCert's 2027h1 logs (Sphinx 2027h1 and Wyvern 2027h1) don't appear at all in Apple's log list. Since Apple doesn't use a public bug tracker for their CT log program like Chrome, I have no idea what went wrong. Did DigiCert forget to tell Apple about their new logs, or is Apple slow-rolling them for some reason? Certificates which rely on either DigiCert log won't work at all on Apple platforms. (They are already Usable in Chrome's list.)

While the late addition of logs is not ideal, it should not have been a problem, because there are plenty of other 2027h1 logs which became Usable for both Apple and Chrome in time.

I first became aware of issues last Tuesday when Arabella Barks posted a message to Mozilla's dev-security-policy mailing list referencing a certificate issued by Certum with SCTs from DigiCert Wyvern 2027h1. Sensing that this could be a widespread problem, I decided to investigate. My company, SSLMate, maintains a 51TB PostgreSQL database with the contents of every Certificate Transparency log. The database's primary purpose is to power our Certificate Transparency monitoring service, Cert Spotter, and our Certificate Transparency Search API, but it's also very handy for investigating ecosystem issues.

I ran a query to find all precertificates logged to Google's and DigiCert's 2027h1 logs. This alone was not sufficient to identify broken certificates, since CAs could be submitting precertificates to these logs but not including the SCTs in the final certificate, or including more than the minimum number of required SCTs. Therefore, for every precertificate, I looked to see if the corresponding final certificate had been logged anywhere. If it had, I ran it through SSLMate's CT Policy Analyzer to see if it had enough SCTs from broadly Usable logs. If the final certificate wasn't available for analysis, I counted how many other logs the precertificate was logged to. If fewer than three of these logs were Usable, then there was no way the corresponding certificate could have enough SCTs.

I posted my findings to the ct-policy mailing list later that day, alerting CAs to the problem. Since then, I've found even more certificates relying on logs that are not broadly Usable. As of publication time, the following CAs have issued such certificates:

• Certum
• Cybertrust Japan (fixed)
• Disig
• GDCA
• GlobalSign (fixed)
• HARICA
• IdenTrust (fixed)
• Izenpe (fixed)
• Microsec
• NAVER
• SECOM
• SSL.com
• SHECA
• TWCA (fixed)
• certSIGN
• emSign

Of those, only the five indicated above have fixed their systems. The others have all issued broken certificates within the last two days, even though it has been a week since my first public posting.

Unfortunately, logging to non-Usable logs wasn't the only problem. Last Wednesday, Cert Spotter began alerting me about certificates issued by Cybertrust Japan containing SCTs with invalid signatures. I noticed that the SCTs with invalid signatures were all from static-ct-api logs.

To address shortcomings with the original Certificate Transparency specification (RFC6962), the ecosystem has been transitioning to logs based on the static-ct-api specification. Almost half of the 2027h1 logs use static-ct-api. However, while static-ct-api requires major changes for log monitors, it uses the exact same protocol for CAs to submit (pre)certificates. This was an intentional decision to make static-ct-api easier to adopt, so that it wouldn't suffer the same fate as RFC9162, which was intended to replace RFC6962 but was dead-on-arrival in part because it completely broke compatibility with the existing ecosystem.

However, there is one teeny tiny difference with static-ct-api: whereas RFC6962 logs always return SCTs with an empty extensions field, static-ct-api logs return SCTs with non-empty extensions. This should not be problem - the extensions field is just an opaque byte array and CAs do not need to understand what static-ct-api logs place it in it. They just need to copy it through to the final certificate, which they should have been doing anyways with RFC6962 logs. But Cybertrust Japan was always leaving the extension field empty regardless of what the log returned, breaking the SCT's signature. Since SCTs with invalid signatures are disregarded by browsers, this left their certificates with an insufficient number of SCTs, dooming them to rejection.

After publication of this post, Cert Spotter alerted me to invalid SCT signatures in certificates issued by NAVER. In this case, the SCT extensions were non-empty but encoded in base64, indicating that NAVER wasn't decoding the base64 from the JSON response when copying it to the SCT. On one hand, I don't love RFC6962's wording about the extensions field: while the other JSON fields, like id and signature, are clearly indicated as "base64 encoded", it's only implied that extensions is base64-encoded (it says "Clients should decode the base64-encoded data and include it in the SCT"). On the other hand, if NAVER were verifying the signature of SCTs before embedding them in certificates, they almost certainly would have caught this mistake, since successful verification relies on correctly decoding the JSON response. And we know from past incidents that it's very important for CAs to verify SCT signatures.

Unfortunately, we'll probably never learn the root cause of these failures or what CAs are doing to prevent them from happening again. Normally, when a CA violates a policy, they are required to publish a public incident report, answer questions from the community, and note the failure in their next audit. If their incident response is bad or they keep having the same incident, they run the risk of being distrusted. However, Certificate Transparency is not a policy requirement in the traditional sense - CAs are free to issue certificates which violate CT requirements; those certificates just won't work in CT-enforcing browsers. This allows to CAs to issue unlogged certificates to customers who don't want their certificates to be public knowledge (and don't need them to work in browsers). Of course, that's not what the CAs here were doing - they were clearly trying to issue certificates that work in browsers; they just did a bad job of it.

Previously:

• GoDaddy Issues Thousands of Certificates That Don't Work in Safari (again)
• The Story Behind Last Week's Let's Encrypt Downtime
• Why Your Website Is Failing With ERR_CERTIFICATE_TRANSPARENCY_REQUIRED (March 2023 Edition)
• Tens of Thousands of GoDaddy Certificates Will Break in macOS 11.4 and iOS 14.6