Andrew Ayer

My preferred method for deploying webapps is to have the webapp listen directly on port 443, without any sort of standalone web server or HTTP reverse proxy in front. I have had it with standalone web servers: they're all over-complicated and I always end up with an awkward bifurcation of logic between my app's code and the web server's config. Meanwhile, my preferred language, Go, has a high-quality, memory-safe HTTPS server in the standard library that is well suited for direct exposure on the Internet.

However, only one process at a time can listen on a given IP address and port number. In a world of ubiquitous IPv6, this wouldn't be a problem - each of my servers has literally trillions of IPv6 addresses, so I could easily dedicate one IPv6 address per webapp. Unfortunately, IPv6 is not ubiquitous, and due to the shortage of IPv4 addresses, it would be too expensive to give each app its own IPv4 address.

The conventional solution for this problem is HTTP reverse proxying, but I want to do better. I want to be able to act like IPv6 really is ubiquitous, but continue to support IPv4-only clients with a minimum amount of complexity and mental overhead. To accomplish this, I've turned to SNI-based proxying.

I've written about SNI proxying before, but in a nutshell: a proxy server can use the first message in a TLS connection (the Client Hello message, which is unencrypted and contains the server name (SNI) that the client wants to connect to) to decide where to route the connection. Here's how I'm using it:

• My webapps listen on port 443 of a dedicated IPv6 address. They do not listen on IPv4.
• Each of my servers runs snid, a Go daemon which listens on port 443 of the server's single IPv4 address.
• When snid receives a connection, it peeks at the first TLS message to get the desired server name. It does a DNS lookup for the server name's IPv6 address, and proxies the TCP connection there. To prevent snid from being used as an open proxy, snid only forwards the connection if the IPv6 address in within my server's IPv6 range.
• The AAAA record for a webapp is the dedicated IPv6 address, and the A record is the shared IPv4 address. Thus, IPv6 clients connect directly to the webapp, while IPv4 clients are proxied via snid.

Preserving the Client's IP Address

One of the headaches caused by proxies is that the backend doesn't see the client's IP address - connections appear to be from the proxy server instead. With HTTP proxying, this problem is typically solved by stuffing the client's IP address in a header field, which is a minefield of security problems that allow client IP addresses to be spoofed if you're not careful. With TCP proxying, a common solution is to use the PROXY protocol, which puts the client's IP address at the beginning of the proxied connection. However, this requires backends to understand the PROXY protocol.

snid can do better. Since IPv6 addresses are 128 bits long, but IPv4 addresses are only 32 bits, it's possible to embed IPv4 addresses in IPv6 addresses. snid embeds the client's IP address in the lower 32 bits of the source address which it uses to connect to the backend. It's trivial for the backend to translate the source address back to the IPv4 address, but this is purely a user interface concern. If a backend doesn't do the translation, it's possible for the human operator to do the translation manually when viewing log entries, configuring access control, etc.

For the IPv6 prefix, I use 64:ff9b:1::/48, which is a non-publicly-routed prefix reserved for IPv4/IPv6 translation mechanisms. For example, the IPv4 address 192.0.2.10 translates to:

64:ff9b:1::c000:20a

Conveniently, it can also be written using embedded IPv4 notation:

64:ff9b:1::192.0.2.10

O(1) Config

snid's configuration is just a few command line arguments. Here's the command line for the instance of snid that's running on the server that serves www.agwa.name and src.agwa.name:

snid -listen tcp:18.220.42.202:443 -mode nat46 -nat46-prefix 64:ff9b:1:: -backend-cidr 2600:1f16:719:be00:5ba7::/80

-listen tells snid to listen on port 443 of 18.220.42.202, which is the IPv4 address for www.agwa.name and src.agwa.name. -mode nat46 tells snid to forward connections over IPv6, with the source IPv4 address embedded using the prefix specified by -nat46-prefix. -backend-cidr tells snid to only forward connections to addresses within the 2600:1f16:719:be00:5ba7::/80 subnet, which includes the IPv6 addresses for www.agwa.name and src.agwa.name (2600:1f16:719:be00:5ba7::2 and 2600:1f16:719:be00:5ba7::1, respectively).

The best thing about snid's configuration is that I only have to touch it once. I don't have to change it when deploying new webapps. Deploying a new webapp only requires assigning it an IPv6 address and publishing DNS records for it, just like it would be in my dream world of ubiquitous IPv6. I call this O(1) configuration since it doesn't get longer or more complex with the number of webapps I run.

Guaranteed Secure

HTTP reverse proxying is a minefield of security concerns. In addition to the IP address spoofing problems discussed above, you have to contend with request smuggling and HTTP desync vulnerabilities. This is a class of vulnerability that will never truly be solved: you can patch vulnerabilities as they're discovered, but thanks to the inherent ambiguity in parsing HTTP, you can never be sure there won't be more.

I don't have to worry about any of this with snid. Since snid doesn't decrypt the TLS connection (and lacks the necessary keys to do so), proxying with snid is guaranteed to be secure as long as TLS is secure. It can harm security no more than any untrusted router on the Internet can. This helps put snid out of my mind so I can forget that it even exists.

Compatible with ACME ALPN

Since ACME's TLS-ALPN challenge uses SNI to convey the hostname being validated, snid will forward TLS-ALPN requests from the certificate authority to the appropriate backend. Automatic certificate acquisition, such as with Go's autocert package, Just Works.

What About Encrypted Client Hello?

Since the SNI hostname is in plaintext, a network eavesdropper can determine what hostname a client is connecting to. This is bad for privacy and censorship resistance, so there is an effort underway to encrypt not just the SNI hostname, but the entire Client Hello message. How does this affect snid?

First, it's important to note that the destination IP address in the IP header is always going to be unencrypted, so by putting my webapps on different IPv6 addresses, I'm giving eavesdroppers the ability to find out which webapp clients are connecting to, regardless of SNI. However, a single webapp might handle multiple hostnames, and I'd like to hide the specific hostname from eavesdroppers, so Encrypted Client Hello still has some value. Fortunately, Encrypted Client Hello works with snid.

Encrypted Client Hello doesn't actually encrypt the initial Client Hello message. It's still sent in the clear, but with a decoy SNI hostname. The actual Client Hello message, with the true SNI hostname, is encrypted and placed in an extension of the unencrypted Client Hello. To make Encrypted Client Hello work with snid, I just need to ensure that the decoy SNI hostname resolves to the IPv6 address of the backend server. snid will see this hostname and route the connection to the correct backend server, as usual. The backend will decrypt the true Encrypted Client Hello to determine which specific hostname the client wants.

For additional detail about this approach, see my comment on Hacker News.

What About Port 80?

Obviously, I can't proxy unencrypted HTTP traffic using SNI-based proxying. But at this point, port 80 exists solely to redirect clients to HTTPS. To handle this, I plan to run a tiny, zero-config daemon on port 80 of all IPv4 and IPv6 addresses that will redirect the client to the same URL but with http:// replaced with https://. (For now, I'm using Apache for this.)

Installing snid

If you have Go, you can install snid by running:

go install src.agwa.name/snid@latest

You can also download a statically-linked binary.

See the README for the command line usage.

Rejected Approach: UNIX Domain Sockets

Before settling on the approach described above, I had snid listen on port 443 of all interfaces (both IPv4 and IPv6) and forward connections to a UNIX domain socket whose path contained the SNI hostname. For example, connections to example.com would be forwarded to /var/tls/example.com. The client's IP address was preserved using the PROXY protocol.

This had some nice properties. I could use filesystem permissions to control who was allowed to create sockets, either by setting permissions on /var/tls, or by symlinking specific hostnames under /var/tls to other locations on the filesystem which non-root users could write to. It felt really elegant that applications could listen on an SNI hostname rather than on an IP address and port.

However, few server applications support the PROXY protocol or listening on UNIX domain sockets. I could make sure my own apps had support, but I really wanted to be able to use off-the-shelf apps with snid. I did write an amazing LD_PRELOAD library that intercepts the bind system call and transparently replaces binding to a TCP port with binding to a UNIX domain socket. It even intercepts getpeername and makes it returns the IP address received via the PROXY protocol. Although this worked with every application I tried it with, it felt like a hack.

Additionally, UNIX domain sockets have some annoying semantics: if the socket file already exists (perhaps because the application crashed without removing the socket file), you can't bind to it - even if no other program is actually bound to it. But if you remove the socket file, any program bound to it continues running, completely unaware that it will never again accept a client. The semantics of TCP port binding feel more robust in comparison.

For these reasons I switched to the IPv6 approach described above, allowing me to use standard, unmodified TCP-listening server apps without any hacks that might compromise robustness. However, support for UNIX domain sockets lives on in snid with the -mode unix flag.