Caddy Reverse Proxy: Troubleshooting TLS Handshake Failed Verification on macOS Local

Resolve Caddy's 'TLS handshake failed verification' error on macOS local setups. Covers untrusted certificates, hostname issues, and configuration fixes.

Resolve Caddy's 'TLS handshake failed verification' error on macOS local setups. Covers untrusted certificates, hostname issues, and configuration fixes.

Introduction

As a Systems Administrator, encountering TLS/SSL errors in local development environments is a common challenge. When Caddy, acting as a reverse proxy, reports a "TLS handshake failed verification" error on your macOS machine, it typically means Caddy cannot establish a trusted secure connection to your upstream (backend) server. This guide will walk you through the diagnostic steps and provide robust solutions tailored for a macOS local environment.

Symptom & Error Signature

Users will typically observe a 502 Bad Gateway error in their browser when trying to access the Caddy-proxied service, or a connection refused/timeout message. The most indicative information, however, comes from Caddy's logs.

Here's what a typical error signature looks like in Caddy's output, often found in stderr if running directly, or in journalctl if running as a service, or Docker logs:

ERROR   reverse_proxy   handshake_error error="remote error: tls: handshake failure" request={"proto":"HTTP/2.0","remote_addr":"127.0.0.1:56819","server_name":"localhost","host":"mylocalservice.test","method":"GET","uri":"/"}
ERROR   http.log.error  reverse proxying for http://mylocalservice.test failed: remote error: tls: handshake failure
ERROR   http.log.access 127.0.0.1:56819 - "GET / HTTP/2.0" 502 {"error":"reverse proxying for http://mylocalservice.test failed: remote error: tls: handshake failure"}

Or a more explicit verification error:

ERROR   reverse_proxy   handshake_error error="x509: certificate signed by unknown authority" request={"proto":"HTTP/2.0","remote_addr":"127.0.0.1:56819","server_name":"localhost","host":"mylocalservice.test","method":"GET","uri":"/"}

Root Cause Analysis

The "TLS handshake failed verification" error means that when Caddy tried to initiate a secure (HTTPS) connection to your backend server, the backend presented a TLS certificate that Caddy, acting as a client, could not validate. This typically stems from one of the following reasons in a local macOS environment:

  1. Self-Signed Certificate: The backend server is using a self-signed TLS certificate. By default, Caddy (like most TLS clients) does not trust self-signed certificates because they lack a chain of trust back to a publicly recognized Certificate Authority (CA).
  2. Custom/Development CA: The backend server's certificate was issued by a custom Certificate Authority (e.g., one generated by mkcert, minica, or similar tools for local development). While mkcert does a good job of adding its root CA to your system's trust store on macOS, Caddy might not automatically pick up these system-wide additions if it's running in an isolated environment (like Docker) or if its own trust store isn't correctly configured.
  3. Hostname Mismatch: The hostname Caddy is attempting to connect to (e.g., my-backend-app:8443) does not match any of the Subject Alternative Names (SANs) or the Common Name (CN) specified in the backend server's TLS certificate.
  4. Expired or Invalid Certificate: The backend server's TLS certificate has expired, is not yet valid, or has been revoked.
  5. Intermediate Certificate Chain Incompleteness: The backend server is not sending its full certificate chain, leading Caddy to fail building a trusted path to a root CA.
  6. Firewall/Proxy Interference: Less common for a "local environment" but a system-wide proxy or firewall could be intercepting TLS traffic, presenting its own untrusted certificate.

On macOS, specifically, the interaction between system-wide trust stores (managed by Keychain Access) and individual application trust stores can sometimes lead to confusion. Caddy, being a Go application, typically relies on the Go runtime's TLS mechanisms, which usually defer to the system's root CAs, but can be explicitly configured.

Step-by-Step Resolution

Follow these steps to diagnose and resolve the TLS handshake verification error.

1. Inspect the Backend Server's TLS Certificate

First, verify the details of the certificate presented by your backend server. This will help identify issues like self-signed status, expiration, or hostname mismatches.

# Replace 'your-backend-host.local' and '8443' with your actual backend's hostname and port
openssl s_client -connect your-backend-host.local:8443 -servername your-backend-host.local </dev/null 2>/dev/null | openssl x509 -noout -text

Examine the output for:

  • Issuer:: If it's the same as Subject:, it's a self-signed certificate. If it's a custom CA, note its name.
  • Validity: Check Not Before and Not After dates.
  • Subject: and X509v3 Subject Alternative Name:: Ensure the hostname Caddy is trying to reach is listed here.

2. Ensure Custom CA is Trusted on macOS (If Applicable)

If your backend uses a certificate signed by a custom CA (like mkcert), ensure that CA's root certificate is trusted by your macOS system. mkcert automates this, but manual verification is useful.

# List mkcert's installed root CAs (if mkcert is used)
mkcert -CAROOT
ls "$(mkcert -CAROOT)"

You can also check Keychain Access.app (under "System" or "login" keychains) for your custom root CA and ensure it's marked as "Always Trust".

While mkcert installs the CA into the macOS system trust store, Caddy running in a Docker container or a specific Go environment might not automatically inherit this. This step ensures the system trusts the CA, which is a prerequisite for some Caddy configurations to work correctly.

3. Configure Caddy to Trust the Backend Certificate

This is the most critical step. You have several options, ranging from most secure to least secure, depending on your environment.

Option A: Trust a Specific Custom CA Certificate (Recommended for Local Dev)

If your backend certificate is signed by a custom CA (e.g., mkcert's root CA), explicitly tell Caddy to trust that CA. This is secure because Caddy will only trust certificates signed by your specified CA.

  1. Locate your CA certificate: If using mkcert, it's typically in $(mkcert -CAROOT)/rootCA.pem.

    mkcert -CAROOT # This will output the path to your CA root
ROOT_CA_PATH=$(mkcert -CAROOT)/rootCA.pem
echo "Your CA certificate is at: $ROOT_CA_PATH"

  2. Add trusted_ca_certs to your Caddyfile: Modify your Caddyfile for the reverse_proxy directive.

    mylocalservice.test {
    reverse_proxy https://your-backend-host.local:8443 {
        transport http {
            tls_trusted_ca_certs /path/to/your/rootCA.pem
            # Example for mkcert: tls_trusted_ca_certs /Users/youruser/Library/Application Support/mkcert/rootCA.pem
        }
    }
}

    Replace /path/to/your/rootCA.pem with the actual path obtained above. If Caddy is in Docker, you'll need to mount this file into the container.

Option B: Insecurely Skip TLS Verification (Use with Extreme Caution)

This option disables certificate verification entirely. Only use this for strictly local, non-sensitive development environments where you fully understand the risks. Never use this in production.

mylocalservice.test {
    reverse_proxy https://your-backend-host.local:8443 {
        transport http {
            tls_insecure_skip_verify
        }
    }
}

tls_insecure_skip_verify bypasses all TLS certificate validation. This leaves your connection vulnerable to Man-in-the-Middle (MITM) attacks. Only use this for temporary debugging or highly controlled local development where security is not a concern.

4. Verify Hostname Match

Ensure the hostname Caddy is connecting to in the reverse_proxy directive exactly matches a Subject Alternative Name (SAN) or Common Name (CN) in the backend server's certificate.

If your backend's certificate is issued for backend.local, but you're proxying to 127.0.0.1:8443, Caddy will see a mismatch. You might need to specify the hostname within the transport http block.

mylocalservice.test {
    reverse_proxy https://127.0.0.1:8443 {
        transport http {
            # This tells Caddy to present 'backend.local' as the SNI hostname,
            # and to expect a certificate for 'backend.local'
            tls_servername backend.local
            tls_trusted_ca_certs /path/to/your/rootCA.pem # Or tls_insecure_skip_verify
        }
    }
}

In this example, backend.local is the hostname that the upstream server's certificate is issued for.

5. Check Certificate Expiry and Validity

If openssl s_client output showed an expired certificate:

  • Generate a new certificate for your backend server.
  • If using mkcert, ensure your local CA is still valid, and then re-generate the backend certificate using mkcert.

6. Reload/Restart Caddy

After making changes to your Caddyfile, you need to apply them.

For Caddy running directly (e.g., caddy run) or via brew services:

If you're running Caddy directly, you might need to stop and restart it. If caddy reload doesn't work, a full restart is usually necessary for TLS trust changes.

# If Caddy is running in the foreground, Ctrl+C to stop, then:
caddy run -config Caddyfile -adapter caddyfile

# If using brew services on macOS:
brew services restart caddy
For Caddy running as a Systemd service (e.g., on a Linux VM/Docker Desktop VM):
sudo systemctl reload caddy
# If reload doesn't work or for more significant changes:
sudo systemctl restart caddy
sudo systemctl status caddy
For Caddy running in Docker:
docker compose restart caddy-service # If using docker compose
# Or if running a single container:
docker restart <caddy-container-name-or-id>
docker logs <caddy-container-name-or-id>

When running Caddy in Docker and using tls_trusted_ca_certs, ensure that the /path/to/your/rootCA.pem file is properly mounted into the Caddy container at the specified path. For example, in docker-compose.yml:

services:
  caddy:
    image: caddy:latest
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./caddy_data:/data
      - /Users/youruser/Library/Application Support/mkcert/rootCA.pem:/etc/caddy/rootCA.pem:ro # Mount your CA
    # Ensure Caddy user can read the mounted CA file
    user: root # Or ensure file permissions are correct for caddy user

Adjust the host path for rootCA.pem to match your system.

By systematically working through these steps, you should be able to identify and resolve the "TLS handshake failed verification" error with Caddy acting as a reverse proxy on your macOS local development environment.