Caddy Reverse Proxy TLS Handshake Failed on Alpine Linux: Verification Error Troubleshooting Guide

Resolve Caddy reverse proxy 'TLS handshake failed verification' errors on Alpine Linux. Troubleshoot common issues like CA certificates, network, and clock skew.

Caddy is a powerful, automatic HTTPS web server and reverse proxy. When used as a reverse proxy, it establishes a separate TLS connection to your backend service. A "TLS handshake failed verification error" indicates that Caddy, running on Alpine Linux, encountered issues while trying to establish this secure connection to your upstream server. This guide provides a highly technical approach to diagnose and resolve this common, yet often elusive, problem.

Symptom & Error Signature

Users attempting to access a service proxied by Caddy will typically experience browser errors such as ERR_SSL_PROTOCOL_ERROR, SSL_ERROR_RX_RECORD_TOO_LONG, or simply a connection timeout. On the Caddy server itself, you will observe specific errors in your logs.

Access Caddy logs via journalctl (if running as a systemd service) or your container logs (if running via Docker):

# For Systemd installations
journalctl -u caddy.service -f

# For Docker containers
docker-compose logs -f caddy
# OR
docker logs -f <caddy_container_id>

Typical error signatures you might encounter:

{"level":"error","ts":"2023-10-27T10:30:00.123Z","logger":"http.log.error","msg":"x509: certificate signed by unknown authority","stacktrace":"go.uber.org/[email protected]/zapcore/entry.go:529ngo.uber.org/[email protected]/zapcore/entry.go:529ngithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/reverseproxy/reverseproxy.go:1210ngithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/reverseproxy/reverseproxy.go:1210n..."}

Or a more generic handshake failure:

{"level":"error","ts":"2023-10-27T10:30:05.456Z","logger":"http.log.error","msg":"remote error: tls: handshake failure","stacktrace":"go.uber.org/[email protected]/zapcore/entry.go:529ngithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/reverseproxy/reverseproxy.go:1210n..."}

The key message is often x509: certificate signed by unknown authority or remote error: tls: handshake failure.

Root Cause Analysis

The "TLS handshake failed verification error" typically occurs when Caddy, acting as a TLS client to your backend, cannot establish a secure, trusted connection. The underlying reasons can be diverse, but they generally fall into these categories:

1. Untrusted Certificate Authority (CA): This is the most prevalent issue. The certificate presented by your backend server is signed by a Certificate Authority that Caddy (running on Alpine Linux) does not implicitly trust. Alpine's minimal ca-certificates package might lack certain root CAs common in other distributions.
2. Invalid or Expired Backend Certificate: The certificate on your backend server itself might be expired, revoked, or have a hostname mismatch (the common name/SANs in the certificate do not match the hostname Caddy is trying to connect to).
3. Missing Intermediate Certificates: The backend server might not be sending its full certificate chain, leading to Caddy being unable to build a trusted path back to a root CA.
4. Time Skew: A significant difference in system time between the Caddy server and the backend server can cause certificate validity checks to fail, as certificates have specific validity periods.
5. Network/Firewall Interference: An intermediary device, firewall, or network policy might be inspecting, blocking, or altering TLS traffic between Caddy and the backend, causing the handshake to fail.
6. Backend TLS Misconfiguration: The backend server might be using deprecated TLS versions, unsupported cipher suites, or have other non-standard TLS configurations that Caddy's client-side TLS stack cannot negotiate.
7. Self-Signed Certificates: If your backend uses a self-signed certificate, Caddy will not trust it by default and requires explicit configuration.

Step-by-Step Resolution

Follow these steps to diagnose and resolve the TLS handshake failure.

1. Verify Backend Server's TLS Certificate

The first step is to ensure the backend server is presenting a valid and complete TLS certificate chain. Use openssl s_client from the Caddy host (or a network-adjacent diagnostic machine) to inspect the backend's certificate.

openssl s_client -connect backend.example.com:443 -showcerts -servername backend.example.com

Replace backend.example.com:443 with your actual backend hostname and port.

Key things to look for in the openssl output:

• Verify return code: 0 (ok): This indicates the certificate chain is valid and trusted by the system running the openssl command. If you see 18 (self signed certificate) or 21 (unable to verify the first certificate), this points to a trust issue.
• notBefore and notAfter dates: Ensure the certificate is within its validity period.
• Subject and Subject Alternative Name (SAN): Verify that the hostname Caddy is connecting to (e.g., backend.example.com) is listed in the certificate's Subject or SAN fields.
• Certificate Chain: Check that all intermediate certificates are present. If you only see one -----BEGIN CERTIFICATE----- block, the backend might not be sending the full chain.

If openssl s_client reports an x509 error or a non-zero verify return code, the problem likely lies with the backend server's certificate or the Caddy host's ability to trust its CA.

2. Update Alpine Linux CA Certificates

Alpine Linux is known for its minimal footprint, which extends to its default ca-certificates package. It might not contain all the root CAs found in larger distributions.

Ensure your ca-certificates package is installed and up-to-date:

apk update
apk add ca-certificates
update-ca-certificates

After updating, if Caddy is running directly on Alpine, restart it:

systemctl restart caddy.service # If using systemd
# Or if in Docker:
docker-compose restart caddy
# Or manually restart the container:
docker restart <caddy_container_id>

3. Configure Caddy to Trust Custom CA Certificates (If Necessary)

If your backend uses a private Certificate Authority (CA) or a self-signed certificate, Caddy needs to be explicitly configured to trust it.

1. Obtain the CA certificate: Get the .crt file for your private CA or the self-signed certificate of your backend.

2. Mount/Copy the CA certificate:

   • For Docker: Mount the CA certificate into your Caddy container (e.g., /etc/caddy/certs/my_private_ca.crt).

     # docker-compose.yml
     version: '3.8'
     services:
       caddy:
         image: caddy:2-alpine
         ports:
           - "80:80"
           - "443:443"
         volumes:
           - ./Caddyfile:/etc/caddy/Caddyfile
           - ./caddy_data:/data
           - ./certs/my_private_ca.crt:/etc/caddy/certs/my_private_ca.crt:ro # Mount your CA cert
         restart: unless-stopped
     

   • For Systemd/Direct Alpine: Place the certificate in a well-known location, e.g., /etc/ssl/certs/my_private_ca.crt.

3. Update Caddyfile: Use the tls_trusted_ca_certs option within your reverse_proxy block.

   # Caddyfile
   my.caddy.host {
       reverse_proxy https://backend.example.com {
           # Explicitly trust your custom CA certificate
           tls_trusted_ca_certs /etc/caddy/certs/my_private_ca.crt
       }
   }
   

   AVOID tls_insecure_skip_verify IN PRODUCTION. While tls_insecure_skip_verify would bypass verification:

   reverse_proxy https://backend.example.com {
       tls_insecure_skip_verify # DANGER ZONE! Only for dev/debug.
   }
   

   This option completely disables TLS certificate validation, making your connection to the backend vulnerable to Man-in-the-Middle (MITM) attacks. Only use this for temporary debugging in isolated development environments. Never use it in production.

After modifying the Caddyfile, reload or restart Caddy:

systemctl reload caddy.service # For systemd
# Or for Docker:
docker-compose restart caddy

4. Check System Time Synchronization (NTP)

Inaccurate system time can cause certificate validity checks to fail. Ensure your Alpine Linux host (where Caddy is running) has accurate time synchronization using NTP.

# Install NTP client on Alpine
apk add ntpsec-ntp

# Enable and start NTP service
rc-update add ntpsec-ntpd default
rc-service ntpsec-ntpd start

# Optionally, force a time sync and check status
ntpd -q -d

If Caddy is running in a Docker container, ensure the host machine's time is correct, as containers typically inherit time from the host.

5. Verify Network Connectivity and Firewall Rules

Ensure there are no network restrictions preventing Caddy from connecting to your backend on its HTTPS port (typically 443).

From the Alpine host running Caddy:

# Test connectivity to the backend's HTTPS port
nc -vz backend.example.com 443

# Expected output for success:
# Connection to backend.example.com 443 port [tcp/https] succeeded!

• Check Firewalls: Review iptables, ufw, or firewalld rules on both the Caddy server and the backend server. Ensure traffic on the backend's HTTPS port (e.g., 443) is permitted from the Caddy server's IP address.
• Security Groups/ACLs: If you're in a cloud environment (AWS, Azure, GCP), verify security groups or network ACLs allow the necessary traffic.

6. Examine Backend Server's TLS Configuration

Sometimes, the issue is not Caddy but the backend itself. The backend might be serving:

• Weak/Deprecated Cipher Suites: Caddy's default client TLS configuration is quite modern. If the backend only offers very old or insecure cipher suites, Caddy might refuse to connect.
• Unsupported TLS Protocols: An extremely old backend might not support TLS 1.2 or 1.3, which Caddy prefers.
• Incomplete Chain: As noted in Step 1, the backend might not be sending its full certificate chain, which is required for Caddy to verify the certificate.

Use tools like testssl.sh (for internal servers) or ssllabs.com/ssltest/ (for publicly accessible backends) to perform a comprehensive analysis of your backend's TLS configuration.

7. Caddyfile TLS Configuration for reverse_proxy Transport

While Caddy typically infers TLS for https:// backends, you can explicitly configure the transport block within reverse_proxy for more control. This is less common for "TLS handshake failed" but can be useful for advanced debugging or specific requirements.

# Caddyfile
my.caddy.host {
    reverse_proxy backend.example.com:443 {
        transport http {
            # Explicitly enable TLS for the upstream connection
            tls
            # Optionally set min/max TLS versions for upstream, e.g.:
            # tls_client_auth {
            #     # ... client certificate for backend
            # }
        }
        # Other reverse_proxy options...
    }
}

This configuration ensures Caddy uses TLS for the backend connection. The tls directive under transport http is usually implied by https://backend.example.com but being explicit can clarify intent.

8. Caddy Server Reload/Restart

After any configuration changes, ensure Caddy picks them up.

# For systemd installations
systemctl reload caddy.service
# If a full restart is needed (e.g., for major changes or if reload fails)
systemctl restart caddy.service

# For Docker containers
docker-compose restart caddy
# Or
docker restart <caddy_container_id>

Continuously monitor Caddy's logs after each change (journalctl -u caddy.service -f or docker-compose logs -f caddy) to observe if the error signature changes or disappears.

By systematically working through these steps, you should be able to pinpoint and resolve the "Caddy reverse proxy TLS handshake failed verification error" on your Alpine Linux environment.