Resolve Caddy's 'TLS handshake failed verification' error on Debian 12 Bookworm. This guide covers common causes like untrusted certificates, expired chains, and misconfigurations.

Caddy Reverse Proxy: Troubleshooting TLS Handshake Failed Verification on Debian 12 Bookworm

As an expert Systems Administrator, encountering TLS handshake failures with a reverse proxy like Caddy is a critical issue that immediately impacts service availability. When Caddy, acting as a reverse proxy, attempts to establish a secure (HTTPS) connection to an upstream backend server, a "TLS handshake failed verification" error signifies that Caddy cannot trust or validate the certificate presented by the upstream server. This guide provides a highly technical, step-by-step approach to diagnose and resolve this common problem on Debian 12 (Bookworm) systems.

Symptom & Error Signature

Users attempting to access your service through Caddy will typically experience a 502 Bad Gateway error in their browser. On the Caddy server itself, checking the Caddy service logs will reveal the underlying TLS verification failure.

Typical Browser Output: A 502 Bad Gateway page, often indicating the reverse proxy could not connect to the upstream server.

Caddy Service Log Output: You can inspect Caddy's logs using journalctl:

sudo journalctl -u caddy -f

Look for log entries similar to these:

{
  "level": "error",
  "ts": 1678886400.0,
  "logger": "http.log.error",
  "msg": "upstream connect error or disconnect/reset before headers. retried and failed twice. last upstream request: {"remote_addr":"127.0.0.1:53940","proto":"HTTP/2.0","method":"GET","host":"myapp.example.com","uri":"/","headers":{"Accept":["text/html,..."],"User-Agent":["Mozilla/5.0..."]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":""}}",
  "error": "remote error: tls: handshake failure",
  "stacktrace": "..."
}

Or, more specifically indicating a verification failure:

{
  "level": "error",
  "ts": 1678886401.5,
  "logger": "http.handlers.reverse_proxy",
  "msg": "aborting with incomplete response",
  "error": "tls: failed to verify certificate: x509: certificate signed by unknown authority",
  "stacktrace": "..."
}

Other common x509 errors you might see:

• x509: certificate has expired or is not yet valid
• x509: certificate is not valid for any requested host
• x509: cannot verify certificate: host, IP, or DNS name does not match any of the subject alternative names in the certificate

Root Cause Analysis

The "TLS handshake failed verification" error indicates that Caddy could not establish a trusted secure connection to your upstream (backend) server. This typically stems from issues with the upstream server's TLS certificate or how it's presented.

The most common root causes include:

1. Untrusted Certificate Authority (CA):

   • The upstream server uses a self-signed certificate.
   • The certificate is issued by a private or internal Certificate Authority (CA) that is not recognized or trusted by the Debian 12 system where Caddy is running.
   • The certificate is issued by a public CA, but its root or intermediate certificates are not present in the Caddy server's trust store.

2. Expired or Invalid Certificate:

   • The upstream server's certificate has expired or its validity period has not yet begun.
   • The system clock on the Caddy server is significantly out of sync with reality (time skew), causing certificate validity checks to fail prematurely.

3. Hostname Mismatch (Common Name/Subject Alternative Name – SAN):

   • The hostname Caddy is using to connect to the upstream server (e.g., backend.internal) does not match the Common Name (CN) or any Subject Alternative Names (SAN) listed in the upstream server's certificate.

4. Incomplete Certificate Chain:

   • The upstream server is not sending its full certificate chain, meaning it's missing intermediate certificates required for Caddy to build a trusted path back to a root CA it already trusts.

5. Firewall or Network Interruption:

   • While less likely to specifically cause a verification error, network issues can lead to an incomplete handshake, which might manifest as a general TLS failure. Ensure Caddy can reach the upstream server on its HTTPS port.

Step-by-Step Resolution

Follow these steps meticulously to diagnose and resolve the TLS handshake verification error.

1. Inspect the Upstream Server's TLS Certificate

The first crucial step is to directly inspect the certificate presented by your upstream server. This will reveal if it's expired, self-signed, missing intermediates, or has a hostname mismatch.

Replace your-upstream-host.com and 443 with the actual hostname/IP and port of your backend server.

# Install openssl if not already present
sudo apt update
sudo apt install openssl -y

# Inspect the certificate details (validity, subject, issuer, SANs)
openssl s_client -connect your-upstream-host.com:443 -servername your-upstream-host.com </dev/null 2>/dev/null | openssl x509 -noout -text

Carefully examine the output for:

• Not Before and Not After: Ensure the certificate is currently valid.
• Subject: This is the identity of the certificate holder.
• Issuer: This indicates who signed the certificate. If it's the same as the Subject, it's likely self-signed.
• X509v3 Subject Alternative Name: Check if the hostname Caddy is using to connect is listed here.

Next, check the full certificate chain and verification status:

openssl s_client -connect your-upstream-host.com:443 -servername your-upstream-host.com -showcerts

Look for:

• Verify return code: 0 (ok): This indicates that openssl on your Caddy server can successfully verify the certificate. If you see this, the issue might be Caddy-specific configuration rather than a trust store problem.
• Verify return code: 20 (unable to get local issuer certificate): This typically means a self-signed certificate or a private CA whose root is not in your system's trust store.
• Verify return code: 21 (unable to verify the first certificate): Often indicates missing intermediate certificates in the chain sent by the upstream.

2. Synchronize System Clocks

An out-of-sync system clock can cause certificates to appear expired or not yet valid. Ensure your Debian 12 server's time is accurate.

# Check current time status
timedatectl

# Enable NTP synchronization if not already active
sudo timedatectl set-ntp true

# Verify status again
timedatectl

The NTP service: active status confirms synchronization.

3. Trust the Upstream Certificate Authority (CA)

If openssl s_client indicated an untrusted issuer (e.g., Verify return code: 20) because of a self-signed certificate or an internal CA, you need to add the upstream's CA certificate to your Debian system's trust store.

Only trust CAs from sources you explicitly control or verify. Adding arbitrary CA certificates can introduce security risks.

Steps to Add a Custom CA:

1. Obtain the Upstream CA Certificate: You need the public certificate of the Certificate Authority that signed your upstream server's certificate. This is not the server's leaf certificate, but the CA's certificate. You can often extract it from the openssl s_client -showcerts output. Look for the certificate in the chain whose Subject matches the Issuer of your server's certificate, or the root CA itself. Copy the entire block including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----. Alternatively, the administrator of the upstream service should provide you with their CA's public certificate (e.g., internal_ca.crt).

2. Place the CA Certificate in the Trust Store: Create a new file for your CA certificate in /usr/local/share/ca-certificates/. Use a .crt extension.

   # Example: Create a new file for your internal CA
   sudo nano /usr/local/share/ca-certificates/my-internal-ca.crt
   

   Paste the entire content of your CA certificate into this file and save it.

3. Update the System CA Trust Store: Run the update-ca-certificates command to integrate your new CA into the system's trusted list.

   sudo update-ca-certificates
   

   You should see output indicating that your certificate was added: Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done.

4. Verify the new CA is trusted (Optional): You can re-run the openssl s_client command from Step 1. It should now show Verify return code: 0 (ok).

4. Address Hostname Mismatch (If Applicable)

If openssl x509 -text showed that the certificate's Subject Alternative Names do not include the hostname Caddy is using to connect to the upstream, you have two primary options:

1. Correct the Upstream Certificate: The ideal solution is to regenerate the upstream server's certificate to include the correct hostname(s) in its SANs. This requires intervention on the backend server.

2. Adjust Caddy's Upstream Host: Configure Caddy to connect to the upstream using a hostname that is valid according to the upstream's certificate. This might involve updating your /etc/hosts file on the Caddy server or adjusting Caddy's reverse_proxy directive:

   # Caddyfile example:
   your.caddy.domain {
       reverse_proxy https://valid-upstream-cert-hostname.com:443 {
           # Ensure Caddy sends the correct SNI for the upstream
           # Caddy usually handles this well by default using the upstream hostname
       }
   }
   

   If your reverse_proxy target is an IP address (e.g., https://192.168.1.100:8443), Caddy will not send an SNI hostname during the TLS handshake, and the upstream server might serve a default certificate or reject the connection if it relies on SNI. Always use a hostname in the reverse_proxy directive if the upstream server expects a specific hostname for its certificate.

5. Bypass Upstream Certificate Verification (Use with Extreme Caution!)

As a last resort, or for specific development/testing environments, Caddy can be configured to ignore upstream certificate verification errors.

Using tls_insecure_skip_verify DISABLES all TLS certificate verification for the upstream connection. This means Caddy will connect to ANY server presenting ANY certificate on the upstream, making your connection vulnerable to Man-in-the-Middle attacks. DO NOT USE THIS IN PRODUCTION unless you fully understand and accept the security implications.

To enable this, modify your Caddyfile:

your.caddy.domain {
    reverse_proxy https://your-upstream-host.com:443 {
        # DANGER: Disables upstream certificate validation
        tls_insecure_skip_verify
    }
}

6. Restart Caddy Service and Monitor Logs

After making any changes to your Caddyfile or system trust store, restart the Caddy service and monitor its logs for new errors.

# Test Caddyfile configuration for syntax errors
sudo caddy validate --config /etc/caddy/Caddyfile

# Restart Caddy service
sudo systemctl restart caddy

# Check service status
sudo systemctl status caddy

# Monitor logs for successful connection or new errors
sudo journalctl -u caddy -f

If the issue persists, carefully review the new log entries. They might point to a different problem or provide more specific details about the TLS handshake failure.

By systematically applying these troubleshooting steps, you should be able to identify and resolve the "Caddy reverse proxy TLS handshake failed verification" error, restoring secure and reliable access to your backend services.