Caddy Reverse Proxy TLS Handshake Failed Verification Error on Ubuntu 22.04 LTS

Troubleshoot 'TLS handshake failed verification' errors with Caddy reverse proxies on Ubuntu 22.04 LTS. Resolve certificate trust, expiry, and hostname issues effectively.

Caddy Reverse Proxy TLS Handshake Failed Verification Error on Ubuntu 22.04 LTS

When Caddy is configured as a reverse proxy to an upstream backend service over HTTPS, encountering a "TLS handshake failed verification" error means Caddy, acting as a TLS client, failed to validate the certificate presented by the backend server. This typically results in a broken connection, preventing Caddy from forwarding requests to your application. Users accessing your Caddy frontend will likely see a 502 Bad Gateway error.

Symptom & Error Signature

Users accessing the Caddy-proxied domain will typically see:

502 Bad Gateway

On the server hosting Caddy, inspecting the Caddy service logs will reveal the underlying issue. You can check these logs using journalctl:

sudo journalctl -u caddy.service --since "10 minutes ago" -f

Typical error signatures you might encounter include:

{"level":"error","ts":1678886400.000000,"logger":"http.reverse_proxy","msg":"upstream roundtrip: reverse proxy: upstream TLS handshake failed: x509: certificate signed by unknown authority"}
{"level":"error","ts":1678886400.000000,"logger":"http.reverse_proxy","msg":"upstream roundtrip: reverse proxy: upstream TLS handshake failed: x509: certificate is not valid for any requested host"}
{"level":"error","ts":1678886400.000000,"logger":"http.reverse_proxy","msg":"upstream roundtrip: reverse proxy: upstream TLS handshake failed: x509: certificate has expired or is not yet valid"}
{"level":"error","ts":1678886400.000000,"logger":"http.reverse_proxy","msg":"upstream roundtrip: reverse proxy: upstream TLS handshake failed: remote error: tls: handshake failure"}

Root Cause Analysis

The "TLS handshake failed verification" error signifies that Caddy could not establish a secure, trusted connection to your upstream backend. The primary reasons for this verification failure are:

1. Untrusted Certificate Authority (CA): The most common cause. The backend server's TLS certificate is either self-signed or issued by a Private Certificate Authority (CA) that Caddy's underlying system (or Caddy itself) does not implicitly trust. Caddy, by default, relies on the operating system's trusted CA store.
2. Hostname Mismatch: The hostname Caddy is using to connect to the upstream server (e.g., mybackend.local or 192.168.1.100) does not match the Common Name (CN) or any Subject Alternative Name (SAN) listed in the backend's TLS certificate.
3. Expired or Not Yet Valid Certificate: The backend server's certificate has either passed its expiration date or its validity period has not yet begun.
4. Incomplete Certificate Chain: The backend server is not sending its full certificate chain, meaning intermediate certificates required to establish trust back to a root CA are missing. Caddy cannot build a complete trust path.
5. Client Certificate Requirement (Less common): The upstream backend server is configured to require Caddy to present a client certificate for mutual TLS authentication, but Caddy is not configured to do so.

Step-by-Step Resolution

Follow these steps to diagnose and resolve the Caddy TLS handshake verification error.

1. Inspect Caddy Logs for Specific Error Details

Start by thoroughly examining the Caddy service logs. The exact x509 error message provides crucial clues.

sudo journalctl -u caddy.service --since "30 minutes ago" -f

Look for lines containing upstream TLS handshake failed: followed by specific x509 errors. This will guide your troubleshooting path.

2. Verify Upstream Backend Certificate Status

Use openssl s_client from the Caddy server to directly inspect the backend's certificate. This simulates Caddy's attempt to connect.

Replace <UPSTREAM_HOST> with the actual hostname or IP Caddy uses to connect to your backend, and <UPSTREAM_PORT> with the HTTPS port (e.g., 443, 8443).

openssl s_client -connect <UPSTREAM_HOST>:<UPSTREAM_PORT> -servername <UPSTREAM_HOST> < /dev/null 2>/dev/null | openssl x509 -text -noout

Key things to look for in the output:

• Issuer:: Is it a known CA (e.g., Let's Encrypt, DigiCert) or a private/self-signed one?
• Validity: Not Before and Not After dates. Is the certificate currently valid?
• Subject:: Does the CN (Common Name) match UPSTREAM_HOST?
• X509v3 Subject Alternative Name:: Does this list include UPSTREAM_HOST or its IP address?
• Chain completeness: You can also use openssl s_client -showcerts -connect <UPSTREAM_HOST>:<UPSTREAM_PORT> to see if the full chain is sent. Look for multiple certificates, ideally ending with a trusted root.

3. Address Untrusted CA / Self-Signed Certificates

If openssl s_client indicates Verify return code: 21 (unable to verify the first certificate) or similar, it means the certificate's issuer is not trusted.

Option A: Add Upstream CA to System Trust Store (Recommended for Production Backends)

This is the most secure and robust solution, making the CA trusted system-wide.

1. Retrieve the Upstream CA Certificate: If your backend uses a self-signed certificate, or a custom private CA, you'll need its public certificate. If the backend is accessible, you can try to extract it:

   openssl s_client -showcerts -connect <UPSTREAM_HOST>:<UPSTREAM_PORT> < /dev/null 2>/dev/null | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{print $0}' > /tmp/upstream_backend_cert.pem
   

   Inspect /tmp/upstream_backend_cert.pem. If it's a self-signed cert, this is the cert you need to trust. If it's issued by a private CA, you'll need the CA's public certificate, not the backend's leaf certificate itself. Often, this means obtaining the root.pem or intermediate.pem file from your private CA.

   Ensure you obtain the Root CA certificate or the Intermediate CA certificate that signed your backend's certificate, not just the backend's leaf certificate itself (unless it's self-signed). Placing the leaf cert into ca-certificates is generally incorrect for CA trust.

2. Copy the CA Certificate: Move your CA certificate file (e.g., private_root_ca.crt) to the system's CA certificates directory.

   sudo cp /path/to/your/private_root_ca.crt /usr/local/share/ca-certificates/
   

3. Update System Trust Store: Tell the system to update its list of trusted certificates.

   sudo update-ca-certificates
   

   You should see output indicating your certificate was added.

4. Restart Caddy Service:

   sudo systemctl restart caddy
   sudo systemctl status caddy
   sudo journalctl -u caddy.service -f
   

   Monitor the logs to confirm the error is resolved.

Option B: Configure Caddy to Trust Specific Certificates (Per-Backend Basis)

If you only want Caddy to trust a specific CA or self-signed certificate for one particular upstream, you can configure this directly in your Caddyfile.

1. Place the Certificate: Copy the public CA certificate (or self-signed certificate) for your upstream backend to a path accessible by Caddy (e.g., /etc/caddy/certs/upstream_ca.pem).

2. Modify Caddyfile: Add the tls_trusted_ca_certs option to your reverse_proxy directive.

   your.domain {
       reverse_proxy https://<UPSTREAM_HOST>:<UPSTREAM_PORT> {
           transport http {
               tls_trusted_ca_certs /etc/caddy/certs/upstream_ca.pem
           }
       }
   }
   

3. Apply Caddyfile Changes:

   sudo systemctl reload caddy
   # Or for a full restart if reload doesn't pick up changes:
   # sudo systemctl restart caddy
   sudo journalctl -u caddy.service -f
   

Option C: Disable TLS Verification (Use with Extreme Caution!)

Disabling TLS verification (tls_insecure_skip_verify) bypasses a fundamental security mechanism and makes your connection vulnerable to Man-in-the-Middle (MITM) attacks. Only use this for development, testing, or in highly controlled, isolated internal networks where you fully trust all network components and data being transmitted. It is NOT recommended for production environments or over public networks.

To disable verification, add tls_insecure_skip_verify to your reverse_proxy block:

your.domain {
    reverse_proxy https://<UPSTREAM_HOST>:<UPSTREAM_PORT> {
        transport http {
            tls_insecure_skip_verify
        }
    }
}

Then reload/restart Caddy:

sudo systemctl reload caddy
sudo journalctl -u caddy.service -f

4. Resolve Hostname Mismatch Issues

If the openssl s_client output showed that the certificate's CN/SAN does not match the hostname Caddy is trying to connect to, you have a hostname mismatch.

1. Correct Caddyfile Upstream Address: Ensure the hostname used in your reverse_proxy directive matches a CN or SAN entry in the backend's certificate.

   # If the certificate is for "mybackend.local"
   your.domain {
       reverse_proxy https://mybackend.local:8443
   }
   
   # If the certificate is for "192.168.1.100" and you're using it in reverse_proxy
   your.domain {
       reverse_proxy https://192.168.1.100:8443 {
           # You might still need to specify the server_name if the cert expects a hostname
           # transport http {
           #    tls_server_name mybackend.local
           # }
       }
   }
   

2. Update DNS//etc/hosts: If you're using a hostname (e.g., mybackend.local), ensure that Caddy's server can correctly resolve this hostname to the backend's IP address. You might need to add an entry to /etc/hosts on the Caddy server:

   # Example /etc/hosts entry on Caddy server
   192.168.1.100   mybackend.local
   

3. Use tls_server_name in Caddyfile: If Caddy connects by IP address but the certificate is only valid for a specific hostname, you can instruct Caddy to present that hostname during the TLS handshake.

   your.domain {
       reverse_proxy https://192.168.1.100:8443 {
           transport http {
               tls_server_name mybackend.local # This is the hostname the cert expects
           }
       }
   }
   

4. Reload Caddy:

   sudo systemctl reload caddy
   sudo journalctl -u caddy.service -f
   

5. Renew or Fix Expired/Invalid Certificates

If openssl s_client shows the certificate is expired (Not After date passed) or not yet valid (Not Before date not reached), the issue is with the backend server's certificate.

This issue must be resolved on the upstream backend server, not on the Caddy server.

• For Let's Encrypt certificates: Ensure Certbot (or your ACME client) is properly installed, configured, and its renewal cron job is running correctly on the backend server. Manually trigger a renewal if needed:

  sudo certbot renew --force-renewal
  sudo systemctl restart <backend_webserver_service> # e.g., nginx, apache2
  

• For privately issued or self-signed certificates: Regenerate and redeploy a new certificate with a valid date range on your backend server.

After renewing/fixing the backend certificate, restart the backend web server and then restart Caddy to ensure it picks up the changes.

6. Ensure Complete Certificate Chain on Upstream

If openssl s_client -showcerts shows only one certificate (the leaf certificate) and not the intermediate CA certificates, the backend server's configuration might be incomplete. The full chain is needed for clients to establish trust.

This issue must be resolved on the upstream backend server.

• Nginx: Ensure your ssl_certificate directive points to a file that contains both your server certificate and any intermediate certificates, usually concatenated.

  ssl_certificate /etc/nginx/ssl/your_domain_bundle.crt; # This file should contain server cert + intermediates
  ssl_certificate_key /etc/nginx/ssl/your_domain.key;
  

• Apache: Ensure SSLCertificateFile points to your server certificate and SSLCertificateChainFile (for older versions) or SSLCertificateFile (for newer versions, with bundle) includes intermediates.

  SSLCertificateFile /etc/apache2/ssl/your_domain.crt
  SSLCertificateChainFile /etc/apache2/ssl/your_intermediate.crt # Or bundle in SSLCertificateFile
  SSLCertificateKeyFile /etc/apache2/ssl/your_domain.key
  

After correcting the chain on the backend, restart the backend web server and then Caddy.

7. Restart Caddy Service

After making any configuration changes, always restart Caddy and monitor its logs.

sudo systemctl restart caddy
sudo systemctl status caddy
sudo journalctl -u caddy.service -f

8. Check Firewall Rules

While less likely to cause a "TLS handshake failed verification" specifically (as this implies the handshake started), ensure that the Caddy server's firewall (UFW, iptables) allows outgoing connections to the upstream backend's HTTPS port, and the backend's firewall allows incoming connections on that port from the Caddy server.

# On Caddy server:
sudo ufw status # Check outgoing rules if UFW is configured for them

# On Backend server:
sudo ufw status # Ensure port <UPSTREAM_PORT> is open to Caddy's IP

By systematically working through these steps, you should be able to identify and resolve the root cause of your Caddy reverse proxy TLS handshake verification error.