Caddy Reverse Proxy TLS Handshake Failed Verification on CentOS Stream / Rocky Linux

Resolve Caddy reverse proxy errors where TLS handshake to the backend fails validation. Troubleshoot common causes like untrusted certificates or misconfigurations on CentOS Stream / Rocky Linux.

As an expert Systems Administrator, encountering a "TLS handshake failed verification" error with Caddy acting as a reverse proxy indicates a critical communication breakdown between Caddy and its backend server. When Caddy is configured to forward requests to a secure (HTTPS) upstream, it acts as a client and must successfully establish a trusted TLS connection with that backend. This guide will walk you through diagnosing and resolving this common issue on CentOS Stream and Rocky Linux environments.

Symptom & Error Signature

Users attempting to access your Caddy-fronted service will typically encounter a 502 Bad Gateway error in their browser, or Caddy's custom error page if configured. The root cause will be visible in Caddy's logs, indicating that it could not establish a secure connection to the backend.

Here's what you'll typically see in your system logs when Caddy encounters this issue:

{
  "level": "error",
  "ts": "2023-10-27T10:30:45.123Z",
  "logger": "http.log.error",
  "msg": "x509: certificate signed by unknown authority",
  "stack": "github.com/caddyserver/caddy/v2/modules/caddyhttp.errCtx.ServeHTTP (caddyhttp.go:217)ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP (reverseproxy.go:123)ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.reverseproxy.roundTrip (reverseproxy.go:107)ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.reverseproxy.ServeHTTP (reverseproxy.go:107)ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.reverseproxy.ServeHTTP (reverseproxy.go:107)ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP (reverseproxy.go:123)n...n"
}

Other common variations might include:

2023/10/27 10:30:45 [ERROR] http.log.error: dial tcp 192.168.1.100:8443: x509: certificate is not valid for any of the following names: backend.example.com, localhost, 127.0.0.1

2023/10/27 10:30:45 [ERROR] http.log.error: remote error: tls: handshake failure

Root Cause Analysis

A "TLS handshake failed verification" error, particularly "x509: certificate signed by unknown authority" or "certificate is not valid", usually points to one of the following underlying issues:

1. Untrusted Backend Certificate: The most frequent cause. Caddy, acting as a client, does not trust the Certificate Authority (CA) that issued the backend server's TLS certificate. This often happens with:
   • Self-signed certificates on the backend.
   • Certificates issued by a private or internal CA whose root certificate is not present in the Caddy server's trusted CA store.
   • Development/testing certificates that are not publicly trusted.
2. Invalid Backend Certificate:
   • Expired Certificate: The backend server's certificate has passed its validity date.
   • Hostname Mismatch: The hostname Caddy is trying to connect to (e.g., backend.example.com) does not match the Common Name (CN) or any Subject Alternative Name (SAN) listed in the backend server's certificate.
   • Incomplete Certificate Chain: The backend server is not sending its full certificate chain, missing intermediate certificates that are necessary for Caddy to build a trusted path to a root CA it does trust.
3. Caddyfile Misconfiguration:
   • The reverse_proxy directive points to an incorrect hostname or IP address for the backend, leading to a hostname mismatch against the certificate.
   • Missing or incorrect tls_insecure_skip_verify (if intending to bypass validation).
4. System Trust Store Issues: The operating system (CentOS Stream / Rocky Linux) on which Caddy is running does not have the necessary root or intermediate CA certificates installed and trusted, which Caddy relies upon.
5. System Time Skew: Although less common, a significant time difference between the Caddy server and the backend server (or the CA that issued the cert) can cause certificate validity checks to fail.

Step-by-Step Resolution

Follow these steps to diagnose and resolve the TLS handshake verification error.

1. Analyze Caddy Logs

The first step is always to check the detailed Caddy logs to pinpoint the exact error message.

sudo journalctl -u caddy.service --since "5 minutes ago" -f

Look for messages containing x509, certificate, TLS handshake, or remote error. The error message will often tell you if it's an "unknown authority", "expired", or "hostname mismatch".

2. Verify Backend TLS Certificate Manually

From your Caddy server, use openssl s_client to connect to the backend and manually inspect its certificate. This will help determine if the certificate itself is the problem.

Replace backend.example.com and 443 with your actual backend's hostname/IP and port.

openssl s_client -connect backend.example.com:443 -showcerts

Analyze the output:

• Verify return code: 0 (ok): Indicates the certificate chain is trusted by OpenSSL's default trust store. If you see Verify return code: 21 (unable to verify the first certificate) or similar, it means the certificate is not trusted.
• subject: Check the CN= (Common Name) and X509v3 Subject Alternative Name fields. Does your Caddyfile's reverse_proxy target exactly match one of these names?
• notBefore / notAfter: Check the validity dates. Is the certificate expired?
• Issuer: Who issued the certificate? If it's your own private CA or C=US, O=Self-Signed, it's untrusted by default.
• Certificate Chain: Ensure all certificates (root, intermediates, and leaf) are present in the output. A missing intermediate can cause unable to get local issuer certificate.

3. Inspect Caddyfile Configuration

Review your Caddyfile for any misconfigurations related to the reverse_proxy directive.

your.frontend.com {
    reverse_proxy https://backend.example.com:8443 {
        # Optional: For hostname mismatch issues,
        # set the Host header to match the certificate's CN/SAN
        header_up Host {http.request.host}

        # Optional: Only use this as a temporary measure or in controlled environments
        # where you fully understand the security implications.
        # This completely disables TLS certificate verification.
        # tls_insecure_skip_verify
    }
}

• Target Hostname/IP: Ensure https://backend.example.com:8443 correctly points to your backend. The hostname backend.example.com must match the CN or SAN in the backend's TLS certificate. If your backend uses an IP address for its certificate, you must use the IP address in your Caddyfile, or ensure the IP is listed as a SAN.
• tls_insecure_skip_verify: If you must connect to a backend with a self-signed or otherwise untrusted certificate and cannot add it to the system trust store (e.g., for development/testing), you can use this option.

Using tls_insecure_skip_verify disables all TLS certificate verification for that upstream. This is a significant security risk and should only be used in highly controlled environments or for temporary debugging. Never use this in production for critical services unless absolutely necessary and consequences are fully understood. It exposes your Caddy instance to man-in-the-middle attacks.

4. Establish Trust for Custom/Private CAs (CentOS Stream / Rocky Linux)

If your backend uses a self-signed certificate or one issued by a private/internal CA, Caddy (and the underlying Go runtime) needs to be able to trust that CA. You achieve this by adding the CA's root certificate to the operating system's trust store.

1. Obtain the CA Certificate: Make sure you have the public root certificate of the CA that signed your backend server's certificate. This should be a .crt or .pem file. If you only have the backend certificate, you might need to extract the CA from it or obtain it from your CA administrator.

   If you used openssl s_client in step 2 and saw an "unknown authority" error, the certificate chain output from openssl will include the issuer. You'll need the root CA certificate that signed the intermediate CA, or the self-signed certificate itself.

2. Copy the CA Certificate: Place the CA certificate file(s) into the /etc/pki/ca-trust/source/anchors/ directory. For example, if your CA certificate is my-internal-ca.crt:

   sudo cp /path/to/my-internal-ca.crt /etc/pki/ca-trust/source/anchors/
   

3. Update the System Trust Store: Run update-ca-trust to integrate the new certificate(s) into the system's trust store.

   sudo update-ca-trust extract
   

   This command updates the /etc/pki/tls/certs/ca-bundle.crt file, which is used by many applications, including Caddy.

4. Restart Caddy: For Caddy to pick up the changes to the system trust store, it must be restarted.

   sudo systemctl restart caddy
   

   Check journalctl -u caddy.service again to confirm the error is gone.

5. Ensure Backend Server Presents Full Certificate Chain

If your openssl s_client output from step 2 indicated an incomplete chain (e.g., you see only the leaf certificate but not the intermediate ones), the issue lies with the backend server's configuration.

• For Nginx backend: Ensure ssl_certificate contains the full chain (leaf + intermediate) or ssl_trusted_certificate (or equivalent for older versions) is correctly configured if separating. The ssl_certificate directive in Nginx often takes the server certificate concatenated with the intermediate certificates.

  ssl_certificate /etc/nginx/certs/fullchain.pem; # Contains server cert + intermediate certs
  ssl_certificate_key /etc/nginx/certs/privkey.pem;
  

• For Apache backend: Ensure SSLCertificateFile points to the server certificate and SSLCertificateChainFile (or including intermediate in SSLCertificateFile for newer versions) points to the intermediate CA certificate(s).

  SSLCertificateFile      /etc/httpd/conf/ssl.crt/server.crt
  SSLCertificateKeyFile   /etc/httpd/conf/ssl.key/server.key
  SSLCertificateChainFile /etc/httpd/conf/ssl.crt/ca-bundle.crt
  

After correcting the backend's certificate configuration, restart the backend web server and then re-test from Caddy.

6. Check System Time Synchronization

While less common, significant time skew on either the Caddy server or the backend server can lead to certificate validation failures if certificates appear to be "not yet valid" or "expired" prematurely.

timedatectl

Ensure System clock synchronized: yes and NTP service: active. If not, configure NTP synchronization.

sudo timedatectl set-ntp true

7. Test and Restart Caddy

After making any changes (especially to the Caddyfile or trust store), always validate your Caddyfile syntax and restart the Caddy service.

1. Validate Caddyfile:

   sudo caddy validate --config /etc/caddy/Caddyfile
   

   (Adjust path if your Caddyfile is elsewhere).
2. Restart Caddy Service:

   sudo systemctl restart caddy
   

3. Verify Service Status:

   sudo systemctl status caddy
   

4. Check Logs for New Errors:

   sudo journalctl -u caddy.service -f
   

8. Verify DNS Resolution

Ensure that the Caddy server can correctly resolve the hostname of your backend server. An incorrect DNS resolution could lead Caddy to connect to the wrong server, resulting in a certificate mismatch.

dig backend.example.com

Or:

nslookup backend.example.com

Confirm that the IP address returned matches the intended backend server.

By systematically working through these steps, you should be able to identify and resolve the "Caddy reverse proxy TLS handshake failed verification" error on your CentOS Stream or Rocky Linux system. Remember to prioritize security by adding trusted CAs to the system store rather than using tls_insecure_skip_verify in production.