Caddy Reverse Proxy: Troubleshooting TLS Certificate Verification Failures & Handshake Errors

Resolve Caddy reverse proxy TLS certificate verification and handshake errors when connecting to upstream servers, covering common causes like self-signed certificates and untrusted CAs.

Caddy is an incredibly powerful and user-friendly web server and reverse proxy, known for its automatic HTTPS capabilities. However, when configuring Caddy to act as a reverse proxy to an upstream backend server that also uses HTTPS (e.g., an internal API, another web server, or a microservice), you might encounter TLS certificate verification failures or handshake errors. These issues prevent Caddy from establishing a secure connection to your backend, resulting in 502 Bad Gateway errors for your users or a broken application.

This guide will walk you through diagnosing and resolving these common TLS issues, focusing on scenarios where Caddy, as a client, cannot trust the backend server’s presented certificate.

Symptom & Error Signature

When Caddy fails to verify the TLS certificate of an upstream backend, end-users will typically see a 502 Bad Gateway error in their browser. In your Caddy logs, accessible via journalctl -u caddy (for systemd installations) or docker logs <caddy-container-name> (for Docker deployments), you’ll find entries similar to these:

{"level":"error","ts":1678886400.000000,"logger":"http.log.error","msg":"x509: certificate signed by unknown authority","request":{"remote_ip":"192.168.1.1","remote_port":"54321","proto":"HTTP/2.0","method":"GET","host":"your-domain.com","uri":"/app/","headers":{"User-Agent":["Mozilla/5.0..."],"Accept":["text/html,..."]}},"duration":0.000000,"status":502,"err_id":"some-uuid","err_trace":"reverseproxy.go:881"}
{"level":"error","ts":1678886400.000000,"logger":"http.reverseproxy","msg":"handshake error: x509: certificate signed by unknown authority","error":"x509: certificate signed by unknown authority"}
{"level":"error","ts":1678886400.000000,"logger":"http.reverseproxy","msg":"tls: failed to verify certificate: x509: certificate is not valid yet","error":"tls: failed to verify certificate: x509: certificate is not valid yet"}
{"level":"error","ts":1678886400.000000,"logger":"http.reverseproxy","msg":"tls: failed to verify certificate: x509: cannot validate certificate for 10.0.0.5 because it doesn't contain any IP SANs","error":"tls: failed to verify certificate: x509: cannot validate certificate for 10.0.0.5 because it doesn't contain any IP SANs"}

Common error messages include:

  • x509: certificate signed by unknown authority
  • tls: failed to verify certificate
  • x509: certificate is not valid yet or x509: certificate has expired
  • x509: cannot validate certificate for <hostname/IP> because it doesn't contain any IP SANs (Subject Alternative Names)

Root Cause Analysis

The “certificate verification fails” error means Caddy, acting as a TLS client, received a certificate from the backend server but could not trust it based on its configured trust stores and policies. Here are the most common underlying reasons:

  1. Untrusted Certificate Authority (CA): This is the most frequent cause.

    • The backend server uses a self-signed certificate.
    • The backend’s certificate is issued by a private or internal CA whose root certificate is not trusted by Caddy (or the underlying operating system).
    • The certificate is issued by a publicly trusted CA, but its intermediate certificates are not sent by the backend, preventing Caddy from building a complete trust chain.

  2. Expired or Not Yet Valid Certificate: The backend server’s certificate is outside its designated validity period. Caddy will correctly reject it.

  3. Hostname Mismatch: The hostname or IP address Caddy uses to connect to the backend (e.g., https://backend-service.internal) does not match any of the Subject Alternative Names (SANs) or the Common Name (CN) specified in the backend’s certificate.

  4. Time Skew: The system clock on the Caddy server is significantly out of sync with the actual time, leading to incorrect validation of certificate validity periods.

  5. Backend Misconfiguration: The backend server itself might be configured to present an incorrect certificate, or its TLS setup is otherwise flawed.

Step-by-Step Resolution

Follow these steps to diagnose and resolve Caddy’s TLS certificate verification issues with your backend.

1. Verify Backend Certificate Status

Before modifying Caddy, inspect the backend server’s certificate. This will reveal if the certificate itself is the problem.

[!NOTE] Replace backend.example.com with your backend’s hostname or IP, and 8443 with its HTTPS port. If your backend uses a hostname for its certificate but you’re connecting via IP, specify the hostname with -servername.

# Connect to the backend and inspect its certificate
openssl s_client -connect backend.example.com:8443 -showcerts -servername backend.example.com

Carefully examine the output:

  • Verify return code: 0 (ok): Indicates the certificate is trusted by your current system’s OpenSSL (which might differ from Caddy’s trust store depending on installation). If it’s not 0 (ok), you have a clear indication of a trust issue.
  • subject= and X509v3 Subject Alternative Name:: Check if the hostname/IP you’re connecting to is listed here.
  • Not Before: and Not After:: Ensure the current date falls within this validity window.
  • Issuer:: Note the issuer. If it’s your own domain, it’s likely self-signed or from an internal CA.
  • The presented certificate chain (multiple ---BEGIN CERTIFICATE--- blocks).

2. Install/Trust Custom CA or Self-Signed Certificate

If openssl s_client shows Verify return code: 21 (unable to verify the first certificate) or 18 (self signed certificate), you need to make Caddy trust the backend’s certificate or its issuing CA.

Method A: System-wide Trust (Recommended for internal CA roots)

If your backend’s certificate is signed by an internal Certificate Authority, add that CA’s root certificate to the operating system’s trust store. Caddy, when installed via systemd, typically respects this.

  1. Obtain the CA Root Certificate:

    • If you have the .crt file for your internal CA’s root certificate (e.g., my-internal-ca-root.crt), copy it to the Caddy server.
    • If your backend is self-signed, you can extract its server certificate (the first one in openssl s_client -showcerts output) and use it as the “CA” for itself. Save the first ---BEGIN CERTIFICATE--- to backend-self-signed.crt.

  2. Add to System Trust Store:

    sudo cp /path/to/my-internal-ca-root.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

  3. Restart Caddy:

    sudo systemctl reload caddy
# If issues persist, try a full restart:
# sudo systemctl restart caddy

Method B: Caddy-specific Trust (For specific upstream certificates/CA)

Caddy allows you to specify trusted CA certificates directly within your Caddyfile for specific upstream reverse proxies. This is often cleaner for isolated scenarios or when you don’t want system-wide changes.

  1. Obtain the CA Root Certificate: Same as Method A (or the self-signed server cert). Place it in a location accessible by Caddy (e.g., /etc/caddy/certs/my-internal-ca-root.crt).

  2. Modify Caddyfile:

    your-domain.com {
    reverse_proxy https://backend.example.com:8443 {
        transport http {
            tls_trusted_ca_certs /etc/caddy/certs/my-internal-ca-root.crt
        }
    }
}

  3. Apply Caddyfile Changes:

    sudo caddy fmt --overwrite /etc/caddy/Caddyfile # Format your Caddyfile
sudo systemctl reload caddy

3. Handle Hostname Mismatch (SAN/CN Issues)

If the backend’s certificate doesn’t list the hostname or IP Caddy uses to connect, you have a hostname mismatch.

  1. Option A: Correct Caddy’s Backend Address: The simplest fix is to ensure the hostname Caddy uses in reverse_proxy matches a SAN or CN in the backend’s certificate.

    your-domain.com {
    # If backend.example.com is in the backend's certificate SANs
    reverse_proxy https://backend.example.com:8443
}

    If you’re connecting to an IP (e.g., 10.0.0.5) but the certificate is for a hostname (e.g., api.internal), you must connect via the hostname. If the certificate only has IP SANs and you’re connecting via hostname, connect via IP.

  2. Option B: Specify tls_server_name: If you must connect to a specific IP or hostname (e.g., 10.0.0.5) but the backend’s certificate is issued for a different hostname (e.g., api.internal), you can tell Caddy which Host header to send during the TLS handshake for SNI and certificate validation.

    your-domain.com {
    reverse_proxy https://10.0.0.5:8443 {
        transport http {
            tls_server_name api.internal
        }
    }
}

    [!IMPORTANT] tls_server_name affects the Host header sent during the TLS handshake (SNI) for certificate validation. It does not change the Host header sent in the HTTP request itself. If your backend relies on the HTTP Host header, you might also need header_up Host api.internal in the reverse_proxy block.

4. Address Expired or Not Yet Valid Certificates

If openssl s_client showed Not Before / Not After issues, the backend’s certificate is invalid by date.

  1. Action: The only proper solution is to renew the backend server’s certificate. This is a configuration task on the backend server itself.

  2. Temporary Workaround (USE WITH EXTREME CAUTION): For development, testing, or truly isolated internal networks where security is less critical, you can instruct Caddy to skip TLS certificate verification.

    [!WARNING] Do not use tls_insecure_skip_verify in production environments unless you fully understand the security implications. It disables crucial security checks, making your connection vulnerable to man-in-the-middle attacks.

    your-domain.com {
    reverse_proxy https://backend.example.com:8443 {
        transport http {
            tls_insecure_skip_verify
        }
    }
}

    After modifying, sudo systemctl reload caddy.

5. Ensure Full Certificate Chain is Sent by Backend

If openssl s_client shows missing intermediate certificates (only the leaf certificate is returned, and Verify return code is not 0 (ok) but indicates a chain issue), the backend server is misconfigured.

  1. Action: Configure the backend server to send its full certificate chain, including any intermediate CA certificates, along with its server certificate.
    • For Nginx backend: Ensure your ssl_certificate directive points to a file containing both your server certificate and all intermediate certificates (usually a fullchain.pem file). 
      # Example Nginx backend config
server {
    listen 443 ssl;
    server_name api.internal;

    ssl_certificate /etc/nginx/ssl/fullchain.pem; # Contains server cert + intermediates
    ssl_certificate_key /etc/nginx/ssl/private.key;
    # ...
}
    • Consult your backend server’s documentation for correct certificate chain configuration.

6. Synchronize System Time

If time skew is suspected as the cause for certificate is not valid yet or has expired messages, ensure your Caddy server’s time is accurate.

  1. Install/Verify NTP/Chrony: Most modern Linux distributions use systemd-timesyncd by default or you can install ntp or chrony. 
    # For systemd-timesyncd (common on Ubuntu 20.04+)
timedatectl status

# For NTP
sudo apt update && sudo apt install ntp
sudo systemctl status ntp
ntpq -p # Check NTP peer synchronization

# For Chrony
sudo apt update && sudo apt install chrony
sudo systemctl status chrony
chronyc sources -v # Check Chrony sources
  2. Restart Caddy after ensuring time synchronization: sudo systemctl restart caddy.

7. Debugging with Caddy Logs

If after all these steps, you are still experiencing issues, increase Caddy’s log verbosity to DEBUG to get more insight into the TLS handshake process.

  1. Modify Caddyfile Global Configuration: 
    {
    # Other global options...
    log {
        output stderr
        level DEBUG
    }
}

your-domain.com {
    reverse_proxy https://backend.example.com:8443 {
        # ...
    }
}
  2. Apply Changes and Monitor Logs: 
    sudo caddy fmt --overwrite /etc/caddy/Caddyfile
sudo systemctl reload caddy
journalctl -u caddy -f # Monitor logs in real-time
    Look for more detailed TLS-related messages during the handshake attempt.

By systematically working through these troubleshooting steps, you should be able to identify and resolve the root cause of your Caddy reverse proxy TLS certificate verification failures.