Troubleshooting OpenSSL SELF_SIGNED_CERT_IN_CHAIN Validation Errors on Ubuntu 20.04 LTS

Fix OpenSSL validation errors caused by self-signed certificates in the chain on Ubuntu 20.04 LTS. Learn to properly manage trusted CAs for secure connections.

Introduction

As a seasoned Systems Administrator, encountering certificate validation failures is a common occurrence, especially when dealing with custom Certificate Authorities (CAs), internal services, or improperly configured SSL/TLS chains. The SELF_SIGNED_CERT_IN_CHAIN error indicates that a certificate within the trust chain presented by a server is self-signed, and this self-signed certificate, or its signing root, is not trusted by your Ubuntu 20.04 LTS system's default CA store. This guide will walk you through diagnosing and resolving this specific OpenSSL validation error.

This issue typically manifests as failed connections to an HTTPS endpoint, whether from a web browser, curl, wget, or a custom application using an SSL/TLS library. The underlying cause is the client's inability to establish a trusted path from the server's certificate back to a known and trusted root CA.

Symptom & Error Signature

When your system or an application attempts to connect to an SSL/TLS endpoint that presents a certificate chain containing an untrusted self-signed certificate, you will observe various errors depending on the client.

Common Error Outputs:

1. curl:

curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

2. wget:

--2023-10-27 10:30:00--  https://your-untrusted-service.example.com/
Resolving your-untrusted-service.example.com (your-untrusted-service.example.com)... 192.168.1.100
Connecting to your-untrusted-service.example.com (your-untrusted-service.example.com)|192.168.1.100|:443... connected.
ERROR: cannot verify your-untrusted-service.example.com's certificate, issued by '/CN=My Internal CA':
  Self-signed certificate encountered.
ERROR: certificate common name 'your-untrusted-service.example.com' doesn't match requested host name 'your-untrusted-service.example.com'.
To connect to your-untrusted-service.example.com insecurely, use '--no-check-certificate'.

3. openssl s_client (Diagnostic Tool):

openssl s_client -connect your-untrusted-service.example.com:443
# ... (snip) ...
---
Certificate chain
 0 s:/CN=your-untrusted-service.example.com
   i:/CN=My Intermediate CA
 1 s:/CN=My Intermediate CA
   i:/CN=My Internal CA
 2 s:/CN=My Internal CA
   i:/CN=My Internal CA   <--- This indicates a self-signed certificate
---
# ... (snip) ...
Verify return code: 19 (self signed certificate in certificate chain)

4. Application Logs (e.g., Python requests):

import requests
requests.exceptions.SSLError: HTTPSConnectionPool(host='your-untrusted-service.example.com', port=443): Max retries exceeded with url: / (Caused by SSLError(CertificateError("hostname 'your-untrusted-service.example.com' doesn't match 'your-untrusted-service.example.com'",), SSL.Error(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1129)')))

While the Python error might be more generic, the underlying cause when debugging often points back to the OpenSSL self signed certificate in certificate chain issue.

Root Cause Analysis

The core of this problem lies in how Public Key Infrastructure (PKI) and Certificate Authorities (CAs) establish trust. A standard, publicly trusted SSL/TLS certificate chain looks like this:

End-Entity Certificate (your website) -> Intermediate CA Certificate -> Root CA Certificate

Each certificate in this chain is cryptographically signed by the one above it, all the way up to a Root CA certificate. Publicly trusted Root CA certificates are pre-installed in the operating systems and browsers of the world, forming a web of trust.

The SELF_SIGNED_CERT_IN_CHAIN error specifically means that during the validation process:

1. A certificate within the chain (often an intermediate or the root itself) is self-signed. This means its Issuer (i) and Subject (s) fields are identical.
2. This self-signed certificate is not explicitly trusted by the client's system. The client's ca-certificates store (on Ubuntu, managed by the ca-certificates package) does not contain this specific self-signed certificate as a trusted root, nor does it contain a trusted root that signed the self-signed intermediate.

This scenario is common in several contexts:

• Internal PKI deployments: Organizations often run their own internal CAs to issue certificates for internal services. These custom Root CAs are, by nature, self-signed and not recognized by public trust stores.
• Development/Staging environments: Developers might use self-signed certificates or certificates issued by a private CA for testing.
• Missing Intermediate Certificates: Sometimes, the server providing the certificate does not send the full chain, omitting necessary intermediate certificates. If an intermediate certificate needed to complete the chain back to a trusted root is missing, and the next certificate in the chain (as perceived by the client) appears self-signed without proper context, this error can occur. However, the most direct interpretation is that a certificate in the chain is self-signed and untrusted.

On Ubuntu 20.04 LTS, the system's trust store is managed by the ca-certificates package, which collects trusted root certificates from various sources and consolidates them into a single bundle that OpenSSL and other applications use. When a certificate in your chain is self-signed and not part of this bundle, validation fails.

Step-by-Step Resolution

The solution involves identifying the untrusted self-signed certificate and adding its corresponding public key (as a .crt file) to your system's trusted CA store.

1. Identify the Problematic Certificate in the Chain

First, use openssl s_client to inspect the certificate chain presented by the server.

openssl s_client -showcerts -connect your-untrusted-service.example.com:443 < /dev/null

This command will output a lot of information. Look for the Certificate chain section. Each certificate will be listed with s: (Subject) and i: (Issuer) lines.

---
Certificate chain
 0 s:/CN=your-untrusted-service.example.com
   i:/CN=My Intermediate CA
-----BEGIN CERTIFICATE-----
... (your service cert) ...
-----END CERTIFICATE-----
 1 s:/CN=My Intermediate CA
   i:/CN=My Internal CA
-----BEGIN CERTIFICATE-----
... (intermediate cert) ...
-----END CERTIFICATE-----
 2 s:/CN=My Internal CA
   i:/CN=My Internal CA   <--- This is the key: s: and i: are identical. This is the self-signed root or intermediate.
-----BEGIN CERTIFICATE-----
... (self-signed cert) ...
-----END CERTIFICATE-----
---
# ... (snip) ...
Verify return code: 19 (self signed certificate in certificate chain)

The certificate where s: and i: are identical and it's not a publicly known root CA is your target. Copy the entire -----BEGIN CERTIFICATE----- to -----END CERTIFICATE----- block for that specific certificate.

2. Obtain the Full Public Certificate of the Untrusted CA

If you've identified the self-signed certificate in the previous step, you already have its content. If you just know the name (e.g., "My Internal CA") and need the full certificate, you must obtain the .crt (PEM format) file for that specific Root CA or Intermediate CA from its source (e.g., your organization's IT department, the service provider, or the system that generated it). Ensure you trust the source of this certificate.

Let's assume you've copied the self-signed certificate content to a file. For example, save it as my_internal_ca.crt.

# Example content for my_internal_ca.crt
cat <<EOF > my_internal_ca.crt
-----BEGIN CERTIFICATE-----
MIIDYzCCAk+gAwIBAgIRAPvF/P/v+v/v/v/v/v/v/v/v/v/v/v/v/v/v/v/v/v/v
... (actual certificate data) ...
-----END CERTIFICATE-----
EOF

Only add certificates from sources you explicitly trust. Adding a malicious certificate to your system's trust store can compromise the security of your system by allowing man-in-the-middle attacks. Always verify the authenticity of the .crt file.

3. Add the CA Certificate to the System Trust Store

Ubuntu uses the ca-certificates package to manage the system-wide collection of trusted CA certificates.

1. Move the certificate file to the ca-certificates directory:

   sudo cp my_internal_ca.crt /usr/local/share/ca-certificates/
   

   Ensure the file has a .crt extension. The update-ca-certificates script specifically looks for files ending in .crt in this directory.

2. Update the CA certificate store:

   sudo update-ca-certificates
   

   You should see output similar to this, indicating your certificate was added:

   Updating certificates in /etc/ssl/certs...
   1 added, 0 removed; done.
   Running hooks in /etc/ca-certificates/update.d...
   done.
   

   This command symlinks your new .crt file into /etc/ssl/certs and regenerates /etc/ssl/certs/ca-certificates.crt, which is the bundle OpenSSL typically uses.

4. (Optional) Configure Specific Applications/Services

While updating the system trust store resolves the issue for most applications that rely on OpenSSL's default configuration, some applications or services might use their own trust stores or require specific configuration.

a. Nginx (as a client/proxy)

If Nginx is acting as a reverse proxy or is making outbound HTTPS requests to an upstream service that uses this certificate, you might need to configure Nginx to trust it.

Edit your Nginx configuration (e.g., /etc/nginx/nginx.conf or a site-specific config in /etc/nginx/conf.d/). In the location block proxying to the upstream, add:

location /my_untrusted_service/ {
    proxy_pass https://your-untrusted-service.example.com;
    proxy_ssl_verify on;
    proxy_ssl_trusted_certificate /etc/ssl/certs/my_internal_ca.crt; # Path to your added CA cert
    proxy_ssl_name your-untrusted-service.example.com; # Optional, for SNI
}

The proxy_ssl_trusted_certificate directive tells Nginx to use only the specified certificate(s) for trusting the upstream. If the upstream's certificate is signed by a chain that also includes public CAs, you might need to concatenate your custom CA with the system's trusted CAs into a single file, or manage this differently based on your specific Nginx version and configuration. For most cases, adding your custom CA to the system store (as done in Step 3) and letting Nginx use the default system store might be sufficient, unless proxy_ssl_trusted_certificate is explicitly set to a custom file. If Nginx is using the system's trust store, then Step 3 is sufficient. If it's configured to use a specific bundle, you might need to merge your cert into that bundle.

After modifying Nginx configuration, test and reload:

sudo nginx -t
sudo systemctl reload nginx

b. Docker Builds and Runtime

If your Docker containers need to trust this CA (e.g., during apt update, npm install, pip install, or when connecting to internal services), you need to make the CA certificate available inside the container.

For Dockerfile builds:

# Assuming my_internal_ca.crt is in the build context
COPY my_internal_ca.crt /usr/local/share/ca-certificates/my_internal_ca.crt
RUN update-ca-certificates

For Docker daemon-level trust (for image pulls from private registries, etc.):

You can configure the Docker daemon to trust your custom CA by placing the .crt file in /etc/docker/certs.d/<your-registry-hostname>:<port>/ca.crt or by placing it in /etc/pki/ca-trust/source/anchors/ (on RHEL/CentOS) or /usr/local/share/ca-certificates/ on Ubuntu/Debian hosts and running update-ca-certificates on the host system. Docker containers often inherit the host's trust store if they are built or run with appropriate mounts or if the ca-certificates package is updated inside the image.

For host-level:

sudo mkdir -p /etc/docker/certs.d/my.private.registry:5000
sudo cp my_internal_ca.crt /etc/docker/certs.d/my.private.registry:5000/ca.crt
sudo systemctl restart docker

c. Python Applications

Python's requests library and other HTTP clients often rely on a CA bundle.

• System-wide (if using certifi and updated correctly): If Python uses certifi and you've updated the system CA store, it might pick up the change.

• Specify CA bundle: For granular control, set the REQUESTS_CA_BUNDLE environment variable or pass the verify argument in requests.

  export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
  # Or for a custom bundle:
  export REQUESTS_CA_BUNDLE=/path/to/my_custom_ca_bundle.pem
  

  In Python code:

  import requests
  response = requests.get("https://your-untrusted-service.example.com", verify="/etc/ssl/certs/ca-certificates.crt")
  # Or, if you prefer to just trust your one cert:
  # response = requests.get("https://your-untrusted-service.example.com", verify="/usr/local/share/ca-certificates/my_internal_ca.crt")
  

5. Verify the Fix

After performing the steps above, re-test the connection that was previously failing.

curl https://your-untrusted-service.example.com

You should now receive the expected output from curl (e.g., HTML content) without any SSL certificate errors. Similarly, your applications should now be able to connect successfully.

You can also use openssl s_client again to confirm the verification status:

openssl s_client -connect your-untrusted-service.example.com:443 < /dev/null
# ... (snip) ...
Verify return code: 0 (ok)

A Verify return code: 0 (ok) confirms that the certificate chain is now fully trusted by OpenSSL on your system.