Fix OpenSSL ‘self signed certificate in certificate chain’ on Ubuntu 22.04 LTS

Resolve 'self signed certificate in certificate chain' errors on Ubuntu 22.04 LTS. This expert guide details how to add custom CA certificates to your system and application trust stores.

Introduction

As a Systems Administrator or DevOps engineer managing services on Ubuntu 22.04 LTS, encountering SSL/TLS certificate validation errors can be a common hurdle, especially when dealing with internal services, development environments, or less conventional Certificate Authorities (CAs). One of the most frequently reported issues is the "self signed certificate in certificate chain" error. This error indicates that a client application, such as curl, wget, git, or even a web server like Nginx acting as a reverse proxy, is unable to establish trust for a server's presented SSL certificate because one of the certificates in its chain is either self-signed and unknown, or an intermediate CA is not trusted by the client's system.

This comprehensive guide will walk you through diagnosing the root cause and providing robust, step-by-step solutions to correctly establish trust for these certificates across your Ubuntu 22.04 system and common applications.

Symptom & Error Signature

Users typically encounter this issue when trying to connect to an HTTPS endpoint, clone a Git repository over HTTPS, or when an application attempts to communicate with another service via TLS. The exact error messages vary depending on the client, but they all point to a problem in validating the certificate chain.

Common Error Outputs:

• curl:

  curl: (60) SSL certificate problem: self signed certificate in certificate chain
  More details here: https://curl.haxx.se/docs/sslcerts.html
  

• wget:

  ERROR: cannot verify <hostname>'s certificate, issued by 'CN=<Common Name of CA>,O=<Organization of CA>': Self-signed certificate encountered.
  To connect to <hostname> insecurely, use '--no-check-certificate'.
  

• Git:

  fatal: unable to access 'https://<repository-url>/': SSL certificate problem: self signed certificate in certificate chain
  

• Python requests module (example in application logs):

  requests.exceptions.SSLError: HTTPSConnectionPool(host='<hostname>', port=443): Max retries exceeded with url: / (Caused by SSLError(CertificateError("hostname '<hostname>' doesn't match '<cert-hostname>'",),))}
  

• Browser (e.g., Chrome/Firefox): You would see a security warning page such as NET::ERR_CERT_AUTHORITY_INVALID or SEC_ERROR_UNKNOWN_ISSUER.

Root Cause Analysis

The "self signed certificate in certificate chain" error fundamentally means that your client system does not possess the necessary trust anchor (a Root CA certificate) to validate the entire chain of trust presented by the server. When a server presents its certificate, it also typically sends a chain of intermediate certificates leading back to a root CA. The client then tries to link this chain to a root CA certificate it already trusts in its local certificate store. If it cannot complete this chain validation, the connection is aborted.

Here are the primary underlying reasons:

1. Internal/Private Certificate Authority (CA): Many organizations operate their own internal CAs to issue certificates for their private services. These CAs are not publicly recognized and thus their root certificates are not present in standard operating system trust stores (like those provided by ca-certificates on Ubuntu).
2. Incomplete Certificate Chain: The server might be configured to send only its leaf certificate and not include the necessary intermediate CA certificates. Without the intermediates, the client cannot build a path back to a trusted root, even if the root itself is trusted. While the error states "self signed certificate in chain", this often points to an untrusted intermediate acting as if it were self-signed from the client's perspective, or an actual self-signed root missing from the trust store.
3. Self-Signed Leaf Certificate: While less common for the "in chain" error, a server might present a completely self-signed certificate directly (not signed by any CA). In this scenario, there is no chain, and the certificate itself is untrusted.
4. System Trust Store Outdated or Corrupted: Although rare on a well-maintained Ubuntu 22.04 system, the ca-certificates package or the generated bundle (/etc/ssl/certs/ca-certificates.crt) could be outdated or corrupted, leading to legitimate public CAs being untrusted.
5. Application-Specific Trust Stores: Certain applications or runtimes (e.g., Java, Docker, some Python environments) maintain their own certificate trust stores, which might not automatically inherit or respect changes made to the system's global trust store.

Step-by-Step Resolution

The resolution involves identifying the missing CA certificate(s) and adding them to the appropriate trust stores.

1. Diagnose the Server's Certificate Chain

First, inspect the certificate chain presented by the server to identify which CA is causing the trust issue.

openssl s_client -showcerts -connect example.com:443 </dev/null

Replace example.com:443 with the actual hostname and port of the problematic service. Examine the output. You'll see several certificate blocks (e.g., -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----). Look for the Issuer and Subject lines.

• The first certificate is usually the server's leaf certificate. Its Issuer should match the Subject of the next certificate in the chain.
• Work your way down the chain until you find a certificate whose Issuer is the same as its Subject (indicating a self-signed certificate, likely a Root CA) or whose Issuer corresponds to an organization you don't recognize or expect to be publicly trusted. This is your target CA certificate.

You can also save the individual certificates from BEGIN to END into separate .crt files and use openssl x509 -in cert.crt -text -noout to inspect them in more detail.

2. Obtain the Missing CA Certificate(s)

Once you've identified the untrusted CA (e.g., your internal company CA, a specific intermediate CA), you need to obtain its .crt or .pem file.

• Internal CA: Request the root or intermediate CA certificate file from your organization's IT security or infrastructure team. They should provide it in PEM format (.crt or .pem extension).
• Missing Public Intermediate: Sometimes, a server simply fails to send a widely trusted intermediate CA. In such cases, you might be able to download the missing intermediate CA from the issuing public CA's website. However, a better fix for this scenario is to properly configure the server to send the full chain.

Only obtain CA certificates from trusted sources. Adding an untrusted CA to your system's trust store can compromise your system's security by allowing attackers to issue seemingly valid certificates for any domain.

3. Add the CA Certificate to the System-Wide Trust Store

Ubuntu systems manage trusted CA certificates using the ca-certificates package. This is the primary method for system-wide trust.

1. Copy the CA certificate: Place the .crt file (e.g., my-internal-ca.crt) into the /usr/local/share/ca-certificates/ directory.

   sudo cp /path/to/my-internal-ca.crt /usr/local/share/ca-certificates/my-internal-ca.crt
   

   Ensure the file has a .crt extension. If it's a .pem file, you can rename it or create a symlink.

2. Update the system CA bundle: This command tells Ubuntu to rebuild the combined CA bundle at /etc/ssl/certs/ca-certificates.crt by including your newly added certificate.

   sudo update-ca-certificates
   

   You should see output similar to this, indicating your certificate was added:

   Updating certificates in /etc/ssl/certs...
   1 added, 0 removed; done.
   Running hooks in /etc/ca-certificates/update.d...
   done.
   

   This step is crucial. Applications that rely on the system's CA bundle (like curl, wget, git, and most system utilities) will now trust certificates issued by your added CA.

4. Verify System-Wide Trust

After updating the CA certificates, test the connection to the problematic service again.

curl https://internal-service.example.com/
wget https://internal-service.example.com/
git clone https://internal-git.example.com/repo.git

If the issue persists, carefully re-examine the openssl s_client output, ensure you have the correct root or intermediate CA certificate, and that it's correctly placed and updated.

5. Handle Application-Specific Trust Stores (If Applicable)

Some applications do not fully rely on the system's ca-certificates store and maintain their own. You may need to configure these separately.

#### 5.1 Docker

Docker environments, especially when dealing with private registries or building images that need to access internal services, often require explicit CA trust configuration.

For the Docker Daemon (host-level trust for docker pull, docker push to private registries):

1. Create a directory for the CA certificate under /etc/docker/certs.d/<registry-hostname>/.

   sudo mkdir -p /etc/docker/certs.d/my-registry.example.com/
   

2. Copy your CA certificate into this directory, naming it ca.crt.

   sudo cp /path/to/my-internal-ca.crt /etc/docker/certs.d/my-registry.example.com/ca.crt
   

3. Restart the Docker daemon.

   sudo systemctl restart docker
   

For Docker Builds (trust within containers during docker build):

If your Dockerfile needs to access internal services during the build process (e.g., apt update, npm install, curl), you must add the CA certificate inside the Docker image.

# Dockerfile snippet
# ...
COPY my-internal-ca.crt /usr/local/share/ca-certificates/my-internal-ca.crt
RUN chmod 644 /usr/local/share/ca-certificates/my-internal-ca.crt && 
    update-ca-certificates
# ...

#### 5.2 Java Applications

Java applications use a trust store called cacerts within their JRE/JDK installation.

1. Locate your Java cacerts file. Common paths include:

   • /etc/ssl/certs/java/cacerts (on Ubuntu for system-wide Java)
   • /usr/lib/jvm/java-XX-openjdk-amd64/lib/security/cacerts
   • <JAVA_HOME>/lib/security/cacerts

2. Import your CA certificate using keytool. The default password for cacerts is changeit.

   sudo keytool -import -trustcacerts -keystore /etc/ssl/certs/java/cacerts 
   -storepass changeit -noprompt -alias my-internal-ca 
   -file /usr/local/share/ca-certificates/my-internal-ca.crt
   

   Replace /etc/ssl/certs/java/cacerts with your actual cacerts path. If the password has been changed, use the correct one.

   Changing the cacerts password from default changeit is a good security practice, especially in production environments.

#### 5.3 Python/Pip

While Python's ssl module generally leverages the system's ca-certificates.crt, pip might require explicit configuration, especially if running in isolated virtual environments.

1. For pip: You can configure pip to use a specific CA bundle by creating or editing ~/.pip/pip.conf (for user-specific) or /etc/pip.conf (for system-wide):

   # ~/.pip/pip.conf or /etc/pip.conf
   [global]
   cert = /etc/ssl/certs/ca-certificates.crt
   

   Ensure that /etc/ssl/certs/ca-certificates.crt contains your added CA after sudo update-ca-certificates.

2. For Python requests library: By default, requests uses certifi which provides a bundle of common CAs. If your system CA is added correctly, requests should usually pick it up. If not, you can explicitly point requests to your system bundle or a custom bundle.

   import requests
   
   response = requests.get('https://internal-service.example.com/', verify='/etc/ssl/certs/ca-certificates.crt')
   print(response.text)
   

#### 5.4 Nginx (as a proxy client)

If Nginx is configured as a reverse proxy to an internal upstream service that uses your custom CA, Nginx needs to trust that CA.

# Nginx server block snippet acting as a reverse proxy
server {
    listen 443 ssl;
    server_name my-frontend.example.com;

    # ... other SSL config ...

    location /internal-app/ {
        proxy_pass https://internal-service.example.com/;
        proxy_ssl_verify on; # Ensure Nginx verifies upstream cert
        proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt; # Point to the system CA bundle
        # Or if you have a specific bundle for Nginx:
        # proxy_ssl_trusted_certificate /etc/nginx/certs/internal_ca_bundle.pem;
        
        # ... other proxy settings ...
    }
}

After modifying Nginx configuration, always test and reload:

sudo nginx -t
sudo systemctl reload nginx

6. Revoke and Remove an Untrusted/Expired CA (if necessary)

If a CA certificate becomes compromised, expires, or is no longer needed, it's critical to remove it from your trust stores.

1. Remove from system-wide store:

   sudo rm /usr/local/share/ca-certificates/my-internal-ca.crt
   

2. Update system CA bundle to remove it:

   sudo update-ca-certificates --fresh
   

   The --fresh flag rebuilds the entire bundle from scratch based on current files, ensuring the removed one is gone.

3. For Java (if previously added):

   sudo keytool -delete -alias my-internal-ca -keystore /etc/ssl/certs/java/cacerts -storepass changeit
   

By following these detailed steps, you should be able to resolve "self signed certificate in certificate chain" errors on your Ubuntu 22.04 LTS systems and ensure reliable, secure communication across your infrastructure.