Troubleshooting: OpenSSL ‘Self Signed Certificate in Certificate Chain’ Validation Error on CentOS Stream / Rocky Linux

Resolve OpenSSL 'self signed certificate in certificate chain' errors on CentOS Stream & Rocky Linux when validating remote servers or API endpoints. Learn to add custom CAs to your system's trust store.

When operating applications or services on CentOS Stream or Rocky Linux, you may encounter issues connecting to remote endpoints, APIs, or Git repositories over HTTPS. This often manifests as an OpenSSL error indicating a problem with certificate chain validation, specifically revolving around a "self signed certificate in certificate chain." This guide will walk you through diagnosing and resolving this common security and connectivity challenge.

Symptom & Error Signature

The core symptom is the inability of an application or a command-line utility (like curl, wget, or git) to establish a secure HTTPS connection to a remote server. You'll typically see errors similar to these:

cURL Output:

$ curl https://your-internal-api.example.com/data
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file or CA cert doesn't work, you can specify an alternate bundle
with the --cacert option.
If this HTTPS server uses a certificate signed by a CA that is not present in
the bundle, you can specify an alternate CA file using the --cacert option.
If you are unable to verify the certificate for some reason, --insecure
option would bypass the verification.

(Note: While curl may sometimes state "Peer's certificate issuer has been marked as not trusted", the underlying OpenSSL error in system logs or more verbose output will often specify SELF_SIGNED_CERT_IN_CERT_CHAIN.)

OpenSSL Error Message (often found in verbose logs or direct OpenSSL calls):

error: unable to get local issuer certificate
error: certificate verification failed
...
OpenSSL Error: SSL_CTX_use_PrivateKey_file:EE_KEY_TOO_SMALL
OpenSSL Error: SSL_CTX_use_PrivateKey_file:system lib
140598858655552:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:ssl/s3_clnt.c:1269:

(Note: The certificate verify failed line is common, but the specific SELF_SIGNED_CERT_IN_CERT_CHAIN is the key differentiator.)

Python requests Library Error:

import requests
try:
    response = requests.get('https://your-internal-api.example.com/data')
    print(response.text)
except requests.exceptions.SSLError as e:
    print(f"SSL Error: {e}")

Output:

SSL Error: HTTPSConnectionPool(host='your-internal-api.example.com', port=443): Max retries exceeded with url: /data (Caused by SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)'))

Git Clone Error:

$ git clone https://your-internal-git.example.com/repo.git
Cloning into 'repo'...
fatal: unable to access 'https://your-internal-git.example.com/repo.git/': SSL certificate problem: self signed certificate in certificate chain

Root Cause Analysis

The error SELF_SIGNED_CERT_IN_CERT_CHAIN signifies that your CentOS Stream or Rocky Linux system's OpenSSL library, acting as a client, failed to validate the certificate chain presented by a remote server. This occurs when:

1. Untrusted Custom/Internal CA: The remote server's certificate is signed by a Certificate Authority (CA) that is not included in your system's trusted CA bundle. This is common in enterprise environments using their own Public Key Infrastructure (PKI) or with self-hosted services that generate certificates signed by a custom root CA. The client receives a chain, and somewhere in that chain, there's a certificate that is self-signed (usually the root CA), but your system doesn't recognize its signature as trustworthy.
2. Incomplete Certificate Chain: The remote server might be presenting an incomplete certificate chain. A proper chain should include the leaf certificate (for the server itself) and all intermediate CA certificates, leading up to a root CA that is widely trusted by default in operating systems and browsers. If intermediate certificates are missing, the client cannot properly connect the leaf certificate to a trusted root, and an intermediate CA might appear "self-signed" and untrusted.
3. Expired or Invalid Certificate: Less common for this specific error, but an expired or otherwise invalid certificate in the chain can sometimes lead to validation failures that manifest ambiguously. However, SELF_SIGNED_CERT_IN_CERT_CHAIN specifically points to a trust issue with a self-signed certificate within the chain.

CentOS Stream and Rocky Linux (like other RHEL-based distributions) manage system-wide trusted CA certificates using the ca-certificates package and the update-ca-trust utility. This system aggregates certificates from various sources into a single bundle (/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem) that OpenSSL and other applications rely on. When a CA certificate is missing from this bundle, connections to servers signed by that CA will fail.

Step-by-Step Resolution

The primary solution involves adding the untrusted self-signed root or intermediate CA certificate to your CentOS Stream / Rocky Linux system's trust store.

1. Inspect the Server's Certificate Chain

First, identify which certificate in the chain is causing the issue. This often reveals if it's a missing intermediate or an untrusted root.

# Replace your-internal-api.example.com and 443 with your actual hostname and port
# The -showcerts option displays the entire certificate chain.
# The 2>/dev/null suppresses OpenSSL output to stderr, ensuring only certs are piped.
# The -servername is crucial for SNI (Server Name Indication) in modern TLS.
echo -n | openssl s_client -connect your-internal-api.example.com:443 -servername your-internal-api.example.com -showcerts 2>/dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > remote_certs.pem

Now, examine the remote_certs.pem file. It will contain all certificates presented by the server, typically in order from leaf to root. You can inspect each certificate:

# To view the first certificate (leaf certificate):
openssl x509 -in remote_certs.pem -text -noout -certopt no_subject,no_header,no_version,no_serial,no_signame,no_issuer,no_pubkey,no_sigdump,no_aux | head -n 20

# To view a specific certificate (e.g., the second one in the file):
# This requires splitting the file into individual certs, then inspecting.
# A simpler way is to grep for "Issuer" and "Subject"
grep -E "Subject:|Issuer:" remote_certs.pem

Look for a certificate where the Issuer and Subject fields are identical, or where the Issuer is a CA you don't recognize and its Subject is the same. This often indicates the self-signed root or an intermediate signed by an untrusted entity.

If you have multiple certificates in remote_certs.pem, you can split them into individual files for easier inspection:

csplit -f cert_ remote_certs.pem '/-----BEGIN CERTIFICATE-----/' '{*}'
for i in cert_*; do openssl x509 -in "$i" -text -noout | grep -E "Subject:|Issuer:|X509v3 Basic Constraints"; done
rm cert_* remote_certs.pem # Clean up

Identify the self-signed certificate (where Issuer == Subject) or the root CA you need to trust.

2. Obtain the Missing Certificate(s)

Once you've identified the specific self-signed CA certificate that needs to be trusted (from step 1 or provided by the service owner), ensure you have it in PEM format (a text file starting with -----BEGIN CERTIFICATE----- and ending with -----END CERTIFICATE-----). Let's assume you've named this file custom_ca.pem.

Only add certificates from sources you explicitly trust. Adding arbitrary or unknown certificates can compromise the security of your system by allowing man-in-the-middle attacks. Always verify the source and integrity of the certificate.

3. Add the Certificate to the System-Wide Trust Store (CentOS Stream / Rocky Linux)

CentOS Stream and Rocky Linux use the update-ca-trust utility to manage trusted CA certificates.

a. Create a directory for custom certificates:

sudo mkdir -p /etc/pki/ca-trust/source/anchors/

b. Copy your custom CA certificate into the directory:

# Replace /path/to/your/custom_ca.pem with the actual path to your certificate file
sudo cp /path/to/your/custom_ca.pem /etc/pki/ca-trust/source/anchors/

The certificate file must have a .pem, .crt, or .cer extension to be recognized by update-ca-trust.

c. Update the system's CA trust store:

sudo update-ca-trust extract

This command gathers all certificates from /etc/pki/ca-trust/source/ (including your new one in anchors/) and generates a unified trust store in /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.

d. Verify the certificate has been added:

You can check if your certificate's subject is now part of the system's trusted bundle:

# Replace "Your Custom CA Name" with a unique part of the Subject or Issuer of your CA cert
grep "Your Custom CA Name" /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

If the output shows your CA's subject or issuer, it has been successfully added.

4. (Optional) Verify Your Own Server's Certificate Chain

If the SELF_SIGNED_CERT_IN_CERT_CHAIN error occurs when external clients (or other services on your CentOS/Rocky system) try to connect to an Nginx or Apache server running on your CentOS/Rocky system, the issue might be with your server's certificate configuration. Ensure your web server is configured to send the full certificate chain.

For Nginx:

Ensure your ssl_certificate directive points to a file containing both your server's leaf certificate and all intermediate CA certificates, concatenated together.

# Example: Combine your server cert and intermediate certs
cat your_domain.crt intermediate_ca.crt > your_domain_fullchain.crt

# In your Nginx configuration (e.g., /etc/nginx/nginx.conf or a site-specific config):
server {
    listen 443 ssl;
    server_name your_domain.com;

    ssl_certificate /etc/nginx/certs/your_domain_fullchain.crt; # This must be the full chain
    ssl_certificate_key /etc/nginx/certs/your_domain.key;
    # ... other SSL settings ...
}

After modifying Nginx configuration, always test and reload:

sudo nginx -t
sudo systemctl reload nginx

If you have a separate ssl_trusted_certificate (or ssl_client_certificate in older Nginx) directive, it's typically used for client certificate verification, not for presenting your server's chain. For standard server cert chain presentation, ssl_certificate should point to the full chain.

5. Test the Resolution

After updating the CA trust store, try your failing command or application again.

# Test with curl
curl https://your-internal-api.example.com/data

# Test with git
git clone https://your-internal-git.example.com/repo.git

If you're running a service (e.g., a Python application managed by Systemd, or a Docker container) that was experiencing the issue, you might need to restart it to pick up the updated CA trust store:

# For a Systemd service
sudo systemctl restart your-application.service

# For a Docker container (if it's using the host's CA bundle, which is less common for isolated containers)
# It's more likely you'd need to rebuild a Docker image to include the CA cert inside it.
sudo systemctl restart docker # Or restart the specific container

For Docker containers, simply updating the host's ca-trust store might not be sufficient, as containers often have their own isolated filesystem and CA bundles. You may need to build the custom CA into the Docker image itself, or mount the host's /etc/pki/ca-trust/extracted into the container.

Example Dockerfile snippet for RHEL-based images:

FROM rockylinux:8
COPY custom_ca.pem /etc/pki/ca-trust/source/anchors/
RUN update-ca-trust extract
# ... rest of your Dockerfile

By following these steps, you should successfully resolve the "self signed certificate in certificate chain" validation error on your CentOS Stream or Rocky Linux systems, enabling secure communication with your internal and external services.