Fixing OpenSSL ‘self signed certificate in certificate chain’ Errors on Alpine Linux

Resolve OpenSSL `self signed certificate in certificate chain` errors on Alpine Linux. This guide covers root causes and step-by-step solutions for validating custom and untrusted SSL certificates securely in minimal environments.

Introduction

As a Systems Administrator or DevOps engineer, encountering SSL/TLS certificate validation errors is a common rite of passage. On Alpine Linux, due to its minimalist design philosophy centered around musl libc and a reduced package set, these issues can manifest differently and require a nuanced approach compared to more feature-rich distributions like Ubuntu or Debian. The "self signed certificate in certificate chain" error specifically indicates that OpenSSL, or an application relying on it, cannot establish a chain of trust back to a recognized Root Certificate Authority (CA) for a presented certificate. This often happens when dealing with internal services, development environments using self-signed certificates, or when consuming APIs protected by non-standard corporate CAs. This guide will walk you through diagnosing and resolving this specific OpenSSL validation failure on Alpine Linux.

Symptom & Error Signature

Users typically experience application failures, connection refused messages, or curl commands failing when trying to connect to a service via HTTPS. The exact error signature can vary slightly depending on the application or tool being used, but the core message points to a certificate validation failure within the trust chain.

Common curl error:

$ curl https://untrusted-service.example.com
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

OpenSSL s_client output:

$ openssl s_client -connect untrusted-service.example.com:443 -showcerts
CONNECTED(00000003)
depth=1 CN = MyInternalCA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=0 CN = untrusted-service.example.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:CN = untrusted-service.example.com
   i:CN = MyInternalCA
-----BEGIN CERTIFICATE-----
MII...
-----END CERTIFICATE-----
 1 s:CN = MyInternalCA
   i:CN = MyInternalCA
-----BEGIN CERTIFICATE-----
MII...
-----END CERTIFICATE-----
---
Server certificate
subject=CN = untrusted-service.example.com
issuer=CN = MyInternalCA
---
No client certificate CA names sent
---
SSL handshake has read 2686 bytes and written 394 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
...

Notice the verify error:num=19:self signed certificate in certificate chain and Verification error: self signed certificate in certificate chain lines, explicitly pointing to the issue.

Root Cause Analysis

The "self signed certificate in certificate chain" error primarily indicates that the Certificate Authority (CA) that issued one of the certificates in the chain (often the root CA itself, or an intermediate CA that is also self-signed or not widely trusted) is not present in the system's trusted certificate store.

1. Missing or Outdated ca-certificates Package: Alpine Linux, in its pursuit of minimal footprint, might ship with a very basic or outdated set of trusted CA certificates. The ca-certificates package provides these essential trust anchors. If it's not installed, or hasn't been updated recently, your system won't recognize many CAs.
2. Untrusted Custom/Internal CA: Many organizations use their own internal Certificate Authorities for issuing certificates to internal services. These CAs are often self-signed or not recognized by public trust stores. For OpenSSL on your Alpine system to trust certificates issued by such a CA, that CA's root certificate must be explicitly added to the system's trust store.
3. Incomplete Certificate Chain from Server: While the error specifically points to a self-signed certificate, sometimes a server might fail to send the full certificate chain, leading to OpenSSL not being able to trace back to a known trusted root. If an intermediate CA is missing, and that intermediate itself is signed by a self-signed or untrusted root, the error can appear. However, the self signed certificate in certificate chain error usually implies the problem is higher up the chain or with the root itself.
4. Docker Container Specifics: When running applications in Docker containers based on Alpine, the trust store must be updated within the container image itself, during the build process. Changes made to the host's /etc/ssl/certs will not affect the container.
5. Application-Specific Trust Stores: Some applications (e.g., specific Python libraries, Java applications) might use their own, separate trust stores or provide environment variables to override system defaults. While rare for OpenSSL directly, it's a consideration if system-wide fixes don't immediately resolve application-specific issues.

Step-by-Step Resolution

The primary solution involves ensuring the system's trust store is up-to-date and, if necessary, adding the specific custom or self-signed CA certificate to it.

1. Update and Install ca-certificates

First, ensure that the ca-certificates package, which contains the common public root CAs, is installed and up-to-date.

apk update
apk add ca-certificates
update-ca-certificates

• apk update: Refreshes the local package index.
• apk add ca-certificates: Installs the package if not present. If it is already installed, apk will check for updates.
• update-ca-certificates: This command rebuilds the /etc/ssl/certs/ca-certificates.crt bundle, which OpenSSL uses as its default trust store. On Alpine, ca-certificates package installation hooks typically run this, but explicitly running it ensures the bundle is fresh.

After running these commands, re-test your application or curl command. If the issue persists, especially with internal services or private CAs, proceed to the next step.

2. Add Custom Root/Intermediate CA Certificates

If the service uses a private or corporate CA, you'll need to obtain its root certificate and add it to Alpine's trust store.

a. Obtain the CA Certificate

You need the public key of the Root CA certificate in PEM format. This is usually provided by your organization's IT/security department. If you can only access the problematic service via a browser on another machine, you might be able to export the root certificate from the browser's certificate viewer (look for the "Certification Path" or "Certificate Chain" and export the root). Ensure it's in PEM format (.pem or .crt extension, containing -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- blocks).

Let's assume you have a file named my-internal-ca.pem.

b. Install the Custom CA Certificate

Alpine Linux handles custom CA certificates by placing them in /usr/local/share/ca-certificates/ and then running update-ca-certificates.

# Create a directory for custom certificates
mkdir -p /usr/local/share/ca-certificates/extra

# Copy your custom CA certificate into the directory
# Ensure the file has a .crt extension for update-ca-certificates to pick it up
cp my-internal-ca.pem /usr/local/share/ca-certificates/extra/my-internal-ca.crt

# Update the system's CA certificate bundle
update-ca-certificates

The update-ca-certificates command on Alpine specifically looks for .crt files within /usr/local/share/ca-certificates/ (and its subdirectories) to include them in the main /etc/ssl/certs/ca-certificates.crt bundle. Ensure your custom CA file has a .crt extension.

3. Verify the Certificate Installation

After adding the custom CA, it's crucial to verify that OpenSSL now trusts the certificate chain.

a. Using openssl verify

You can verify the original server certificate against the updated trust store. You'll need the server's public certificate file for this. If you don't have it, you can extract it using openssl s_client as shown in the symptom section.

# Assuming you have the server certificate (e.g., server.pem) and the CA is in the system store
openssl verify -CApath /etc/ssl/certs/ server.pem

Expected output: server.pem: OK

b. Retest with openssl s_client

Use openssl s_client to connect to your service again. This time, if successful, you should see verify return:0 and Verification: OK.

openssl s_client -connect untrusted-service.example.com:443 -showcerts

Look for these lines indicating success:

...
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
...

c. Retest with curl

Finally, re-run your curl command or restart your application.

curl https://untrusted-service.example.com

This should now execute successfully without the SSL error.

4. Docker-Specific Considerations

If your Alpine application is running inside a Docker container, all the above steps must be performed within the Dockerfile during the image build process.

Here's an example Dockerfile snippet:

FROM alpine:3.18

# 1. Install/Update ca-certificates
#    The --no-cache flag cleans up apk cache, essential for minimal image size.
RUN apk update && 
    apk add --no-cache ca-certificates && 
    update-ca-certificates

# 2. Add custom CA certificate (if needed)
#    Copy your custom CA .pem file into the image
COPY my-internal-ca.pem /usr/local/share/ca-certificates/my-internal-ca.crt

#    Update the system's CA certificate bundle to include your custom CA
RUN update-ca-certificates

# ... rest of your Dockerfile (e.g., installing application dependencies, etc.)
# EXPOSE 8080
# CMD ["./my-app"]

Ensure that update-ca-certificates is run after COPYing your custom CA certificates into the image. If you copy the certificates after update-ca-certificates has already run, they will not be included in the trust store.

Avoid putting apk update and apk add on separate RUN layers if possible. Combining them into a single RUN command along with rm -rf /var/cache/apk/* helps reduce Docker image size by ensuring cached package lists are immediately cleaned up.

5. Application-Specific Overrides (If Still Failing)

In rare cases, an application might still fail even after correctly configuring the system-wide trust store. This usually means the application is using its own method for SSL validation.

• curl / wget: If you must bypass validation (e.g., for quick debugging in a dev environment), use -k or --insecure. Do not use this in production.

  curl -k https://untrusted-service.example.com
  

• Environment Variables: Some libraries respect specific environment variables to point to a custom CA bundle:

  • CURL_CA_BUNDLE: Can point to a custom CA file for curl.
  • REQUESTS_CA_BUNDLE (Python requests library).
  • NODE_EXTRA_CA_CERTS (Node.js).
  • SSL_CERT_FILE / SSL_CERT_DIR (general OpenSSL-aware applications).

  Example:

  export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
  # Or for a specific custom CA if only one is needed:
  export SSL_CERT_FILE=/usr/local/share/ca-certificates/extra/my-internal-ca.crt
  

Relying on application-specific overrides or environment variables should be a last resort or for development/debugging purposes only. The most robust and secure solution is to correctly configure the system's trust store (/etc/ssl/certs/ca-certificates.crt) so that all OpenSSL-reliant applications automatically pick up the trusted CAs. This ensures consistent security policy enforcement across your system or container.