Nginx SSL Certificate Key File Mismatch: Resolving ssl_handshake_alert on CentOS Stream/Rocky Linux

Troubleshoot Nginx `ssl_handshake_alert` due to SSL certificate and key file mismatch on CentOS Stream/Rocky Linux. Fix common configuration errors.

When configuring Nginx for HTTPS, one of the most common and perplexing issues system administrators encounter is the ssl_handshake_alert often stemming from a mismatch between the provided SSL certificate and its corresponding private key file. This problem typically prevents clients from establishing a secure connection to your server, rendering your website inaccessible via HTTPS. This guide will walk you through diagnosing and resolving this critical Nginx SSL error on CentOS Stream and Rocky Linux environments.

Symptom & Error Signature

Users attempting to access your site via HTTPS will typically see browser errors indicating a failure to establish a secure connection. These can vary by browser but often include messages such as:

• NET::ERR_CERT_COMMON_NAME_INVALID
• SSL_ERROR_HANDSHAKE_FAILURE
• ERR_SSL_PROTOCOL_ERROR
• "Your connection is not private" or "Secure Connection Failed"

On the server side, the Nginx error logs (typically /var/log/nginx/error.log on CentOS/Rocky Linux) will provide more specific insights. You might see entries like these:

2023/10/26 10:30:45 [crit] 12345#12345: *1 SSL_do_handshake() failed (SSL: error:140B5047:SSL routines:SSL_do_handshake:key values mismatch) while SSL handshaking, client: 192.168.1.100, server: 0.0.0.0:443
2023/10/26 10:30:45 [crit] 12345#12345: *1 SSL_do_handshake() failed (SSL: error:140A90F1:SSL routines:ssl_do_handshake:deny short primary key) while SSL handshaking
2023/10/26 10:30:45 [error] 12345#12345: *1 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking, client: 192.168.1.100, server: 0.0.0.0:443

If Nginx fails to start due to this issue, systemctl status nginx might show:

$ sudo systemctl status nginx
× nginx.service - The NGINX HTTP and reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Mon 2023-10-26 10:30:45 UTC; 1min 2s ago
    Process: 12340 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=1/FAILURE)
   Main PID: 12340 (code=exited, status=1/FAILURE)

Oct 26 10:30:45 yourhostname.example.com nginx[12340]: nginx: [emerg] SSL_CTX_use_PrivateKey_file("/etc/letsencrypt/live/yourdomain.com/privkey.pem") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:KEY_VALUES_MISMATCH)
Oct 26 10:30:45 yourhostname.example.com nginx[12340]: nginx: configuration file /etc/nginx/nginx.conf test failed
Oct 26 10:30:45 yourhostname.example.com systemd[1]: nginx.service: Main process exited, code=exited, status=1/FAILURE
Oct 26 10:30:45 yourhostname.example.com systemd[1]: nginx.service: Failed with result 'exit-code'.
Oct 26 10:30:45 yourhostname.example.com systemd[1]: Failed to start The NGINX HTTP and reverse proxy server.

The specific error SSL: error:140B5047:SSL routines:SSL_do_handshake:key values mismatch or X509_check_private_key:KEY_VALUES_MISMATCH directly points to the problem.

Root Cause Analysis

The ssl_handshake_alert with a "key values mismatch" error is fundamentally caused by Nginx being configured to use an SSL private key file that does not mathematically correspond to the public certificate file it is also configured to use. When a client attempts to initiate an SSL/TLS handshake, Nginx presents the certificate. The client then expects Nginx to prove ownership of the private key associated with that certificate. If the key file specified in Nginx's configuration is incorrect, Nginx cannot complete this cryptographic proof, leading to the handshake failure.

Common scenarios leading to this mismatch include:

1. Incorrect Key Pair Selection: You generated a Certificate Signing Request (CSR) with one private key, but when you received the signed certificate back from the Certificate Authority (CA), you mistakenly configured Nginx with an older or different private key.
2. Certificate Renewal Issues: During a certificate renewal process, a new private key might have been generated, but only the certificate file (.crt or .pem) was updated in Nginx's configuration, leaving the old, incorrect private key in place.
3. File Management Errors: Accidental overwriting of files, incorrect file paths in nginx.conf, or restoration from an outdated backup.
4. Incomplete Chain: While distinct, sometimes issues with an incomplete certificate chain (missing intermediate certificates) can manifest with similar symptoms, though the primary error log will often be different. However, it's worth checking.
5. Password-Protected Private Keys: If a private key is encrypted with a passphrase, Nginx typically cannot use it directly unless explicitly configured with ssl_password_file or if the key is decrypted. While this usually results in a different Nginx error during startup, it can contribute to confusion.

Step-by-Step Resolution

The core of the resolution involves verifying that the SSL certificate and private key Nginx is using are a valid, matching pair.

1. Verify Nginx Configuration Syntax

Before diving into certificate files, ensure Nginx's configuration itself is syntactically valid. This helps rule out simple typos or structural errors.

sudo nginx -t

If the syntax is OK, you'll see:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

If there are errors, address them first.

2. Identify the Active Nginx SSL Configuration

Locate the Nginx server block responsible for your HTTPS traffic and identify the paths to your ssl_certificate and ssl_certificate_key files.

# Search for SSL directives in your Nginx configuration files
sudo grep -r "ssl_certificate" /etc/nginx/

# Example output:
# /etc/nginx/nginx.conf:    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
# /etc/nginx/nginx.conf:    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;

Make a note of the exact paths found for both the certificate (.pem, .crt) and the private key (.key, .pem). For this guide, we'll use /path/to/your/certificate.crt and /path/to/your/private.key as placeholders.

3. Inspect Certificate and Key Files for Modulus Match

The cryptographic modulus of a certificate and its private key must be identical. We can use openssl to extract and compare these values.

• Extract Modulus from Certificate:

  sudo openssl x509 -noout -modulus -in /path/to/your/certificate.crt | sudo openssl md5
  

• Extract Modulus from Private Key (RSA):

  sudo openssl rsa -noout -modulus -in /path/to/your/private.key | sudo openssl md5
  

  If your private key is password-protected, openssl rsa will prompt you for the password. If you don't know it, you will need to find the correct key or generate a new one. For Nginx to use it without a password, you'd typically decrypt it first (openssl rsa -in encrypted.key -out decrypted.key). For advanced use, Nginx supports ssl_password_file.

• Extract Modulus from Private Key (ECC/EC – Elliptic Curve Cryptography):

  If you are using an ECC certificate (e.g., from Let's Encrypt with certbot --key-type ecdsa), use openssl ec instead of openssl rsa:

  sudo openssl ec -noout -modulus -in /path/to/your/private.key | sudo openssl md5
  

  It's crucial to use the correct openssl command (rsa or ec) based on your private key type. Using the wrong command will lead to errors.

Compare the md5 hashes. The output of the md5 hash for both commands must be identical.

Example of matching output:

(stdin)= 83a0050854c6e61f1c24e65520e17135   # Certificate Modulus
(stdin)= 83a0050854c6e61f1c24e65520e17135   # Private Key Modulus

If these hashes are different, you have a mismatch.

4. Re-establish the Correct Certificate and Key Pair

If the moduli do not match, you need to find the correct pair or generate a new one.

• For Let's Encrypt (Certbot) Users:

  • Certbot usually manages this automatically, storing certificates and keys in /etc/letsencrypt/live/yourdomain.com/. The fullchain.pem (certificate + intermediates) and privkey.pem (private key) files in this directory should always match.
  • If you suspect an issue, try renewing your certificate:

    sudo certbot renew --force-renewal
    

  • After renewal, ensure your Nginx configuration points to the live directory:

    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    

  Using the live directory symlinks ensures your configuration always points to the latest active certificate files after renewal.

• For Manually Managed Certificates:

  • Find the correct key: Search your server for other .key files (e.g., in /etc/ssl/private, or your web application's directory) and test their modulus against your active certificate.
  • Find the correct certificate: If you have multiple .crt or .pem files, test each one against your known private key.
  • Reissue/Regenerate: If you cannot locate the matching private key, you will need to regenerate a new private key, create a new CSR, and request your CA to reissue the certificate. Once you receive the new certificate, ensure you configure Nginx with this new certificate and its corresponding new private key.

  Always handle private key files with extreme care. They should have restrictive permissions (e.g., 600 for the root user) and never be publicly accessible.

5. Consider the Certificate Chain (Intermediate Certificates)

While less common for a "key values mismatch" error, an incorrect or missing intermediate certificate can also cause SSL handshake issues. Ensure your ssl_certificate directive points to a file that contains both your server certificate and any intermediate certificates provided by your CA (often referred to as fullchain.pem or a concatenation of domain.crt and ca_bundle.crt).

# Example for a full chain file (recommended)
ssl_certificate /path/to/your/fullchain.pem;
ssl_certificate_key /path/to/your/private.key;

# If you have separate files:
# sudo cat /path/to/your/domain.crt /path/to/your/intermediate.crt > /path/to/your/fullchain.pem
# Then use the fullchain.pem in Nginx.

To verify the contents of your certificate file:

sudo openssl x509 -in /path/to/your/certificate.crt -text -noout | grep "Issuer:|Subject:"

This will show you the issuer and subject of each certificate in the file. Ensure the subject of one certificate matches the issuer of the next, forming a complete chain up to a trusted root.

6. Test and Reload Nginx

Once you've corrected the ssl_certificate and ssl_certificate_key paths in your Nginx configuration to point to a matching pair, perform a final syntax test and reload Nginx:

sudo nginx -t
sudo systemctl reload nginx

If the nginx -t command returns syntax is ok and test is successful, and systemctl reload nginx executes without errors, your Nginx server should now be serving content over HTTPS correctly.

Finally, verify the fix by attempting to access your website in a web browser. Check for the padlock icon in the address bar and inspect the certificate details to ensure the correct certificate is being served.