Certbot ACME Directory Verification Failed: Server Connection Errors on CentOS Stream/Rocky Linux

Troubleshoot Certbot 'ACME directory verification failed server connection' on CentOS Stream/Rocky Linux, covering firewalls, DNS, web server, and network issues.

Troubleshoot Certbot 'ACME directory verification failed server connection' on CentOS Stream/Rocky Linux, covering firewalls, DNS, web server, and network issues.

Introduction

Obtaining or renewing SSL/TLS certificates with Certbot and Let's Encrypt is usually a straightforward process. However, you may encounter the cryptic error "ACME directory verification failed server connection" which indicates that the Let's Encrypt ACME server could not establish a connection to your server to perform the necessary domain validation. This guide will walk you through diagnosing and resolving these connection issues on CentOS Stream and Rocky Linux environments, ensuring your websites remain secure with valid certificates.

Symptom & Error Signature

When running certbot to obtain or renew a certificate, you'll typically see an error similar to this in your terminal output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Requesting a certificate for yourdomain.com and www.yourdomain.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: yourdomain.com
  Type:   connection
  Detail: During secondary validation: 203.0.113.1: Fetching http://yourdomain.com/.well-known/acme-challenge/YOUR_CHALLENGE_TOKEN: Connection refused

  Domain: www.yourdomain.com
  Type:   connection
  Detail: During secondary validation: 203.0.113.1: Fetching http://www.yourdomain.com/.well-known/acme-challenge/ANOTHER_CHALLENGE_TOKEN: Connection timed out

Hint: The Certificate Authority failed to download the challenge files. Ensure that your webserver is running and that it is accessible from the internet.

Some challenges have failed.

The key indicators are Type: connection, Connection refused, or Connection timed out when fetching the ACME challenge file (.well-known/acme-challenge/). This directly points to an issue preventing the Let's Encrypt validation server from reaching your web server.

Root Cause Analysis

The "ACME directory verification failed server connection" error fundamentally means the Let's Encrypt validation servers cannot communicate with your host on the required port (typically 80 for http-01 challenges or 443 for tls-alpn-01 challenges). The underlying causes can vary:

  1. Firewall Restrictions: This is the most common culprit. Your server's local firewall (firewalld, iptables) or an external cloud/network firewall (security groups, ACLs) is blocking incoming connections on ports 80 and/or 443.
  2. Incorrect DNS Resolution: The domain's A/AAAA records might point to the wrong IP address, or DNS changes haven't fully propagated. The ACME server attempts to connect to the IP address resolved for your domain.
  3. Web Server Issues:
    • The web server (Nginx, Apache) is not running or has crashed.
    • The web server is not configured to listen on port 80 (or 443).
    • The web server's virtual host configuration is incorrect, preventing it from serving content from the .well-known/acme-challenge directory.
    • If using the webroot authenticator, the specified webroot path is incorrect or inaccessible.
  4. Network Connectivity Problems: Less common for inbound connections, but potential underlying network issues on the server itself, or routing problems preventing external access.
  5. SELinux Interference: While less frequent for basic HTTP-01 challenges, SELinux policies could potentially prevent the web server from accessing certain files or directories required by Certbot, or even bind to ports under strict configurations.
  6. CDN/Reverse Proxy Misconfiguration: If your domain uses a CDN (like Cloudflare) or a reverse proxy, it might be intercepting or caching the ACME challenge requests, preventing them from reaching your origin server.
  7. Rate Limiting by Let's Encrypt: If you've attempted certificate issuance or renewal too many times within a short period, Let's Encrypt might temporarily block requests from your IP, although this typically results in a different error message (too many requests).

Step-by-Step Resolution

Follow these steps meticulously to diagnose and resolve the connection failure.

1. Verify Basic Network Connectivity and DNS Resolution

First, ensure your server can reach the internet and that your domain resolves correctly.

  • Check Outbound Connectivity:

    ping -c 4 8.8.8.8
curl -v https://acme-v02.api.letsencrypt.org/directory

    If ping fails, you have a fundamental network issue. If curl fails, your server might have outbound firewall issues or general connectivity problems to the Let's Encrypt API.

  • Verify DNS Resolution: Confirm that your domain's A/AAAA records point to the correct public IP address of your server. Use an external DNS lookup tool (e.g., dig from another machine, or a web-based tool) as well as locally.

    # On your server
dig A +short yourdomain.com
dig AAAA +short yourdomain.com

# Example for external check (from a different machine or online tool)
dig @8.8.8.8 A yourdomain.com

    Ensure the IP address returned matches your server's public IP. DNS changes can take time to propagate globally.

2. Check and Configure Server Firewall (Firewalld)

This is the most common fix. CentOS Stream and Rocky Linux use firewalld by default.

  • Check Firewall Status:

    sudo firewall-cmd --state
sudo firewall-cmd --list-all --zone=public

    Ensure firewalld is running and verify that services http (port 80) and https (port 443) are listed as allowed in the public zone.

  • Open Required Ports: If http or https are not listed, add them. Let's Encrypt typically uses port 80 for the http-01 challenge and port 443 for tls-alpn-01 (if applicable).

    sudo firewall-cmd --zone=public --add-service=http --permanent
sudo firewall-cmd --zone=public --add-service=https --permanent
sudo firewall-cmd --reload

    After running these commands, always run sudo firewall-cmd --reload to apply the changes without dropping existing connections. Verify with sudo firewall-cmd --list-all --zone=public.

  • Cloud Provider Firewalls: If your server is hosted on a cloud platform (AWS EC2, Google Cloud, Azure, DigitalOcean), check its security group rules or network firewall settings. These often override or complement your server's local firewall. Ensure inbound TCP traffic on ports 80 and 443 is allowed from anywhere (0.0.0.0/0).

3. Inspect Web Server Status and Configuration

Ensure your web server is running and correctly configured to serve the ACME challenge.

  • Check Web Server Status:

    sudo systemctl status nginx  # Or apache2 for Apache

    Verify the output shows active (running). If not, start it:

    sudo systemctl start nginx   # Or apache2
sudo systemctl enable nginx  # To ensure it starts on boot

  • Verify Web Server Listeners: Ensure your web server is configured to listen on port 80. For Nginx, check /etc/nginx/nginx.conf and your virtual host files (e.g., /etc/nginx/conf.d/yourdomain.com.conf). Look for listen 80;. For Apache, check /etc/httpd/conf/httpd.conf and virtual host files (e.g., /etc/httpd/conf.d/yourdomain.com.conf). Look for Listen 80.

    Test Nginx configuration:

    sudo nginx -t

    Test Apache configuration:

    sudo apachectl configtest

    Resolve any reported syntax errors.

  • Test ACME Challenge Path: If your web server is running and accessible externally, you should be able to create a dummy file and access it. Create a test file:

    sudo mkdir -p /var/www/html/.well-known/acme-challenge/
echo "test" | sudo tee /var/www/html/.well-known/acme-challenge/testfile

    Then, try to access http://yourdomain.com/.well-known/acme-challenge/testfile from your browser or curl from an external machine. If this fails, your web server or its configuration for the webroot is the issue. If you use a different webroot (e.g., /usr/share/nginx/html), adjust the path accordingly.

    Certbot's webroot authenticator requires the web server to correctly serve files from the specified webroot-path. If using the nginx or apache authenticators, Certbot attempts to temporarily modify your server config.

4. SELinux Policy Review

SELinux can sometimes block web server access, leading to "permission denied" errors when Certbot tries to write or the web server tries to read challenge files.

  • Check SELinux Status:

    sestatus

    If SELinux is enforcing, check the audit logs for denials.

    sudo ausearch -c httpd -ts today

    Look for AVC denials related to httpd (Apache) or nginx.

  • Temporary Test (Not Recommended for Production): If you suspect SELinux, you can temporarily set it to permissive mode to confirm:

    sudo setenforce 0
# Try running certbot again
sudo certbot renew --dry-run
sudo setenforce 1 # Re-enable enforcing mode

    Do NOT leave SELinux in permissive or disabled mode in a production environment. If this fixes the issue, identify the specific SELinux policy that needs adjustment (e.g., using audit2allow) rather than disabling it.

  • Common SELinux Booleans for Web Servers: Sometimes, specific booleans might be needed, though less common for basic .well-known access.

    sudo getsebool -a | grep httpd
# Example: If your webroot is on an unusual path, you might need:
# sudo setsebool -P httpd_enable_homedirs on

5. Check for CDN/Reverse Proxy Interference

If your domain is behind a CDN (like Cloudflare) or a reverse proxy, the Let's Encrypt validator will connect to the CDN/proxy's IP, not your origin server.

  • Cloudflare Specifics: If using Cloudflare, ensure your DNS records for yourdomain.com and www.yourdomain.com have the "Proxy status" set to "DNS only" (the cloud icon should be grey, not orange) during the Certbot validation process. This allows Let's Encrypt to directly connect to your server. After successful validation, you can re-enable the proxy if desired. Alternatively, for http-01 challenges, you can set up a Cloudflare Page Rule to bypass caching/security for the /.well-known/acme-challenge/* path.

  • Other Reverse Proxies: Ensure your reverse proxy is correctly configured to forward requests for /.well-known/acme-challenge/ to your actual web server without modification or caching.

6. Increase Logging and Debugging

If the problem persists, increase Certbot's verbosity and examine system logs.

  • Certbot Debug Output:

    sudo certbot renew --dry-run -v
sudo certbot certonly --webroot -w /var/www/html -d yourdomain.com -d www.yourdomain.com --dry-run -v

    The -v (verbose) flag will provide more detailed output. --dry-run allows you to test the process without hitting Let's Encrypt's rate limits.

  • Review System Logs:

    sudo journalctl -u nginx --since "1 hour ago"
sudo journalctl -u httpd --since "1 hour ago"
sudo tail -f /var/log/letsencrypt/letsencrypt.log

    Look for any errors around the time you tried to run Certbot. This can give clues about web server issues or permission problems.

7. Try Different Certbot Authenticator

If the webroot authenticator struggles, consider the nginx or apache authenticators if you use those web servers directly, as they can temporarily modify configurations to handle challenges.

  • Using Nginx Authenticator:

    sudo certbot renew --nginx --dry-run
# Or for initial cert
sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com --dry-run

    Ensure the python3-certbot-nginx package is installed (sudo dnf install python3-certbot-nginx).

  • Using Apache Authenticator:

    sudo certbot renew --apache --dry-run
# Or for initial cert
sudo certbot --apache -d yourdomain.com -d www.yourdomain.com --dry-run

    Ensure the python3-certbot-apache package is installed (sudo dnf install python3-certbot-apache).

By systematically working through these steps, you should be able to identify and resolve the root cause of your Certbot ACME directory verification failure on CentOS Stream and Rocky Linux. Most commonly, it boils down to an improperly configured firewall or an inactive/misconfigured web server.