Let's Encrypt HTTP-01 Challenge Fails with Cloudflare Proxy Active

Resolve Let's Encrypt HTTP-01 validation errors when Cloudflare's proxy (orange cloud) is enabled, preventing direct access to your origin server.

Welcome to this technical guide for resolving a common Let’s Encrypt HTTP-01 challenge failure when your domain is actively proxied through Cloudflare. This issue typically manifests as a failure to obtain or renew your SSL certificate, leaving your website with security warnings or inaccessible via HTTPS.

Symptom & Error Signature

When attempting to obtain or renew a Let’s Encrypt certificate using an ACME client like Certbot, you will encounter an error indicating that the HTTP-01 challenge failed. Your website might display browser warnings about an insecure connection (NET::ERR_CERT_COMMON_NAME_INVALID, MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT), or simply fail to load over HTTPS.

The typical error output from Certbot will resemble:

sudo certbot --nginx -d example.com -d www.example.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Attempting to renew cert (example.com) from /etc/letsencrypt/renewal/example.com.conf produced an unexpected error: Failed authorization procedure. www.example.com (http-01): urn:ietf:params:acme:error:dns :: DNS problem: SERVFAIL looking up A for www.example.com - the domain's nameservers may be malfunctioning. example.com (http-01): urn:ietf:params:acme:error:dns :: DNS problem: SERVFAIL looking up A for example.com - the domain's nameservers may be malfunctioning.
...
Hint: The Certificate Authority failed to download the challenge files.
Please check your firewall and webserver configuration.

Another common variation, especially when Cloudflare is actively proxying, is:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Attempting to renew cert (example.com) from /etc/letsencrypt/renewal/example.com.conf produced an unexpected error: Failed authorization procedure. www.example.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://www.example.com/.well-known/acme-challenge/YOUR_TOKEN: Connection refused.
...

Notice the DNS problem or Connection refused for the http-01 challenge, often referencing urn:ietf:params:acme:error:dns or urn:ietf:params:acme:error:connection.

Root Cause Analysis

The core of this problem lies in how the Let’s Encrypt HTTP-01 challenge works and how Cloudflare’s proxy interacts with it.

  1. HTTP-01 Challenge Mechanism: To prove you own a domain, Let’s Encrypt’s validation servers attempt to fetch a specific file containing a unique token from http://yourdomain.com/.well-known/acme-challenge/YOUR_TOKEN. This request must reach your origin web server (e.g., Nginx, Apache) on port 80, where your ACME client (like Certbot) temporarily places the token.

  2. Cloudflare Proxy (Orange Cloud): When your domain’s A or AAAA records are proxied through Cloudflare (indicated by an orange cloud icon in your Cloudflare DNS settings), all traffic to your domain is routed through Cloudflare’s global edge network. Cloudflare sits between your users and your origin server, providing CDN, WAF, and DDoS protection.

  3. The Conflict:

    • When Let’s Encrypt’s validators try to access http://yourdomain.com/.well-known/acme-challenge/YOUR_TOKEN, their DNS lookup for yourdomain.com resolves to a Cloudflare IP address, not your actual origin server’s IP.
    • The request reaches Cloudflare’s edge servers. Cloudflare, unaware of the specific ACME challenge token that your origin server is trying to present, cannot fulfill this request. It will either serve a cached page, an error, or, if you don’t have an SSL certificate on Cloudflare’s side for that domain yet, it might even fail to establish a connection to your origin securely (if Flexible SSL is not enabled or full strict is active without a cert).
    • Consequently, Let’s Encrypt’s validators fail to retrieve the challenge file, leading to the urn:ietf:params:acme:error:dns (because Cloudflare is effectively acting as the nameserver for the HTTP request) or urn:ietf:params:acme:error:connection (because Cloudflare isn’t passing the challenge correctly to your origin or the connection from Cloudflare to origin fails).

In essence, Cloudflare’s proxy prevents Let’s Encrypt’s validators from directly communicating with your server to verify domain ownership via the HTTP-01 method.

Step-by-Step Resolution

The most common and effective solution is to temporarily disable Cloudflare’s proxy for the domain(s) you are trying to certify, obtain the certificate, and then re-enable the proxy.

1. Verify Current DNS & Proxy Status

First, confirm that your domain is indeed proxied by Cloudflare and identify your origin server’s IP address.

# Check the A record for your domain
dig +short example.com A
dig +short www.example.com A

If your domain is proxied by Cloudflare, the dig command will return Cloudflare IP addresses (e.g., 104.x.x.x, 172.x.x.x).

Alternatively, check your Cloudflare Dashboard:

  1. Log in to your Cloudflare account.
  2. Select your domain.
  3. Navigate to the DNS section.
  4. Look at the Proxy status column for your A and AAAA records. If it shows an orange cloud, it’s proxied. Make note of your actual origin IP address from the Content column.

2. Temporarily Disable Cloudflare Proxy

You need to temporarily bypass Cloudflare’s proxy so that Let’s Encrypt’s validators can directly reach your origin server.

  1. In your Cloudflare Dashboard, go to the DNS section.

  2. For each A and AAAA record associated with the domain(s) you are trying to certify (e.g., example.com, www.example.com), click the orange cloud icon to change it to a grey cloud (DNS Only).

    • This means traffic will now go directly to your origin server’s IP address.

    [!IMPORTANT] Disabling the Cloudflare proxy will temporarily expose your origin server’s IP address to the public internet. Ensure your server’s firewall is correctly configured to only allow necessary inbound connections (e.g., SSH on port 22, HTTP on port 80, HTTPS on port 443).

  3. Allow a few minutes for DNS changes to propagate. While Cloudflare’s DNS changes are usually instant, some clients might still cache old DNS records. You can verify the change with dig again; it should now return your origin server’s IP.

3. Ensure Nginx/Web Server is Prepared for Challenge

Before running Certbot, ensure your web server (Nginx in this guide) and server’s firewall are configured to allow direct access to port 80.

  1. Check Firewall (UFW): Ensure port 80 is open on your server.

    sudo ufw status verbose

    If 80/tcp is not allowed, enable it:

    sudo ufw allow 'Nginx HTTP'
sudo ufw reload

  2. Nginx Configuration: Certbot’s --nginx authenticator plugin usually handles creating a temporary configuration for the /.well-known/acme-challenge/ path. However, it’s crucial that your Nginx server block for the domain in question is listening on port 80.

    Verify your Nginx configuration, typically in /etc/nginx/sites-available/yourdomain.com:

    server {
    listen 80;
    listen [::]:80;
    server_name example.com www.example.com;

    # Ensure no aggressive HTTP to HTTPS redirects here for the challenge to work.
    # If you have a 'return 301 https://$host$request_uri;' for port 80,
    # comment it out temporarily, or add an exception for /.well-known/acme-challenge/

    location /.well-known/acme-challenge/ {
        root /var/www/html; # Or wherever your webroot is
    }

    # Other locations...
}

    [!WARNING] If you have an existing Nginx configuration that aggressively redirects all HTTP traffic to HTTPS (e.g., return 301 https://$host$request_uri; directly within the port 80 server block, before any location /.well-known/acme-challenge directive), Certbot’s HTTP-01 challenge will fail. Temporarily comment out or modify such redirects to allow the challenge path on port 80. Certbot typically adds its own temporary configuration, but this can interfere.

    Test your Nginx configuration for syntax errors and reload if you made changes:

    sudo nginx -t
sudo systemctl reload nginx

4. Run Certbot to Obtain/Renew Certificate

With Cloudflare proxy disabled and your server ready, run Certbot.

  1. Obtain a new certificate:

    sudo certbot --nginx -d example.com -d www.example.com

    Replace example.com and www.example.com with your actual domain(s).

  2. Renew an existing certificate (if it’s just about to expire):

    sudo certbot renew --force-renewal

    Using --force-renewal is generally discouraged for regular use due to rate limits, but can be useful for immediate troubleshooting when you know the environment has changed. Otherwise, a simple sudo certbot renew should work.

Certbot will communicate directly with your Nginx server, place the challenge file, and Let’s Encrypt’s validators will now be able to retrieve it from your origin IP. If successful, Certbot will output a success message and usually offer to configure Nginx to use the new certificate and set up redirects.

5. Re-enable Cloudflare Proxy

Once Certbot confirms the certificate has been successfully obtained or renewed:

  1. Return to your Cloudflare Dashboard and the DNS section.

  2. For the A and AAAA records you previously changed, click the grey cloud icon to turn it back to an orange cloud (Proxied).

    [!IMPORTANT] Re-enabling the proxy is crucial to restore Cloudflare’s security, performance, and caching benefits for your website. Do not skip this step!

Your website will now serve content through Cloudflare’s network with your newly issued Let’s Encrypt SSL certificate installed on your origin server. Cloudflare will also automatically provision its Universal SSL certificate for your domain on its edge network, completing the end-to-end encryption.

6. (Optional/Advanced) Use DNS-01 Challenge with Cloudflare API

For a more automated and robust solution, especially for environments where temporarily disabling Cloudflare proxy is problematic or for domains not serving web content on port 80, consider using the DNS-01 challenge. This method proves domain ownership by creating a specific TXT record in your domain’s DNS.

This requires:

  • Certbot Cloudflare DNS Plugin: certbot-dns-cloudflare.
  • Cloudflare API Token: A token with appropriate permissions to modify DNS records for your domain.

  1. Install the plugin:

    sudo apt update
sudo apt install python3-certbot-dns-cloudflare

  2. Create Cloudflare API Token:

    • Log in to Cloudflare.
    • Go to My Profile > API Tokens > Create Token.
    • Use the “Edit Cloudflare DNS” template or create a custom token with:
      • Permissions: Zone > DNS > Edit
      • Zone Resources: Include > Specific zone > [your domain]
    • Save the token securely.

  3. Create API Credentials File: Create a file like /etc/letsencrypt/cloudflare.ini with your token:

    dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN

    Secure the file:

    sudo chmod 600 /etc/letsencrypt/cloudflare.ini

  4. Run Certbot with DNS-01 Challenge:

    sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini -d example.com -d www.example.com

    This method does not require disabling the Cloudflare proxy and is excellent for full automation.

7. (Optional) Troubleshooting Nginx Configuration

If after unproxying Cloudflare and checking your firewall, Certbot still fails, the issue might be with Nginx’s ability to serve the challenge file.

  • Temporary Manual Test: Manually create a test file:

    sudo mkdir -p /var/www/html/.well-known/acme-challenge/
echo "test" | sudo tee /var/www/html/.well-known/acme-challenge/testfile

    Try to access it from your browser (or curl from a different machine/VPN): http://example.com/.well-known/acme-challenge/testfile If this fails (e.g., 404 Not Found), your Nginx configuration for the webroot or the .well-known path is incorrect.

  • Ensure Correct Webroot: Make sure your Nginx server block’s root directive points to the correct directory where certbot will place its challenge files. If you use a different webroot (e.g., /var/www/yourdomain), adjust the root directive or ensure certbot is configured to use the correct path. When using certbot --nginx, it usually attempts to deduce the correct webroot.

By following these steps, you should successfully resolve the Let’s Encrypt HTTP-01 challenge failure caused by an active Cloudflare proxy and secure your website with a valid SSL certificate.