Troubleshooting Certbot DNS-01 Challenge TXT Record Mismatch on Ubuntu 20.04 LTS

Fix 'TXT record mismatch' errors during Certbot DNS-01 challenges on Ubuntu 20.04 LTS. This guide covers diagnosing DNS propagation, incorrect records, and Certbot configuration for successful SSL acquisition.

The Certbot DNS-01 challenge is a powerful method for obtaining Let's Encrypt SSL certificates, especially for wildcard domains or when direct HTTP access to the server is not feasible. It relies on verifying domain ownership by placing a specific TXT record in your domain's DNS zone. When Certbot reports a "TXT record mismatch," it indicates that the value it found in DNS for the _acme-challenge record does not match the value it expects, preventing certificate issuance or renewal. This guide will walk you through diagnosing and resolving this common issue on an Ubuntu 20.04 LTS system.

Symptom & Error Signature

When attempting to obtain or renew a certificate using certbot with a DNS-01 authenticator (e.g., --dns-cloudflare, --manual, etc.), you will encounter an error message similar to the following in your terminal output and Certbot logs (/var/log/letsencrypt/letsencrypt.log):

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator dns-cloudflare, Installer nginx
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Attempting to renew cert (example.com) from /etc/letsencrypt/renewal/example.com.conf produced an unexpected error: An unexpected error occurred: urn:ietf:params:acme:error:unauthorized :: The TXT record found did not match the expected value. See https://letsencrypt.org/docs/challenge-types/ for more information.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenge failed for domain example.com
dns-01 challenge for example.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cleaning up challenges
Failed to renew certificate example.com with error: An unexpected error occurred: urn:ietf:params:acme:error:unauthorized :: The TXT record found did not match the expected value. See https://letsencrypt.org/docs/challenge-types/ for more information.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/example.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log for more details.

Root Cause Analysis

The "TXT record mismatch" error typically stems from one or more of the following issues:

1. DNS Propagation Delays: The most frequent cause. After creating or updating a TXT record, it takes time for the changes to propagate across DNS servers globally. Certbot, or the ACME server, might check for the record before it has fully propagated, leading to a mismatch (either finding an old record, no record, or a partially propagated record).
2. Incorrect TXT Record Value:
   • Typos: A simple copy-paste error or manual transcription mistake when entering the TXT record value into your DNS provider's interface.
   • Trailing/Leading Spaces: Invisible characters can cause a mismatch.
   • Certbot Generated a New Value: For each challenge attempt, Certbot generates a unique token. If you previously tried and failed, then manually re-added an old record, it won't match the new expected value.
3. Multiple Stale TXT Records: If previous _acme-challenge TXT records for the same domain or subdomain were not removed, Certbot or the ACME server might pick up an outdated record, leading to a mismatch. Only one active _acme-challenge TXT record should exist at any given time for the domain being challenged.
4. DNS Provider Caching Issues: While less common for direct mismatches, your DNS provider might be caching old values, or the recursive DNS servers Let's Encrypt queries might be serving cached, outdated information.
5. Certbot DNS Plugin Misconfiguration (for automated renewals):
   • Incorrect API Credentials: The API key/token might be invalid, expired, or lack the necessary permissions to create/update TXT records in your DNS zone.
   • Incorrect Zone/Domain ID: The plugin might be trying to update the wrong DNS zone.
   • Propagation Timeout Too Short: The default propagation delay configured for the plugin might be too short for your specific DNS provider.

Step-by-Step Resolution

Follow these steps to diagnose and resolve the Certbot DNS-01 TXT record mismatch error.

1. Obtain the Expected TXT Record Value

First, you need to know exactly what value Certbot expects. The --dry-run or --staging flags are invaluable here to avoid hitting production rate limits.

If you are using a manual method:

sudo certbot certonly --manual --preferred-challenges dns --staging -d example.com -d *.example.com

Certbot will pause and prompt you, displaying the exact TXT record name (_acme-challenge.example.com) and the value it expects you to add. Copy this value precisely.

If you are using an automated DNS plugin (e.g., Cloudflare, Route 53):

The plugin will automatically attempt to create the record. You won't see the value directly in the terminal, but it will be logged in /var/log/letsencrypt/letsencrypt.log during the AddTXTRecord phase if you enable verbose logging (-v). You can often infer the expected value by temporarily failing the challenge and observing what Certbot attempts to put in place.

2. Manually Verify DNS Propagation

After you've created or updated the _acme-challenge TXT record with the expected value in your DNS provider's control panel, use dig or nslookup to confirm its presence and value from various resolvers.

1. Check your authoritative DNS server: Replace example.com with your domain.

   dig -t TXT _acme-challenge.example.com @$(dig +short NS example.com | head -n 1) +short
   

   This command queries one of your domain's authoritative name servers directly.

2. Check public DNS resolvers (e.g., Google, Cloudflare):

   dig -t TXT _acme-challenge.example.com @8.8.8.8 +short
   dig -t TXT _acme-challenge.example.com @1.1.1.1 +short
   nslookup -type=TXT _acme-challenge.example.com 8.8.8.8
   

   The output of these commands should precisely match the value Certbot expects, enclosed in double quotes. If you see multiple values, outdated values, or no values, proceed to the next steps.

3. Remove Stale or Duplicate TXT Records

Log in to your DNS provider's control panel and navigate to your domain's DNS records.

• Locate all _acme-challenge TXT records for the domain(s) you are trying to certify.
• Delete any old or duplicate _acme-challenge TXT records. There should only be one TXT record for _acme-challenge.example.com (or _acme-challenge.sub.example.com if challenging a subdomain) at any given time.
• Ensure the current record's value is an exact match to what Certbot expects, with no leading/trailing spaces or extra characters.

4. Adjust DNS Provider TTL and Certbot Wait Times

DNS propagation can take minutes or even hours, depending on your DNS provider and the record's Time To Live (TTL).

1. Reduce TXT record TTL (if possible): Temporarily set the TTL for your _acme-challenge TXT records to a lower value (e.g., 60 to 300 seconds) during troubleshooting. Remember to change it back to a more sensible value (e.g., 3600 seconds) after you've successfully issued/renewed your certificate.

2. Increase Certbot's propagation wait time (for automated plugins): Many automated DNS plugins allow you to specify a longer propagation delay. For example, with certbot-dns-cloudflare:

   sudo certbot renew --cert-name example.com --dns-cloudflare --dns-cloudflare-credentials ~/.secrets/certbot/cloudflare.ini --dns-cloudflare-propagation-seconds 60 --post-hook "systemctl reload nginx"
   

   Adjust 60 seconds to 120 or 180 if you still face issues. Refer to your specific plugin's documentation for the correct flag (e.g., --dns-route53-propagation-seconds).

3. For manual challenges, simply wait longer. After adding the TXT record, wait 5-10 minutes, verify propagation with dig, and then press Enter in the Certbot prompt.

5. Verify Certbot DNS Plugin Configuration (if applicable)

If you're using an automated DNS plugin, double-check its configuration.

1. Credentials File: Ensure your credentials file (e.g., ~/.secrets/certbot/cloudflare.ini, ~/.aws/credentials) is correctly configured and accessible.

   Example for Cloudflare (~/.secrets/certbot/cloudflare.ini):

   dns_cloudflare_email = [email protected]
   dns_cloudflare_api_key = your_global_api_key_or_token
   

   Always protect your API credentials. Set strict file permissions:

   sudo chmod 600 ~/.secrets/certbot/cloudflare.ini
   

2. API Key/Token Permissions: Verify that your API key/token has the necessary permissions to edit DNS records for the specific zone in question. For Cloudflare, this typically requires Zone:DNS:Edit permission.

3. Zone ID/Account ID: Confirm that the plugin is targeting the correct domain/zone. If you manage multiple domains, a misconfiguration could point to the wrong zone.

6. Use the Certbot Staging Environment for Testing

Let's Encrypt has strict rate limits. To avoid hitting them during troubleshooting, use the staging environment:

# To test a renewal
sudo certbot renew --dry-run

# To test a new certificate issuance with a plugin
sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials ~/.secrets/certbot/cloudflare.ini --staging -d example.com -d *.example.com

The --dry-run flag performs a full renewal simulation without actually requesting or installing certificates, making it ideal for checking if the challenge process succeeds.

7. Clean Up and Retry

After making any changes:

1. Clean up previous challenges: If you added a TXT record manually, ensure it's removed before the next Certbot attempt (unless the plugin handles cleanup automatically).
2. Run Certbot again:
   • For renewals:

     sudo certbot renew --nginx --post-hook "systemctl reload nginx"
     

     The --post-hook ensures your Nginx configuration is reloaded to use the new certificate if certbot successfully renews.
   • For new certificates:

     sudo certbot certonly --nginx --dns-cloudflare --dns-cloudflare-credentials ~/.secrets/certbot/cloudflare.ini -d example.com -d *.example.com
     

Troubleshooting with Docker

If you are running Certbot in a Docker container, ensure your volumes, environment variables, and network configurations are correct.

1. Mount Credentials: Make sure your API credentials file is correctly mounted into the container.

   docker run -it --rm 
     -v "/etc/letsencrypt:/etc/letsencrypt" 
     -v "/var/lib/letsencrypt:/var/lib/letsencrypt" 
     -v "~/.secrets/certbot:/secrets"  # Mount your secrets directory
     --name certbot 
     certbot/certbot 
     certonly --dns-cloudflare --dns-cloudflare-credentials /secrets/cloudflare.ini 
     -d example.com -d *.example.com --staging
   

2. Permissions within Container: The certbot user inside the container needs to be able to read the credentials file. Ensure chmod 600 is applied to the file on the host before mounting.

By systematically going through these steps, verifying DNS propagation, correcting record values, and ensuring Certbot's configuration is precise, you should be able to resolve the "TXT record mismatch" error and successfully obtain or renew your SSL certificates.