Troubleshooting GitLab CI Runner Jobs Stuck Pending: Registration Token Issues on Ubuntu 20.04 LTS

Resolve GitLab CI runner jobs stuck in pending state due to registration token problems on Ubuntu 20.04 LTS. A comprehensive guide for sysadmins and DevOps engineers.

When GitLab CI/CD jobs remain in a "pending" state indefinitely, it often indicates an issue with the GitLab Runner's ability to communicate with the GitLab instance, specifically regarding its registration and authentication. This guide provides a highly technical, step-by-step approach to diagnose and resolve such issues on an Ubuntu 20.04 LTS system, focusing on problems related to registration tokens.

Symptom & Error Signature

The primary symptom is that CI/CD jobs submitted to GitLab never start, remaining in a "Pending" status within the GitLab UI, even when runners are registered and appear to be online.

While there might not always be a single, explicit "registration token error" log, the runner's logs will often show it repeatedly checking for jobs without success, or struggling to connect.

Typical GitLab UI symptom:

Pipeline #12345
Status: Pending
Job: build_app
Status: Pending

Common gitlab-runner service log entries (via journalctl -u gitlab-runner or /var/log/gitlab-runner/runner.log):

# Runner repeatedly checking for jobs, but not picking any up:
Aug 25 10:00:01 runnerhost gitlab-runner[1234]: Checking for jobs... nothing
Aug 25 10:00:11 runnerhost gitlab-runner[1234]: Checking for jobs... nothing
Aug 25 10:00:21 runnerhost gitlab-runner[1234]: Checking for jobs... nothing

# Potential errors during connection attempts (if token or URL are severely wrong, or network issues):
Aug 25 10:00:01 runnerhost gitlab-runner[1234]: FATAL: Failed to register runner. Trying again...  runner=XXX
Aug 25 10:00:01 runnerhost gitlab-runner[1234]: PANIC: Failed to register runner. Error: POST https://your.gitlab.com/api/v4/runners: 401 Unauthorized
Aug 25 10:00:01 runnerhost gitlab-runner[1234]: ERROR: Failed to update runner: Put "https://your.gitlab.com/api/v4/runners/123": dial tcp 1.2.3.4:443: connect: connection refused

Root Cause Analysis

Jobs getting stuck in "pending" due to registration token issues primarily stem from the runner's inability to authenticate or communicate correctly with the GitLab instance. The underlying reasons can be diverse:

1. Incorrect or Expired Registration Token: The runner was registered with an invalid, revoked, or expired token. GitLab registration tokens are intended for one-time use during runner setup.
2. Network Connectivity Issues:
   • Firewall Rules: Host-based (ufw, iptables) or network-level firewalls blocking TCP port 443 (HTTPS) or 80 (HTTP) to the GitLab instance.
   • DNS Resolution Failure: The runner host cannot resolve the GitLab instance's hostname.
   • Proxy Configuration: If the runner is behind a corporate proxy, HTTP_PROXY/HTTPS_PROXY environment variables or config.toml proxy settings are incorrect or missing.
   • Routing Issues: Network routes between the runner and GitLab are misconfigured.
3. GitLab Instance Misconfiguration:
   • CI/CD Disabled: CI/CD features are disabled at the project, group, or instance level.
   • Instance Runners Disabled: Shared runners are disabled globally, and no specific project runners are available.
   • Runner Locked/Paused: The specific runner has been locked or paused in the GitLab UI.
4. Runner Configuration (config.toml) Corruption: The /etc/gitlab-runner/config.toml file might be corrupted, have incorrect GitLab URL, or an invalid runner token (token field under [[runners]]).
5. Systemd Service Issues: The gitlab-runner systemd service is not running, crashing, or lacks necessary permissions/environment variables.
6. SSL/TLS Certificate Issues: If GitLab uses a self-signed certificate, the runner host might not trust it, leading to TLS handshake errors.
7. Outdated Runner Version: An significantly outdated runner might have compatibility issues with a newer GitLab instance.

Step-by-Step Resolution

Follow these steps meticulously to diagnose and resolve the pending job issue.

1. Verify GitLab Instance Health and Runner Status in UI

First, ensure GitLab itself is operational and that the runner is not locked or paused.

1. Check GitLab Instance Health:
   • Log into your GitLab instance as an administrator.
   • Navigate to Admin Area -> Overview -> Dashboard to check system health.
   • Ensure CI/CD is enabled for the relevant project (Project -> Settings -> CI/CD -> General pipelines).
2. Check Runner Status in GitLab UI:
   • Go to Admin Area -> CI/CD -> Runners (for instance runners) or Project -> Settings -> CI/CD -> Runners (for project-specific runners).
   • Locate your runner. Check its status: it should be "online" (green circle).
   • Ensure it is not "locked" or "paused." If it is, unlock/unpause it.

2. Basic System and Network Connectivity Checks on Runner Host

Verify fundamental communication paths.

1. Check gitlab-runner Service Status: Ensure the runner service is active and running without errors.

   sudo systemctl status gitlab-runner
   

   If it's not running, start it:

   sudo systemctl start gitlab-runner
   

   If it's in a failed state, proceed to step 3 to check logs.

2. Network Connectivity to GitLab Instance: Replace your.gitlab.com with your actual GitLab instance URL.

   # Test DNS resolution
   dig +short your.gitlab.com
   
   # Test basic reachability (ICMP)
   ping -c 4 your.gitlab.com
   
   # Test TCP port 443 (HTTPS) connectivity
   nc -vz your.gitlab.com 443
   
   # Test HTTPS connection with curl (useful for proxy and SSL issues)
   curl -v https://your.gitlab.com/api/v4/version
   

   If curl fails with SSL errors (CERT_HAS_EXPIRED, UNABLE_TO_GET_ISSUER_CERT_LOCALLY), your runner host might not trust GitLab's SSL certificate. Ensure your system's certificate authorities are up-to-date (sudo apt update && sudo apt install ca-certificates) or configure custom CA certificates for the runner.

   If you are using a proxy, ensure the curl command also uses the proxy. For example: http_proxy="http://your.proxy.com:8080" https_proxy="http://your.proxy.com:8080" curl -v https://your.gitlab.com/api/v4/version

3. Inspect gitlab-runner Logs

Detailed logs are crucial for understanding the problem.

# For Systemd managed service logs:
sudo journalctl -u gitlab-runner -f --since "5 minutes ago"

# Alternatively, if file logging is configured in /etc/gitlab-runner/config.toml:
sudo tail -f /var/log/gitlab-runner/runner.log

Look for keywords like ERROR, FATAL, Failed, Unauthorized, TLS Handshake, connection refused, timeout. These will pinpoint network, authentication, or certificate issues.

4. Verify and Correct config.toml

The primary configuration file for the runner is /etc/gitlab-runner/config.toml.

1. Backup config.toml:

   sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak
   

2. Inspect config.toml: Open the file and verify the url and token for each runner entry.

   # /etc/gitlab-runner/config.toml
   concurrent = 1
   check_interval = 0
   
   [[runners]]
     name = "my-ubuntu-runner"
     url = "https://your.gitlab.com/" # MUST match your GitLab instance URL
     token = "glrt-XXXXXXXXXXXXXXXXXXXX" # This is the runner token, NOT the registration token
     executor = "shell" # Or docker, kubernetes, etc.
     [runners.custom_build_dir]
     [runners.cache]
       [runners.cache.s3]
       [runners.cache.gcs]
   

   • url: Ensure this is the exact, correct URL to your GitLab instance. A common mistake is a trailing slash or incorrect protocol (http vs https).
   • token: This is the runner authentication token, automatically generated when the runner is successfully registered. It is not the one-time registration token used during setup. If this token is wrong or compromised, the runner cannot authenticate.
   • executor: While not directly related to registration, ensure the executor is correctly configured (e.g., shell, docker). If docker is used, ensure Docker is installed and the gitlab-runner user has permissions to access the Docker socket.

   Do not manually change the token field unless you are re-registering the runner entirely. Any manual modification here will likely break authentication.

5. Re-register the GitLab Runner

If the previous steps don't resolve the issue, especially if logs suggest authentication problems or the runner has been deleted/re-created in GitLab UI, a full re-registration is often the most effective solution.

1. Unregister the Runner from GitLab UI:

   • Go to Admin Area -> CI/CD -> Runners or Project -> Settings -> CI/CD -> Runners.
   • Find the problematic runner and delete it. This removes its authentication token from GitLab.

   Deleting a runner removes its historical job data linked to that specific runner ID. If you need historical data, consider pausing or locking it first, then re-registering a new runner instead of re-using the old ID.

2. Stop the gitlab-runner Service:

   sudo systemctl stop gitlab-runner
   

3. Clean Up Local Runner Configuration: You can unregister all runners configured on the host or manually edit config.toml.

   • Option A: Unregister all (clean slate)

     sudo gitlab-runner unregister --all-runners
     # This will interactively ask for confirmation for each runner.
     # It will also clean up /etc/gitlab-runner/config.toml
     

   • Option B: Manual Edit (if only one runner or specific cleanup) Manually edit /etc/gitlab-runner/config.toml and remove the [[runners]] block(s) corresponding to the deleted runner(s). Save the file, ensuring it's either empty or only contains global settings.

4. Obtain a New Registration Token:

   • In GitLab UI, go back to Admin Area -> CI/CD -> Runners or Project -> Settings -> CI/CD -> Runners.
   • Click on "Register a runner" and copy the new registration token.

5. Register the Runner with the New Token: Execute the registration command. Replace https://your.gitlab.com with your GitLab URL and YOUR_NEW_REGISTRATION_TOKEN with the one you just copied.

   sudo gitlab-runner register 
     --url https://your.gitlab.com/ 
     --registration-token YOUR_NEW_REGISTRATION_TOKEN 
     --executor shell 
     --description "My Ubuntu 20.04 Shell Runner" 
     --tag-list "ubuntu,shell,production" 
     --run-untagged="true" 
     --locked="false"
   

   • --executor: Choose your executor (e.g., shell, docker). If using Docker, ensure Docker is installed.
   • --tag-list: Assign relevant tags for job selection.
   • You might be prompted for additional configuration, like a Docker image if you chose the docker executor.

   If your runner needs to use a proxy, ensure the proxy environment variables are set before running the register command, or directly configure them in the config.toml after registration.

   export HTTP_PROXY="http://proxy.example.com:8080"
   export HTTPS_PROXY="http://proxy.example.com:8080"
   export NO_PROXY="localhost,127.0.0.1,your.gitlab.com" # Crucial for internal networks
   sudo -E gitlab-runner register ... # -E preserves environment variables
   

6. Start the gitlab-runner Service:

   sudo systemctl start gitlab-runner
   sudo systemctl enable gitlab-runner # Ensure it starts on boot
   

7. Verify New Runner Status:

   • Check sudo systemctl status gitlab-runner for a running service.
   • Check sudo journalctl -u gitlab-runner -f for successful "Checking for jobs…" messages.
   • Verify the runner appears online in the GitLab UI.

6. Configure Proxy Settings (If Applicable)

If your runner host is behind a proxy, the gitlab-runner service needs to be aware of it.

1. Edit Systemd Service File Overrides: Create or edit a Systemd override file for the gitlab-runner service to inject proxy environment variables.

   sudo systemctl edit gitlab-runner
   

   This will open an editor (usually nano or vi) for /etc/systemd/system/gitlab-runner.service.d/override.conf. Add the following content, replacing with your proxy details:

   [Service]
   Environment="HTTP_PROXY=http://your.proxy.com:8080"
   Environment="HTTPS_PROXY=http://your.proxy.com:8080"
   Environment="NO_PROXY=localhost,127.0.0.1,your.gitlab.com,.localdomain" # Customize NO_PROXY carefully
   

   The NO_PROXY variable is critical. It should include localhost, 127.0.0.1, your GitLab instance hostname (e.g., your.gitlab.com), and any other internal domains that should bypass the proxy.

2. Reload Systemd and Restart Runner:

   sudo systemctl daemon-reload
   sudo systemctl restart gitlab-runner
   

   Verify logs with sudo journalctl -u gitlab-runner -f.

7. Update GitLab Runner

Ensure your runner is running a compatible and recent version.

# For debian/ubuntu:
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt update
sudo apt install gitlab-runner

After updating, restart the service: sudo systemctl restart gitlab-runner.

By systematically following these steps, you should be able to identify and resolve the root cause of GitLab CI runner jobs stuck in a pending state due to registration or communication issues on Ubuntu 20.04 LTS.