GitLab CI Runner Jobs Stuck Pending: Registration Token Issues on Ubuntu 22.04 LTS

Resolve GitLab CI runner jobs stuck pending on Ubuntu 22.04 LTS. This guide addresses registration token issues, network blocks, and misconfigurations for reliable CI/CD.

GitLab CI/CD is an indispensable toolchain for modern DevOps practices, automating the build, test, and deployment phases of your software lifecycle. When GitLab CI jobs remain perpetually in a "pending" state, it's a critical bottleneck that halts your entire CI/CD pipeline. One of the most common yet frustrating causes for this is an issue with the GitLab Runner's registration token, preventing it from effectively polling the GitLab instance for new jobs. This guide provides a highly technical, accurate, and actionable approach to diagnose and resolve such "stuck pending" issues on Ubuntu 22.04 LTS, drawing upon 16 years of web hosting and DevOps expertise.

Symptom & Error Signature

The primary symptom you'll observe is in the GitLab UI: your CI/CD jobs will indefinitely display a "pending" status, often showing "Waiting for a runner to pick up this job…" despite an active runner being configured.

While there might not always be an explicit "registration token error" directly in the logs, the underlying problem often manifests through a lack of successful job polling or connectivity issues.

GitLab UI (Jobs view):

Job #12345 (main)
Status: Pending
Duration: -
Runner: #<RunnerID> (If assigned, else blank)
Description: Waiting for a runner to pick up this job...

Typical GitLab Runner logs (journalctl -u gitlab-runner.service -f): In cases of a bad token or connectivity, the runner will fail to register or poll for jobs. You might see messages indicating connection failures or unauthorized access:

gitlab-runner[1234]: Checking for jobs... nothing
gitlab-runner[1234]: WARNING: Checking for jobs failed runner=xxxxxx status=403 Forbidden
gitlab-runner[1234]: ERROR: Failed to request new job: runner is not healthy
gitlab-runner[1234]: WARNING: Failed to get new job... error
gitlab-runner[1234]: WARNING: Retrying...

Or, if there's a network issue preventing initial registration or subsequent polling:

gitlab-runner[1234]: ERROR: Registering runner... failed                 runner=xxxxx status=couldn't execute POST against https://gitlab.yourdomain.com/api/v4/runners: Post "https://gitlab.yourdomain.com/api/v4/runners": dial tcp <gitlab_ip>:443: connect: connection refused

Root Cause Analysis

The "Gitlab CI runner jobs pending stuck registration token" issue primarily points to a breakdown in communication or authentication between the GitLab Runner and your GitLab instance. The root causes can be multifaceted:

1. Incorrect or Expired Registration Token: This is the most prevalent cause. A typo during registration, using a token from the wrong project/group/instance, or a token that has been revoked or expired (though less common for registration tokens) will prevent the runner from authenticating with GitLab.
2. Network Connectivity Blockade: The runner server cannot reach the GitLab instance. This could be due to:
   • Firewall Rules: Local UFW/iptables on the runner server, cloud security groups, or corporate network firewalls blocking outbound HTTPS (port 443) traffic to the GitLab instance.
   • DNS Resolution Failure: The runner server cannot resolve the GitLab instance's hostname.
   • Proxy Issues: If the runner is behind an HTTP/HTTPS proxy, it might not be correctly configured to use it.
   • SSL/TLS Certificate Problems: The GitLab instance might be using a self-signed or custom CA certificate not trusted by the runner's operating system, leading to handshake failures.
3. GitLab Runner Service Malfunction:
   • Service Not Running: The gitlab-runner systemd service is stopped, failed, or hasn't started correctly.
   • Corrupted Configuration: The /etc/gitlab-runner/config.toml file contains incorrect GitLab URL, executor settings, or other misconfigurations that prevent proper operation.
   • Permissions Issues: The gitlab-runner user lacks necessary permissions to read its configuration file or execute commands.
4. Resource Constraints: While less common for pending state directly, severe resource exhaustion (CPU, memory, disk I/O) on the runner server can make the gitlab-runner process unresponsive, leading to job polling failures.
5. GitLab Instance Issues: The GitLab instance itself might be under heavy load, experiencing issues, or its API might be temporarily unavailable.

Step-by-Step Resolution

Follow these steps meticulously to diagnose and resolve your GitLab CI runner issues.

1. Verify GitLab Runner Service Status and Logs

First, ensure the GitLab Runner service is running and check its recent logs for immediate clues.

# Check the service status
sudo systemctl status gitlab-runner

# View real-time logs for the service
sudo journalctl -u gitlab-runner.service -f

• If the service is inactive (dead) or failed, start it:

  sudo systemctl start gitlab-runner
  

• If the service is active (running) but logs show Failed to request new job, status=403 Forbidden, or network errors, proceed to the next steps.

2. Confirm Network Connectivity to GitLab Instance

Ensure your runner server can reach your GitLab instance over HTTPS (port 443).

# Test DNS resolution
ping gitlab.yourdomain.com

# Test basic HTTPS connectivity (replace with your GitLab URL)
curl -v https://gitlab.yourdomain.com/api/v4/version

• DNS Issues: If ping fails or resolves to an incorrect IP, check /etc/resolv.conf and your DNS server configuration.
• Connection Refused/Timeout: If curl fails with Connection refused, Failed to connect, or Timeout, a firewall is likely blocking access.
  • Local Firewall (UFW):

    sudo ufw status
    # If active, check rules. Temporarily allow outgoing HTTPS (443) if necessary:
    # sudo ufw allow out 443
    

  • Cloud Firewall/Security Groups: Verify outbound rules for port 443 on your cloud provider (AWS Security Groups, Azure Network Security Groups, Google Cloud Firewall Rules).
  • Corporate Proxy: If you're behind a corporate proxy, you'll need to configure it in the runner's environment variables or config.toml. See step 6.
• SSL/TLS Certificate Issues: If curl shows SSL certificate problem: unable to get local issuer certificate or similar, the runner doesn't trust your GitLab instance's certificate. Proceed to Step 5.

3. Re-Register the GitLab Runner (The Most Common Fix for Token Issues)

An incorrect or stale registration token is a primary culprit. The most robust solution is to unregister the old runner entry and register a new one.

a. Identify and Unregister the Existing Runner

# List existing runners registered on this machine
sudo gitlab-runner list

You'll see output like:

Listing configured runners                          ConfigFile=/etc/gitlab-runner/config.toml
Runner ID=xxxxxxxxx url=https://gitlab.yourdomain.com/ name=my-ubuntu-runner executor=shell token=xxxxxxxxx

Unregistering a runner will immediately stop it from picking up new jobs. Ensure no critical CI/CD jobs are currently active on this runner before proceeding.

# Unregister the specific runner by URL and token or ID.
# Using --name and --url is often safer if you have multiple runners on one machine.
# Replace with your runner's details:
sudo gitlab-runner unregister --url https://gitlab.yourdomain.com/ --token xxxxxxxxx

# Alternatively, if you only have one runner or want to clear all, you can use:
# sudo gitlab-runner unregister --all-runners

After unregistration, verify its removal from /etc/gitlab-runner/config.toml.

b. Obtain a New Registration Token

1. Log in to your GitLab instance.
2. Navigate to your Project or Group where the runner is (or will be) associated.
3. Go to Settings > CI/CD.
4. Expand the "Runners" section.
5. Click on "Register a runner" (or "New project runner" / "New group runner").
6. Copy the new registration token displayed.

The registration token is sensitive. Treat it like a password. Do not share it publicly. Ensure you copy it precisely without any leading/trailing spaces or typos.

c. Register the New Runner

Use the gitlab-runner register command, providing the new token and appropriate configuration.

sudo gitlab-runner register

Follow the prompts:

• Enter your GitLab instance URL: https://gitlab.yourdomain.com/ (Crucial: ensure https:// and no trailing slashes if your instance doesn't expect it).
• Enter the registration token: Paste the new token you copied.
• Enter a description for the runner: Ubuntu 22.04 Shell Runner (or a descriptive name).
• Enter tags for the runner: ubuntu,shell,production (comma-separated, no spaces). These tags determine which jobs this runner will pick up.
• Enter an executor: shell (for direct command execution on the host) or docker (if you plan to use Docker containers for jobs). For most basic setups, shell is sufficient. If docker, you'll be prompted for a default image.

After successful registration, restart the service:

sudo systemctl restart gitlab-runner

Check journalctl -u gitlab-runner.service -f again. You should see Checking for jobs... nothing without errors, indicating successful communication. Your pending jobs should now start processing.

4. Validate config.toml Permissions and Content

The runner's configuration is stored in /etc/gitlab-runner/config.toml. Review its contents and permissions.

# View the configuration file
sudo cat /etc/gitlab-runner/config.toml

# Check permissions
sudo ls -l /etc/gitlab-runner/config.toml

• Permissions: The file should typically be owned by root:root with permissions 0644 or 0600. The gitlab-runner user should have read access.
• Content:
  • Ensure the url under [[runners]] matches your GitLab instance URL exactly.
  • Verify the token matches the new token you registered.
  • concurrent: Make sure this is not 0 (unless you explicitly want to disable job processing). A value of 1 or more allows jobs to run.
  • limit: Ensure this is not 0.
  • If using docker executor, verify privileged = true if your jobs require it (e.g., building Docker images).

5. Address SSL/TLS Certificate Issues

If your GitLab instance uses a self-signed certificate or a certificate from a custom Certificate Authority (CA), the runner needs to trust it.

a. For Self-Signed or Custom CA Certificates:

1. Obtain your GitLab instance's root CA certificate file (ca.crt). This is usually provided by your security team or can be extracted from your GitLab server.
2. Copy this .crt file to the runner server:

   sudo mkdir -p /etc/gitlab-runner/certs/
   sudo cp /path/to/your/gitlab_ca.crt /etc/gitlab-runner/certs/gitlab.yourdomain.com.crt
   

   Replace gitlab.yourdomain.com.crt with a descriptive name, often the hostname of your GitLab instance.
3. Update the system's trusted certificates store (Ubuntu 22.04 LTS):

   sudo cp /etc/gitlab-runner/certs/gitlab.yourdomain.com.crt /usr/local/share/ca-certificates/
   sudo update-ca-certificates
   

4. Restart the GitLab Runner service:

   sudo systemctl restart gitlab-runner
   

While gitlab-runner register --tls-verify=false or configuring clone_url: "http://..." might bypass SSL/TLS errors, it's highly insecure and should never be used in production environments. Always aim for proper SSL/TLS configuration and certificate trust.

6. Configure Proxy Settings (If Applicable)

If your runner server needs to access the internet via an HTTP/HTTPS proxy, you must configure it.

a. Environment Variables (Recommended for Service-wide)

Edit the systemd service file to include proxy environment variables.

sudo systemctl edit gitlab-runner.service

This will open an editor for an override file. Add the following, replacing with your proxy details:

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080"
Environment="HTTPS_PROXY=http://proxy.example.com:8080"
Environment="NO_PROXY=localhost,127.0.0.1,.yourdomain.com,gitlab.yourdomain.com"

Save and exit. Then, reload systemd and restart the runner:

sudo systemctl daemon-reload
sudo systemctl restart gitlab-runner

b. In config.toml (Less common for system-wide, but possible for specific runners)

You can also add proxy_url under the [runners.docker] or [runners.shell] section depending on your executor. However, environment variables are generally more robust for the runner process itself.

7. Update GitLab Runner

Ensure your GitLab Runner is up-to-date. Outdated versions can sometimes have compatibility issues or bugs that have been patched.

# Download the latest Debian package for AMD64
curl -LJO "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb"

# Install the package
sudo dpkg -i gitlab-runner_amd64.deb

# Restart the service to apply updates
sudo systemctl restart gitlab-runner

8. Clear Runner Cache (General Troubleshooting)

While less likely to resolve registration token issues directly, a corrupted runner cache can sometimes lead to unexpected behavior.

# This command attempts to clean up old job caches if configured.
# You might need to adjust based on your cache configuration in config.toml.
sudo gitlab-runner clean

If your runner uses specific cache directories, you might need to manually inspect and clear them.

By systematically working through these steps, you should be able to pinpoint and resolve the underlying issues causing your GitLab CI runner jobs to be stuck in a "pending" state due to registration token or related connectivity problems. Always monitor the runner's logs (journalctl -u gitlab-runner.service -f) during and after applying changes to confirm the resolution.