Troubleshooting GitLab CI Runner Jobs Pending: Stuck Registration Token on CentOS Stream / Rocky Linux

Resolve GitLab CI jobs stuck in 'pending' on CentOS/Rocky Linux due to registration token issues, network blocks, or misconfigurations. Get your pipelines running!

Introduction

As a seasoned Systems Administrator, few things are as frustrating as a seemingly operational GitLab CI runner failing to pick up jobs, leaving them indefinitely stuck in a "pending" state. This often points to a fundamental communication breakdown between the runner and the GitLab instance, frequently rooted in an incomplete or flawed registration process, particularly concerning the registration token. This guide provides a highly technical, step-by-step resolution for such issues specifically on CentOS Stream and Rocky Linux environments, tackling common culprits from network woes to SELinux policies.

Symptom & Error Signature

The primary symptom is that your GitLab CI jobs remain in the "Pending" state within the GitLab UI, even when gitlab-runner status reports the runner service as running. While there might not always be a clear error message directly in the job logs (as the job hasn't started), the lack of activity is the key indicator.

You might observe:

• GitLab UI: Jobs appearing indefinitely in "Pending".
• On the Runner Server:

  # Check runner service status
  systemctl status gitlab-runner.service
  # Expected output:
  # ● gitlab-runner.service - GitLab Runner
  #    Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: disabled)
  #    Active: active (running) since ...
  #    Main PID: XXXX (gitlab-runner)
  #    Tasks: 8 (limit: 11181)
  #    Memory: 112.5M
  #    CGroup: /system.slice/gitlab-runner.service
  #            └─XXXX /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner
  
  # Verification command may show issues or not list the expected runner
  gitlab-runner verify
  # Output might be empty, or show an error like:
  # ERROR: Verifying runner... is not healthy!  runner=XXXXXXX status=couldn't execute POST against https://gitlab.example.com/api/v4/runners/verify: Post "https://gitlab.example.com/api/v4/runners/verify": dial tcp ...: connect: connection refused
  # Or simply:
  # Runners online: X
  # Runners of this system: 0 (if not registered or misconfigured)
  
  # Key log entries indicating connectivity or token issues:
  journalctl -u gitlab-runner.service -f
  # Look for messages similar to:
  # ERROR: Failed to get new job...  error=couldn't execute POST against https://gitlab.example.com/api/v4/jobs/request: Post "https://gitlab.example.com/api/v4/jobs/request": x509: certificate signed by unknown authority
  # ERROR: Failed to get new job...  error=couldn't execute POST against https://gitlab.example.com/api/v4/jobs/request: Post "https://gitlab.example.com/api/v4/jobs/request": dial tcp ...: connect: connection refused
  # WARNING: Checking for jobs... failed                runner=XXXXXXX status=403 Forbidden
  

Root Cause Analysis

The "pending" state, especially when the runner service appears active, typically indicates a failure in the runner's ability to communicate with the GitLab instance to request new jobs. This can stem from several underlying issues:

1. Network Connectivity Issues:

   • Firewall: The server's local firewall (firewalld) or an external network firewall blocking outbound connections from the runner to the GitLab instance (usually port 443/TCP or 80/TCP).
   • DNS Resolution: The runner server unable to resolve the GitLab instance's hostname.
   • Proxy Configuration: Incorrect or missing proxy settings if the runner is behind a proxy server.
   • TLS/SSL Certificate Issues: GitLab instance using a self-signed certificate not trusted by the runner's system, or an expired/invalid certificate.

2. Incorrect or Expired Registration Token/URL:

   • The runner was registered with an incorrect token, or the token has since expired or been revoked from the GitLab UI.
   • The GitLab instance URL provided during registration (or in config.toml) is incorrect.

3. SELinux Policies:

   • SELinux on CentOS Stream/Rocky Linux can prevent gitlab-runner from making network connections, writing to necessary directories, or executing scripts, even if standard file permissions are correct.

4. Corrupted Runner Configuration (config.toml):

   • Manual edits to /etc/gitlab-runner/config.toml could introduce syntax errors or misconfigurations that prevent proper operation.

5. Resource Constraints (Less Common for Registration):

   • While less likely to cause registration issues, severe resource starvation could theoretically impact background processes.

Step-by-Step Resolution

Follow these steps meticulously to diagnose and resolve the issue on your CentOS Stream / Rocky Linux server.

1. Verify Basic Network Connectivity & GitLab URL

First, ensure the runner host can reach your GitLab instance and that the URL in the runner's configuration is correct.

# 1. Ping the GitLab instance hostname
ping -c 4 gitlab.example.com

# 2. Test HTTPS connectivity to GitLab (assuming HTTPS)
curl -v https://gitlab.example.com/api/v4/version
# Look for successful connection and HTTP 200/201 response.
# If you see "Could not resolve host" -> DNS issue.
# If "Connection refused" or "Failed to connect" -> Firewall or GitLab instance down/misconfigured.
# If "SSL certificate problem" -> TLS/SSL issue (see Step 5).

# 3. Check the GitLab Runner configuration file for the correct URL
sudo cat /etc/gitlab-runner/config.toml | grep 'url ='
# Ensure the URL matches your GitLab instance (e.g., https://gitlab.example.com/).

2. Check and Configure Firewalld

firewalld is the default firewall management tool on CentOS Stream and Rocky Linux. It's a very common culprit.

# 1. List all active firewall rules for the default zone
sudo firewall-cmd --list-all

# Look for output that indicates if services like 'https' or port '443/tcp' are allowed.
# If not present, you need to add it:
> [!IMPORTANT]
> Replace `443/tcp` with the actual port your GitLab instance listens on if it's not standard HTTPS.

sudo firewall-cmd --zone=public --add-port=443/tcp --permanent
sudo firewall-cmd --reload

# 2. Retest connectivity with curl after firewall changes.
curl -v https://gitlab.example.com/api/v4/version

3. Validate and Re-register GitLab Runner

An incorrect, expired, or revoked registration token is a frequent cause. The safest approach is to re-register the runner.

# 1. Stop the GitLab Runner service
sudo systemctl stop gitlab-runner.service

# 2. Unregister the existing runner(s)
# First, find the runner token or description from config.toml to identify which one to remove.
# You can also use `gitlab-runner list` or check the GitLab UI.
# If you want to unregister ALL runners on this machine:
sudo gitlab-runner unregister --all-runners

# Or, to unregister a specific runner using its token (from config.toml)
# sudo gitlab-runner unregister --token <YOUR_RUNNER_TOKEN>

> [!WARNING]
> Unregistering a runner permanently removes its association with your GitLab instance. Any jobs assigned to it will be lost or need to be re-queued. Ensure you have the necessary permissions to create a new runner token in GitLab.

# 3. Obtain a NEW registration token from your GitLab instance:
# Navigate to your GitLab project or group's CI/CD settings:
# `Settings` -> `CI/CD` -> `Runners` -> `Register a new runner`.
# Copy the token provided.

# 4. Re-register the GitLab Runner
# This command will prompt you for the GitLab URL and the new registration token.
sudo gitlab-runner register
# Enter the GitLab instance URL (e.g., https://gitlab.example.com)
# Enter the new registration token
# Enter a description for the runner (e.g., "CentOS Stream Shared Runner")
# Enter tags (e.g., "linux,shell,centos") - important for job selection
# Enter the executor (e.g., "shell", "docker", "kubernetes"). "shell" is common for basic setups.

# Example interaction:
# Enter the GitLab instance URL (for example, https://gitlab.com/):
# https://gitlab.example.com
# Enter the registration token:
# XXXXXXXXXX_YOUR_NEW_TOKEN_XXXXXXXXXX
# Enter a description for the runner:
# my-rocky-linux-runner
# Enter tags for the runner (comma-separated):
# rocky,shell,production
# Enter an executor: shell, docker, docker-ssh, docker+machine, docker-ssh+machine, custom, ssh, virtualbox, parallels, kubernetes:
# shell
# Runner registered successfully. Feel free to start it, but if it's a concurrent-limited runner, register more of them to increase the number of concurrent jobs that can be run.

# 5. Start the GitLab Runner service
sudo systemctl start gitlab-runner.service
sudo systemctl enable gitlab-runner.service # Ensure it starts on boot

# 6. Verify the runner status and check GitLab UI for pending jobs
sudo gitlab-runner verify
journalctl -u gitlab-runner.service -f

4. Examine GitLab Runner Logs for Deeper Insights

If the problem persists, detailed logs are crucial.

# View the live logs of the GitLab Runner service
journalctl -u gitlab-runner.service -f

# Look for:
# - Connection errors (e.g., "connection refused", "timeout")
# - TLS/SSL certificate errors ("x509: certificate signed by unknown authority")
# - HTTP 403 Forbidden errors (often indicative of a token issue or invalid URL)
# - Any messages related to "Failed to get new job" or "Checking for jobs... failed"

# If you encounter TLS/SSL errors with a self-signed certificate, you need to configure the runner to trust it:
# 1. Place your CA certificate (e.g., ca.crt) into `/etc/gitlab-runner/certs/`
#    sudo mkdir -p /etc/gitlab-runner/certs/
#    sudo cp /path/to/your/ca.crt /etc/gitlab-runner/certs/gitlab.example.com.crt
# 2. Restart the runner
#    sudo systemctl restart gitlab-runner.service

5. Address SELinux Policies

SELinux can silently block network connections or file access. Temporarily setting SELinux to permissive mode can help diagnose if it's the culprit.

# 1. Check current SELinux status
sudo getenforce

# 2. Temporarily set SELinux to permissive mode (for testing)
# This will allow operations that would normally be blocked, but will log them.
sudo setenforce 0

# 3. Restart GitLab Runner and retest
sudo systemctl restart gitlab-runner.service
# Check GitLab UI for jobs, and runner logs.

# If jobs now run, SELinux is the issue. Re-enable enforcing mode:
sudo setenforce 1

> [!WARNING]
> Running with SELinux disabled or in permissive mode indefinitely is a security risk. If SELinux is the cause, you must create a permanent policy.

# 4. Generate a custom SELinux policy (if SELinux was the culprit)
# Look at the audit log for denial messages:
sudo ausearch -c gitlab-runner --raw | audit2allow -M gitlab-runner-custom
# This command will create `gitlab-runner-custom.pp` and `gitlab-runner-custom.te`.

# 5. Install the custom policy
sudo semodule -i gitlab-runner-custom.pp

# 6. Verify the policy is loaded
sudo semodule -l | grep gitlab-runner-custom

# 7. Restart the GitLab Runner service and test
sudo systemctl restart gitlab-runner.service

6. Check Proxy Settings (If Applicable)

If your runner server requires a proxy to access the internet, these settings must be configured for gitlab-runner.

# 1. Edit the GitLab Runner service unit file to include proxy variables
sudo systemctl edit gitlab-runner.service

# Add the following lines in the `[Service]` section (adjust proxy details):
# [Service]
# Environment="http_proxy=http://proxy.example.com:8080"
# Environment="https_proxy=http://proxy.example.com:8080"
# Environment="no_proxy=localhost,127.0.0.1,gitlab.example.com"

# 2. Reload systemd and restart the service
sudo systemctl daemon-reload
sudo systemctl restart gitlab-runner.service

# Alternatively, configure proxy within config.toml (less common for global system proxy, but an option):
# sudo vi /etc/gitlab-runner/config.toml
#
# Add or modify:
# [[runners]]
#   ...
#   [runners.proxy]
#     no_proxy = ["localhost", "127.0.0.1", "gitlab.example.com"]
#     http_proxy = "http://proxy.example.com:8080"
#     https_proxy = "http://proxy.example.com:8080"

By systematically working through these steps, you should be able to identify and resolve the root cause of your GitLab CI runner jobs being stuck in a pending state on CentOS Stream or Rocky Linux. Remember to always check logs (journalctl -u gitlab-runner.service -f) after each change to observe its effect.