Troubleshooting GitLab CI Runner Jobs Pending: Stuck Registration Token on Debian 12 Bookworm

Resolve GitLab CI jobs stuck in 'pending' on Debian 12 due to runner registration token issues. This guide covers network, TLS, and configuration fixes.

When deploying or managing GitLab CI runners, encountering jobs that remain perpetually in a "pending" state can be a significant roadblock in your CI/CD pipeline. On Debian 12 Bookworm, this often points to issues during the runner's registration process, particularly around establishing a secure connection to the GitLab instance or correctly utilizing the registration token. This guide provides an expert-level, step-by-step approach to diagnose and resolve such problems.

Symptom & Error Signature

Users will primarily observe jobs stuck in the "pending" state within the GitLab UI, often with the runner's status appearing offline or unregistered. When attempting to register a new runner or inspecting existing runner logs, you might encounter output similar to these examples:

GitLab UI:

• Jobs listed with a status of pending indefinitely.
• The "Runners" section of your project/group might show the runner as "Never contacted" or "Disconnected."

Terminal/Runner Logs (e.g., journalctl -u gitlab-runner.service or during manual registration):

# During manual registration via `gitlab-runner register`
Please enter the GitLab instance URL (e.g. https://gitlab.com/): https://your-gitlab.example.com
Please enter the registration token for this runner: your_registration_token
Please enter a description for the runner: my-debian12-runner
Please enter the tags for the runner (comma separated): debian,docker
Please enter the executor: docker, shell, ssh, kubernetes, custom: docker
Please enter the default Docker image (e.g. ruby:2.7): alpine:latest
Registering runner... failed                 runner=your_registration_token status=couldn't execute POST against https://your-gitlab.example.com/api/v4/runners: Post "https://your-gitlab.example.com/api/v4/runners": dial tcp [IP_OF_GITLAB]:443: connect: connection refused

# From journalctl -u gitlab-runner.service
Sep 07 09:00:01 debian12-host gitlab-runner[12345]: ERROR: Failed to create runner service                     error=Couldn't register runner. Registering runner... failed                 runner=your_registration_token status=couldn't execute POST against https://your-gitlab.example.com/api/v4/runners: Post "https://your-gitlab.example.com/api/v4/runners": x509: certificate signed by unknown authority
Sep 07 09:00:05 debian12-host gitlab-runner[12345]: WARNING: Checking for jobs... failed                      runner=abcdefgh version=16.4.0 error=couldn't execute POST against https://your-gitlab.example.com/api/v4/runners/verify: Post "https://your-gitlab.example.com/api/v4/runners/verify": dial tcp [IP_OF_GITLAB]:443: i/o timeout

Root Cause Analysis

The "stuck registration token" symptom primarily points to the runner's inability to establish a successful and trusted connection with the GitLab instance using the provided token. Common underlying reasons include:

1. Network Connectivity Issues:

   • Firewall Blockage: The runner host's outgoing connection to the GitLab instance (typically port 443 for HTTPS) is blocked by a local firewall (ufw, iptables) or an intermediate network firewall.
   • DNS Resolution Failure: The runner host cannot resolve the GitLab instance's hostname to an IP address.
   • Proxy Configuration: If the runner is behind an HTTP/S proxy, the proxy settings might be incorrect or missing.
   • Incorrect GitLab URL: A typo in the GitLab instance URL provided during registration.

2. TLS/SSL Certificate Verification Failures:

   • Self-Signed Certificates: If your GitLab instance uses a self-signed SSL certificate, or a certificate issued by an internal Certificate Authority (CA), the Debian 12 host might not trust it by default.
   • Expired or Invalid Certificates: The GitLab instance's certificate might be expired, revoked, or incorrectly configured.
   • Man-in-the-Middle (MITM) Proxy: An intercepting proxy might be presenting its own certificate, which the runner host doesn't trust.

3. Incorrect Registration Token:

   • Typo or Copy-Paste Error: The registration token copied from GitLab might have a typo or include leading/trailing whitespace.
   • Expired/Revoked Token: GitLab registration tokens can be revoked or may have an expiration if configured.

4. gitlab-runner Service Misconfiguration or Malfunction:

   • The gitlab-runner service might not be running or might be configured incorrectly in /etc/gitlab-runner/config.toml.
   • Permissions issues preventing the runner from writing to its configuration or log files.

5. Docker Daemon Issues (if using Docker executor):

   • The Docker service is not running or is not accessible by the gitlab-runner user. While less common for registration issues, it can cause jobs to pend indefinitely even if registration succeeds, as the runner can't execute tasks.

Step-by-Step Resolution

Follow these steps to systematically diagnose and resolve the "stuck registration token" issue on Debian 12.

1. Verify Basic Network Connectivity and DNS Resolution

Ensure your Debian 12 runner host can reach the GitLab instance.

# Replace your-gitlab.example.com with your actual GitLab instance URL
GITLAB_HOST="your-gitlab.example.com"
GITLAB_PORT="443" # Or 80 if not using HTTPS

echo "--- Pinging GitLab host ---"
ping -c 4 "${GITLAB_HOST}"

echo -e "n--- Checking DNS resolution for GitLab host ---"
dig +short "${GITLAB_HOST}"

echo -e "n--- Testing TCP connectivity to GitLab host on port ${GITLAB_PORT} ---"
# Using nc for a quick port check
nc -zv "${GITLAB_HOST}" "${GITLAB_PORT}"

echo -e "n--- Attempting cURL to GitLab instance ---"
# A simple curl to verify HTTPS connectivity (expect HTTP 302 or similar for root path)
curl -vvv "https://${GITLAB_HOST}/"

• Expected ping output: Successful replies from the GitLab IP address.
• Expected dig output: One or more IP addresses for your GitLab instance.
• Expected nc output: Connection to your-gitlab.example.com 443 port [tcp/https] succeeded!
• Expected curl output: A successful HTTP response, even if it's a redirect or an error page, indicating a connection was made. If you see curl: (6) Could not resolve host, curl: (7) Failed to connect, or curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL, it points to DNS, network, or SSL issues respectively.

If any of these network tests fail, investigate local firewall rules (sudo ufw status or sudo iptables -L), network routing, or DNS server configuration on your Debian 12 host first. If behind a proxy, ensure http_proxy, https_proxy, and no_proxy environment variables are correctly set for the gitlab-runner user or service.

2. Inspect GitLab Runner Service Logs

The gitlab-runner service logs are your primary source of error information.

sudo journalctl -u gitlab-runner.service --follow --no-pager

Look for ERROR messages, specifically those mentioning certificate signed by unknown authority, i/o timeout, connection refused, or failed to connect. These provide crucial clues about the exact point of failure.

3. Address SSL/TLS Certificate Issues

This is a very common cause of "stuck registration" on new systems, especially with internal or self-signed CAs.

a. Install Custom CA Certificates

If your GitLab instance uses an internal CA:

# Example: Download your CA certificate (e.g., your_internal_ca.crt)
# This might involve accessing an internal URL or a shared network drive.
# Place it in a trusted system location.
sudo mkdir -p /usr/local/share/ca-certificates/extra
sudo wget -O /usr/local/share/ca-certificates/extra/your_internal_ca.crt "http://internal-ca.example.com/your_internal_ca.crt"

# Update the system's CA store
sudo update-ca-certificates

# Verify the certificate is now trusted by trying curl again with no extra flags
curl -vvv "https://your-gitlab.example.com/"

After updating, restart the gitlab-runner service:

sudo systemctl restart gitlab-runner.service

b. Configure gitlab-runner to Trust a Specific CA File

If you prefer to configure the runner directly, or if the system-wide update isn't feasible, you can point the runner to a specific CA file.

# Assuming your CA cert is at /etc/ssl/certs/your_internal_ca.crt
sudo gitlab-runner register 
  --url https://your-gitlab.example.com 
  --registration-token YOUR_REGISTRATION_TOKEN 
  --description "Debian 12 Runner with Custom CA" 
  --executor docker 
  --docker-image "alpine:latest" 
  --tls-ca-file "/etc/ssl/certs/your_internal_ca.crt"

Or, if already registered, edit /etc/gitlab-runner/config.toml and add/modify the tls-ca-file entry under the [[runners]] section or [global_section]:

# /etc/gitlab-runner/config.toml
# ...
[[runners]]
  name = "my-debian12-runner"
  url = "https://your-gitlab.example.com"
  token = "YOUR_REGISTERED_TOKEN"
  executor = "docker"
  tls-ca-file = "/etc/ssl/certs/your_internal_ca.crt" # Add this line
  [runners.docker]
    # ...

After editing config.toml, restart the service:

sudo systemctl restart gitlab-runner.service

Avoid --tls-skip-verify in Production! While --tls-skip-verify can bypass certificate errors during registration and get jobs running, it creates a severe security vulnerability by disabling critical TLS verification. This makes your connections susceptible to Man-in-the-Middle attacks. Only use this for temporary debugging in isolated, non-production environments. Always strive to properly configure certificate trust.

If you must use it for testing:

sudo gitlab-runner register 
  --url https://your-gitlab.example.com 
  --registration-token YOUR_REGISTRATION_TOKEN 
  --description "Debian 12 Test Runner" 
  --executor docker 
  --docker-image "alpine:latest" 
  --tls-skip-verify # <-- DANGEROUS FOR PROD!

4. Re-register the Runner with Correct Details

If you suspect an incorrect URL or token, it's best to clean up and re-register.

a. Remove Existing Runner Configuration

First, get the runner's current status and UUID/token.

sudo gitlab-runner status
sudo gitlab-runner list

Note the runner's token or UUID from the list command. Then, unregister it.

# Option 1: Unregister by token (recommended)
sudo gitlab-runner unregister --token YOUR_REGISTERED_TOKEN

# Option 2: Unregister by name (if unique)
# sudo gitlab-runner unregister --name "my-debian12-runner"

# Option 3: If unregistration fails, manually remove the entry from config.toml
# You might need to use `sudo nano /etc/gitlab-runner/config.toml` and delete the corresponding [[runners]] block.

b. Obtain a New Registration Token

Go to your GitLab project or group's CI/CD settings (Settings > CI/CD > Runners), copy a new registration token. Using a fresh token helps rule out token expiration or previous accidental revocation.

c. Register the Runner

Use the newly obtained token and verify the GitLab URL.

sudo gitlab-runner register 
  --url https://your-gitlab.example.com 
  --registration-token YOUR_NEW_REGISTRATION_TOKEN 
  --description "Debian 12 Bookworm Runner" 
  --tag-list "debian,bookworm,docker" 
  --executor "docker" 
  --docker-image "alpine:latest" 
  --non-interactive 
  --maintenance-note "Managed via automation" # Optional, for visibility in GitLab UI

Replace https://your-gitlab.example.com and YOUR_NEW_REGISTRATION_TOKEN with your actual values. If using a custom CA, include --tls-ca-file "/path/to/your_ca.crt" as discussed in Step 3.

The --non-interactive flag is great for scripting, but if you want to be prompted for details, omit it.

5. Verify gitlab-runner Service Status

Ensure the runner service is active and running after any configuration changes.

sudo systemctl status gitlab-runner.service

The output should show Active: active (running). If it's failed or inactive, investigate with sudo journalctl -xeu gitlab-runner.service.

6. Check Docker Daemon Status (if using Docker executor)

If your runner uses the docker executor, ensure the Docker daemon is running and the gitlab-runner user has permissions to interact with it.

sudo systemctl status docker.service

If Docker is not running, start it:

sudo systemctl start docker.service
sudo systemctl enable docker.service

Also, ensure the gitlab-runner user is part of the docker group (this is typically handled by the gitlab-runner installation script).

sudo usermod -aG docker gitlab-runner
# You may need to restart the gitlab-runner service after this to apply group changes
sudo systemctl restart gitlab-runner.service

7. Review config.toml for Errors

Open the runner's configuration file for manual inspection.

sudo nano /etc/gitlab-runner/config.toml

Verify:

• concurrent setting is appropriate for your server resources.
• url under [[runners]] is correct.
• token under [[runners]] matches the registered token (this is the registered token, not the registration token from GitLab UI).
• executor is set correctly (e.g., docker).
• Any [runners.docker] or [runners.kubernetes] sections have correct configurations (e.g., privileged, volumes, pull_policy).
• No extraneous characters or malformed TOML syntax.

After any manual edits, always restart the gitlab-runner service:

sudo systemctl restart gitlab-runner.service

By methodically working through these steps, you should be able to identify and rectify the underlying cause of your GitLab CI runner jobs getting stuck in "pending" due to registration token or connectivity issues on Debian 12 Bookworm.