Troubleshooting GitLab CI Runner Jobs Stuck in Pending on Alpine Linux

Resolve GitLab CI runner jobs stuck in pending state on Alpine Linux. This guide covers common causes like networking, invalid tokens, and Docker setup, ensuring your CI/CD pipelines run smoothly.

When your GitLab CI/CD pipeline jobs remain stuck in a "Pending" state, it indicates that your GitLab CI Runner is either unable to connect to the GitLab instance to fetch new jobs, or it's failing to properly execute them. This guide specifically addresses scenarios encountered when running GitLab Runners on Alpine Linux, a common choice for its lightweight footprint, often within Docker containers. A common misconception is that the "registration token" itself is literally stuck; rather, it often points to a failure in the runner's initial handshake or ongoing communication with the GitLab coordination server.

Symptom & Error Signature

The primary symptom is that jobs initiated within your GitLab project's CI/CD pipeline remain indefinitely in the "Pending" status within the GitLab UI, never transitioning to "Running" or "Failed."

Typical error signatures, which can be found in the GitLab Runner logs, include:

# Runner failing to connect to GitLab instance
ERROR: Failed to dial GitLab: Get "https://gitlab.example.com/api/v4/version": dial tcp 192.0.2.10:443: connect: connection refused
ERROR: Check for jobs failed: error

# Runner failing due to SSL/TLS certificate issues (e.g., self-signed or expired certs)
ERROR: Failed to dial GitLab: Get "https://gitlab.example.com/api/v4/runners/verify": x509: certificate signed by unknown authority
ERROR: Runner registration failed: Post "https://gitlab.example.com/api/v4/runners": x509: certificate signed by unknown authority

# Runner configured with an invalid or expired registration token
ERROR: Failed to register runner. A 401 response was returned by the API indicating that the token is invalid or has insufficient permissions.
WARNING: Checking for jobs... failed                 runner=xxxxxxxx status=401 Unauthorized

# Runner attempting to register but encountering network issues
WARNING: Checking for jobs...                        runner=xxxxxxxx
ERROR: Checking for jobs failed: Get "https://gitlab.example.com/api/v4/jobs/request": EOF

Root Cause Analysis

The "pending stuck" state, especially when associated with "registration token" issues, typically stems from one of the following underlying problems:

1. Network Connectivity Issues: The most common culprit. The runner host (or the container it resides in) cannot establish a stable network connection to your GitLab instance. This could be due to:
   • Firewall rules: Host-based firewalls (e.g., iptables on Alpine) or network firewalls blocking outbound traffic on port 443 (or 80).
   • DNS Resolution: Incorrect or failed DNS lookups for the GitLab instance hostname.
   • Proxy Configuration: If the runner is behind a corporate proxy, it might not be correctly configured to use it.
   • Routing Problems: Incorrect network routes preventing access to the GitLab server.
2. Incorrect or Expired Registration Token: The token used during runner registration is invalid, has expired, or belongs to a different GitLab instance, group, or project than intended.
3. Misconfigured config.toml: The primary configuration file for GitLab Runner (/etc/gitlab-runner/config.toml) contains incorrect url, token, executor, or tags settings.
4. Docker Daemon Issues (for Docker Executor): If the runner is configured to use the Docker executor, the Docker daemon might not be running, the runner user might lack permissions to access the Docker socket (/var/run/docker.sock), or the Docker daemon itself is experiencing issues.
5. SSL/TLS Certificate Problems: If your GitLab instance uses self-signed or internally managed SSL certificates, the Alpine-based runner might not trust the Certificate Authority (CA), leading to handshake failures.
6. System Time Skew: A significant time difference between the runner host and the GitLab instance can cause SSL certificate validation failures, even with valid certificates.
7. Resource Constraints: While less common for initial "pending" states, a runner host critically low on disk space, memory, or CPU can prevent the runner service from operating correctly or spawning job environments.
8. GitLab Runner Service Status: The gitlab-runner service itself might not be running, or it might be in a crashed/restarting loop.

Step-by-Step Resolution

Follow these steps to diagnose and resolve your GitLab CI runner pending jobs issue on Alpine Linux.

1. Verify GitLab Runner Service Status

First, confirm that the gitlab-runner service is actually running on your Alpine host.

If you installed GitLab Runner directly on Alpine (using apk):

# Check service status using OpenRC (Alpine's default init system)
rc-service gitlab-runner status

# If not running, attempt to start it
rc-service gitlab-runner start

# View recent logs (if available through OpenRC or direct log files)
grep "gitlab-runner" /var/log/messages # or similar log file location

If your GitLab Runner is running as a Docker container (common for Alpine):

# List running containers and find your runner container
docker ps

# Check the logs of the specific runner container (replace <CONTAINER_ID>)
docker logs <CONTAINER_ID> --tail 50 -f

2. Verify Network Connectivity to GitLab Instance

Ensure your Alpine host or runner container can reach your GitLab instance.

# Replace gitlab.example.com with your GitLab instance URL
GITLAB_URL="gitlab.example.com"

# Check DNS resolution
apk add bind-tools # Install dig on Alpine if not present
dig +short $GITLAB_URL

# Test basic connectivity to port 443 (HTTPS)
apk add busybox-extras # Install telnet on Alpine if not present
telnet $GITLAB_URL 443

# Test HTTP/HTTPS reachability and certificate validity
curl -vI https://$GITLAB_URL/api/v4/version

Look for Connected to... in telnet and a successful HTTP status code (e.g., 200 OK) in curl. If curl shows curl: (60) SSL certificate problem: unable to get local issuer certificate, you have an SSL certificate trust issue.

If you are behind a corporate proxy, ensure the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables are correctly set for the gitlab-runner service or container. For Alpine, this might involve editing /etc/profile.d/proxy.sh or passing --env flags to docker run.

3. Validate GitLab Runner Configuration (config.toml)

The runner's core configuration is in /etc/gitlab-runner/config.toml.

# View the configuration file
cat /etc/gitlab-runner/config.toml

Pay close attention to the following sections:

• url: Must exactly match your GitLab instance URL.
• token: This is the runner's authentication token. It must be valid and correspond to the url.
• executor: Common executors are docker, shell, kubernetes. Ensure the chosen executor is appropriate and configured correctly.
• concurrent: Defines how many jobs the runner can execute simultaneously. If set to 0 or too low, it can appear stuck.

If the token in config.toml is incorrect or expired, you'll need to re-register the runner (Step 5). Do NOT manually edit the token in config.toml; it's securely managed.

4. Address SSL/TLS Certificate Trust Issues

If curl or runner logs indicate x509: certificate signed by unknown authority, your Alpine system does not trust your GitLab instance's SSL certificate.

# Install ca-certificates if not already present
apk add ca-certificates

# If using a custom or self-signed certificate, you need to add it to the system trust store.
# First, obtain your GitLab server's certificate (e.g., gitlab.crt).
# Copy the certificate to the trusted certificates directory:
cp /path/to/your/gitlab.crt /usr/local/share/ca-certificates/gitlab.crt

# Update the CA certificate store
update-ca-certificates

# For Dockerized runners, you might need to mount the custom certificate into the container
# or build a custom runner image with the certificate pre-installed.
# Example docker run option:
# -v /path/to/your/gitlab.crt:/etc/gitlab-runner/certs/gitlab.example.com.crt:ro

After updating certificates, restart the gitlab-runner service or container.

5. Re-register the GitLab Runner

If network connectivity and SSL certificates are confirmed, but the runner still fails to pick up jobs or shows "401 Unauthorized" errors, the registration token might be invalid. The safest approach is to re-register the runner.

Re-registering a runner will generate a new token and update config.toml. Make sure you have the correct registration URL and token from your GitLab project/group/instance settings. Delete the old runner entry in the GitLab UI before re-registering if possible.

Steps to re-register:

1. Stop the gitlab-runner service/container:

   If installed directly on Alpine:

   rc-service gitlab-runner stop
   

   If running as a Docker container:

   docker stop <CONTAINER_ID>
   docker rm <CONTAINER_ID> # Remove the old container
   

2. Remove the existing config.toml (or back it up):

   mv /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak
   

3. Perform the registration process:

   If installed directly on Alpine:

   # Run the interactive registration command
   gitlab-runner register
   # Follow the prompts:
   # 1. Enter your GitLab instance URL (e.g., https://gitlab.example.com/)
   # 2. Enter the registration token (from GitLab UI -> CI/CD -> Runners -> New runner -> Show runner registration token)
   # 3. Enter a description for the runner
   # 4. Enter tags (comma-separated, e.g., "alpine,docker")
   # 5. Enter the executor (e.g., "docker")
   # 6. Enter the default Docker image (e.g., "alpine/git")
   

   If running as a Docker container, you will typically register it by passing environment variables to docker run:

   docker run -d --name gitlab-runner --restart always 
     -v /etc/gitlab-runner:/etc/gitlab-runner 
     -v /var/run/docker.sock:/var/run/docker.sock 
     gitlab/gitlab-runner:alpine 
     register 
     --non-interactive 
     --url "https://gitlab.example.com/" 
     --registration-token "YOUR_REGISTRATION_TOKEN" 
     --executor "docker" 
     --description "My Alpine Docker Runner" 
     --tag-list "alpine,docker" 
     --docker-image "alpine/git" 
     --locked="false" 
     --access-level="not_protected" # or "ref_protected"
   

   Replace YOUR_REGISTRATION_TOKEN and gitlab.example.com with your actual values.

4. Start the gitlab-runner service/container:

   If installed directly on Alpine:

   rc-service gitlab-runner start
   

   If running as a Docker container (it would have started automatically with -d --restart always from the docker run command).

6. Address Docker Daemon Issues (for Docker Executor)

If your runner uses the docker executor, ensuring Docker is healthy is crucial.

1. Verify Docker Daemon Status:

   If Docker is installed directly on Alpine (the host OS for the runner):

   rc-service docker status
   # If not running:
   rc-service docker start
   

   If Docker is on an Ubuntu/Debian host (where your Alpine runner container might run):

   systemctl status docker
   # If not running:
   systemctl start docker
   

2. Check Docker Socket Permissions: The gitlab-runner user needs access to /var/run/docker.sock.

   ls -l /var/run/docker.sock
   

   The output should show root docker or similar. Ensure the user running the gitlab-runner process (or the container) is part of the docker group.

   # If gitlab-runner user is not in the docker group:
   apk add shadow # For usermod on Alpine
   usermod -aG docker gitlab-runner
   # Restart the gitlab-runner service after modifying groups
   rc-service gitlab-runner restart
   

   Adding users to the docker group grants root-level privileges over the Docker daemon. This should be done with caution and only for trusted services.

7. Verify System Time Synchronization

Significant time discrepancies can invalidate SSL certificates or tokens.

date

If the time is incorrect, synchronize it:

# Install NTP client
apk add ntp

# Manually synchronize time
ntpdate pool.ntp.org

# Consider enabling ntpd for continuous sync
rc-update add ntpd default
rc-service ntpd start

8. Check System Resources and Disk Space

Ensure your Alpine host has sufficient resources.

df -h # Check disk space
free -h # Check memory usage
top # or htop (apk add htop) for CPU/memory/process overview

Lack of disk space can prevent job artifacts from being stored or new Docker images from being pulled, leading to failures that might initially appear as pending jobs.

9. Update GitLab Runner and Docker

Outdated software can sometimes lead to obscure compatibility issues.

To update GitLab Runner on Alpine:

apk update
apk upgrade gitlab-runner
rc-service gitlab-runner restart

To update a Dockerized GitLab Runner:

docker pull gitlab/gitlab-runner:alpine
# Then stop and remove the old container, and start a new one with the updated image
# (refer to Step 5 for removal and re-registration, or simply replace the image if configuration is persistent)

By systematically going through these steps, you should be able to identify and resolve the root cause of GitLab CI runner jobs getting stuck in a pending state on Alpine Linux. Always check logs thoroughly after each step for new errors or successful operations.