Troubleshooting GitLab CI Runner Jobs Stuck Pending on macOS: Registration Token & Connectivity Issues

Fix GitLab CI runner jobs stuck in pending state on macOS. This guide covers registration token errors, network connectivity, SSL, and local firewall configurations.

When running GitLab CI/CD pipelines in a macOS local development environment, it's a common and frustrating experience to see your jobs perpetually stuck in a "Pending" state. This usually indicates that your local GitLab Runner instance is unable to register with the GitLab server or, once registered, cannot pick up new jobs. The root cause often lies in network connectivity, incorrect registration tokens, SSL certificate issues, or local macOS firewall settings preventing proper communication. This guide will walk you through a systematic troubleshooting process to resolve these persistent "Pending" job issues.

Symptom & Error Signature

You will primarily observe the following symptoms:

1. GitLab UI: Your CI/CD jobs, whether triggered manually or by a push, remain in a "Pending" state indefinitely, even if a runner is theoretically available and online.

   (Placeholder for a screenshot showing a job stuck in pending)

2. Runner Registration Terminal Output: When attempting to register a new runner using gitlab-runner register, you might encounter errors indicating connection failures or inability to POST to the GitLab API.

   Running in system-mode.
   
   Please enter the GitLab instance URL (for example, https://gitlab.com/):
   https://gitlab.example.com
   Please enter the registration token:
   <YOUR_REGISTRATION_TOKEN>
   Please enter a description for the runner:
   My macOS Local Runner
   Please enter the tags for the runner (comma separated):
   macos,local
   Registering runner... failed                 runner=<TOKEN_PREFIX> status=couldn't execute POST against https://gitlab.example.com/api/v4/runners: Post "https://gitlab.example.com/api/v4/runners": dial tcp <GITLAB_IP>:443: connect: connection refused
   FATAL: Failed to register the runner. You may be having network problems.
   

   Or, a timeout:

   Registering runner... failed                 runner=<TOKEN_PREFIX> status=couldn't execute POST against https://gitlab.example.com/api/v4/runners: Post "https://gitlab.example.com/api/v4/runners": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
   FATAL: Failed to register the runner. You may be having network problems.
   

3. Runner Logs (Running in Debug Mode): When running gitlab-runner run --debug, you might see repetitive connection errors, SSL handshake failures, or messages indicating the runner cannot fetch new jobs.

   DEBU[0005] Checking for jobs... nothing                      runner=aBcDeFgH
   ERRO[0005] Failed to get the Go-GitLab client for https://gitlab.example.com/api/v4 with a timeout of 10s. error="Post "https://gitlab.example.com/api/v4/runners/verify": dial tcp <GITLAB_IP>:443: connect: operation timed out"
   ERRO[0005] Failed to send the runner.verify request to GitLab API, failed to verify runner. runner=aBcDeFgH status=403 Forbidden
   

Root Cause Analysis

The "pending stuck" issue typically stems from one or more of the following underlying problems:

1. Network Connectivity Issues: This is the most prevalent cause.

   • Local Firewall (macOS/Third-Party): macOS's built-in firewall or third-party solutions (e.g., Little Snitch, LuLu) blocking outgoing connections from the gitlab-runner process or Docker containers to the GitLab instance.
   • Corporate/VPN Firewalls: If you're on a corporate network or VPN, it might restrict access to your GitLab instance's URL/IP.
   • DNS Resolution: The macOS environment failing to correctly resolve the GitLab instance's hostname to an IP address.
   • Proxy Configuration: Incorrect or missing proxy settings for the gitlab-runner process or Docker daemon.

2. Incorrect GitLab Instance URL or Registration Token:

   • Typos: Simple errors in the GitLab instance URL (e.g., http instead of https).
   • Invalid Token: The registration token copied from the GitLab UI is incorrect, expired, or belongs to a different project/group.

3. SSL/TLS Certificate Problems:

   • Self-Signed Certificates: If your GitLab instance uses a self-signed SSL certificate, the macOS environment or Docker containers might not trust it, leading to handshake failures.
   • Missing CA Certificates: Intermediate CA certificates not being properly installed or recognized.

4. Docker Daemon and Environment Issues (if using Docker executor):

   • Docker Desktop Not Running: The Docker daemon required by the runner's Docker executor is not active.
   • Docker Network Problems: Internal Docker network issues preventing containers from reaching the internet or the host.
   • Resource Constraints: Docker Desktop running out of memory or CPU resources.

5. Corrupted Runner Configuration: The config.toml file might be misconfigured or corrupted, preventing the runner from starting or picking up jobs correctly.

Step-by-Step Resolution

Follow these steps systematically to diagnose and resolve your "pending stuck" GitLab CI runner jobs on macOS.

1. Verify Network Connectivity to GitLab Instance

This is the most critical first step. Ensure your macOS machine can reach your GitLab instance.

• Ping Test:

  ping gitlab.example.com
  

  Look for successful replies. If you see Request timeout for icmp_seq..., there's a connectivity issue.

• Traceroute:

  traceroute gitlab.example.com
  

  This command helps identify where the connection is failing (e.g., at your router, a corporate firewall, or the GitLab server itself).

• HTTPS Connection Test with curl:

  curl -v https://gitlab.example.com/api/v4/version
  

  Look for an HTTP 200 OK status code and version information. If you get curl: (7) Failed to connect... or curl: (35) SSL peer certificate or -with-long-error..., it indicates a network or SSL issue.

• Test from within a Docker Container (if using Docker executor):

  docker run --rm alpine:latest ping -c 3 gitlab.example.com
  docker run --rm alpine/git curl -v https://gitlab.example.com/api/v4/version
  

  This confirms if the Docker environment itself has network access.

If any of these network tests fail, your runner will not be able to communicate with GitLab. Prioritize resolving network connectivity before proceeding.

2. Confirm GitLab Instance URL and Registration Token

Simple errors here are surprisingly common.

• GitLab Instance URL: Double-check the URL in your browser when accessing your GitLab. Ensure it includes https:// if your GitLab instance is secured.
• Registration Token:
  1. Navigate to your GitLab project or group's Settings > CI/CD > Runners.
  2. Find the specific runner registration token (either for "Project runners" or "Group runners" depending on your setup).
  3. Copy the token carefully. Regenerate the token if you suspect it's been exposed or is incorrect, and then use the new one.

3. Address SSL/TLS Certificate Issues

If your GitLab instance uses a self-signed certificate or one not widely trusted, your macOS machine or Docker containers might reject the connection.

• For gitlab-runner on macOS:

  1. Obtain the CA certificate:

     openssl s_client -showcerts -connect gitlab.example.com:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > gitlab.example.com.crt
     

  2. Install the certificate into macOS Keychain: Open gitlab.example.com.crt, double-click, and add it to your keychain, trusting it for SSL.
  3. Configure config.toml: Edit your runner's config.toml (usually located at ~/.gitlab-runner/config.toml or /etc/gitlab-runner/config.toml) and add the tls-ca-file entry under the [[runners]] section:

     [[runners]]
       # ... other runner configuration
       tls-ca-file = "/path/to/your/gitlab.example.com.crt"
       # ...
     

• For Docker Executor (if tls-ca-file isn't enough): If jobs still fail within Docker containers, you need to ensure Docker containers trust the certificate.

  1. Make the certificate available to Docker containers by mounting it as a volume in config.toml:

     [[runners]]
       # ...
       executor = "docker"
       [runners.docker]
         # ...
         volumes = ["/cache", "/path/to/your/gitlab.example.com.crt:/etc/ssl/certs/gitlab.example.com.crt:ro"]
         # ...
     

     Replace /path/to/your/gitlab.example.com.crt with the actual path on your macOS host.
  2. Alternatively, for specific images, you might need to build a custom Docker image that includes your CA certificate.

You can bypass SSL verification by setting CI_SERVER_SKIP_VERIFY_SSL=true in the config.toml under the [runners] section or as an environment variable in your CI/CD pipeline. This is highly discouraged for production environments as it opens you to Man-in-the-Middle attacks. Use only for temporary debugging in isolated local environments.

[[runners]]
  environment = ["CI_SERVER_SKIP_VERIFY_SSL=true"]
  # ...

4. Re-register the GitLab Runner

A fresh registration often resolves issues caused by corrupted configurations or outdated tokens.

1. Delete Existing Runner (from GitLab UI):

   • Navigate to your GitLab project/group's Settings > CI/CD > Runners.
   • Find your macOS runner, click the "Edit" button (pencil icon), and then click "Remove runner" at the bottom. This ensures GitLab no longer expects the old runner.

2. Stop and Clean Runner Configuration (on macOS):

   • Stop the running gitlab-runner process:

     gitlab-runner stop
     

   • Delete the old configuration file.

     rm -rf ~/.gitlab-runner/config.toml # For user-specific runner
     # OR
     sudo rm -rf /etc/gitlab-runner/config.toml # For system-wide runner
     

3. Register the Runner Again:

   • Run the registration command using the correct URL and the (potentially new) registration token. Choose your preferred executor (e.g., docker or shell).

   gitlab-runner register 
     --url https://gitlab.example.com/ 
     --registration-token <YOUR_REGISTRATION_TOKEN> 
     --executor docker 
     --description "macOS Local Runner" 
     --tag-list "macos,local,docker" 
     --docker-image "alpine:latest" 
     --non-interactive
   

   • For a shell executor:

   gitlab-runner register 
     --url https://gitlab.example.com/ 
     --registration-token <YOUR_REGISTRATION_TOKEN> 
     --executor shell 
     --description "macOS Local Shell Runner" 
     --tag-list "macos,local,shell" 
     --non-interactive
   

4. Start the Runner:

   gitlab-runner start
   # OR to run interactively and see logs:
   gitlab-runner run
   

Ensure the gitlab-runner process has the necessary permissions to access its configuration and execute jobs. If installed via Homebrew, it typically runs under your user context.

5. Verify Docker Daemon Status and Connectivity (if using Docker executor)

If you've configured your runner to use the docker executor, Docker Desktop must be running and healthy.

• Check Docker Desktop: Ensure Docker Desktop is running on your macOS machine.
• Check Docker Info:

  docker info
  

  Look for "Server Version" and other healthy statistics. If it fails, restart Docker Desktop.
• Check Docker Processes:

  docker ps
  

  Confirm no unexpected issues.
• Docker Network: If you suspect internal Docker network issues, try restarting Docker Desktop, or resetting Docker to factory defaults (Docker Desktop -> Settings -> Troubleshoot -> Reset to factory defaults, warning: this removes all images/containers).

6. Check macOS Firewall & Proxy Settings

macOS and third-party firewalls can silently block connections.

• macOS Built-in Firewall:

  1. Go to System Settings -> Network -> Firewall.
  2. Ensure it's not blocking outgoing connections for gitlab-runner or Docker. You might need to temporarily disable it for testing.
  3. You can inspect pfctl rules via terminal, but the GUI is usually sufficient:

     sudo pfctl -s rules # View current pf rules
     

• Third-Party Firewalls (e.g., Little Snitch, LuLu): These tools require explicit rules to allow gitlab-runner and Docker processes (e.g., com.docker.backend, com.docker.hyperkit) to connect to your GitLab instance's IP address and port 443. Check their respective configuration panels and add appropriate "Allow" rules.

• Proxy Settings:

  1. Go to System Settings -> Network -> (Your Active Network Interface) -> Details... -> Proxies.
  2. Ensure no unintended proxy configurations are active, or if a proxy is required, ensure it's correctly configured for HTTP/HTTPS.
  3. If a proxy is needed, configure it in your config.toml under the [runners] section or as environment variables:

     [[runners]]
       # ...
       environment = ["HTTP_PROXY=http://proxy.example.com:8080", "HTTPS_PROXY=http://proxy.example.com:8080", "NO_PROXY=localhost,127.0.0.1,gitlab.example.com"]
       # ...
     

     And for Docker daemon itself, configure it in Docker Desktop settings.

7. Inspect Runner Logs with Debug Mode

Running the runner in debug mode provides verbose output, which can be invaluable for pinpointing specific errors.

• Stop any running runner instances:

  gitlab-runner stop
  

• Run in foreground with debug:

  gitlab-runner run --debug
  

  Observe the output carefully. Look for specific error messages related to dial tcp, SSL handshake, connection refused, or certificate verify failed. These messages will often directly point to the underlying problem.

8. Remove and Reinstall GitLab Runner (Last Resort)

If all else fails, a clean reinstallation can sometimes resolve cryptic issues.

1. Uninstall via Homebrew:

   brew uninstall gitlab-runner
   

2. Ensure all related files are gone:

   sudo rm -rf ~/.gitlab-runner/ /etc/gitlab-runner/ /usr/local/bin/gitlab-runner
   

3. Reinstall:

   brew install gitlab-runner
   

4. Then, proceed with Step 4 (Re-register the GitLab Runner).

By following these systematic steps, you should be able to diagnose and resolve the "GitLab CI runner jobs pending stuck" issue on your macOS local environment and get your CI/CD pipelines flowing smoothly again.