Kubernetes ImagePullBackOff: Resolving Private Registry Authentication on CentOS Stream / Rocky Linux

Troubleshoot Kubernetes ImagePullBackOff due to private registry authentication failures on CentOS Stream and Rocky Linux. Fix secret issues quickly.

Introduction

The ImagePullBackOff error in Kubernetes is a common symptom indicating that a Pod cannot start because it's unable to pull its specified container image from a registry. While this error can stem from various causes like incorrect image names, network issues, or registry downtime, one of the most frequent and frustrating culprits, especially in production environments, is private registry authentication failure.

This guide specifically addresses ImagePullBackOff errors occurring when your Kubernetes cluster, running on CentOS Stream or Rocky Linux nodes, fails to authenticate with a private container image registry (e.g., Docker Hub Private Repos, AWS ECR, Azure Container Registry, Google Container Registry, Harbor, etc.). You'll learn how to diagnose the underlying authentication problem and implement robust solutions.

Symptom & Error Signature

When a Pod experiences ImagePullBackOff due to authentication issues, you'll typically observe the following:

1. Pod status: Your Pod will repeatedly show ImagePullBackOff or ErrImagePull in its status, often cycling through Pending or CrashLoopBackOff if previous attempts failed.

   kubectl get pods -n my-app-namespace
   

   NAME                         READY   STATUS             RESTARTS      AGE
   my-app-deployment-78f9xxxx   0/1     ImagePullBackOff   0             2m
   another-pod-xxxxxxxx         1/1     Running            0             10m
   

2. Pod Events: Detailed events for the problematic Pod will clearly indicate authentication failures. This is the most crucial diagnostic output.

   kubectl describe pod my-app-deployment-78f9xxxx -n my-app-namespace
   

   ...
   Events:
     Type     Reason     Age                  From               Message
     ----     ------     ----                 ----               -------
     Normal   Scheduled  2m                   default-scheduler  Successfully assigned my-app-deployment-78f9xxxx to worker-node-01.example.com
     Normal   Pulled     1m (x2 over 2m)      kubelet            Container image "my-private-registry/my-image:latest" already present on machine
     Warning  Failed     1m (x2 over 2m)      kubelet            Failed to pull image "my-private-registry/my-image:latest": rpc error: code = Unknown desc = Error response from daemon: Get "https://my-private-registry/v2/": unauthorized: authentication required
     Warning  Failed     1m (x2 over 2m)      kubelet            Error: ErrImagePull
     Normal   BackOff    40s (x3 over 2m)     kubelet            Back-off pulling image "my-private-registry/my-image:latest"
     Warning  Failed     40s (x3 over 2m)     kubelet            Error: ImagePullBackOff
   

   Look specifically for messages like "unauthorized: authentication required" or "401 Unauthorized".

Root Cause Analysis

The ImagePullBackOff error, when specifically related to "unauthorized: authentication required," indicates that the Kubernetes node's container runtime (e.g., containerd or cri-o) cannot successfully log in to the specified private image registry using the provided credentials. This can be attributed to several underlying issues:

1. Missing or Incorrect imagePullSecrets:

   • The Pod or Deployment manifest may not specify imagePullSecrets.
   • The imagePullSecrets specified may refer to a Secret that does not exist in the same namespace.
   • The imagePullSecrets value might be misspelled or reference an incorrect Secret name.

2. Invalid or Malformed Kubernetes Secret:

   • The Kubernetes Secret of type kubernetes.io/dockerconfigjson contains incorrect, expired, or malformed registry credentials.
   • The dockerconfigjson content within the Secret is not a valid base64 encoded JSON string, or its structure is incorrect.
   • The auth token or password in the Secret has expired, been revoked, or is simply wrong.

3. Incorrect Namespace for the Secret:

   • Kubernetes Secrets are namespace-scoped. If the Secret providing the registry credentials is not in the same namespace as the Pod attempting to pull the image, authentication will fail.

4. Network Connectivity or DNS Issues to the Registry:

   • While less common for authentication-specific errors (which imply a successful initial connection), network path issues or incorrect DNS resolution for the registry hostname can prevent the authentication handshake from completing. This can sometimes manifest ambiguously.

5. Container Runtime Configuration Issues (Less Common for Authentication):

   • For containerd or cri-o on CentOS Stream/Rocky Linux, if there's a misconfiguration in config.toml or registries.conf.d that overrides or interferes with how imagePullSecrets are handled, it could cause issues. This is rare when imagePullSecrets are correctly defined.

Step-by-Step Resolution

Follow these steps to systematically diagnose and resolve the ImagePullBackOff authentication error.

1. Verify Pod Status and Events

Start by confirming the exact error messages as described in the "Symptom & Error Signature" section.

kubectl get pods -n my-app-namespace
kubectl describe pod my-app-deployment-78f9xxxx -n my-app-namespace

Carefully review the Events section for "unauthorized: authentication required" or similar messages indicating a credential problem.

2. Inspect the imagePullSecrets Configuration in Your Deployment

Ensure your Pod, Deployment, or StatefulSet manifest correctly references the imagePullSecrets.

kubectl get deployment my-app-deployment -n my-app-namespace -o yaml

Look for the imagePullSecrets field under spec.template.spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
  namespace: my-app-namespace
spec:
  # ... other deployment spec ...
  template:
    spec:
      containers:
      - name: my-app
        image: my-private-registry/my-image:latest
        # ... container config ...
      imagePullSecrets:
      - name: my-registry-secret # <--- This is the secret name to verify

Make a note of the imagePullSecrets name (e.g., my-registry-secret) and the namespace (e.g., my-app-namespace).

3. Validate the Kubernetes Secret

This is the most critical step. You need to ensure the referenced Secret exists, is in the correct namespace, and contains valid credentials.

a. Check if the Secret Exists

kubectl get secret my-registry-secret -n my-app-namespace

If the output says Error from server (NotFound): secrets "my-registry-secret" not found, then the secret is missing, or you're looking in the wrong namespace. Proceed to create or recreate it.

b. Inspect the Secret Content

Kubernetes Secrets store credentials as base64 encoded strings. When decoding, be extremely cautious not to expose sensitive information in public logs or outputs. Use this step for diagnostic purposes only.

Inspect the Secret's content to verify the dockerconfigjson.

kubectl get secret my-registry-secret -n my-app-namespace -o yaml

You'll see a data field with config.json or .dockerconfigjson (depending on how it was created) which is base64 encoded. Decode it:

kubectl get secret my-registry-secret -n my-app-namespace -o jsonpath='{.data..dockerconfigjson}' | base64 --decode

The output should look similar to this (formatted for readability; it will be a single line in actual output):

{
  "auths": {
    "my-private-registry": {
      "username": "your-username",
      "password": "your-password-or-token",
      "email": "[email protected]",
      "auth": "base64-encoded-username:password"
    }
  }
}

Verify the following:

• The registry URL (my-private-registry) precisely matches the one used in your image name (my-private-registry/my-image:latest). Trailing slashes or port differences matter.
• The username and password (or auth token) are correct and have not expired.
• The entire JSON structure is valid.

c. Recreate the Secret if Necessary

If the secret is missing, incorrect, or you suspect the credentials are bad, recreate it.

First, delete the old secret (if it exists):

kubectl delete secret my-registry-secret -n my-app-namespace

Then, create a new one using your actual registry credentials.

kubectl create secret docker-registry my-registry-secret 
  --docker-server=my-private-registry 
  --docker-username=YOUR_REGISTRY_USERNAME 
  --docker-password='YOUR_REGISTRY_PASSWORD_OR_TOKEN' 
  [email protected] 
  -n my-app-namespace

Replace my-private-registry, YOUR_REGISTRY_USERNAME, YOUR_REGISTRY_PASSWORD_OR_TOKEN, [email protected], and my-app-namespace with your actual values.

When using docker-password, enclose it in single quotes if it contains special characters. Some registries use long access tokens for passwords.

Alternatively, using an existing ~/.docker/config.json: If you have successfully logged in via podman login or docker login on a workstation, you can generate the secret directly from your ~/.docker/config.json file.

# On your workstation where podman/docker login works:
# First, ensure you have a valid ~/.docker/config.json with the registry credentials
podman login my-private-registry

# Then, create the secret in Kubernetes
kubectl create secret generic my-registry-secret 
  --from-file=.dockerconfigjson=$HOME/.docker/config.json 
  -n my-app-namespace

4. Ensure Correct Namespace

As mentioned, Secrets are namespace-scoped. Always double-check that the Secret lives in the exact same namespace as the Pod trying to pull the image. If not, delete and recreate it in the correct namespace.

5. Test Registry Credentials Independently on a Node (CentOS Stream / Rocky Linux Specific)

To definitively rule out credential issues versus Kubernetes configuration, try to log in and pull the image directly from one of your Kubernetes worker nodes. This step is crucial for isolating the problem.

a. SSH into a Kubernetes Worker Node

ssh [email protected]

b. Log in to the Private Registry using podman or docker

On CentOS Stream / Rocky Linux, podman is the default container runtime tool. docker-ce can also be installed. Use the one your cluster is configured to interact with, or just podman for testing.

# If using podman (recommended on RHEL-based systems)
podman login my-private-registry
# Enter YOUR_REGISTRY_USERNAME and YOUR_REGISTRY_PASSWORD_OR_TOKEN when prompted.

# If using docker-ce (if installed)
docker login my-private-registry
# Enter YOUR_REGISTRY_USERNAME and YOUR_REGISTRY_PASSWORD_OR_TOKEN when prompted.

If podman login (or docker login) fails with "unauthorized" or similar errors, your credentials are definitively incorrect or expired. You must fix them with your registry provider.

c. Attempt to Pull the Image

If podman login was successful, try to pull the image:

podman pull my-private-registry/my-image:latest

If this succeeds, then the credentials themselves are fine, and the problem likely lies within the Kubernetes Secret or its association with the Pod. If it fails, your credentials are still the primary issue.

6. Check Node Container Runtime Configuration (Advanced/Less Common)

On CentOS Stream / Rocky Linux, Kubernetes typically uses containerd as its Container Runtime Interface (CRI). While imagePullSecrets should handle authentication, a misconfigured containerd could sometimes interfere.

a. Access Containerd Configuration

grep -r "registry" /etc/containerd/config.toml

Look for any [plugins."io.containerd.grpc.v1.cri".registry.configs."my-private-registry".auth] sections. These are generally used for hardcoding credentials or specific registry behaviors, which should not interfere with imagePullSecrets. However, ensure there are no conflicting or incorrect entries.

b. Check for Registry Mirrors or Insecure Registries

Also, check for [plugins."io.containerd.grpc.v1.cri".registry.mirrors] or insecure_skip_verify_v2 under [plugins."io.containerd.grpc.v1.cri".registry]. If you have a private registry that uses self-signed certificates and is not properly configured for trusted CAs, you might also see related errors, though they usually manifest differently than "unauthorized."

If you made changes to config.toml, restart containerd:

sudo systemctl restart containerd

7. Address Potential DNS/Network Issues

Although authentication errors usually imply a successful handshake, it's good practice to quickly rule out DNS or basic network connectivity to the registry if other steps fail.

a. Check DNS Resolution on Nodes

# On a worker node:
dig my-private-registry
nslookup my-private-registry

Ensure these commands resolve to the correct IP address for your registry. Check /etc/resolv.conf on the node if DNS lookups fail or are incorrect.

b. Test Network Connectivity

# On a worker node:
curl -I https://my-private-registry/v2/

You should see an HTTP 401 Unauthorized response (which is good, it means the registry was reached and demanded authentication). If you get connection refused, timeout, or DNS errors, then network connectivity is the root cause.

8. Redeploy the Pod

After making any changes to the Kubernetes Secret or your Deployment manifest, you need to force Kubernetes to re-pull the image. Simply deleting the problematic Pod will achieve this.

kubectl delete pod my-app-deployment-78f9xxxx -n my-app-namespace

Kubernetes will create a new Pod (via the Deployment controller), which will attempt to pull the image using the updated Secret. Monitor its status:

kubectl get pods -n my-app-namespace -w

You should now see the new Pod transition to ContainerCreating and then Running. If it still fails, repeat the troubleshooting steps, carefully reviewing each detail.