Troubleshooting Kubernetes CrashLoopBackOff on CentOS Stream / Rocky Linux

Resolve Kubernetes CrashLoopBackOff errors on CentOS Stream or Rocky Linux. This guide details root causes and step-by-step solutions for container startup failures.

Resolve Kubernetes CrashLoopBackOff errors on CentOS Stream or Rocky Linux. This guide details root causes and step-by-step solutions for container startup failures.

A CrashLoopBackOff status in Kubernetes indicates that a container within a pod is repeatedly starting and then crashing. Kubernetes attempts to restart the container, backs off with increasing delays, and repeats the cycle. This typically signifies a fundamental issue preventing the application from initializing successfully, leading to an unresponsive or unavailable service. Understanding the underlying cause is crucial for a swift resolution.

Symptom & Error Signature

When you encounter CrashLoopBackOff, your Kubernetes pods will show this status when inspected:

kubectl get pods -n <namespace>

Example Output:

NAME                                   READY   STATUS             RESTARTS         AGE
my-app-deployment-78f9c78f9f-abcde     0/1     CrashLoopBackOff   5 (50s ago)      2m15s
another-service-pod-xyz12-fghij        1/1     Running            0                5m30s

Further details can be found by describing the problematic pod, which reveals events related to the container's lifecycle:

kubectl describe pod my-app-deployment-78f9c78f9f-abcde -n <namespace>

Example Output (excerpt from Events section):

...
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Pulled     2m15s (x5 over 3m)   kubelet            Container image "myregistry/my-app:v1.0" already present on machine
  Normal   Created    2m15s (x5 over 3m)   kubelet            Created container my-app
  Normal   Started    2m15s (x5 over 3m)   kubelet            Started container my-app
  Warning  BackOff    2m14s (x5 over 3m)   kubelet            Back-off restarting failed container
  Warning  Failed     1m45s (x3 over 2m)   kubelet            Error: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "npm": executable file not found in $PATH: unknown

The most critical information, however, is almost always within the container's own startup logs, which will reveal why the application crashed:

kubectl logs my-app-deployment-78f9c78f9f-abcde -n <namespace>

Example Output (actual crash log):

/docker-entrypoint.sh: line 10: npm: command not found

Root Cause Analysis

A CrashLoopBackOff state almost always points to an issue with the containerized application itself or its immediate Kubernetes environment. The underlying reasons can be categorized as follows:

  1. Application-Level Issues:

    • Incorrect Startup Command/Arguments: The command or args specified in the Pod spec (or inherited from the Dockerfile's ENTRYPOINT/CMD) might be incorrect, point to a non-existent executable, or have incorrect syntax.
    • Missing Environment Variables: The application might fail to start if crucial environment variables (e.g., database connection strings, API keys) are not correctly injected.
    • Configuration Errors: Misconfigured application files (e.g., nginx.conf, appsettings.json, database connection files) or missing configuration files preventing the application from initializing.
    • Dependency Issues: The application might try to connect to an unavailable database, message queue, or another external service during startup and crash when it fails.
    • Application Bugs: A fatal error in the application code itself that causes it to exit immediately upon startup.
    • Insufficient Permissions: The application might lack necessary file system or network permissions to start correctly.

  2. Container Image Issues:

    • Missing Dependencies: The base image might not include all necessary runtime libraries or binaries that the application expects (e.g., npm, python3, specific shared libraries).
    • Corrupted Image: A rare but possible scenario where the container image itself is corrupted.

  3. Kubernetes Resource & Configuration Issues:

    • Insufficient Resource Requests/Limits:
      • Memory: If the container attempts to allocate more memory than specified in its limits.memory, it can be terminated by the kernel (OOMKilled – Out Of Memory Killed). This is a very common cause.
      • CPU: While less common for startup crashes, extremely low CPU limits combined with a CPU-intensive startup process could theoretically cause issues, though typically it results in slow performance, not a crash.
    • Incorrect Liveness/Readiness Probes: If a livenessProbe is configured too aggressively, or points to an incorrect path/port, Kubernetes might kill the container before it has a chance to fully start up.
    • Volume Mounting Issues: Problems with mounting ConfigMap, Secret, or PersistentVolume into the container can lead to missing configuration or data, causing startup failure.
    • Service Account Permissions: If the application requires specific RBAC permissions during startup (e.g., to interact with the Kubernetes API), and the assigned serviceAccount lacks these, it could fail.

  4. Host-Level Issues (less common but can contribute):

    • Disk Full: The node's disk being full might prevent logs from being written or temporary files from being created, leading to application failure.
    • Kernel Panics/Issues: Underlying OS instability, though this would typically affect many pods or the entire node.

Step-by-Step Resolution

Solving CrashLoopBackOff requires a systematic approach, starting with identifying the specific error message from the container logs.

1. Initial Triage: Identify the Crashing Pod

First, confirm which pod is experiencing the CrashLoopBackOff state and in which namespace.

kubectl get pods --all-namespaces -o wide | grep -E 'CrashLoopBackOff|ContainerCreating'

Note the pod name and its namespace. For the rest of the steps, replace <pod-name> and <namespace> with these values.

2. Inspect Pod Events for Clues

The kubectl describe pod command provides a wealth of information, especially the Events section, which can sometimes give a high-level reason for the crash (e.g., OOMKilled).

kubectl describe pod <pod-name> -n <namespace>

Look for Warning or Error events. If you see OOMKilled, it's a strong indicator of memory issues.

3. Examine Container Logs for the Root Cause

This is the most crucial step. The logs will reveal the exact error message that caused your application to terminate.

# Get logs from the current (crashing) container instance
kubectl logs <pod-name> -n <namespace>

# If the container crashed and restarted, get logs from the previous instance
kubectl logs <pod-name> -n <namespace> -p

# If your pod has multiple containers, specify the container name
kubectl logs <pod-name> -n <namespace> -c <container-name>
kubectl logs <pod-name> -n <namespace> -c <container-name> -p

Carefully read through the logs. The actual error message might be buried among other startup messages. Look for keywords like Error, Failed, Exception, Segmentation fault, command not found, permission denied, or specific stack traces. The example exec: "npm": executable file not found in $PATH from the symptom section is a clear indicator.

4. Verify Container Image and Configuration

Based on the logs, cross-reference with your Deployment, StatefulSet, or DaemonSet YAML.

  • Check image:: Is the correct image tag being used? Is it accessible?
  • Check command and args: These override the Dockerfile's ENTRYPOINT and CMD. Ensure the command exists within the container and its arguments are correct.
  • Check env variables: Are all necessary environment variables defined and correctly populated (e.g., from ConfigMapKeyRef or SecretKeyRef)?
    env:
- name: DB_HOST
  value: "mydb-service"
- name: API_KEY
  valueFrom:
    secretKeyRef:
      name: my-app-secrets
      key: api-key
  • Check volumeMounts and volumes: Ensure ConfigMaps and Secrets are correctly mounted, and the application expects them at the mounted paths. Verify that the ConfigMap or Secret itself exists and contains the expected data.
    kubectl get configmap <configmap-name> -n <namespace> -o yaml
# Be careful when inspecting secrets, as they contain sensitive data.
kubectl get secret <secret-name> -n <namespace> -o yaml

5. Analyze Resource Requests and Limits

If kubectl describe pod showed OOMKilled or your logs hint at memory exhaustion, you need to adjust resource limits.

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

Increasing limits.memory too much without optimizing your application can lead to resource contention on the node. Start with a modest increase and monitor. If OOMKilled persists, your application might have a memory leak or simply require more memory than you initially estimated.

6. Review Liveness and Readiness Probes

Aggressive or incorrect probes can cause premature termination.

  • initialDelaySeconds: Ensure this is long enough for your application to fully start before the first probe.
  • periodSeconds: How often the probe runs.
  • timeoutSeconds: How long to wait for a probe response.
  • path: For httpGet probes, ensure the path exists and returns a success status.
  • exec command: For exec probes, ensure the command exits with status 0 for success.
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

During initial deployment or complex application startup, it's sometimes useful to temporarily disable liveness probes to confirm the application can start without intervention. Re-enable and fine-tune them once stable.

7. Check for Dependency Issues

If your application depends on external services (databases, caches, other microservices), ensure they are reachable and healthy.

  • Network Connectivity: Use a debug container to check connectivity from within the cluster.
    kubectl debug -it <pod-name> --image=busybox --target=<container-name> -- /bin/sh
# Inside the debug container
ping <db-host>
telnet <db-host> <db-port>
  • Credentials: Verify database user/passwords, API keys, etc., stored in Secrets or ConfigMaps are correct.

8. Redeploy the Application

After identifying and fixing the issue in your Deployment, StatefulSet, or DaemonSet YAML, apply the changes:

# Apply your updated manifest
kubectl apply -f my-app-deployment.yaml -n <namespace>

# Alternatively, if you just want to force a restart without manifest changes
# (e.g., if a ConfigMap/Secret was updated or external dependency fixed)
kubectl rollout restart deployment <deployment-name> -n <namespace>

Monitor the new pods (kubectl get pods -n <namespace> -w) and their logs (kubectl logs <new-pod-name> -n <namespace>) to confirm the issue is resolved.

9. Advanced Debugging on CentOS Stream / Rocky Linux (if all else fails)

If the container logs are unhelpful or you suspect deeper host-level issues specific to CentOS Stream/Rocky Linux:

  • Examine Kubelet Logs: The Kubelet running on the node is responsible for managing containers. Its logs can reveal issues with image pulling, volume mounting, or other runtime problems.
    # SSH to the node where the problematic pod is running
journalctl -u kubelet -f
  • Check System Dmesg: For OOM events not immediately visible in kubectl describe, the kernel ring buffer is a good source.
    # SSH to the node
dmesg -T | grep -i "oom-killer"
  • Container Runtime Logs: If you are using containerd (default for recent Kubernetes versions), its logs might offer further insights.
    # SSH to the node
journalctl -u containerd -f
  • Ephemeral Containers (Kubernetes 1.23+): For complex debugging, you can add an ephemeral container to a running pod for inspection without restarting the original container.
    kubectl debug -it <pod-name> -n <namespace> --image=busybox --target=<container-name>
    This allows you to run commands, inspect files, or test network connectivity directly within the pod's environment.

By systematically working through these steps, starting with the container logs, you should be able to pinpoint and resolve most CrashLoopBackOff scenarios in your Kubernetes environment on CentOS Stream or Rocky Linux.