Troubleshooting Kubernetes PVC Stuck in Pending on Debian 12 Bookworm

Resolve Kubernetes Persistent Volume Claims (PVCs) stuck in pending status on Debian 12. Diagnose common causes like missing PVs, misconfigured StorageClasses, and provisioner issues.

Resolve Kubernetes Persistent Volume Claims (PVCs) stuck in pending status on Debian 12. Diagnose common causes like missing PVs, misconfigured StorageClasses, and provisioner issues.

A Kubernetes Persistent Volume Claim (PVC) stuck in a "Pending" state is a common yet critical issue that prevents applications from starting or functioning correctly, as they cannot mount their required storage. This guide provides a highly technical, step-by-step approach to diagnose and resolve PVCs perpetually pending on a Kubernetes cluster running on Debian 12 Bookworm.

Symptom & Error Signature

When a PVC is stuck in pending, applications requiring that storage will fail to start or remain in a "ContainerCreating" or "Pending" state themselves. You will typically observe the following when inspecting the PVC:

kubectl get pvc -n <namespace>

NAME                STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS        AGE
my-app-pvc          Pending                                      standard-ssd        5m
another-pvc         Pending                                      nfs-storage         2m

A more detailed inspection using kubectl describe pvc will often reveal the underlying reason in the "Events" section:

kubectl describe pvc my-app-pvc -n my-app-namespace

Name:          my-app-pvc
Namespace:     my-app-namespace
StorageClass:  standard-ssd
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-class: standard-ssd
               volume.beta.kubernetes.io/storage-provisioner:
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Gi
Access Modes:
Volume Mode:   Filesystem
DataSource:    <unset>
Events:
  Type     Reason              Age                From                         Message
  ----     ------              ----               ----                         -------
  Warning  ProvisioningFailed  14s (x3 over 4m3s) persistentvolume-controller  failed to provision volume with StorageClass "standard-ssd": storageclass.storage.k8s.io "standard-ssd" not found
  Warning  FailedBinding       14s (x3 over 4m3s) persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

Or, if a StorageClass is specified but a provisioner is missing or misconfigured:

Events:
  Type     Reason              Age                From                         Message
  ----     ------              ----               ----                         -------
  Warning  ProvisioningFailed  2m                 persistentvolume-controller  failed to provision volume with StorageClass "nfs-storage": failed to get provisioner for StorageClass "nfs-storage": provisioner "cluster.local/nfs-subdir-external-provisioner" not found
  Warning  FailedBinding       2m                 persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

Another common message, especially for dynamically provisioned volumes without immediate consumers:

Events:
  Type     Reason              Age                From                         Message
  ----     ------              ----               ----                         -------
  Normal   WaitForFirstConsumer  2m                 persistentvolume-controller  waiting for first consumer to be created before binding

Root Cause Analysis

A Kubernetes PVC enters a "Pending" state when the Kubernetes control plane cannot find or provision a suitable PersistentVolume (PV) to satisfy the claim's requirements. The underlying reasons are typically one or a combination of the following:

  1. Missing or Misconfigured StorageClass: The PVC requests a storageClassName that either does not exist, or the specified StorageClass is improperly configured (e.g., incorrect provisioner name, missing parameters).
  2. No Matching PersistentVolume (PV) for Static Provisioning: If using static provisioning (i.e., you manually create PVs), no existing PV matches the PVC's criteria (capacity, access modes, volume mode, StorageClass name, labels, selectors). All suitable PVs might already be bound.
  3. Dynamic Provisioner Issues:
    • Provisioner Not Deployed: The specified provisioner (e.g., nfs-subdir-external-provisioner, local-path-provisioner, a CSI driver) is not running or correctly installed in the cluster.
    • Provisioner Misconfiguration: The provisioner's deployment, RBAC, or configuration parameters (e.g., NFS server IP, share path, cloud provider credentials) are incorrect, preventing it from creating volumes.
    • Underlying Storage System Problems: The external storage backend (NFS server, iSCSI target, cloud storage service) is inaccessible, out of capacity, or experiencing issues.
  4. Insufficient Storage Capacity: Even if a StorageClass or PVs exist, there might not be enough available capacity on the underlying storage to fulfill the PVC's request.
  5. Access Mode Mismatch: The PVC's requested accessModes (e.g., ReadWriteOnce, ReadOnlyMany, ReadWriteMany) do not match any available PV or the capabilities of the storage provisioner.
  6. WaitForFirstConsumer Policy: If the StorageClass (or the PVC's annotation) uses volumeBindingMode: WaitForFirstConsumer, the PVC will intentionally remain pending until a Pod is scheduled to use it. This is a normal behavior, but can be confusing if not understood.
  7. RBAC/Permissions: The Kubernetes ServiceAccount used by the provisioner or controller-manager might lack the necessary permissions to interact with the underlying storage system or Kubernetes API.
  8. Control Plane Health: While less common, an unhealthy kube-controller-manager (responsible for PVC-PV binding) could lead to this issue.

Step-by-Step Resolution

Follow these steps to diagnose and resolve a PVC stuck in pending status.

1. Inspect the Persistent Volume Claim (PVC) Events

This is always the first step. The events in kubectl describe pvc usually point directly to the problem.

kubectl describe pvc <pvc-name> -n <namespace>

Carefully examine the Events section. Look for Warning messages like "failed to provision volume with StorageClass…", "no persistent volumes available…", or "provisioner '…' not found".

2. Verify StorageClass Configuration

If the PVC specifies a storageClassName, ensure it exists and is correctly configured.

kubectl get sc

If the StorageClass is missing, you'll need to create it. If it exists, describe it:

kubectl describe sc <storageclass-name>

Common issues:

  • Missing StorageClass: If kubectl get sc does not list the storageClassName specified in your PVC, create it.
  • Incorrect provisioner name: The provisioner field in the StorageClass must exactly match the name the deployed provisioner registers itself with.
  • Missing or incorrect parameters: Cloud-specific StorageClasses or custom CSI drivers often require specific parameters (e.g., type, zone, fsType).
  • ReclaimPolicy or volumeBindingMode misconfiguration: While not direct causes for "Pending", they are important for volume lifecycle. WaitForFirstConsumer will keep PVCs pending until a Pod uses them.

Ensure the provisioner name in your StorageClass YAML exactly matches the string your actual storage provisioner (e.g., NFS Subdir External Provisioner, AWS EBS CSI driver) registers with Kubernetes. A mismatch here is a frequent cause of ProvisioningFailed.

Example of a simple NFS-backed StorageClass:

# nfs-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-storage
provisioner: cluster.local/nfs-subdir-external-provisioner # This must match your provisioner deployment
parameters:
  archiveOnDelete: "false" # Set to "true" to archive data instead of deleting
reclaimPolicy: Delete
volumeBindingMode: Immediate # Or WaitForFirstConsumer

Apply with:

kubectl apply -f nfs-storageclass.yaml

3. Check Dynamic Provisioner Health and Logs

If your StorageClass uses dynamic provisioning, confirm the provisioner pod is running and check its logs.

  1. Identify the provisioner pod: Find the provisioner string from kubectl describe sc <storageclass-name>. Then, search for pods that match:

    kubectl get pods -A | grep -i <provisioner-substring>
# Example for NFS:
kubectl get pods -A | grep -i nfs-subdir-external-provisioner
# Example for local-path:
kubectl get pods -A | grep -i local-path-provisioner

    Note the namespace and pod name.

  2. Check pod status:

    kubectl get pod <provisioner-pod-name> -n <provisioner-namespace>

    Ensure it's Running and Ready. If not, investigate why (e.g., ImagePullBackOff, CrashLoopBackOff) by describing the pod:

    kubectl describe pod <provisioner-pod-name> -n <provisioner-namespace>

  3. Examine provisioner logs:

    kubectl logs <provisioner-pod-name> -n <provisioner-namespace> --tail=50

    Look for error messages related to volume creation, connection failures to the underlying storage, or permission issues.

If using local-path-provisioner, ensure the host path /var/lib/rancher/k3s/storage (or /mnt/data if manually configured) exists on your worker nodes and has appropriate permissions (e.g., 0777).

4. Verify Underlying Storage System

If the provisioner logs indicate issues with the backend storage, investigate that system directly.

  • NFS:

    • Verify the NFS server is running and accessible from your Kubernetes nodes.
    • Check showmount -e <nfs-server-ip> from a node to confirm exports.
    • Check firewall rules on both the Kubernetes nodes and the NFS server.
    • Ensure the NFS share has enough free space.
    • Try manually mounting the NFS share from a node to test connectivity and permissions:
      # On a Kubernetes worker node (Debian 12)
sudo apt update && sudo apt install -y nfs-common
sudo mkdir -p /mnt/nfs_test
sudo mount <nfs-server-ip>:/<path/to/share> /mnt/nfs_test
df -h /mnt/nfs_test
sudo umount /mnt/nfs_test

  • iSCSI:

    • Verify the iSCSI target is running and configured correctly.
    • Check iscsiadm -m discovery -t sendtargets -p <iscsi-target-ip> from a node.
    • Ensure appropriate firewall rules are in place.

  • Cloud Providers (AWS EBS, Azure Disk, GCP Persistent Disk):

    • Check your cloud provider console for disk quotas, region availability, and service health.
    • Ensure the Kubernetes cluster's IAM roles/Service Accounts have the necessary permissions to create and manage disk volumes.

5. Check for Existing Persistent Volumes (PVs) (Static Provisioning)

If you are using static provisioning or dynamic provisioning has failed to create a PV, inspect the existing PVs.

kubectl get pv

Look for PVs with Status Available. Then, describe them:

kubectl describe pv <pv-name>

Compare the PV's properties (Capacity, Access Modes, StorageClass, Volume Mode, Node Affinity if local) against the PVC's requirements.

For a PVC to bind to an existing PV, the following must match:

  • accessModes (e.g., ReadWriteOnce, ReadWriteMany)
  • capacity (PV must be equal or larger than PVC request)
  • storageClassName (must be identical, or both empty)
  • volumeMode (Filesystem or Block)

If no matching PV exists and dynamic provisioning is not used or failing, you may need to manually create a PV.

Example of a static PV definition for an NFS share:

# nfs-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-nfs-pv
spec:
  capacity:
    storage: 5Gi # Must be >= PVC requested capacity
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-storage # Must match the StorageClass used by the PVC
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    path: /path/on/nfs/server/my-app-data # The actual path on your NFS server
    server: 192.168.1.100 # Your NFS server IP address

Apply with:

kubectl apply -f nfs-pv.yaml

6. Address WaitForFirstConsumer Policy

If kubectl describe pvc shows Events: Normal WaitForFirstConsumer, this indicates the PVC is intentionally pending until a Pod requests it. This is normal.

# storageclass-wait-for-first-consumer.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: delayed-binding
provisioner: example.com/my-provisioner
volumeBindingMode: WaitForFirstConsumer # The key setting
reclaimPolicy: Delete

To resolve this "pending" status, you simply need to deploy the Pod that uses this PVC. The binding will happen automatically when the scheduler attempts to place the Pod.

# my-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: my-storage
          mountPath: /usr/share/nginx/html
      volumes:
      - name: my-storage
        persistentVolumeClaim:
          claimName: my-app-pvc # This PVC will trigger the binding

7. Check for RBAC Issues

Ensure your storage provisioner's ServiceAccount has the necessary permissions. This usually involves ClusterRole and ClusterRoleBinding resources. Refer to the specific provisioner's documentation for required RBAC.

Example (common for many provisioners):

# rbac.yaml for a generic provisioner
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-provisioner-sa
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: my-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-provisioner-runner-binding
subjects:
  - kind: ServiceAccount
    name: my-provisioner-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: my-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

8. Restart Affected Components

After making configuration changes, it's often beneficial to restart the affected components.

  1. Delete and recreate the PVC: If the initial binding attempt failed due to a misconfiguration, sometimes deleting and recreating the PVC can trigger a fresh binding attempt.

    kubectl delete pvc <pvc-name> -n <namespace>
# Recreate from your YAML file
kubectl apply -f my-pvc.yaml

  2. Restart the provisioner pod: If you changed the provisioner's configuration, delete the pod to force a restart.

    kubectl delete pod <provisioner-pod-name> -n <provisioner-namespace>

  3. Restart kube-controller-manager (Advanced/Last Resort): For self-hosted Kubernetes or if running in a VM/bare metal, you might need to restart the kube-controller-manager process on your control plane node. This is a powerful step and should only be done if other methods fail and you suspect a controller-manager issue. For kubeadm deployments on Debian 12, kube-controller-manager runs as a static pod. You would typically restart the kubelet service which manages static pods:

    # On the Kubernetes control plane node
sudo systemctl restart kubelet

    Restarting kube-controller-manager or kubelet on a control plane node can temporarily impact cluster stability. Perform this during a maintenance window.

By systematically going through these steps, you should be able to identify and resolve the root cause of a Kubernetes PVC stuck in pending status on your Debian 12 Bookworm cluster.