Kubernetes PVC Stuck in Pending on Ubuntu 22.04 LTS: A Deep Dive Troubleshooting Guide

Resolve Kubernetes Persistent Volume Claims (PVCs) stuck in pending status on Ubuntu 22.04 LTS. Diagnose StorageClass, CSI driver, and provisioner issues.

Introduction

Kubernetes Persistent Volume Claims (PVCs) are a critical abstraction layer that allows workloads to request and consume storage without knowing the underlying infrastructure details. However, it's a common scenario for a PVC to get "stuck" in a Pending status, preventing your application pods from starting or operating correctly. This guide provides a highly technical, step-by-step approach to diagnose and resolve PVCs stuck in pending state on a Kubernetes cluster running on Ubuntu 22.04 LTS. We'll explore the common root causes, from misconfigured StorageClasses to underlying CSI driver issues, equipping you with the expertise to debug and restore your storage provisioning.

Symptom & Error Signature

When a PVC is stuck in Pending status, your applications will fail to provision their required storage, often leading to pods remaining in Pending or ContainerCreating states if they depend on the PVC.

You will typically observe this using kubectl:

kubectl get pvc

NAME                STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS      AGE
my-app-pvc          Pending                                     standard          5m
another-pvc         Bound     pvc-xyz  10Gi       RWO            fast-storage      2h

A more detailed inspection will reveal the specific reason for the Pending status in the Events section:

kubectl describe pvc my-app-pvc

Name:          my-app-pvc
Namespace:     default
StorageClass:  standard
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/no-provisioner
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
DataSource:    <none>
Events:
  Type     Reason              Age                  From                         Message
  ----     ------              ----                 ----                         -------
  Warning  ProvisioningFailed  2m (x15 over 5m)     persistentvolume-controller  waiting for a volume to be created, either by a user or a storage provisioner.
  Warning  ProvisioningFailed  2m (x15 over 5m)     persistentvolume-controller  no persistent volumes available for this claim and no storage class is set that can satisfy this claim
  Warning  ProvisioningFailed  1m (x16 over 5m)     persistentvolume-controller  storageclass.storage.k8s.io "standard" not found
  Warning  ProvisioningFailed  1m (x16 over 5m)     persistentvolume-controller  waiting for a volume to be created, either by a user or a storage provisioner.

(Note: The Message field will vary significantly based on the exact root cause.)

Root Cause Analysis

A PVC stuck in Pending status indicates that the Kubernetes control plane is unable to find or provision a suitable Persistent Volume (PV) to bind to the PVC. This can stem from several underlying issues:

1. Missing or Incorrect StorageClass: The PVC might request a StorageClass that does not exist or is misconfigured. Kubernetes relies on StorageClass objects to define how storage is dynamically provisioned. If the specified StorageClass is absent, or its provisioner field is invalid, the PVC cannot bind.
2. No Dynamic Provisioner for the StorageClass: Even if a StorageClass exists, its associated CSI (Container Storage Interface) driver or in-tree provisioner might not be deployed, configured, or running correctly within the cluster. This is crucial for dynamic provisioning, where PVs are created on demand.
3. Insufficient or Unavailable Storage (Static Provisioning): If you're using static provisioning (i.e., not relying on a StorageClass or dynamic provisioner), there might not be a pre-created PersistentVolume (PV) that matches the PVC's requirements (size, access mode, StorageClass name, labels).
4. Access Mode Mismatch: The PVC requests a specific accessMode (e.g., ReadWriteOnce, ReadOnlyMany, ReadWriteMany) that no available PV or the configured StorageClass provisioner can satisfy. For instance, a PVC requesting ReadWriteMany for an EBS-backed StorageClass would fail as EBS only supports ReadWriteOnce.
5. Resource Constraints or Underlying Storage System Issues: The dynamic provisioner might be unable to create the PV due to issues on the underlying storage system (e.g., out of disk space, network connectivity issues to NFS/iSCSI target, cloud provider API limits/errors, incorrect credentials).
6. CSI Driver Pod Failures: The pods responsible for the CSI driver (controller and/or node plugins) might be failing, restarting, or stuck in a non-ready state due preventing them from provisioning storage. This could be due to misconfiguration, missing dependencies, or resource limitations on the nodes.
7. RBAC Permissions: The CSI driver's service account might lack the necessary Kubernetes RBAC permissions to create PVs or interact with the storage backend API.
8. Node Taints/Tolerations: While less common for PVCs directly, if the CSI provisioner pods are scheduled on nodes with taints they cannot tolerate, they will fail to run and thus fail to provision storage.

Step-by-Step Resolution

Follow these steps meticulously to pinpoint and resolve the root cause of your Pending PVCs.

1. Inspect the PVC Events for Specific Clues

The kubectl describe pvc command is your primary diagnostic tool. Pay close attention to the Events section.

kubectl describe pvc my-app-pvc -n default

Look for messages like:

• no persistent volumes available for this claim and no storage class is set that can satisfy this claim (No StorageClass, or Static Provisioning with no matching PV).
• storageclass.storage.k8s.io "standard" not found (StorageClass does not exist).
• waiting for a volume to be created, either by a user or a storage provisioner. (StorageClass exists, but provisioner not active or failing).
• Failed to provision volume with StorageClass "slow": rpc error: code = Internal desc = volume creation failed: ... (CSI driver specific error message).

2. Verify StorageClass Existence and Configuration

Ensure the StorageClass requested by your PVC actually exists and is correctly configured.

kubectl get sc

If your PVC specifies a StorageClass (e.g., standard) but it's not listed, this is a clear indication that the StorageClass needs to be created.

If it exists, inspect its details:

kubectl describe sc <STORAGECLASS_NAME>

Name:            standard
IsDefaultClass:  No
Annotations:     storageclass.kubernetes.io/is-default-class=false
Provisioner:     csi.example.com  # <--- IMPORTANT: This tells you which provisioner should be running
Parameters:      type=gp2
AllowVolumeExpansion: true
MountOptions:    <none>
ReclaimPolicy:   Delete
VolumeBindingMode: Immediate
Events:          <none>

The Provisioner field is critical. It defines which CSI driver or in-tree provisioner is responsible for creating volumes for this StorageClass. Note this value for subsequent steps.

If the Provisioner is incorrect or a generic kubernetes.io/no-provisioner, you either need to define a correct StorageClass or explicitly create PersistentVolume objects if you intend to use static provisioning.

Example: Creating a StorageClass (if missing)

Let's say you're using hostpath for local testing with storage.k8s.io/csi-hostpath.

# hostpath-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hostpath-provisioner
provisioner: hostpath.csi.k8s.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

kubectl apply -f hostpath-sc.yaml

3. Verify the Storage Provisioner (CSI Driver) Status

This is often the most complex part. You need to ensure the CSI driver corresponding to your StorageClass's Provisioner field is installed and running correctly.

a. Identify the CSI Driver Namespace and Pods: CSI drivers typically run in their own namespaces (e.g., kube-system, ceph-csi, aws-ebs-csi-driver). Common provisioners and their potential namespaces:

• kubernetes.io/aws-ebs: kube-system or aws-ebs-csi-driver
• disk.csi.azure.com: kube-system or azure-disk-csi
• pd.csi.storage.gke.io: kube-system
• cephfs.csi.ceph.com / rbd.csi.ceph.com: ceph-csi
• hostpath.csi.k8s.io: kube-system (or a dedicated namespace if manually installed)
• nfs.csi.k8s.io (for NFS CSI driver): often nfs-csi

# Example for AWS EBS CSI driver
kubectl get pods -n aws-ebs-csi-driver

You should see pods for controller components (e.g., csi-ebs-controller-*) and potentially node components (e.g., csi-ebs-node-*) running and in Running or Completed status.

b. Check Logs and Events of Provisioner Pods: If any provisioner pods are not in Running state, or if your PVC is still pending, check their logs and describe them for errors.

# For controller pods (usually the ones provisioning PVs)
kubectl get pods -n <CSI_DRIVER_NAMESPACE> -l app=csi-ebs-controller # Adjust label as needed
kubectl logs <CSI_CONTROLLER_POD_NAME> -n <CSI_DRIVER_NAMESPACE>

# For node pods (less likely to cause PVC pending, but good to check)
kubectl get pods -n <CSI_DRIVER_NAMESPACE> -l app=csi-ebs-node # Adjust label as needed
kubectl logs <CSI_NODE_POD_NAME> -n <CSI_DRIVER_NAMESPACE>

kubectl describe pod <CSI_CONTROLLER_POD_NAME> -n <CSI_DRIVER_NAMESPACE>

Look for errors related to API access, connection to the storage backend, permissions, or configuration. Common issues here include:

• Failed to get credentials (Cloud provider access issues)
• Failed to connect to <storage_backend_ip> (Network issues to NFS/iSCSI)
• RBAC: access denied (CSI service account lacks permissions)

4. Address Access Mode Mismatch

Ensure the accessModes requested by your PVC are supported by the StorageClass's provisioner or the available PersistentVolume (for static provisioning).

# Check your PVC's access modes
kubectl get pvc my-app-pvc -o yaml | grep accessModes

  accessModes:
  - ReadWriteOnce

Consult your CSI driver documentation to understand which accessModes it supports for specific StorageClass configurations. For example, most cloud block storage (EBS, Azure Disk, GCE PD) only supports ReadWriteOnce. Network file systems (NFS, CephFS) typically support ReadWriteMany.

If there's a mismatch, you'll either need to:

• Modify your PVC to request a supported access mode (if the application can handle it).
• Use a different StorageClass that provisions storage supporting the required access mode.
• Configure your CSI driver or storage backend to provide the desired access mode.

5. Check Underlying Storage System Health and Quotas

If the CSI driver pods are running seemingly fine but still report provisioning failures in their logs (e.g., Failed to create volume: out of resources), the issue might be with the actual storage backend.

• Cloud Providers: Check your cloud provider's console for storage quotas, regional availability issues, or API rate limits.
• NFS: Verify the NFS server is running, reachable from your Kubernetes nodes, and has sufficient free space.

  # On a Kubernetes node, try to mount the NFS share manually
  sudo mount -t nfs <NFS_SERVER_IP>:/<NFS_SHARE> /mnt/test
  df -h /mnt/test
  sudo umount /mnt/test
  

• iSCSI/Ceph: Check the status of your iSCSI targets or Ceph cluster.

6. Review Kubernetes Node Taints and Tolerations

While less common, if your CSI driver's controller or node pods are prevented from scheduling due to node taints, they cannot perform their duties.

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

If you see taints that might affect your CSI driver pods (e.g., NoSchedule taints on all nodes where the CSI driver expects to run), verify that the CSI driver's Deployment or DaemonSet has corresponding tolerations configured.

7. Clean Up Stale Resources (If applicable)

Sometimes, failed provisioning attempts can leave behind stale VolumeAttachment objects or other orphan resources that interfere with subsequent attempts. While Kubernetes usually handles this, a manual check can be beneficial if repeated provisioning fails.

Exercise extreme caution when deleting Kubernetes resources. Ensure you understand the impact before proceeding. This step is for advanced users and only if other diagnostics yield no results.

# Check for VolumeAttachment related to your PVC (if it ever partially provisioned)
kubectl get volumeattachment | grep <PV_NAME_IF_ANY>
# If a stale VA is found and the PV no longer exists or is unbindable, you might need to delete it.
# kubectl delete volumeattachment <VA_NAME>

# Check for orphan PVs (less likely to cause PVC pending, but good for cleanup)
kubectl get pv | grep Released # Or other non-Bound/Available states

8. Recreate the PVC (Last Resort)

If all diagnostic steps fail to identify a clear problem and you suspect some internal state corruption, as a last resort, you can try deleting and recreating the PVC.

Deleting a PVC will de-allocate the storage and can lead to data loss if reclaimPolicy is Delete and the underlying PV still exists. Ensure you have a backup strategy in place or are certain the data is not critical before deleting a PVC.

1. Backup Data (if any): If the PVC was ever bound and had data, ensure it's backed up.
2. Delete Dependent Pods: Ensure no pods are actively using the PVC.

   kubectl delete pod <POD_NAME> -n default --force --grace-period=0 # Use force/grace-period only if pod is stuck
   

3. Delete the PVC:

   kubectl delete pvc my-app-pvc -n default
   

4. Recreate the PVC:

   kubectl apply -f my-pvc.yaml -n default
   

By systematically working through these steps, inspecting events, logs, and configurations, you can diagnose and resolve most instances of Kubernetes PVCs stuck in Pending status on your Ubuntu 22.04 LTS cluster.